Information processing device and information processing method

Abstract

This information processing device comprises: an acquisition unit that acquires a captured image including a first article image that is an image of a first article; and a control unit that displays, on a display unit, the first article image and a recognition result for the first article. When a selection made for the first article image or the recognition result for the first article is received from an input unit, the control unit causes the other of the first article image and the recognition result for the first article that was not selected to be displayed with emphasis.

Description

Cross-reference to related applications

[0001] This application claims priority to Japanese Patent Application No. 2022-136048, filed on August 29, 2022, the entire disclosure of which is incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0003] Conventionally, there is known a technology for processing payment for goods by capturing an image of an object using an imaging device or the like and recognizing which item the object is. For example, Patent Document 1 discloses an item recognition device equipped with an image interface that acquires an image of a predetermined location where multiple items are placed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-129038 Summary of the Invention

[0005] An information processing device according to an embodiment of the present disclosure includes: an acquisition unit that acquires a captured image including a first article image that is an image of a first article; a control unit that displays the first article image and the recognition result of the first article on a display unit, When the control unit receives a selection of either the first item image or the recognition result of the first item from the input unit, the control unit highlights and displays the other of the first item image and the recognition result of the first item that has not been selected.

[0006] An information processing method according to an embodiment of the present disclosure includes: acquiring an image including at least one item; Recognizing which item is included in the image; Output the recognition result and the image. When a user input is received to select either the recognition result or the article to be output, the other not selected by the user is displayed in an emphasized manner. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic configuration of a settlement system according to an embodiment of the present disclosure. [Figure 2] 2 is a schematic configuration of the information processing system shown in FIG. [Figure 3] FIG. 3 is a block diagram of the information processing device shown in FIG. 2. [Figure 4] 10 is an image of a recognition result displayed on a display unit according to an embodiment. [Figure 5] 1 is an image emphasizing a first item among items displayed on a display unit according to one embodiment. [Figure 6] 10 is an image emphasizing a second item among the items displayed on the display unit according to one embodiment. [Figure 7] 10 is an image emphasizing a third item among the items displayed on the display unit according to one embodiment. [Figure 8] 10 is an image for correcting an incorrect item name among the item names displayed on the display unit according to an embodiment. [Figure 9] 10 is a flowchart illustrating a display process according to an embodiment. [Figure 10] 10 is an image of the recognition result displayed on a display unit according to the first modification. [Figure 11] 10 is an image emphasizing a plurality of recognizable articles among the articles displayed on the display unit according to the first modification. [Figure 12] 10 is an image emphasizing only one of a plurality of recognized articles displayed on a display unit according to Modification 1. FIG. [Figure 13] 10 is a flowchart illustrating a display process according to Modification 1. [Figure 14] 10 is an image emphasizing the names of a plurality of recognized articles among the articles displayed on the display unit according to the second modification. [Figure 15] 10 is an image in which only one of the names of a plurality of recognized articles among the articles displayed on the display unit according to the second modification is emphasized. [Figure 16] 10 is an image emphasizing the names of a plurality of recognized articles among the articles displayed on the display unit according to the second modification. [Figure 17] 10 is a flowchart illustrating a display process according to Modification 2. [Figure 18] 10 is an image emphasizing a first item name among the item names displayed on the display unit according to Modification 3. [Figure 19] 10 is a flowchart illustrating a display process according to Modification 3. [Figure 20] 10 is an image that highlights one of the articles and one of the article names displayed on the display unit according to Modification 4. [Figure 21] 13 is a flowchart illustrating a display process according to Modification 4. [Figure 22] 10 is an image emphasizing an item and its name displayed on a display unit according to Modification 5. [Figure 23] 13 is a flowchart illustrating a display process according to Modification 5. [Figure 24] 13 is an image emphasizing an item and its name displayed on a display unit according to Modification 6. [Figure 25] 13 is a flowchart illustrating a display process according to Modification 6. [Figure 26] 13 is an image emphasizing one of the articles displayed on the display unit according to the seventh modification. [Figure 27] 13 is a flowchart illustrating a display process according to Modification 7. [Figure 28] 13 is an image emphasizing one of the articles displayed on the display unit according to Modification 8. [Figure 29] 13 is a flowchart illustrating a display process according to Modification 8. DETAILED DESCRIPTION OF THE INVENTION

[0008] In the past, when using an item recognition device with an image interface, if multiple items were recognized from a single image and the recognition results were displayed to the user, the user had to simultaneously check the multiple items in the image and the recognition results. This made it difficult for the user to determine whether the correspondence between the multiple items and the recognition results was correct. This traditionally slowed down the user's payment process.

[0009] According to one embodiment of the present disclosure, the user's payment process can be accelerated.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] (System configuration) The settlement system 1 shown in Figure 1 is configured as a POS (Point Of Sales) system. The settlement system 1 includes at least one information processing system 3 and a server 4. In this embodiment, the settlement system 1 includes multiple information processing systems 3.

[0012] The information processing system 3 and the server 4 can communicate with each other via a network 2. The network 2 may be any network including the Internet.

[0013] The information processing system 3 may be installed in any store, such as a convenience store, a supermarket, or a restaurant.

[0014] The information processing system 3 according to this embodiment is configured as an unmanned cash register. An unmanned cash register is also called a self-checkout. Note that "cash register" is an abbreviation of "cash register" and is used, for example, to settle the payment for an item to be purchased. For example, in the information processing system 3, a user who is a customer of a store places an item they wish to purchase on a table 10 as shown in FIG. 2. The information processing system 3 captures an image of the item placed by the user. The information processing system 3 recognizes whether an object included in the captured image corresponds to any item in the store. By recognizing the item using the information processing system 3, the total amount of the item to be charged to the user can be calculated. Here, when capturing an image of an item using an imaging device such as a camera, multiple items may be placed on the table 10. The information processing system 3 is not limited to an unmanned cash register, but may also be a manned cash register or a semi-self-checkout.

[0015] The information processing system 3 is configured as a cash register of a POS system. The information processing system 3 transmits the processing results, such as the total price of the items, information on each item, and the price of each item, to the server 4 via the network 2.

[0016] The server 4 receives the processing results of the information processing system 3 from the information processing system 3 via the network 2. Based on the received processing results, the server 4 manages the inventory status of the store where the information processing system 3 is installed.

[0017] 2, the information processing system 3 includes a first imaging unit 12 and an information processing device 20. The information processing system 3 may further include a mounting table 10, a support column 11, a display unit 13, and a second imaging unit 14. In this embodiment, the information processing device 20 is configured as a device separate from the first imaging unit 12 and the display unit 13. However, the information processing device 20 may be configured integrally with at least one of the display unit 13, the first imaging unit 12, the support column 11, the mounting table 10, and the second imaging unit 14, for example.

[0018] The platform 10 includes an upper surface 10s. The upper surface 10s can be used to place items that the user wishes to purchase. In this embodiment, the user places only the items that the user wishes to purchase on the upper surface 10s, but this is not limited thereto and the items may be placed in a shopping basket or shopping bag. In this embodiment, the upper surface 10s is substantially rectangular, but this is not limited thereto and the upper surface 10s may have any shape.

[0019] The support pillar 11 supports the first imaging unit 12. The support pillar 11 extends from the side of the mounting table 10 toward above the upper surface 10s. Note that the method for supporting the first imaging unit 12 is not limited to the support pillar 11. The support pillar 11 may support the first imaging unit 12 in any manner that allows imaging of at least a portion of the upper surface 10s of the mounting table 10.

[0020] The first imaging unit 12 is capable of generating an image by imaging. The first imaging unit 12 is fixed to the support column 11 so as to be able to image at least a portion of the surface of the mounting table 10, for example, at least a portion of the upper surface 10s. The first imaging unit 12 may be fixed so that its optical axis is perpendicular to the upper surface 10s. For example, the first imaging unit 12 is capable of imaging the entire upper surface 10s of the mounting table 10, and is fixed so that its optical axis is perpendicular to the upper surface 10s. The first imaging unit 12 may be fixed to the tip of the support column 11. The first imaging unit 12 may perform continuous imaging at any frame rate. The first imaging unit 12 may capture moving images. As an example, the first imaging unit 12 may be a camera.

[0021] The display unit 13 may be any display. The display unit 13 displays an image transmitted from the information processing device 20. The display unit 13 may function as a touch screen of an input unit, which will be described later.

[0022] The second imaging unit 14 can generate an image by capturing an image. The second imaging unit 14 is fixed to the ceiling above the mounting table 10 so as to be able to capture images of the mounting table 10 and its surroundings. The image captured by the second imaging unit 14 includes the user in front of the mounting table 10 and other users lining up to use the information processing system 3. The second imaging unit 14 may capture an image of the mounting table 10 from an angle different from that of the first imaging unit 12. By capturing images of the mounting table 10 from multiple angles, an item that cannot be recognized in the image captured by the first imaging unit 12 may be recognized in the image captured by the second imaging unit 14. The second imaging unit 14 may continuously capture images at any frame rate. The second imaging unit 14 may capture moving images. As an example, the second imaging unit 14 may be a camera.

[0023] As shown in FIG. 3, the information processing device 20 includes a communication unit 21, an input unit 22, a storage unit 23, and a control unit 24.

[0024] The communication unit 21 includes at least one communication module connectable to the network 2. The communication module may be, for example, a communication module compatible with standards such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 21 is connected to the network 2 via the wired LAN or wireless LAN by this communication module.

[0025] The communication unit 21 includes a communication module capable of communicating with the first imaging unit 12, the display unit 13, and the second imaging unit 14. The communication module is a communication module compatible with the standards of communication lines. Communication by the communication unit 21 may be performed at least either wired or wirelessly. The communication unit 21 functions as an acquisition unit that acquires an image of an item placed on the upper surface 10s of the platform 10 from at least one of the first imaging unit 12 and the second imaging unit 14 via this communication module. The communication unit 21 may also communicate with a weight sensor 81, which will be described later, via this communication module.

[0026] The input unit 22 is capable of receiving input from a user. The input unit 22 includes at least one input interface capable of receiving input from a user. The input interface is, for example, a physical key, a capacitance key, a pointing device, a microphone, or a touch screen provided integrally with the display unit 13. In this embodiment, the input unit 22 acquires input from the touch screen provided integrally with the display unit 13.

[0027] The storage unit 23 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 23 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 23 stores data used in the operation of the information processing device 20 and data obtained by the operation of the information processing device 20.

[0028] The control unit 24 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 24 executes processes related to the operation of the information processing device 20 while controlling each unit of the information processing device 20.

[0029] The control unit 24 acquires a selection made by the input unit 22 of either the item included in the image or the item recognition result. This selection is input from the input unit 22, for example, by a user who wishes to confirm the item recognition result. Upon receiving this selection, the control unit 24 transmits a signal to the display unit 13 via the communication unit 21 to highlight and display the item included in the image or the item recognition result not selected by the user. When the display unit 13 and the information processing device 20 are configured as a single device, the control unit 24 may transmit a signal to the display unit 13 without going through the communication unit 21.

[0030] (Display processing) The process from the start of transaction to the process of highlighting and displaying the recognition results will be explained below using FIGS. 4, 5, and 6.

[0031] The control unit 24 can receive an input instructing the start of a transaction via the input unit 22. This input may be input via the input unit 22 by a user who wishes to make a transaction for an item. After inputting this input via the input unit 22, the user places the item they wish to purchase on the upper surface 10s of the platform 10. After placing the item on the upper surface 10s of the platform 10, the user may input an input instructing the start of a transaction into the input unit 22.

[0032] The control unit 24 acquires the captured image captured by the first imaging unit 12 or the second imaging unit 14. The control unit 24 may acquire the captured images captured by the first imaging unit 12 and the second imaging unit 14. In this embodiment, the control unit 24 acquires the captured image by receiving the image captured by the first imaging unit 12 via the communication unit 21. This captured image includes an image of an item placed on the upper surface 10s of the platform 10 by the user.

[0033] The control unit 24 recognizes which item an object included in the acquired captured image is. In the present embodiment, the control unit 24 recognizes which item the object is by performing object recognition processing on the captured image. The object recognition processing is processing for detecting an image of an object included in the captured image and identifying which item the object is. The control unit 24 may perform the object recognition processing using a learning model generated by machine learning such as deep learning. In the present embodiment, the control unit 24 identifies the item name of the object through object recognition and acquires the recognition result. After identifying the item name, the control unit 24 may identify the number of recognized items. In the present embodiment, the recognition result of the object recognition processing includes data on the item name and the number of recognized items. Note that the object recognition processing is not limited to being performed by the control unit 24 of the information processing device 20, but may also be performed by an external server or the like. When the object recognition processing is performed by an external server, more specifically, the external server receives the captured image via the communication unit 21 and performs the object recognition processing. When an external server performs object recognition processing, the control unit 24 acquires the recognition result from the external server via the communication unit 21. Furthermore, the object recognition only requires that information such as an ID that can identify the object can be identified instead of the object name. In this case, the control unit 24 may acquire the object name corresponding to the identified ID using a correspondence table or the like stored in the storage unit 23.

[0034] The control unit 24 outputs the recognition result to the display unit 13. In this embodiment, the control unit 24 notifies the user of the recognition result by outputting an image showing the recognition result to the display unit 13. In this embodiment, the control unit 24 outputs the recognition result to the display unit 13 when it recognizes which item an object included in the captured image is, but this is not limited to this, and the recognition result may also be output to the display unit 13 when the input unit 22 receives an instruction from the user to display a confirmation screen. The instruction to display the confirmation screen may be an action of pressing a button to transition to the confirmation screen.

[0035] For example, as shown in FIG. 4, the control unit 24 displays an image 30 on the display unit 13. The image 30 includes an image 31 and an image 32. The image 31 may be displayed on the left side of the display unit 13, and the image 32 may be displayed on the right side of the display unit 13. The image 31 is an image captured by the first imaging unit 12. The image 31 includes an item placed on the platform 10 by the user. The image 32 displays the item name, which is the recognition result of the item placed on the platform 10 by the user. In this case, the image 32 may display information such as the number of items or the price of the items together with the item name. The item names displayed in the image 32 may be displayed in the order in which they were recognized by the information processing device 20. The item names displayed in the image 32 may be displayed in order of the highest or lowest price of the item. In this case, the user can easily identify items that have a large impact on the total amount. The item names displayed in the image 32 may be displayed in order of the highest or lowest recognition accuracy of the item. In this case, the user can easily check items that have a large impact on the accuracy of the total amount. The item names displayed in image 32 may be displayed in ascending or descending order of the items in image 31. In this case, the user can easily check items that are large and eye-catching, or small and difficult to find, first. The item names displayed in image 32 may be displayed in descending or descending order of the number of the same item. In this case, the user can easily check items that have been purchased in large quantities first. The display order of the item names displayed in image 32 may be changeable according to any criteria by the user operating the input unit. Information regarding the prices of the items may be stored in advance in memory unit 23 in association with the item names.

[0036] When an image 30 such as that shown in FIG. 4 is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. To check whether the recognition result displayed in the image 32 is correct, the user may select a first item name 32a using a cursor 33 as shown in FIG. 5. Selecting the first item name 32a may involve hovering the cursor 33 over the first item name 32a and then pressing a button. When the user selects the first item name 32a, an emphasis image 34a is displayed superimposed on a first item image 31a, which is an image of an item corresponding to the first item name 32a. The emphasis image 34a may be, for example, an image that forms a frame along the outline of the first item image 31a. By displaying the emphasis image 34a in this manner, the first item image 31a is highlighted relative to the other item images, allowing the user to check the correspondence between the highlighted item image and the first item name 32a. In this embodiment, the user selects the first item name 32a using the cursor 33, but this is not limited thereto. The first item name 32a may be selected by directly touching it on a touch screen, or by pointing at the first item name 32a included in the image 32 with the user's hand. When using a touch screen, the cursor 33 does not need to be displayed. In addition, in this embodiment, the highlighted image 34a is displayed according to the outline of the corresponding first item image 31a, but this is not limited thereto. The highlighted image 34a may overlap the entire first item image 31a. For example, it may be a circumscribing rectangle of the first item image 31a. In addition, in this embodiment, the highlighted image 34a has the same size as the first item image 31a, but this is not limited thereto. An image larger or smaller than the first item image 31a may be displayed.

[0037] After the user confirms the first item image 31a highlighted by the highlighted image 34a, the user uses the cursor 33 to select a new item, a second item name 32b, as shown in FIG. 6 . At this time, the control unit 24 erases the highlighted image 34a superimposed on the previously confirmed first item image 31a and superimposes the highlighted image 34b on the second item image 31b corresponding to the newly selected second item name 32b. At this time, the control unit 24 may change the position of the first item name 32a to display it at the bottom of the column. Alternatively, the control unit 24 may vertically divide the image 32 into a first area and a second area, displaying item names 32 that have not been selected in the first area and moving the selected first item name 32a to the second area. By distinguishing between previously selected and previously unselected item names 32, the user can easily identify the item name 32 to be confirmed. The timing for erasing the superimposed emphasis image 34a is not limited to when the second item name 32b is newly selected. The superimposed emphasis image 34a may be erased when the user moves the cursor used to select the first item name 32a away from the first item name 32a. The superimposed emphasis image 34a may be erased when the user presses a button to delete the display of the emphasis image 34a. The superimposed emphasis image 34a may be erased when the user presses a button to proceed to a payment process for paying for the item. The superimposed emphasis image 34a may be erased a predetermined time after the user inputs selection of the first item name 32a. In the example shown in FIG. 6, the second item image 31b is located below the item in the first item image 31a. Therefore, a portion of the second item image 31b is obscured by the first item image 31a. In such a case, the outline of the second article image 31b may be drawn as an emphasized image in the area where the second article image 31b and the first article image 31a overlap. Alternatively, the second article image 31b', which was captured before the article was placed on the platform 10, may be displayed superimposed on the second article image 31b.

[0038] In this way, by highlighting and displaying the corresponding first article image 31a or second article image 31b, the user can easily compare the recognition result with the article placed on the platform 10. With this configuration, the user can easily check whether the recognition result is correct or not.

[0039] (correction process) The process of correcting erroneous recognition results among the displayed items will be described with reference to FIGS.

[0040] When the user selects the incorrectly recognized item name 32c' displayed on the display unit 13, a highlighted image 34c is superimposed on the third item image 31c that the information processing device 20 has determined corresponds to the item name 32c'. At this time, the item name corresponding to the third item image 31c is the correct third item name 32c, and the third item name 32c' is incorrect. When the user notices that the incorrect third item name 32c' is displayed, the user may make an input to correct the incorrect third item name 32c'. The input to correct the incorrect third item name 32c' may be an input to select the incorrect third item name 32c' again, an input to select the incorrect third item name 32c' for a predetermined period of time or longer, an input to press a button to correct the item name included in the information processing device 20 or the image 30, or an input to select the third item image 31c that is determined to correspond to the incorrect third item name 32c'.

[0041] When the control unit 24 receives a user input to correct the incorrect third item name 32c′, it displays an image 35 as shown in FIG. 8 on the display unit 13. The image 35 includes multiple item names, as correction candidates 35a, 35b, and 35c, that are presumed to correspond to the third item image 31c. Among these correction candidates, the correction candidate 35a is identical to the third item name 32c that correctly corresponds to the third item image 31c. The correction candidates 35a, 35b, and 35c may be displayed in descending or ascending order of likelihood as a recognition result, in alphabetical order of the item names, or in ascending or descending order of price. If the user determines that the correction candidate 35a is the correct third item name 32c, the user inputs a selection of the correction candidate 35a in the image 35. When the control unit 24 receives an input to select the correction candidate 35a included in the image 35, it corrects the incorrect third item name 32c′ included in the image 32 to the correction candidate 35a. When the control unit 24 receives an input to select the correction candidate 35a, the control unit 24 may send an instruction to output an image and sound to confirm the change of the third item name 32c' to the correction candidate 35a. Furthermore, if the correction candidates 35a, 35b, and 35c do not include an item name corresponding to the second item image 31b, the user may directly input a corresponding item name.

[0042] (System Operation) 9 is a flowchart showing the steps of an information processing method according to an embodiment of the present disclosure. This information processing method includes steps S101 to S112. For example, when control unit 24 receives an input instructing the start of a transaction via input unit 22, control unit 24 starts the process of step S101.

[0043] The control unit 24 receives an image from the first imaging unit 12 via the communication unit 21, thereby acquiring data of the captured image (step S101). The control unit 24 executes an object recognition process on the captured image acquired in the process of step S101 (step S102). The control unit 24 executes the object recognition process, thereby acquiring a recognition result.

[0044] The control unit 24 causes the display unit 13 to display an image showing the recognition result acquired in the process of step S102 (step S103). For example, the control unit 24 causes the display unit 13 to display an image 30 as shown in FIG.

[0045] The control unit 24 determines whether or not an input for selecting the first item name 32a of the image 32 has been received by the input unit 22 (step S104).

[0046] When the control unit 24 determines that the input for selecting the first item name 32a has been received by the input unit 22 (step S104: YES), the control unit 24 proceeds to the process of step S105. For example, when the control unit 24 receives a touch input on the first item name 32a as shown in FIG. 4 by the input unit 22, the control unit 24 determines that the input for selecting the first item name 32a has been received by the input unit 22.

[0047] When the control unit 24 determines that the input for selecting the first item name 32a has not been received by the input unit 22 (step S104: NO), the control unit 24 proceeds to the processing of step S108. For example, when the control unit 24 receives an input from the user via the input unit 22 to transition to the payment process, the control unit 24 may determine that the input for selecting the first item name 32a has not been received by the input unit 22. Note that, when a predetermined time has passed without any operation after the image 30 as shown in FIG. 4 is displayed, the control unit 24 may determine that the input for selecting the first item name 32a has not been received by the input unit 22.

[0048] In the processing of step S105, the control unit 24 sends an instruction to the display unit 13 to display an emphasized image 34a to be superimposed on the first item image 31a based on input from the user selecting the first item name 32a received from the input unit 22.

[0049] The control unit 24 determines whether or not an input for selecting the second item name 32b of the image 32 has been received by the input unit 22 (step S106).

[0050] When the control unit 24 determines that the input for selecting the second item name 32b has been received by the input unit 22 (step S106: YES), the control unit 24 proceeds to the processing of step S107. For example, when the control unit 24 receives a touch input on the second item name 32b as shown in FIG. 6 by the input unit 22, the control unit 24 determines that the input for selecting the second item name 32b has been received by the input unit 22.

[0051] When the control unit 24 determines that the input for selecting the second item name 32b has not been received by the input unit 22 (step S106: NO), the control unit 24 proceeds to step S112. For example, when the control unit 24 receives an input by the user to proceed to the payment process by the input unit 22, the control unit 24 determines that the input for selecting the second item name 32b has not been received by the input unit 22. Note that when a predetermined time has elapsed without any operation, the control unit 24 may determine that the input for selecting the second item name 32b has not been received by the input unit 22.

[0052] In the process of step S107, the control unit 24 transmits an instruction to the display unit 13 to delete the highlighted image 34a superimposed on the already-displayed first item image 31a, based on an input from the user selecting the second item name 32b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S105.

[0053] The control unit 24 determines whether or not an input to correct the third item name 32c' in the image 32 has been accepted by the input unit 22 (step S108).

[0054] When the control unit 24 determines that the input to correct the third item name 32c′ has been received by the input unit 22 (step S108: YES), the control unit 24 proceeds to step S109. For example, when the control unit 24 receives a touch input of pressing and holding the third item name 32c′ for more than a predetermined time as shown in FIG. 7 through the input unit 22, the control unit 24 determines that the input to correct the third item name 32c′ has been received by the input unit 22.

[0055] When the control unit 24 determines that the input to correct the third item name 32c' has not been accepted by the input unit 22 (step S108: NO), the control unit 24 proceeds to step S112. For example, when the control unit 24 accepts an input by the user to proceed to the payment process by the input unit 22, the control unit 24 determines that the input to correct the third item name 32c' has not been accepted by the input unit 22. Note that when a predetermined time has elapsed without any operation by the user, the control unit 24 may determine that the input to correct the third item name 32c' has not been accepted by the input unit 22.

[0056] In the process of step S109, the control unit 24 displays the image 35. The image 35 includes, as correction candidates 35a, 35b, and 35c, product names that are presumed to correspond to the third product image 31c.

[0057] The control unit 24 receives, as a correction input, an input from the user selecting a correction candidate included in the image 35 displayed in step S109 via the input unit 22 (step S110). The correction input is not limited to the selection of a correction candidate, and may be, for example, an input of an item name by the user.

[0058] The control unit 24 corrects the third item image 31c and the corresponding item name 31c' based on the correction input in step S110 (step S111), and then the control unit 24 returns to the process of step S103.

[0059] In the process of step S112, a payment process for paying for the goods is executed based on the recognition result of the process of step S102.

[0060] In this way, in the information processing device 20, the control unit 24 displays the recognition result to the user via the display unit 13, and then instead of immediately executing the payment process, highlights the item image to allow the user to confirm the recognized item. This configuration allows the user to smoothly check whether the displayed item image has been correctly recognized. In addition, the user can quickly complete the payment process.

[0061] (Display Processing Variation 1) Variation 1 of the display process will be described below. Note that the description of the process that overlaps with the display process described above will be omitted.

[0062] As shown in FIG. 10, the control unit 24 causes the display unit 13 to display an image 30. At this time, unlike FIG. 4, the control unit 24 may display one item name along with the number of items, rather than displaying multiple instances of the same item name. As an example, when two bean-jam buns are recognized, the control unit 24 may display that there are two bean-jam buns. Furthermore, for items of the same type, the control unit 24 may display one item name of a superordinate concept of the item along with the number of items. As an example, when bean-jam buns and cream buns are recognized, the control unit 24 may display that there are two superordinate concepts of sweet buns or bread.

[0063] When image 30 as shown in Fig. 10 is displayed on display unit 13, the user checks the recognition result displayed on display unit 13. At this time, to check whether the recognition result displayed on image 32 is correct, the user selects first item name 32a using cursor 33 as shown in Fig. 11. In this modified example, one first item name 32a is displayed, indicating that there are two corresponding items. When the user selects first item name 32a, highlighted images 34a, 34a' are displayed superimposed on first item images 31a, 31a' corresponding to first item name 32a. At this time, the highlighted images 34a and 34a' are displayed simultaneously, but the present invention is not limited to this. As shown in FIG. 12, when the user selects the first item name 32a, the first item names 32a' and 32a" corresponding to the first item name 32a may be displayed in the image 30, and when the user further selects the first item name 32a', the highlighted image 34a corresponding to the corresponding item image 32a may be displayed. For example, when the first item name 32a is bean-jam bun, bean-jam bun may be displayed as both the first item names 32a' and 32a", and when the user selects the first item name 32a, the highlighted image 34a may be displayed as the first item name 32a' and 32a". By selecting any of the bean-jam bun displays (first item name 32a′ or 32a″), the corresponding bean-jam bun (first item image 31a, 31a′) may be superimposed and displayed with an emphasis image 34a. Also, if the first item name 32a is a sweet bread, which is a higher-level concept, bean-jam bun may be displayed as the first item name 32a′ and cream bun as the first item name 32a″, and by the user selecting bean-jam bun (first item name 32a′), the emphasis image 34a may be superimposed and displayed with an emphasis image 34a on the bean-jam bun (first item image 31a).

[0064] (System Operation of Modification 1) 13 is a flowchart showing the procedure of an information processing method according to Modification 1 of the present disclosure. This information processing method includes steps S201 to S212. Note that descriptions that overlap with the above-described steps S101 to S112 will be omitted.

[0065] Steps S201 to S204 are the same as or similar to the above-described steps S101 to S104, and therefore will not be described here.

[0066] In the processing of step S205, the control unit 24 sends an instruction to the display unit 13 to display highlighted images 34a, 34a' superimposed on the first item images 31a, 31a', respectively, based on input from the user selecting the first item name 32a received from the input unit 22.

[0067] Steps S206 to S212 are the same as steps S106 to S112 described above, and therefore will not be described here.

[0068] (Display Processing Variation 2) The second modification of the display process will be described below. Note that steps that overlap with the above-described display process will be omitted.

[0069] 10, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above, and will not be described again.

[0070] When an image 30 as shown in FIG. 10 is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. At this time, to check whether the recognition result displayed in image 32 is correct, the user selects a first item image 31a using a cursor 33 as shown in FIG. 14. When the user selects the first item image 31a, an emphasis image 34a′ is displayed superimposed on a first item image 31a′ classified as the same item, and an emphasis image 34a is displayed superimposed on a first item name 32a. At this time, the emphasis image 34a′ superimposed on the first item image 31a′ does not have to be displayed. Furthermore, when the user selects the first item image 31a, in addition to the emphasis image 34a being displayed superimposed on the first item name 32a, an emphasis image 34a′ may be displayed on a corresponding first item name 32a′ among the multiple first item names 32a′ included in the first item name 32a, as shown in FIG. 15. Furthermore, when a user selects a first item image 31a, as shown in FIG. 16, the first item image 31a, highlighted images 34a, 34a′ superimposed on the first item image 31a′ of an item of the same type as the first item image 31a, and highlighted images 34a, 35a′, 35a″ superimposed on the first item names 32a, 32a′, 32a″ corresponding to the first item images 31a, 31a′ may be displayed. The first item names 32a′, 32a″ are the item names of the items included in the first item name 32a. In this case, the highlighted image 34a superimposed on the first item image 31a and the highlighted image 34a′ superimposed on the first item name 32a′ corresponding to the first item image 31a may be the same color. Furthermore, the highlighted image 34a′ superimposed on the first item image 31a and the highlighted image 34a″ superimposed on the first item name 32a″ corresponding to the first item image 31a may be the same color. Furthermore, the highlighted image 34a and the highlighted image 34a′ may be different colors from the highlighted image 34a′ and the highlighted image 34a″.

[0071] (System operation of variant 2) 17 is a flowchart showing the procedure of an information processing method according to Modification 1 of the present disclosure. This information processing method includes steps S301 to S312. Note that descriptions that overlap with steps S201 to S212 described above will be omitted.

[0072] Steps S301 to S303 are the same as or similar to the above-described steps S201 to S203, and therefore description thereof will be omitted.

[0073] The control unit 24 determines whether or not an input for selecting the first item image 31a of the image 31 has been received by the input unit 22 (step S304).

[0074] When the control unit 24 determines that the input for selecting the first item image 31a has been received by the input unit 22 (step S304: YES), the control unit 24 proceeds to the process of step S305. For example, when the control unit 24 receives a touch input on the first item image 31a as shown in FIG. 10, the control unit 24 determines that the input for selecting the first item image 31a has been received by the input unit 22.

[0075] When the control unit 24 determines that the input for selecting the first item image 31a has not been received by the input unit 22 (step S304: NO), the control unit 24 proceeds to the processing of step S308. For example, when the control unit 24 receives an input from the user to proceed to the payment process by the input unit 22, the control unit 24 may determine that the input for selecting the first item image 31a has not been received by the input unit 22. Note that, when a predetermined time has passed without any operation after the image 30 as shown in FIG. 4 is displayed, the control unit 24 may determine that the input for selecting the first item image 31a has not been received by the input unit 22.

[0076] In the processing of step S305, the control unit 24 sends an instruction to the display unit 13 to display an emphasis image 34a superimposed on the first item name 32a based on input from the user selecting the first item image 31a received from the input unit 22.

[0077] The control unit 24 determines whether or not an input for selecting the second item image 31b of the image 31 has been received by the input unit 22 (step S306).

[0078] When the control unit 24 determines that the input for selecting the second item image 31b has been received by the input unit 22 (step S306: YES), the control unit 24 proceeds to the processing of step S307. For example, when the control unit 24 receives a touch input on the second item image 31b by the input unit 22, the control unit 24 determines that the input for selecting the second item image 31b has been received by the input unit 22.

[0079] When the control unit 24 determines that the input for selecting the second item image 31b has not been received by the input unit 22 (step S306: NO), the control unit 24 proceeds to step S312. For example, when the control unit 24 receives an input from the user to proceed to payment processing by the input unit 22, the control unit 24 determines that the input for selecting the second item image 31b has not been received by the input unit 22. Note that when a predetermined time has elapsed without any operation being performed, the control unit 24 may determine that the input for selecting the second item image 31b has not been received by the input unit 22.

[0080] In the process of step S307, the control unit 24 transmits an instruction to the display unit 13 to delete the highlighted image 34a superimposed on the already-displayed first article image 31a, based on an input from the user selecting the second article image 31b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S305.

[0081] The processing from step S308 to step S312 is the same as the processing from step S208 to step S212, and therefore the description thereof will be omitted.

[0082] (Display Processing Variation 3) The third variation of the display process will be described below. Note that steps that overlap with the above-described display process will be omitted.

[0083] 4, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above and will not be repeated here.

[0084] When image 30 as shown in FIG. 4 is displayed on display unit 13, the user checks the recognition result displayed on display unit 13. At this time, to confirm whether the recognition result displayed on image 32 is correct, the user selects first item image 31a using cursor 33 as shown in FIG. 18. When the user selects first item image 31a, an emphasis image 34a is displayed superimposed on first item name 32a corresponding to first item image 31a. Displaying emphasis image 34a emphasizes first item name 32a. When prompting the user to emphasize first item name 32a, not only may the emphasis image 34a be superimposed, but also the font color, thickness, or size of first item name 32a may be changed. In this embodiment, the user selects first item name 32a using cursor 33, but this is not limited thereto. First item name 32a may also be directly selected using a touch screen. At this time, cursor 33 does not need to be displayed.

[0085] In this way, by highlighting and displaying the corresponding first item name 32a, the user can easily compare the recognition result with the item placed on the mounting table 10. With this configuration, the user can easily check whether the recognition result is correct or not.

[0086] (System Operation of Modification 2) 19 is a flowchart showing the procedure of an information processing method according to Modification 2 of the present disclosure. This information processing method includes steps S401 to S412. Note that descriptions that overlap with the above-described steps S101 to S112 will be omitted.

[0087] Steps S401 to S403 are the same as or similar to the above-described steps S101 to S103, and therefore a description thereof will be omitted.

[0088] The control unit 24 determines whether or not an input for selecting the first item image 31a of the image 32 has been received by the input unit 22 (step S404).

[0089] When the control unit 24 determines that the input for selecting the first item image 31a has been received by the input unit 22 (step S404: YES), the control unit 24 proceeds to the processing of step S305. For example, when the control unit 24 receives a touch input on the first item name 32a as shown in FIG. 18, the control unit 24 determines that the input for selecting the first item name 32a has been received by the input unit 22.

[0090] When the control unit 24 determines that the input for selecting the first item image 31a has not been received by the input unit 22 (step S404: NO), the control unit 24 proceeds to the processing of step S308. For example, when the control unit 24 receives an input from the user to proceed to payment processing by the input unit 22, the control unit 24 determines that the input for selecting the first item image 31a has not been received by the input unit 22. Note that the case where the control unit 24 determines that the input for selecting the first item image 31a has not been received by the input unit 22 is not limited to the above, and may be when a predetermined time has elapsed after displaying the image 30 as shown in FIG.

[0091] In the processing of step S405, the control unit 24 sends an instruction to the display unit 13 to display an emphasis image 34a superimposed on the first item name 32a based on input from the user selecting the first item image 31a received from the input unit 22.

[0092] The control unit 24 determines whether or not an input for selecting the second item image 31b of the image 32 has been received by the input unit 22 (step S406).

[0093] When the control unit 24 determines that the input for selecting the second item image 31b has been received by the input unit 22 (step S406: YES), the control unit 24 proceeds to the processing of step S307. For example, when the control unit 24 receives a touch input on the second item image 31b by the input unit 22, the control unit 24 determines that the input for selecting the second item image 31b has been received by the input unit 22.

[0094] When the control unit 24 determines that the input for selecting the second item image 31b has not been received by the input unit 22 (step S406: NO), the control unit 24 proceeds to step S208. For example, when the control unit 24 receives an input from the user to proceed to payment processing by the input unit 22, the control unit 24 determines that the input for selecting the second item image 31b has not been received by the input unit 22. Note that the cases in which the control unit 24 determines that the input for selecting the second item image 31b has not been received by the input unit 22 are not limited to those described above, and may be when a predetermined time has elapsed after displaying the image 30 as shown in FIG.

[0095] In the process of step S407, the control unit 24 transmits an instruction to delete the highlighted image 34a superimposed on the first item name 32a to the display unit 13, based on an input from the user selecting the second item image 31b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S405.

[0096] The control unit 24 determines whether or not an input for correcting the third product name 32c' determined to correspond to the third product image 31c has been received by the input unit 22 (step S408).

[0097] When the control unit 24 determines that the input to correct the third item name 32c' has been received by the input unit 22 (step S408: YES), the control unit 24 proceeds to step S409. For example, when the control unit 24 receives a touch input to select the third item name 32c' on which the highlighted image 34c is superimposed, the control unit 24 determines that the input to correct the third item name 32c' has been received by the input unit 22.

[0098] Steps S409 to S412 overlap with the above-described steps S109 to S112, and therefore description thereof will be omitted.

[0099] (Display Processing Variation 4) Variation 4 of the display process will be described below. Note that steps that overlap with the display process described above will be omitted.

[0100] 4, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above and will not be repeated here.

[0101] When image 30 as shown in Fig. 4 is displayed on display unit 13, the user checks the recognition result displayed on display unit 13. At this time, to check whether the recognition result displayed in image 32 is correct, the user selects first item image 31a with cursor 33 as shown in Fig. 20. When the user selects first item image 31a, an emphasized image 34a superimposed on first item image 31a and an emphasized image 34a superimposed on first item name 32a corresponding to first item image 31a are displayed, respectively. At this time, emphasized image 34a and emphasized image 34b are displayed in the same color, but this is not limited to this.

[0102] As described above, by simultaneously displaying an emphasized image 34a superimposed on the first item image 31a and an emphasized image 34b superimposed on the first item name 32a corresponding to the item image 32a, it becomes easier to confirm the correspondence between the first item image 31a and the first item name 32a.

[0103] (System operation of variant 4) 21 is a flowchart showing the procedure of the recognition method according to Modification 3 of the present disclosure. This information processing method includes steps S501 to S512. Note that descriptions that overlap with the above-described steps S401 to S412 will be omitted.

[0104] Steps S501 to S504 are the same as or similar to the above-described steps S401 to S404, and therefore a description thereof will be omitted.

[0105] In the processing of step S505, the control unit 24 sends an instruction to the display unit 13 to display both an emphasis image 34a to be superimposed on the first item image 31a and an emphasis image 34a to be superimposed on the first item name 32a, based on input from the user selecting the first item image 31a received from the input unit 22.

[0106] Step S506 overlaps with step S406, so the description will be omitted.

[0107] In the process of step S507, the control unit 24 transmits an instruction to the display unit 13 to delete the highlighted image 34a superimposed on the first article image 31a and the highlighted image 34a superimposed on the first article name 32a, based on an input from the user selecting the second article image 31b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S505.

[0108] Steps S508 to S512 are the same as or similar to steps S408 to S412, and therefore will not be described here.

[0109] (Display Processing Variation 5) Variation 5 of the display process will be described below. Note that steps that overlap with the display process described above will be omitted.

[0110] 4, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above and will not be repeated here.

[0111] When an image 30 such as that shown in FIG. 4 is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. At this time, to check whether the recognition result displayed on the image 32 is correct, the user selects a first item image 31a using a cursor 33 as shown in FIG. 22. When the user selects the first item image 31a, an emphasized image 34a superimposed on the first item image 31a, an emphasized image 34a superimposed on a first item name 32a corresponding to the first item image 31a, and a correspondence line 36a connecting the first item image 31a and the first item name 32a are displayed. At this time, the emphasized images 34a and 34b are displayed in the same color, but this is not a limitation. In this embodiment, the correspondence line 36a is drawn as a solid line, but is not limited to this and may be a dotted line, for example.

[0112] As described above, the highlighted image 34a superimposed on the first item image 31a, the highlighted image 34b superimposed on the first item name 32a corresponding to the item image 32a, and the correspondence line 36a connecting the first item image 31a and the first item name 32a are simultaneously displayed, making it easier to confirm the correspondence between the first item image 31a and the first item name 32a.

[0113] (System operation of variant 5) 23 is a flowchart showing the procedure of the recognition method according to Variation 4 of the present disclosure. This information processing method includes steps S601 to S612. Note that descriptions that overlap with steps S501 to S512 described above will be omitted.

[0114] Steps S601 to S604 are the same as or similar to the above-described steps S501 to S504, and therefore a description thereof will be omitted.

[0115] In the processing of step S605, based on input from the user selecting the first item image 31a received from the input unit 22, the control unit 24 sends an instruction to the display unit 13 to display an emphasis image 34a to be superimposed on the first item image 31a, an emphasis image 34a to be superimposed on the first item name 32a, and a correspondence line 36a connecting the first item image 31a and the first item name 32a.

[0116] Step S606 overlaps with step S506, so the description will be omitted.

[0117] In the process of step S607, the control unit 24 transmits to the display unit 13 an instruction to delete the highlighted image 34a superimposed on the first item image 31a, the highlighted image 34a superimposed on the first item name 32a, and the corresponding line 36a connecting the first item image 31a and the first item name 32a, based on an input from the user selecting the second item image 31b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S605.

[0118] Steps S608 to S612 are the same as or similar to steps S508 to S512, and therefore will not be described here.

[0119] (Display Processing Variation 6) Variation 6 of the display process will be described below. Note that steps that overlap with the display process described above will be omitted.

[0120] 4, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above and will not be repeated here.

[0121] When image 30 as shown in Fig. 4 is displayed on display unit 13, the user checks the recognition result displayed on display unit 13. At this time, in order to check whether the recognition result displayed on image 32 is correct, the user selects first item image 31a with cursor 33 as shown in Fig. 24. When the user selects first item image 31a, an emphasis image 34a superimposed on first item image 31a and an emphasis image 34a superimposed on first item name 32a corresponding to first item image 31a are displayed. Details are as described above and will not be repeated here.

[0122] After the user confirms the first item name 32a highlighted by the highlighted image 34a, the user uses the cursor 33 to newly select a second item image 31b, which is another item, as shown in FIG. 20 . At this time, the control unit 24 changes the color of the highlighted image 34a superimposed on the already confirmed first item image 31a and the highlighted image 34a superimposed on the first item name 32a, and superimposes the highlighted image 34b on the newly selected second item image 31b and the highlighted image 34b on the second item name 32b corresponding to the second item image 31b. At this time, the highlighted image 34a and the highlighted image 34b may be displayed in different colors. The highlighted image 34a and the highlighted image 34b may be displayed in the same or similar colors. By differentiating the display forms of the highlighted images 34a and 34b and the highlighted images 34a and 34b, the user can easily recognize, for example, that the first item name 32a has already been confirmed.

[0123] When the user selects a new second item image 31b after checking the first item image 31a and the first item name 32a, the first item image 31a and the first item name 32a are highlighted in a color different from that of the second item image 31b, making it easy to determine whether the displayed item has already been checked.

[0124] (System operation of variant 6) 25 is a flowchart showing the procedure of a recognition method according to Modification 6 of the present disclosure. This information processing method includes steps S701 to S712. Note that descriptions that overlap with steps S501 to S512 described above will be omitted.

[0125] Steps S701 to S706 are the same as or similar to the above-described steps S501 to S506, and therefore description thereof will be omitted.

[0126] In the process of step S707, the control unit 24 transmits to the display unit 13 an instruction to display the highlighted image 34a superimposed on the first item image 31a and the highlighted image 34a superimposed on the first item name 32a in a color different from the color displayed in step S705, based on an input from the user selecting the second item image 31b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S705.

[0127] Steps S708 to S712 are the same as or similar to steps S708 to S712, and therefore will not be described here.

[0128] (Display Processing Variation 7) Variation 7 of the display process will be described below. Note that steps that overlap with the display process described above will be omitted.

[0129] 4, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above and will not be repeated here.

[0130] When image 30 as shown in FIG. 4 is displayed on display unit 13, the user checks the recognition result displayed on display unit 13. At this time, to check whether the recognition result displayed on image 32 is correct, the user selects first item image 31a using cursor 33 as shown in FIG. 26. When the user selects first item image 31a, an emphasized image 34a is displayed superimposed on first item name 32a corresponding to first item image 31a. When emphasized image 34a is displayed, registered image 37a corresponding to first item image 31a is also displayed. Registered image 37a is an image of an item stored in storage unit 23. In this embodiment, registered image 37a is displayed in the lower right corner of image 30, but this is not limiting. Furthermore, in FIG. 26, registered image 37a is displayed together with emphasized image 34a, but this is not limiting, and may be displayed together with emphasized image 34a. Furthermore, the registered image 37a may be changed according to the orientation or tilt of the first item image 31a, and the registered image 37a may be displayed so as to have the same orientation or tilt as the first item image 31a.

[0131] In this way, by displaying the registered image 37a in addition to the emphasized image 34a, the accuracy with which the user can check the recognition result is improved.

[0132] (System operation of Modification 7) 27 is a flowchart showing the procedure of a recognition method according to Variation 7 of the present disclosure. This information processing method includes steps S801 to S812. Note that descriptions that overlap with the above-described steps S401 to S412 will be omitted.

[0133] Steps S801 to S804 are the same as or similar to the above-described steps S401 to S404, and therefore a description thereof will be omitted.

[0134] In the processing of step S805, based on input from the user selecting the first item image 31a received from the input unit 22, the control unit 24 sends an instruction to the display unit 13 to display both an emphasis image 34a superimposed on the first item name 32a corresponding to the first item image 31a, and a registered image 37a corresponding to the first item image 31a.

[0135] Step S806 overlaps with step S406, so the description will be omitted.

[0136] In the process of step S807, the control unit 24 transmits an instruction to the display unit 13 to delete the highlighted image 34a and the registered image 37a superimposed on the first item name 32a, based on an input from the user selecting the second item image 31b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S805.

[0137] Steps S808 to S812 are the same as or similar to steps S408 to S412, and therefore will not be described here.

[0138] (Display Processing Variation 8) Variation 8 of the display process will be described below. Note that steps that overlap with the display process described above will be omitted.

[0139] 4, the control unit 24 causes the display unit 13 to display an image 30. Details are as described above and will not be repeated here.

[0140] When image 30 as shown in FIG. 4 is displayed on display unit 13, the user checks the recognition result displayed on display unit 13. At this time, to check whether the recognition result displayed on image 32 is correct, the user selects second item name 32b using cursor 33 as shown in FIG. 28. When the user selects second item name 32b, registered image 37b corresponding to second item name 32b is displayed and superimposed on second item image 31b corresponding to second item name 32b. In this embodiment, second item image 31b overlaps another first item image 31a, and second item image 31b is located below. At this time, registered image 37b is superimposed on the corresponding second item image 31b and also on the overlapping portion of first item image 31a and second item image 31b.

[0141] By superimposing the registered image 37b on the second article image 31b in this way, even if the second article images 31b are superimposed on each other and it is difficult to confirm whether the article is correct or not, it can be confirmed smoothly.

[0142] (System Operation of Modification 8) 29 is a flowchart showing the procedure of an information processing method according to Modification 7 of the present disclosure. This information processing method includes steps S901 to S912. Note that descriptions that overlap with the above-described steps S101 to S112 will be omitted.

[0143] Steps S901 to S904 overlap with the above-described steps S101 to S104, and therefore description thereof will be omitted.

[0144] In the processing of step S905, the control unit 24 sends an instruction to the display unit 13 to display a registered image 37a to be superimposed on the first item image 31a based on input from the user selecting the first item name 32a received from the input unit 22.

[0145] Step S906 overlaps with step S106, so the description will be omitted.

[0146] In the process of step S907, the control unit 24 transmits an instruction to the display unit 13 to delete the registered image 37a superimposed on the first item image 31a, based on an input from the user selecting the second item name 32b received from the input unit 22. Thereafter, the control unit 24 returns to the process of step S905.

[0147] Steps S908 to S912 overlap with steps S108 to S112, and therefore description thereof will be omitted.

[0148] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications, alterations, or combinations of the embodiments of the present disclosure are included in the scope of the present disclosure.

[0149] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, the configurations distinguished by descriptions such as "first" and "second" can have their numbers exchanged. For example, a first item can exchange the identifiers "first" and "second" with a second item. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. The identifiers may be deleted. A configuration from which the identifiers have been deleted is distinguished by a symbol. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number.

[0150] In one embodiment, (1) an information processing device includes: an acquisition unit that acquires a captured image including a first article image that is an image of a first article; a control unit that displays the first article image and the recognition result of the first article on a display unit, When the control unit receives a selection of either the first item image or the recognition result of the first item from the input unit, the control unit highlights and displays the other of the first item image and the recognition result of the first item that has not been selected.

[0151] (2) In the information processing device of (1), the captured image includes a second article image that is an image of a second article; the control unit causes a display unit to display the second article image and the recognition result of the second article; The control unit highlights and displays either the first item image or the recognition result of the first item, and then, when a selection of either the second item image or the recognition result of the second item is received from the input unit, removes the display highlighting the first item image or the recognition result of the first item and highlights and displays the other of the second item image and the recognition result of the second item that has not been selected.

[0152] (3) In the information processing device of (2), After the control unit has highlighted and displayed either the first item image or the recognition result of the first item, when the control unit receives input into the input unit of a new selection of either the second item image or the recognition result of the second item, the control unit changes the display format of the display that highlights the first item image or the recognition result of the first item, and highlights and displays the other of the second item and the recognition result of the second item that has not been selected.

[0153] (4) In the information processing device according to (1) to (3) above, The control unit displays a line connecting the first item image and the recognition result of the first item, along with a display that highlights either the first item image or the recognition result of the first item.

[0154] (5) The information processing device of (1) to (3) above is Further, a storage unit that stores a registered image of the article is provided, The control unit further causes the display unit to display the registered image of the first article together with a display that highlights either the first article image or the recognition result of the first article.

[0155] (6) In the information processing device of (5), The control unit causes the display unit to display the registered image superimposed on the first item image.

[0156] (7) In the information processing device of (6), The control unit displays the registered image according to the posture of the article corresponding to the first article image.

[0157] (8) In the information processing device according to any one of (1) to (4) above, the captured image includes a plurality of the first article images, The control unit causes a display unit to display the recognition result of the first items and the number of the first items recognized from the captured image.

[0158] (9) In the information processing device of (8), When the control unit receives a selection of the recognition result of the first object from the input unit, the control unit further causes the display unit to display each of the recognition results of the first object.

[0159] (10) In the information processing device according to any one of (1) to (9), the captured image includes a second article image that is an image of a second article that is the same type of article as the first article, The control unit causes the display unit to display an article name that is a superordinate concept of the first article and the second article.

[0160] (11) In the information processing device according to any one of (1) to (10) above, When the control unit receives a selection of either the first item image or the recognition result of the first item from the input unit, the control unit displays both the first item image and the recognition result of the first item in an highlighted manner.

[0161] (12) In the information processing device according to any one of (1) to (11), The control unit displays correction candidates for correcting the recognition result of the first article.

[0162] In one embodiment, the information processing method (13) includes: acquiring an image including at least one item; Recognizing which item is included in the image; Output the recognition result and the image. When a user input is received to select either the recognition result or the article to be output, the other not selected by the user is displayed in an emphasized manner.

[0163] In one embodiment, the information processing device (14) includes: The device includes a display unit that displays an image including a first item and a recognition result of the first item, and when either the first item or the recognition result is selected by a user, highlights and displays the other of the first item and the recognition result that has not been selected. [Explanation of symbols]

[0164] 1. Payment system 2 Network 3. Information Processing Systems 4 Server 10. Mounting table 10s top 11 Support pillar 12 Imaging unit 13 Display section 14 Imaging device 20 Information processing equipment 21 Communications Department 22 Input section 23 Memory section 24 Control Unit 34 34a 34b Enhanced image 35 35a 35b Enhanced image 36 36a Corresponding lines 37 37a Registered image

Claims

1. an acquisition unit that acquires a captured image including a first article image that is an image of a first article; a control unit that displays the first article image and the recognition result of the first article on a display unit, When the control unit receives a selection of either the first article image or the recognition result of the first article from the input unit, the control unit highlights and displays the other of the first article image and the recognition result of the first article that is not selected. Information processing device.

2. the captured image includes a second article image that is an image of a second article; the control unit causes a display unit to display the second article image and the recognition result of the second article; The control unit, after highlighting and displaying either the first article image or the recognition result of the first article, when receiving a selection of either the second article image or the recognition result of the second article from the input unit, deletes the display highlighting the first article image or the recognition result of the first article and highlights the other of the second article image and the recognition result of the second article that is not selected. The information processing device according to claim 1 .

3. When the control unit receives an input of a new selection of either the second article image or the recognition result of the second article to the input unit after highlighting and displaying either the first article image or the recognition result of the first article, the control unit changes the display form of the display highlighting the first article image or the recognition result of the first article without deleting the display highlighting the first article image or the recognition result of the first article, and highlights and displays the other of the second article image and the recognition result of the second article that has not been selected. The information processing device according to claim 2 .

4. The control unit displays a line connecting the first article image and the recognition result of the first article together with a display that highlights either the first article image or the recognition result of the first article. The information processing device according to claim 1 .

5. Further, a storage unit that stores a registered image of the article is provided, The control unit further causes the display unit to display the registered image of the first article together with a display that highlights either the first article image or the recognition result of the first article. The information processing device according to claim 1 .

6. The control unit causes the display unit to display the registered image superimposed on the first article image.

6. An information processing device according to claim 5.

7. The control unit displays the registered image according to the posture of the article corresponding to the first article image.

7. An information processing device according to claim 6.

8. the captured image includes a plurality of the first article images, the control unit causes a display unit to display a recognition result of the first articles and the number of the first articles recognized from the captured image. The information processing device according to claim 1 .

9. when the control unit receives a selection of the recognition result of the first object from the input unit, the control unit further displays each of the recognition results of the first object on the display unit. The information processing device according to claim 8 .

10. the captured image includes a second article image that is an image of a second article that is the same type of article as the first article, the control unit causes the display unit to display names of items that are generic concepts of the first item and the second item. The information processing device according to claim 1 .

11. When the control unit receives a selection of either the first article image or the recognition result of the first article from the input unit, the control unit highlights and displays both the first article image and the recognition result of the first article. The information processing device according to claim 1 .

12. the control unit displays correction candidates for correcting the recognition result of the first article. The information processing device according to claim 1 .

13. An information processing method executed by an information processing device, comprising: acquiring an image including at least one item; Recognizing which item is included in the image; Output the recognition result and the image. When a user input is received to select either the recognition result or the image of the article to be output, the other image not selected by the user is displayed in an emphasized manner. Information processing methods.

14. a display unit that displays an image including a first item and a recognition result of the first item, and when either the image of the first item or the recognition result is selected by a user, highlights and displays the other of the image of the first item and the recognition result that is not selected. Information processing device.

Images