Information processing device

Abstract

This can speed up the user's payment processing. [Solution] The information processing device includes an acquisition unit that acquires an image including a first article image which is an image of a first article, and a control unit that displays the first article image and the recognition result of the first article on a display unit. When the control unit receives a selection of either the first article image or the recognition result of the first article from an input unit, it highlights and displays the other of the first article image and the recognition result of the first article that has not been selected.

Description

Cross-reference to related applications

[0001] This application claims the priority of Japanese Patent Application No. 2022-136048, filed in Japan on August 29, 2022, and incorporates the entire disclosure of the prior application herein by reference.

Technical Field

[0002] This disclosure relates to an information processing apparatus and an information processing method.

Background Art

[0003] Conventionally, for the settlement processing of the price of an article, a technique of imaging an object with an imaging device or the like and recognizing what article the object is has been known. For example, Patent Document 1 discloses an article recognition apparatus including an image interface that acquires an image obtained by photographing a predetermined location where a plurality of articles are arranged.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] An information processing apparatus according to an embodiment of the present disclosure includes an acquisition unit that acquires an imaging image including a first article image that is an image of a first article, and a control unit that causes a display unit to display the first article image and a recognition result of the first article, and when the control unit receives an input of a selection of either one of the first article image and the recognition result of the first article from an input unit, the control unit causes the other one that is not selected among the first article image and the recognition result of the first article to be highlighted and displayed.

[0006] An information processing method according to an embodiment of the present disclosure includes acquiring an image including at least one article, To recognize which item is included in the aforementioned image, The recognized recognition result and the aforementioned image are output. When the system receives input from a user selecting either the outputted recognition result or the item, it outputs a message that highlights the other item that the user has not selected. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic configuration of a settlement system according to one embodiment of the present disclosure. [Figure 2] Figure 1 shows the schematic configuration of the information processing system. [Figure 3] Figure 2 is a block diagram of the information processing device shown. [Figure 4] This is an image of the recognition result displayed on the display unit according to one embodiment. [Figure 5] This is an image that emphasizes a first item among the items displayed on the display unit according to one embodiment. [Figure 6] This is an image that emphasizes a second item among the items displayed on the display unit according to one embodiment. [Figure 7] This is an image that emphasizes a third item among the items displayed on the display unit according to one embodiment. [Figure 8] This is an image used to correct an incorrect item name among the item names displayed on the display unit according to one embodiment. [Figure 9] This is a flowchart illustrating the display process according to one embodiment. [Figure 10] This is an image of the recognition result displayed on the display unit related to the modified example 1. [Figure 11] This image highlights multiple items that are recognized among the items displayed on the display unit according to Modification Example 1. [Figure 12] This image, according to Modification 1, highlights only one of the multiple items recognized on the display unit. [Figure 13] This is a flowchart illustrating the display process related to Modification Example 1. [Figure 14] It is an image that emphasizes the item names of multiple recognized items among the items projected onto the display unit according to Modified Example 2. [Figure 15] It is an image that emphasizes only one of the item names among the items projected onto the display unit according to Modified Example 2. [Figure 16] It is an image that emphasizes the item names of multiple recognized items among the items projected onto the display unit according to Modified Example 2. [Figure 17] It is a flowchart for explaining the display process according to Modified Example 2. [Figure 18] It is an image that emphasizes the first item name among the item names projected onto the display unit according to Modified Example 3. [Figure 19] It is a flowchart for explaining the display process according to Modified Example 3. [Figure 20] It is an image that emphasizes one item among the items projected onto the display unit according to Modified Example 4, and also emphasizes one item among the item names. [Figure 21] It is a flowchart for explaining the display process according to Modified Example 4. [Figure 22] It is an image that emphasizes the item and the item name projected onto the display unit according to Modified Example 5. [Figure 23] It is a flowchart for explaining the display process according to Modified Example 5. [Figure 24] It is an image that emphasizes the item and the item name projected onto the display unit according to Modified Example 6. [Figure 25] It is a flowchart for explaining the display process according to Modified Example 6. [Figure 26] It is an image that emphasizes one item among the items projected onto the display unit according to Modified Example 7. [Figure 27] It is a flowchart for explaining the display process according to Modified Example 7. [Figure 28] It is an image that emphasizes one item among the items projected onto the display unit according to Modified Example 8. [Figure 29] It is a flowchart for explaining the display process according to Modified Example 8. [Modes for carrying out the invention]

[0008] Traditionally, when using an object recognition device with an image interface, if multiple items were to be recognized from a single image and the recognition results were displayed to the user, the user had to simultaneously check all the items in the image and the recognition results. This made it difficult for the user to determine the correctness of the correspondence between the multiple items and the recognition results. Consequently, this often resulted in delays in the user's payment processing.

[0009] According to one embodiment of this disclosure, the user's payment processing can be accelerated.

[0010] The embodiments of this disclosure will be described below with reference to the drawings.

[0011] (System configuration) The settlement system 1 shown in Figure 1 is configured as a POS (Point of Sales) system. The settlement system 1 includes at least one information processing system 3 and a server 4. In this embodiment, the settlement system 1 includes multiple information processing systems 3.

[0012] The information processing system 3 and the server 4 can communicate with each other via network 2. Network 2 may be any network, including the Internet.

[0013] Information processing system 3 may be installed in any store. For example, information processing system 3 may be installed in a convenience store, supermarket, or restaurant.

[0014] The information processing system 3 according to this embodiment is configured as an unmanned cash register. An unmanned cash register is also called a self-checkout register. The term "cash register" is an abbreviation of "cash register" and is used, for example, to settle the payment for purchased goods. For example, in the information processing system 3, a customer of the store places the items they wish to purchase on a display stand 10 as shown in Figure 2. The information processing system 3 takes an image of the items placed by the user. The information processing system 3 recognizes whether the objects included in the captured image are any of the items in the store. By recognizing the items, the information processing system 3 can calculate the total amount of the items to be charged to the user. Here, when an item is photographed using an imaging device such as a camera, multiple items may be placed on the display stand 10. The information processing system 3 is not limited to an unmanned cash register, but may also be a manned cash register or a semi-self-checkout register.

[0015] Information processing system 3 is configured as a cash register for a POS system. Information processing system 3 transmits the processing results, such as the total price of the goods, information on individual goods, and the price of those goods, to server 4 via network 2.

[0016] Server 4 receives processing results from information processing system 3 via network 2. Based on the received processing results, Server 4 manages the inventory status of the stores where information processing system 3 is installed.

[0017] As shown in Figure 2, the information processing system 3 comprises a first imaging unit 12 and an information processing device 20. The information processing system 3 may further comprise a mounting base 10, a support column 11, a display unit 13, and a second imaging unit 14. In this embodiment, the information processing device 20 is configured as a separate device from the first imaging unit 12 and the display unit 13. However, the information processing device 20 may be integrally configured with, for example, at least one of the display unit 13, the first imaging unit 12, the support column 11, the mounting base 10, and the second imaging unit 14.

[0018] The display stand 10 includes an upper surface 10s. The upper surface 10s can be used to place items that the user wishes to purchase. In this embodiment, the user places only the items they wish to purchase on the upper surface 10s, but is not limited to this, and may place items in a shopping basket or shopping bag. Also, in this embodiment, the upper surface 10s is substantially rectangular, but is not limited to this, and may have any shape.

[0019] The support column 11 supports the first imaging unit 12. The support column 11 extends from the side of the mounting table 10 upward toward the upper surface 10s. Note that the method of supporting the first imaging unit 12 is not limited to the support column 11. The support column 11 may support the first imaging unit 12 in any way that allows imaging of at least a portion of the upper surface 10s of the mounting table 10.

[0020] The first imaging unit 12 is capable of generating images through imaging. The first imaging unit 12 is fixed to the support column 11 so as to be able to image at least a portion of the surface of the mounting table 10, for example, at least a portion of the upper surface 10s. The first imaging unit 12 may be fixed so that its optical axis is perpendicular to the upper surface 10s. For example, the first imaging unit 12 is capable of imaging the entire upper surface 10s of the mounting table 10, and is fixed so that its optical axis is perpendicular to the upper surface 10s. The first imaging unit 12 may be fixed to the tip of the support column 11. The first imaging unit 12 may perform imaging continuously at any frame rate. The first imaging unit 12 may capture moving images. As an example, the first imaging unit 12 may be a camera.

[0021] The display unit 13 may be any display. The display unit 13 displays images transmitted from the information processing device 20. The display unit 13 may function as a touchscreen for the input unit described later.

[0022] The second imaging unit 14 is capable of generating images through imaging. The second imaging unit 14 is fixed to the ceiling above the mounting table 10 so as to be able to image the mounting table 10 and its surroundings. The images captured by the second imaging unit 14 include the user in front of the mounting table 10 and other users lined up to use the information processing system 3. The second imaging unit 14 may image the mounting table 10 from a different angle than the first imaging unit 12. By image the mounting table 10 from multiple angles, it may be possible to recognize items in the images captured by the second imaging unit 14 that cannot be recognized in the images captured by the first imaging unit 12. The second imaging unit 14 may continuously capture images at any frame rate. The second imaging unit 14 may capture moving images. As an example, the second imaging unit 14 may be a camera.

[0023] As shown in Figure 3, the information processing device 20 is composed of a communication unit 21, an input unit 22, a storage unit 23, and a control unit 24.

[0024] The communication unit 21 includes at least one communication module that can connect to the network 2. The communication module may be, for example, a communication module that complies with standards such as wired LAN (Local Area Network) or wireless LAN. The communication unit 21 is connected to the network 2 via the wired LAN or wireless LAN by this communication module.

[0025] The communication unit 21 includes a communication module capable of communicating with the first imaging unit 12, the display unit 13, and the second imaging unit 14. The communication module is a communication module that conforms to the communication line standard. Communication by the communication unit 21 may be performed by wired or wireless means. The communication unit 21 functions as an acquisition unit that acquires an image of an object placed on the upper surface 10s of the mounting table 10 from at least one of the first imaging unit 12 and the second imaging unit 14 using this communication module. The communication unit 21 may also communicate with the weight sensor 81, which will be described later, using this communication module.

[0026] The input unit 22 is capable of receiving input from the user. The input unit 22 includes at least one input interface capable of receiving input from the user. The input interface is, for example, a physical key, a capacitive key, a pointing device, a microphone, or a touchscreen integrated with the display unit 13. In this embodiment, the input unit 22 acquires input from a touchscreen integrated with the display unit 13.

[0027] The storage unit 23 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The ROM is, for example, EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 23 may function as a main memory, auxiliary memory, or cache memory. The storage unit 23 stores data used in the operation of the information processing device 20 and data obtained by the operation of the information processing device 20.

[0028] The control unit 24 is comprised of at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 24 controls each part of the information processing device 20 and executes processes related to the operation of the information processing device 20.

[0029] The control unit 24 obtains the input unit 22's selection of either an object included in the image or the recognition result of an object. This selection is input from the input unit 22 by, for example, a user who wants to check the recognition result of an object. Upon receiving this selection, the control unit 24 transmits a signal to the display unit 13 via the communication unit 21, highlighting the object included in the image or the recognition result of an object that was not selected by the user. If the display unit 13 and the information processing device 20 are configured as a single device, the control unit 24 may transmit the signal to the display unit 13 without going through the communication unit 21.

[0030] (Display processing) The process from the start of accounting to the highlighting and display of the recognition results will be explained below using Figures 4, 5, and 6.

[0031] The control unit 24 can receive an input from the input unit 22 instructing the start of accounting. This input may be entered from the input unit 22 by a user who wants to account for goods. When the user enters this input from the input unit 22, they place the goods they want to purchase on the upper surface 10s of the display stand 10. The user may also enter an input from the input unit 22 instructing the start of accounting after placing the goods on the upper surface 10s of the display stand 10.

[0032] The control unit 24 acquires the captured image captured by the first imaging unit 12 or the second imaging unit 14. The control unit 24 may acquire the captured image captured by both the first imaging unit 12 and the second imaging unit 14. In this embodiment, the control unit 24 acquires the captured image by receiving the image captured by the first imaging unit 12 via the communication unit 21. This captured image includes an image of an item placed on the upper surface 10s of the mounting table 10 by the user.

[0033] The control unit 24 recognizes which article an object in the acquired image belongs to. In this embodiment, the control unit 24 recognizes which article an object belongs to by performing object recognition processing on the image. Object recognition processing is the process of detecting an image of an object in the image and identifying which article the object belongs to. The control unit 24 may perform object recognition processing using a learning model generated by machine learning such as deep learning. In this embodiment, the control unit 24 identifies the article name of the object through object recognition and obtains the recognition result. Once the article name is identified, the control unit 24 may identify the number of recognized articles. In this embodiment, the recognition result of the object recognition processing includes data on the article name and the number of recognized articles. Note that the object recognition processing is not limited to being performed by the control unit 24 of the information processing device 20, but may be performed by an external server or the like. When the object recognition processing is performed by an external server, more specifically, the external server receives the image captured via the communication unit 21, and the server performs the object recognition processing. When object recognition processing is performed by an external server, the control unit 24 obtains the recognition results from the external server via the communication unit 21. Furthermore, object recognition only requires the identification of information such as an ID that can identify an item, rather than the item name. In this case, the control unit 24 may use a correspondence table or the like stored in the storage unit 23 to obtain the item name corresponding to the identified ID.

[0034] The control unit 24 outputs the recognition result to the display unit 13. In this embodiment, the control unit 24 notifies the user of the recognition result by outputting an image showing the recognition result to the display unit 13. In this embodiment, the control unit 24 outputs the recognition result to the display unit 13 when it recognizes which item an object included in the captured image is, but is not limited to this, and may also output the recognition result to the display unit 13 when the input unit 22 receives an instruction from the user to display a confirmation screen. The instruction to display a confirmation screen may be the action of pressing a button to proceed to the confirmation screen.

[0035] For example, as shown in Figure 4, the control unit 24 displays image 30 on the display unit 13. Image 30 includes image 31 and image 32. Image 31 may be displayed on the left side of the display unit 13, and image 32 may be displayed on the right side of the display unit 13. Image 31 is an image captured by the first imaging unit 12. Image 31 includes items placed on the platform 10 by the user. Image 32 displays the names of the items, which are the recognition results of the items placed on the platform 10 by the user. In this case, image 32 may also display information such as the number of items or the price of the items along with the item names. Furthermore, the item names displayed in image 32 may be displayed in the order in which they were recognized by the information processing device 20. The item names displayed in image 32 may be displayed in descending order of price, from highest to lowest. In this case, it becomes easier for the user to identify items that have a large impact on the total amount. The item names displayed in image 32 may be displayed in descending order of recognition accuracy, from highest to lowest. In this case, the user can more easily identify items that have a significant impact on the accuracy of the total amount. The item names displayed in Image 32 may be displayed in descending order of size or quantity within Image 31. In this case, the user can more easily identify large, conspicuous items or small, hard-to-find items first. The item names displayed in Image 32 may be displayed in descending order of quantity or quantity of the same item. In this case, the user can more easily identify items that were purchased in large quantities first. The display order of the item names in Image 32 may be changed by the user through operations on the input unit according to any criteria. Information regarding the price of items may be stored in the storage unit 23 in advance, associated with the item names.

[0036] When an image 30, as shown in Figure 4, is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. To confirm whether the recognition result displayed on image 32 is correct, the user may select the first item name 32a using the cursor 33, as shown in Figure 5. Selecting the first item name 32a may require the user to place the cursor 33 over the first item name 32a and then press a button. When the user selects the first item name 32a, an emphasis image 34a is displayed superimposed on the first item image 31a, which is an image of the item corresponding to the first item name 32a. The emphasis image 34a may be, for example, an image that forms a frame along the outline of the first item image 31a. With the display of this emphasis image 34a, the first item image 31a is highlighted relative to other item images, and the user can confirm the correspondence between the highlighted item image and the first item name 32a. In this embodiment, the user selects the first item name 32a using the cursor 33, but is not limited to this. The user may also select the first item name 32a by directly touching it on the touchscreen, or by pointing to the first item name 32a included in the image 32 with their hand. When using a touchscreen, the cursor 33 does not need to be displayed. In this embodiment, the highlighted image 34a is displayed to match the outline of the corresponding first item image 31a, but is not limited to this. It may overlap the entire first item image 31a. For example, it may be the bounding rectangle of the first item image 31a. In this embodiment, the highlighted image 34a is the same size as the first item image 31a, but is not limited to this. An image larger or smaller than the first item image 31a may be displayed.

[0037] After the user confirms the first item image 31a highlighted by the highlighting image 34a, they select a second item name 32b, which is another item, using the cursor 33, as shown in Figure 6. At this time, the control unit 24 erases the highlighting image 34a that is superimposed on the first item image 31a that has already been confirmed, and superimposes the highlighting image 34b onto the second item image 31b that corresponds to the newly selected second item name 32b. At this time, the control unit 24 may change the position of the first item name 32a and display it at the bottom of the column. Alternatively, the control unit 24 may divide the area of ​​the image 32 into a first area and a second area in the vertical direction, displaying item names 32 that have not been selected in the first area and moving the selected first item name 32a to the second area. By distinguishing between item names 32 that have been selected and item names 32 that have not been selected, it becomes easier for the user to understand which item names 32 should be confirmed. The timing for deleting the superimposed highlight image 34a is not limited to when a second item name 32b is newly selected. The superimposed highlight image 34a may be deleted when the user moves the cursor that selects the first item name 32a away from the first item name 32a. The superimposed highlight image 34a may be deleted when the user presses the button to delete the display of the highlight image 34a. The superimposed highlight image 34a may be deleted when the user presses the button to proceed to the settlement process for settling the price of the items. The superimposed highlight image 34a may be deleted after a predetermined time has elapsed since the user selected the first item name 32a. In addition, in the example shown in Figure 6, the second item image 31b is located below the item in the first item image 31a. Therefore, part of the second item image 31b is not visible due to the first item image 31a. In such cases, as an enhanced image, the outline of the second item image 31b may be drawn in the area where the second item image 31b and the first item image 31a overlap. Alternatively, the second item image 31b', which was taken before the item was placed on the mounting table 10, may be superimposed on the second item image 31b and displayed.

[0038] In this way, by highlighting and displaying the corresponding first item image 31a or second item image 31b, the user can easily compare the recognition result with the item placed on the mounting table 10. With this configuration, the user can easily verify whether or not the recognition result is correct.

[0039] (Correction process) The process of correcting incorrect recognition results for the displayed items will be explained using Figures 7 and 8.

[0040] When the user selects an incorrectly recognized item name 32c' displayed on the display unit 13, the information processing device 20 superimposes an enhanced image 34c onto the third item image 31c that it determines corresponds to the item name 32c'. At this time, the item name corresponding to the third item image 31c is correct as the third item name 32c, and incorrect as the third item name 32c'. When the user notices the incorrect display of the third item name 32c', the user may make an input to correct the incorrect third item name 32c'. The input to correct the incorrect third item name 32c' may be an input to select the incorrect third item name 32c' again, an input to select the incorrect third item name 32c' for a predetermined time or longer, an input to correct the item name included in the information processing device 20 or image 30, or a selection of the third item image 31c that is considered to correspond to the incorrect third item name 32c'.

[0041] When the control unit 24 receives input from the user to correct an incorrect third item name 32c′, it displays an image 35 on the display unit 13, as shown in Figure 8. Image 35 contains multiple item names that are presumed to correspond to the third item image 31c, as correction candidates 35a, 35b, and 35c. Of these correction candidates, correction candidate 35a is identical to the third item name 32c that correctly corresponds to the third item image 31c. The correction candidates 35a, 35b, and 35c may be displayed in order of increasing or decreasing likelihood of recognition, alphabetical order of item names, or order of increasing or decreasing price of items. If the user determines that correction candidate 35a is the correct third item name 32c, they input to select correction candidate 35a in image 35. When the control unit 24 receives input to select correction candidate 35a included in image 35, it corrects the incorrect third item name 32c′ included in image 32 to correction candidate 35a. When the control unit 24 receives input to select a correction candidate 35a, it may send an instruction to output an image and sound confirming the change from the third item name 32c' to the correction candidate 35a. If the correction candidates 35a, 35b, and 35c do not include the item name corresponding to the second item image 31b, the user may directly input the corresponding item name.

[0042] (System operation) Figure 9 is a flowchart showing the procedure of an information processing method according to one embodiment of the present disclosure. This information processing method includes steps S101 to S112. For example, when the control unit 24 receives an input from the input unit 22 instructing the start of accounting, it starts the processing of step S101.

[0043] The control unit 24 acquires image data by receiving the image from the first imaging unit 12 via the communication unit 21 (step S101). The control unit 24 performs object recognition processing on the image acquired in step S101 (step S102). The control unit 24 acquires the recognition result by performing the object recognition processing.

[0044] The control unit 24 displays an image representing the recognition result obtained in step S102 on the display unit 13 (step S103). For example, the control unit 24 displays an image 30, as shown in Figure 4, on the display unit 13.

[0045] The control unit 24 determines whether or not it has received an input from the input unit 22 to select the first item name 32a in the image 32 (step S104).

[0046] If the control unit 24 determines that it has received an input from the input unit 22 to select the first item name 32a (step S104: YES), it proceeds to step S105. For example, if the control unit 24 receives a touch input for the first item name 32a from the input unit 22, as shown in Figure 4, it determines that it has received an input from the input unit 22 to select the first item name 32a.

[0047] If the control unit 24 determines that it did not receive an input from the input unit 22 to select the first item name 32a (step S104: NO), it proceeds to the process in step S108. For example, if the control unit 24 receives an input from the user from the input unit 22 to proceed to the settlement process, it may determine that it did not receive an input from the input unit 22 to select the first item name 32a. Furthermore, if a predetermined time elapses without any operation after displaying an image 30 as shown in Figure 4, the control unit 24 may determine that it did not receive an input from the input unit 22 to select the first item name 32a.

[0048] In step S105, the control unit 24 sends an instruction to the display unit 13 to display an enhanced image 34a superimposed on the first item image 31a, based on the input from the user to select a first item name 32a received from the input unit 22.

[0049] The control unit 24 determines whether or not it has received an input from the input unit 22 to select the second item name 32b in the image 32 (step S106).

[0050] If the control unit 24 determines that it has received an input from the input unit 22 to select the second item name 32b (step S106: YES), it proceeds to the process in step S107. For example, if the control unit 24 receives a touch input for the second item name 32b as shown in Figure 6 from the input unit 22, it determines that it has received an input from the input unit 22 to select the second item name 32b.

[0051] If the control unit 24 determines that the input unit 22 did not receive an input to select the second item name 32b (step S106: NO), it proceeds to step S112. For example, if the control unit 24 receives an input from the user to proceed to payment processing via the input unit 22, it determines that the input unit 22 did not receive an input to select the second item name 32b. In addition, if a predetermined time elapses without any operation being performed, the control unit 24 may determine that the input unit 22 did not receive an input to select the second item name 32b.

[0052] In step S107, the control unit 24 sends an instruction to the display unit 13 to delete the enhanced image 34a superimposed on the already displayed first item image 31a, based on the input from the user to select the second item name 32b received from the input unit 22. After that, the control unit 24 returns to the process of step S105.

[0053] The control unit 24 determines whether or not it has received input from the input unit 22 to modify the third item name 32c' in image 32 (step S108).

[0054] If the control unit 24 determines that it has received an input from the input unit 22 to modify the third item name 32c' (step S108: YES), it proceeds to step S109. For example, if the control unit 24 receives a long-press touch input for the third item name 32c' exceeding a predetermined time, as shown in Figure 7, from the input unit 22, it determines that it has received an input from the input unit 22 to modify the third item name 32c'.

[0055] If the control unit 24 determines that it did not receive an input from the input unit 22 to modify the third item name 32c' (step S108: NO), it proceeds to step S112. For example, if the control unit 24 receives an input from the input unit 22 to proceed to payment processing by the user, it determines that it did not receive an input from the input unit 22 to modify the third item name 32c'. In addition, if a predetermined time has elapsed without any operation by the user, the control unit 24 may determine that it did not receive an input from the input unit 22 to modify the third item name 32c'.

[0056] In step S109, the control unit 24 displays image 35. Image 35 includes the names of items that are presumed to correspond to the third item image 31c as modification candidates 35a, 35b, and 35c.

[0057] The control unit 24 receives user input as modification input via the input unit 22 (step S110) for selecting modification candidates included in the image 35 displayed in step S109. The modification input is not limited to the selection of modification candidates, but may also be the user's input of an item name, etc.

[0058] Based on the modification input entered in step S110, the control unit 24 modifies the third item image 31c and the corresponding item name 31c' (step S111). After that, the control unit 24 returns to the process in step S103.

[0059] In step S112, a settlement process is executed to settle the price of the goods based on the recognition result from step S102.

[0060] Thus, in the information processing device 20, the control unit 24 does not immediately execute the settlement process after displaying the recognition result to the user via the display unit 13, but instead highlights the item image to confirm the recognized item. With this configuration, the user can smoothly confirm whether the displayed item image has been correctly recognized. In addition, the user can perform the settlement process quickly.

[0061] (Example of display processing 1) The first variation of the display process is described below. Note that processes that overlap with the display process described above are omitted.

[0062] As shown in Figure 10, the control unit 24 displays the image 30 on the display unit 13. In this case, unlike in Figure 4, the control unit 24 may display only one item name along with the number of items, rather than displaying multiple identical item names. For example, if two anpan (sweet bean paste buns) are recognized, the control unit 24 may display that there are two anpan. The control unit 24 may also display the number of items along with the name of a higher-level item. For example, if anpan and cream buns are recognized, the control unit 24 may display that there are two sweet buns or two breads, which are higher-level items.

[0063] When an image 30, as shown in Figure 10, is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. At this time, in order to check whether the recognition result displayed on image 32 is correct, the user selects the first item name 32a using the cursor 33, as shown in Figure 11. In this modified example, one first item name 32a is displayed, indicating that there are two corresponding items. When the user selects the first item name 32a, enhanced images 34a and 34a' are displayed superimposed on the first item images 31a and 31a' corresponding to the first item name 32a. In this case, the highlighted images 34a and 34a' are displayed simultaneously, but are not limited to this. As shown in Figure 12, when the user selects a first item name 32a, the first item names 32a' and 32a'' corresponding to the first item name 32a may be displayed in image 30, and when the user selects a first item name 32a', the highlighted image 34a corresponding to the corresponding item image 32a may be displayed. For example, if the first item name 32a is anpan (sweet bean paste bun), then anpan may be displayed as both the first item name 32a' and 32a'', and the user By selecting either of the anpan (first item name 32a' or 32a'') displays, the corresponding anpan (first item image 31a, 31a') may be displayed with the emphasis image 34a superimposed. Also, if the first item name 32a is a broader concept of sweet bread, the first item name 32a' may be anpan and the first item name 32a'' may be a cream bun, and by the user selecting anpan (first item name 32a'), the anpan (first item image 31a) may be displayed with the emphasis image 34a superimposed.

[0064] (Operation of the system in variation 1) Figure 13 is a flowchart showing the procedure of an information processing method according to Modification 1 of this disclosure. This information processing method includes steps S201 to S212. Note that descriptions that overlap with steps S101 to S112 described above are omitted.

[0065] Steps S201 to S204 are the same or similar to steps S101 to S104 described above, so their description is omitted.

[0066] In step S205, the control unit 24 sends an instruction to the display unit 13 to display enhanced images 34a and 34a' superimposed on the first item images 31a and 31a', respectively, based on the input from the user to select the first item name 32a received from the input unit 22.

[0067] Steps S206 to S212 are the same as steps S106 to S112 described above, so their description is omitted.

[0068] (Example of display processing 2) The second variation of the display process is described below. Processes that overlap with the display process described above will be omitted.

[0069] As shown in Figure 10, the control unit 24 displays the image 30 on the display unit 13. Details are omitted as they are described above.

[0070] When an image 30, as shown in Figure 10, is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. At this time, in order to check whether the recognition result displayed on image 32 is correct, the user selects the first item image 31a using the cursor 33, as shown in Figure 14. When the user selects the first item image 31a, an enhanced image 34a' superimposed on the first item image 31a' which is classified as the same item, and an enhanced image 34a superimposed on the first item name 32a are displayed. At this time, the enhanced image 34a' superimposed on the first item image 31a' does not need to be displayed. In addition, when the user selects the first item image 31a, in addition to the enhanced image 34a superimposed on the first item name 32a being displayed, an enhanced image 34a' may also be displayed on the corresponding first item name 32a' among the multiple first item names 32a' included in the first item name 32a, as shown in Figure 15. Furthermore, when a user selects a first item image 31a, the first item image 31a, the highlight images 34a, 34a' superimposed on the first item image 31a' of the same type of item, and the highlight images 34a, 35a', 35a'' superimposed on the first item names 32a, 32a', 32a'' corresponding to the first item images 31a, 31a' may be displayed, as shown in Figure 16. The first item names 32a' and 32a'' are the item names of the items included in the first item name 32a. In this case, the highlighting image 34a superimposed on the first item image 31a and the highlighting image 34a' superimposed on the first item name 32a' corresponding to the first item image 31a may be the same color. Furthermore, the highlighting image 34a' superimposed on the first item image 31a and the highlighting image 34a'' superimposed on the first item name 32a'' corresponding to the first item image 31a may be the same color. Moreover, the highlighting images 34a and 34a' may be different colors.

[0071] (System operation in modified example 2) Figure 17 is a flowchart showing the procedure of an information processing method according to Modification 1 of this disclosure. This information processing method includes steps S301 to S312. Note that descriptions that overlap with steps S201 to S212 described above are omitted.

[0072] Steps S301 to S303 are the same or similar to steps S201 to S203 described above, so their description is omitted.

[0073] The control unit 24 determines whether or not it has received an input from the input unit 22 to select the first item image 31a of image 31 (step S304).

[0074] If the control unit 24 determines that it has received an input from the input unit 22 to select the first item image 31a (step S304: YES), it proceeds to the process in step S305. For example, if the control unit 24 receives a touch input for the first item image 31a as shown in Figure 10 from the input unit 22, it determines that it has received an input from the input unit 22 to select the first item image 31a.

[0075] If the control unit 24 determines that it did not receive an input from the input unit 22 to select the first item image 31a (step S304: NO), it proceeds to the process in step S308. For example, if the control unit 24 receives an input from the user via the input unit 22 to proceed to the payment process, it may determine that it did not receive an input from the input unit 22 to select the first item image 31a. Furthermore, if a predetermined amount of time elapses without any operation after displaying an image 30 as shown in Figure 4, the control unit 24 may determine that it did not receive an input from the input unit 22 to select the first item image 31a.

[0076] In step S305, the control unit 24 sends an instruction to the display unit 13 to display an enhanced image 34a superimposed on the first item name 32a, based on the input from the user to select the first item image 31a received from the input unit 22.

[0077] The control unit 24 determines whether or not it has received an input from the input unit 22 to select the second item image 31b of image 31 (step S306).

[0078] If the control unit 24 determines that it has received an input from the input unit 22 to select the second item image 31b (step S306: YES), it proceeds to the process in step S307. For example, if the control unit 24 receives a touch input for the second item image 31b from the input unit 22, it determines that it has received an input from the input unit 22 to select the second item image 31b.

[0079] If the control unit 24 determines that the input unit 22 did not receive an input to select the second item image 31b (step S306: NO), it proceeds to step S312. For example, if the control unit 24 receives an input from the user to proceed to payment processing via the input unit 22, it determines that the input unit 22 did not receive an input to select the second item image 31b. In addition, if a predetermined time elapses without any operation being performed, the control unit 24 may determine that the input unit 22 did not receive an input to select the second item image 31b.

[0080] In step S307, the control unit 24 sends an instruction to the display unit 13 to delete the enhanced image 34a superimposed on the already displayed first item image 31a, based on the input from the user to select the second item image 31b received from the input unit 22. After that, the control unit 24 returns to the process of step S305.

[0081] The processing from step S308 to step S312 is the same as the processing from step S208 to step S212, so it is omitted from the description.

[0082] (Example of display processing 3) The third variation of the display process is described below. Processes that overlap with the display process described above will be omitted.

[0083] As shown in Figure 4, the control unit 24 displays the image 30 on the display unit 13. Details are omitted here as they are described above.

[0084] When an image 30, as shown in Figure 4, is displayed on the display unit 13, the user confirms the recognition result displayed on the display unit 13. At this time, in order to confirm whether the recognition result displayed on image 32 is correct, the user selects the first item image 31a using the cursor 33, as shown in Figure 18. When the user selects the first item image 31a, an emphasis image 34a is displayed superimposed on the first item name 32a corresponding to the first item image 31a. The display of the emphasis image 34a emphasizes the first item name 32a. When the first item name 32a is emphasized to the user, not only is the emphasis image 34a superimposed, but the font color, thickness, or size of the first item name 32a may also be changed. In this embodiment, the user selects the first item name 32a using the cursor 33, but is not limited to this, and the first item name 32a may be directly selected using the touchscreen. In this case, the cursor 33 does not need to be displayed.

[0085] In this way, by highlighting the corresponding first item name 32a, the user can easily compare the recognition result with the item placed on the mounting base 10. With this configuration, the user can easily verify whether the recognition result is correct or not.

[0086] (Operation of the system in variation 2) Figure 19 is a flowchart showing the procedure of an information processing method according to Modification 2 of this disclosure. This information processing method includes steps S401 to S412. Note that descriptions that overlap with steps S101 to S112 described above are omitted.

[0087] Steps S401 to S403 are identical or similar to steps S101 to S103 described above, so their description is omitted.

[0088] The control unit 24 determines whether or not it has received an input from the input unit 22 to select the first item image 31a of image 32 (step S404).

[0089] If the control unit 24 determines that it has received an input from the input unit 22 to select the first item image 31a (step S404: YES), it proceeds to the process in step S305. For example, if the control unit 24 receives a touch input for the first item name 32a, as shown in Figure 18, from the input unit 22, it determines that it has received an input from the input unit 22 to select the first item name 32a.

[0090] If the control unit 24 determines that it did not receive an input from the input unit 22 to select the first item image 31a (step S404: NO), it proceeds to the process in step S308. For example, if the control unit 24 receives an input from the input unit 22 to proceed to the payment process by the user, it determines that it did not receive an input from the input unit 22 to select the first item image 31a. However, the control unit 24 does not determine that it did not receive an input from the input unit 22 to select the first item image 31a, and may determine that a predetermined amount of time has elapsed after displaying the image 30 as shown in Figure 4.

[0091] In step S405, the control unit 24 sends an instruction to the display unit 13 to display an enhanced image 34a superimposed on the first item name 32a, based on the input from the user to select the first item image 31a received from the input unit 22.

[0092] The control unit 24 determines whether or not it has received an input from the input unit 22 to select the second item image 31b of image 32 (step S406).

[0093] If the control unit 24 determines that it has received an input from the input unit 22 to select the second item image 31b (step S406: YES), it proceeds to the process in step S307. For example, if the control unit 24 receives a touch input for the second item image 31b from the input unit 22, it determines that it has received an input from the input unit 22 to select the second item image 31b.

[0094] If the control unit 24 determines that it did not receive an input from the input unit 22 to select the second item image 31b (step S406: NO), it proceeds to step S208. For example, if the control unit 24 receives an input from the input unit 22 to proceed to payment processing, it determines that it did not receive an input from the input unit 22 to select the second item image 31b. However, the control unit 24 does not determine that it did not receive an input from the input unit 22 to select the second item image 31b, and may also determine that a predetermined amount of time has elapsed after displaying the image 30 as shown in Figure 18.

[0095] In step S407, the control unit 24 sends an instruction to the display unit 13 to delete the enhanced image 34a superimposed on the first item name 32a, based on the input from the user receiving the second item image 31b from the input unit 22. After that, the control unit 24 returns to the process of step S405.

[0096] The control unit 24 determines whether or not it has received an input from the input unit 22 to modify the third item name 32c' which has been determined to correspond to the third item image 31c (step S408).

[0097] If the control unit 24 determines that it has received an input from the input unit 22 to modify the name of the third item 32c' (step S408: YES), it proceeds to step S409. For example, if the control unit 24 receives a touch input from the input unit 22 to select the name of the third item 32c' on which the enhanced image 34c is superimposed, it determines that it has received an input from the input unit 22 to modify the name of the third item 32c'.

[0098] Steps S409 to S412 are omitted because they overlap with steps S109 to S112 described above.

[0099] (4 variations of the display process) The fourth variation of the display process is described below. Processes that overlap with the display process described above are omitted.

[0100] As shown in Figure 4, the control unit 24 displays the image 30 on the display unit 13. Details are omitted here as they are described above.

[0101] When an image 30, as shown in Figure 4, is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. At this time, in order to check whether the recognition result displayed on image 32 is correct, the user selects the first item image 31a using the cursor 33, as shown in Figure 20. When the user selects the first item image 31a, an enhanced image 34a superimposed on the first item image 31a and an enhanced image 34a superimposed on the first item name 32a corresponding to the first item image 31a are displayed. At this time, the enhanced images 34a and 34b are displayed in the same color, but are not limited to this.

[0102] As described above, by displaying simultaneously the enhanced image 34a superimposed on the first item image 31a and the enhanced image 34b superimposed on the first item name 32a corresponding to the item image 32a, the correspondence between the first item image 31a and the first item name 32a becomes easier to confirm.

[0103] (System operation of modified example 4) Figure 21 is a flowchart showing the procedure of the recognition method according to Modification 3 of this disclosure. This information processing method includes steps S501 to S512. Note that descriptions that overlap with steps S401 to S412 described above are omitted.

[0104] Steps S501 to S504 are identical or similar to steps S401 to S404 described above, therefore their description is omitted.

[0105] In step S505, the control unit 24, based on the input from the user receiving from the input unit 22 to select a first item image 31a, sends an instruction to the display unit 13 to display both an enhanced image 34a superimposed on the first item image 31a and an enhanced image 34a superimposed on the first item name 32a.

[0106] Step S506 is omitted because it overlaps with step S406.

[0107] In step S507, the control unit 24, based on the input from the user receiving the input from the input unit 22 to select a second item image 31b, sends an instruction to the display unit 13 to delete the enhanced image 34a superimposed on the first item image 31a and the enhanced image 34a superimposed on the first item name 32a. After that, the control unit 24 returns to the process of step S505.

[0108] Steps S508 to S512 are identical or similar to steps S408 to S412, and therefore their description is omitted.

[0109] (5 variations of the display process) The fifth variation of the display process is described below. Processes that overlap with the display process described above are omitted.

[0110] As shown in Figure 4, the control unit 24 displays the image 30 on the display unit 13. Details are omitted here as they are described above.

[0111] When an image 30, as shown in Figure 4, is displayed on the display unit 13, the user checks the recognition result displayed on the display unit 13. At this time, in order to check whether the recognition result displayed on image 32 is correct, the user selects the first item image 31a using the cursor 33, as shown in Figure 22. When the user selects the first item image 31a, an enhanced image 34a superimposed on the first item image 31a, an enhanced image 34a superimposed on the first item name 32a corresponding to the first item image 31a, and a correspondence line 36a connecting the first item image 31a and the first item name 32a are displayed. At this time, the enhanced images 34a and 34b are displayed in the same color, but are not limited to this. Also, in this embodiment, the correspondence line 36a is drawn as a solid line, but is not limited to this, and may be a dotted line or the like.

[0112] As described above, the simultaneous display of the enhanced image 34a superimposed on the first item image 31a, the enhanced image 34b superimposed on the first item name 32a corresponding to the item image 32a, and the correspondence line 36a connecting the first item image 31a and the first item name 32a makes it easier to confirm the correspondence between the first item image 31a and the first item name 32a.

[0113] (System operation of modified example 5) Figure 23 is a flowchart showing the procedure of the recognition method according to Modification 4 of this disclosure. This information processing method includes steps S601 to S612. Note that descriptions that overlap with steps S501 to S512 described above are omitted.

[0114] Steps S601 to S604 are identical or similar to steps S501 to S504 described above, so their description is omitted.

[0115] In step S605, the control unit 24, based on the input from the user receiving from the input unit 22 to select a first item image 31a, sends an instruction to the display unit 13 to display an enhanced image 34a superimposed on the first item image 31a, an enhanced image 34a superimposed on the first item name 32a, and a correspondence line 36a connecting the first item image 31a and the first item name 32a.

[0116] Step S606 is omitted because it overlaps with step S506.

[0117] In step S607, the control unit 24, based on the input from the user receiving from the input unit 22 to select a second item image 31b, sends an instruction to the display unit 13 to delete the enhanced image 34a superimposed on the first item image 31a, the enhanced image 34a superimposed on the first item name 32a, and the correspondence line 36a connecting the first item image 31a and the first item name 32a. After that, the control unit 24 returns to the process of step S605.

[0118] Steps S608 to S612 are identical or similar to steps S508 to S512, and therefore their description is omitted.

[0119] (Example of display processing 6) The sixth variation of the display process is described below. Processes that overlap with the display process described above are omitted.

[0120] As shown in Figure 4, the control unit 24 displays the image 30 on the display unit 13. Details are omitted here as they are described above.

[0121] When the user sees an image 30, as shown in Figure 4, displayed on the display unit 13, they verify the recognition result displayed on the display unit 13. At this time, in order to verify whether the recognition result displayed on image 32 is correct, the user selects the first item image 31a using the cursor 33, as shown in Figure 24. When the user selects the first item image 31a, an enhanced image 34a superimposed on the first item image 31a and an enhanced image 34a superimposed on the first item name 32a corresponding to the first item image 31a are displayed. Details are as described above and will be omitted here.

[0122] After the user confirms the first item name 32a highlighted by the highlighted image 34a, they select a second item image 31b, which is another item, using the cursor 33, as shown in Figure 20. At this time, the control unit 24 changes the color of the highlighted image 34a superimposed on the already confirmed first item image 31a and the highlighted image 34a superimposed on the first item name 32a, and superimposes the highlighted image 34b on the newly selected second item image 31b and on the second item name 32b corresponding to the second item image 31b. At this time, the highlighted image 34a and the highlighted image 34b may be displayed in different colors. They may be displayed in the same or similar colors. By making the display forms of the highlighted image 34a and the highlighted image 34b different, the user can easily recognize, for example, that the first item name 32a has already been confirmed.

[0123] After the user has confirmed the first item image 31a and the first item name 32a, when they select a new second item image 31b, the first item image 31a and the first item name 32a will be highlighted in a different color than the second item image 31b. This makes it easy to determine whether the displayed item has already been confirmed or not.

[0124] (System operation of modified example 6) Figure 25 is a flowchart showing the procedure of the recognition method according to Modification 6 of this disclosure. This information processing method includes steps S701 to S712. Note that descriptions that overlap with steps S501 to S512 described above are omitted.

[0125] Steps S701 to S706 are identical or similar to steps S501 to S506 described above, therefore their description is omitted.

[0126] In step S707, the control unit 24, based on the input from the user receiving the input from the input unit 22 to select a second item image 31b, sends an instruction to the display unit 13 to display the enhanced image 34a superimposed on the first item image 31a and the enhanced image 34a superimposed on the first item name 32a in a color different from the color displayed in step S705. After that, the control unit 24 returns to the process of step S705.

[0127] Steps S708 to S712 are identical or similar to steps S708 to S712, therefore their description is omitted.

[0128] (Variation of display processing 7) The seventh variation of the display process is described below. Processes that overlap with the display process described above are omitted.

[0129] As shown in Figure 4, the control unit 24 displays the image 30 on the display unit 13. Details are omitted here as they are described above.

[0130] When the user sees an image 30, as shown in Figure 4, displayed on the display unit 13, they verify the recognition result displayed on the display unit 13. At this time, in order to verify whether the recognition result displayed on image 32 is correct, the user selects the first item image 31a using the cursor 33, as shown in Figure 26. When the user selects the first item image 31a, an enhanced image 34a is displayed superimposed on the first item name 32a corresponding to the first item image 31a. When the enhanced image 34a is displayed, the registered image 37a corresponding to the first item image 31a is also displayed. The registered image 37a is an image of an item stored in the storage unit 23. In this embodiment, the registered image 37a is displayed in the lower right of image 30, but is not limited to this. Also, in Figure 26, the registered image 37a is displayed together with the enhanced image 34a, but is not limited to this, and may be displayed together with the enhanced image 34a. Furthermore, the registered image 37a may be changed according to the orientation or tilt of the first item image 31a, so that the registered image 37a is displayed in the same orientation or tilt as the first item image 31a.

[0131] In this way, displaying the registered image 37a in addition to the enhanced image 34a improves the accuracy of user recognition results.

[0132] (System operation of modified example 7) Figure 27 is a flowchart showing the procedure of the recognition method according to Modification 7 of this disclosure. This information processing method includes steps S801 to S812. Note that descriptions that overlap with steps S401 to S412 described above are omitted.

[0133] Steps S801 to S804 are identical or similar to steps S401 to S404 described above, so their description is omitted.

[0134] In step S805, the control unit 24, based on the input from the user receiving from the input unit 22 to select a first item image 31a, sends an instruction to the display unit 13 to display both an enhanced image 34a superimposed on the first item name 32a corresponding to the first item image 31a, and a registered image 37a corresponding to the first item image 31a.

[0135] Step S806 is omitted because it overlaps with step S406.

[0136] In step S807, the control unit 24, based on the input from the user receiving the input from the input unit 22 to select the second item image 31b, sends an instruction to the display unit 13 to delete the enhanced image 34a and the registered image 37a superimposed on the first item name 32a. After that, the control unit 24 returns to the process of step S805.

[0137] Steps S808 to S812 are identical or similar to steps S408 to S412, and therefore their description is omitted.

[0138] (8 variations of the display process) The eighth variation of the display process is described below. Processes that overlap with the display process described above are omitted.

[0139] As shown in Figure 4, the control unit 24 displays the image 30 on the display unit 13. Details are omitted here as they are described above.

[0140] When the user sees an image 30, as shown in Figure 4, displayed on the display unit 13, they verify the recognition result displayed on the display unit 13. At this time, in order to verify whether the recognition result displayed on image 32 is correct, the user selects a second item name 32b using a cursor 33, as shown in Figure 28. When the user selects a second item name 32b, a registered image 37b corresponding to the second item name 32b is displayed, superimposed on the second item image 31b corresponding to the second item name 32b. In this embodiment, the second item image 31b overlaps with another first item image 31a, and the second item image 31b is located below it. At this time, the registered image 37b superimposes on the corresponding second item image 31b, and also on the portion where the first item image 31a and the second item image 31b overlap.

[0141] In this way, the registered image 37b is superimposed on the second item image 31b, allowing for smooth verification even for items where the second item images 31b overlap and it would be difficult to confirm the correctness of the item.

[0142] (Operation of the system in variation 8) Figure 29 is a flowchart showing the procedure of an information processing method according to Modification 7 of this disclosure. This information processing method includes steps S901 to S912. Note that descriptions that overlap with steps S101 to S112 described above are omitted.

[0143] Steps S901 to S904 are omitted from this description as they overlap with steps S101 to S104 described above.

[0144] In step S905, the control unit 24 sends an instruction to the display unit 13 to display a registered image 37a superimposed on the first item image 31a, based on the input from the user to select a first item name 32a received from the input unit 22.

[0145] Step S906 is omitted because it overlaps with step S106.

[0146] In step S907, the control unit 24 sends an instruction to the display unit 13 to delete the registered image 37a superimposed on the first item image 31a, based on the input from the user receiving the second item name 32b from the input unit 22. After that, the control unit 24 returns to the process of step S905.

[0147] Steps S908 to S912 are omitted because they overlap with steps S108 to S112.

[0148] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications, alterations, or combinations of embodiments of this disclosure are included within the scope of this disclosure.

[0149] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, Article 1 may swap the identifiers "First" and "Second" with Article 2. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers.

[0150] In one embodiment, (1) the information processing device is An acquisition unit acquires an image containing a first image of a first article, which is an image of the first article. The system comprises a control unit that causes the first article image and the recognition result of the first article to be displayed on a display unit, When the control unit receives a selection from the input unit for either the first item image or the recognition result of the first item, it highlights and displays the other of the first item image and the recognition result of the first item that was not selected.

[0151] (2) In the information processing device described in (1) above, The captured image includes a second article image, which is an image of a second article. The control unit causes the second article image and the recognition result of the second article to be displayed on the display unit. After the control unit has highlighted and displayed either the first item image or the recognition result of the first item, when it receives a selection from the input unit for either the second item image or the recognition result of the second item, it deletes the display highlighting the first item image or the recognition result of the first item, and highlights and displays the other of the second item image and the recognition result of the second item that was not selected.

[0152] (3) In the information processing device described in (2) above, After the control unit has highlighted and displayed either the first item image or the recognition result of the first item, when it receives input to the input unit for a new selection of either the second item image or the recognition result of the second item, it changes the display format of the highlighting of the first item image or the recognition result of the first item, and highlights and displays the other of the second item and the recognition result of the second item that has not been selected.

[0153] (4) In the information processing devices described in (1) to (3) above, The control unit displays either the first item image or the recognition result of the first item, along with a line connecting the first item image and the recognition result of the first item.

[0154] (5) The information processing devices described in (1) to (3) above are: The system further comprises a storage unit for storing registered images of the aforementioned articles, The control unit further displays the registered image of the first item on the display unit, along with a display that emphasizes either the first item image or the recognition result of the first item.

[0155] (6) In the information processing device described in (5) above, The control unit overlays the registered image onto the first item image and displays it on the display unit.

[0156] (7) In the information processing apparatus described in (6) above, The control unit displays the registered image corresponding to the orientation of the article that corresponds to the first article image.

[0157] (8) In the information processing devices described in (1) to (4) above, The captured image includes a plurality of images of the first article, The control unit displays the recognition result of the first article and the number of the first article recognized from the captured image on the display unit.

[0158] (9) In the information processing device described in (8) above, When the control unit receives a selection of the recognition result for the first item from the input unit, it further displays each of the multiple recognition results for the first item on the display unit.

[0159] (10) In the information processing devices described in (1) to (9) above, The captured image includes a second article image, which is an image of a second article that is the same type of article as the first article. The control unit causes the display unit to display the item name, which is a higher-level concept for the first item and the second item.

[0160] (11) In the information processing devices described in (1) to (10) above, When the control unit receives a selection from the input unit for either the first item image or the recognition result of the first item, it displays both the first item image and the recognition result of the first item with emphasis.

[0161] (12) In the information processing devices described in (1) to (11) above, The control unit displays correction candidates for correcting the recognition result of the first item.

[0162] (13) The information processing method in one embodiment is: Obtain an image containing at least one item, To recognize which item is included in the aforementioned image, The recognized recognition result and the aforementioned image are output. When the system receives input from a user selecting either the outputted recognition result or the item, it outputs a message that highlights the other item that the user has not selected.

[0163] In one embodiment, (14) the information processing device is The system includes a display unit that displays an image containing a first item and the recognition result of the first item, and when either the first item or the recognition result is selected by the user, it highlights and displays the other of the first item and the recognition result that has not been selected. [Explanation of Symbols]

[0164] 1. Settlement System 2 Network 3. Information Processing Systems 4 servers 10 Mounting platform 10s top 11 Support pillar 12 Imaging Unit 13 Display section 14 Imaging device 20 Information Processing Devices 21 Communications Department 22 Input section 23 Memory section 24 Control Unit 34 34a 34b Enhanced image 35 35a 35b Enhanced image 36 36a Corresponding line 37 37a Registered image

Claims

1. An acquisition unit acquires an image containing a first article image, which is an image of the first article. The system comprises a control unit that displays the first article image and the recognition result of the first article on a display unit, The control unit, When the input unit receives the selection of the first item image, the first item image is highlighted and displayed. When the control unit determines that the user has confirmed the recognition result of the first item, it changes the display mode of the image of the first item. Information processing device.

2. The system further includes a storage unit for storing a registered image of the first item, When the control unit receives the selection of the first item image from the input unit, it highlights and displays the first item image, and also displays the registered image of the first item on the display unit. The information processing apparatus according to claim 1.

Images