Information processing systems, information processing methods, information processing devices

Abstract

Conventional technology has room for improvement in terms of user convenience when registering products that users wish to purchase. [Solution] An information processing system according to one embodiment of the present disclosure includes: an acquisition unit that acquires a first image including an image of an object; a control unit that recognizes the object and acquires information about the object; a display unit that includes a first region for displaying the first image, a second region for displaying information about the object, and a third region located between the first region and the second region. When the image of the object moves from the first region to the third region, the control unit causes the information about the object to be displayed in the second region.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, for the settlement processing of product prices, a technique for imaging an object with an imaging device or the like and recognizing which product the object is has been known. For example, Patent Document 1 discloses an article recognition apparatus including an image interface that acquires an image obtained by photographing a predetermined location where a plurality of articles are arranged.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, there is room for improvement in terms of the convenience in registering products that a user wishes to purchase.

[0005] In view of the above-described conventional circumstances, the present disclosure aims to provide an information processing system that can improve user convenience.

Means for Solving the Problems

[0006] An information processing system according to an embodiment of the present disclosure includes an acquisition unit that acquires a first image including an image of an object, a control unit that acquires information on the object based on the first image, and a display unit including a first area for displaying the first image, a second area, and a third area adjacent to the first area, where the control unit causes the information on the object moved from the first area to the third area to be displayed in the second area.

[0007] An information processing method according to one embodiment of this disclosure is: A step of acquiring a first image containing an image of an object, A step of recognizing the object and acquiring information about the object, The process includes the step of displaying information about the object in the second region when the image of the object moves from the first region where the first image is displayed to the third region located between the first region and the second region where information about the object is displayed.

[0008] An information processing apparatus according to one embodiment of this disclosure is An acquisition unit that acquires a first image containing an image of an object, A control unit that acquires information about the object based on the first image, The system comprises a first region for displaying a first image, a second region, and a display unit including a third region adjacent to the first region. The control unit displays information about the object that has moved from the first region to the third region in the second region. [Effects of the Invention]

[0009] According to one embodiment of this disclosure, user convenience can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic configuration of a settlement system according to one embodiment of the present disclosure. [Figure 2] Figure 1 shows the schematic configuration of the information processing system. [Figure 3] Figure 2 is a block diagram of the information processing device shown. [Figure 4] A mounting platform on which a container is placed, according to one embodiment of the present disclosure. [Figure 5] This is an image acquired by the imaging unit. [Figure 6] This shows the platform used during object category recognition processing. [Figure 7]This is an example of a captured image 31 that includes an object lifted by a user. [Figure 8] This is an image with the name of the object displayed. [Figure 9] This is an image of a purchase list with the name of the object displayed. [Figure 10] This is an example in which when an input indicating misrecognition is received, candidates for the correct object are displayed on the display unit 13. [Figure 11] This is an example of an image for displaying a purchase list in which the object is registered. [Figure 12] This is an image prompting the user to photograph the object again. [Figure 13] This is an image after all the objects contained in the container have been recognized. [Figure 14] This is an image prompting confirmation of the presence or absence of unrecognized objects. [Figure 15] This is a flowchart showing the procedure of the information processing method according to an embodiment of the present disclosure. [[ID=2—6]] [Figure 16] This is an example of an information processing system when an object is placed on the upper surface of the mounting table. [Figure 17] This is an example of an image including an object placed on the upper surface of the mounting table. [Figure 18] This is an example of an image for notifying the user that unregistered products remain. [Figure 19] This is a flowchart showing the procedure of the information processing method according to Modification 1 of the present disclosure. [Figure 20] This is an example of an image including an object lifted by a user. [Figure 21] This is an example of the sixth image 237 prompting input of the number of the first object 241a. [Figure 22] This is an example of the second image 232 including information on the first object 241a. [Figure 23] This is a flowchart showing the procedure of the information processing method according to Modification 2 of the present disclosure. [Figure 24] This is an example of the first image 331 including the first object 341a to which discount information is attached. [Figure 25]This is an example of a second image 332 that includes the first object 341a. [Figure 26] This is an example of image 332 containing information about the second object 341b. [Figure 27] This is an example of a correction image 337 that prompts the user to input the items to be corrected. [Figure 28] This is an example of a third image 334b ​​prompting the user to take the discounted image 345b again. [Figure 29] This flowchart shows the procedure for the information processing method relating to Modification 3 of this disclosure. [Figure 30] This is an example image of what happens when the control unit receives an instruction to start accounting. [Figure 31] This is an example of an image displayed on the display unit when the recognition process is started. [Figure 32] This is an example of an image displayed on the display unit when an object is recognized. [Figure 33] This is a flowchart showing the procedure for the information processing method relating to Modification 4 of this disclosure. [Figure 34] This shows a platform on which a container holding an object is placed. [Modes for carrying out the invention]

[0011] The embodiments of this disclosure will be described below with reference to the drawings.

[0012] (System Configuration) Figure 1 illustrates the overview of settlement system 1. Settlement system 1 is, for example, a POS (P It may be a (Official of Sales) system. The settlement system 1 includes at least one information processing system 3 and a server 4. In this embodiment, the settlement system 1 may include a plurality of information processing systems 3.

[0013] Information processing system 3 may be installed in any store. For example, information processing system 3 may be installed in a convenience store, supermarket, or restaurant.

[0014] The information processing system 3 and the server 4 are connected via network 2 and can communicate with each other. Network 2 may be any network, including the Internet.

[0015] The information processing system 3 transmits information such as the total price of the items purchased by the customer, information on each individual item purchased by the customer, and the price of the items purchased by the customer as processing results to the server 4 via the network 2.

[0016] Server 4 receives the processing results from Information Processing System 3 via Network 2. Server 4 runs software that manages inventory status, etc., and manages the inventory status, etc., of the stores where Information Processing System 3 is installed based on the received processing results.

[0017] Figure 2 will be used to explain the overview of the information processing system 3.

[0018] The information processing system 3 according to this embodiment is, for example, a cash register. "Cash register" is an abbreviation for "cash register" and is used, for example, to settle and pay for the amount of goods to be purchased. The information processing system 3 may be an unmanned cash register, a manned cash register, a self-checkout cash register, or a semi-self-checkout cash register.

[0019] As shown in Figure 2, the information processing system 3 comprises a mounting base 10 and an information processing device 20. The information processing system 3 further includes a support column 11, a first imaging unit 12, a display unit 13, a second imaging unit 14, and a measurement unit 15. In this embodiment, the information processing device 20 is configured as a separate device from the first imaging unit 12, the display unit 13, and the measurement unit 15. The information processing device 20 may be integrally configured with at least one of the display unit 13, the first imaging unit 12, the measurement unit 15, the support column 11, the mounting base 10, and the second imaging unit 14.

[0020] The mounting platform 10 includes at least a first upper surface 10s and a second upper surface 10t. The first upper surface 10s and the second upper surface 10t may be adjacent to each other. The first upper surface 10s and the second upper surface 10t are formed flat so that the user can place objects they wish to purchase or containers such as shopping baskets on them. In this embodiment, the first upper surface 10s and the second upper surface 10t are substantially rectangular in shape, but are not limited to this and may be any shape such as a circle. The first upper surface 10s and the second upper surface 10t may be of different heights, and in this embodiment, the first upper surface 10s is configured to be higher than the second upper surface 10t.

[0021] A support column 11 is attached to the second upper surface 10t. The support column 11 is a columnar member made of any material. For example, one end of the support column 11 is attached to the side of the second upper surface 10t, and the other end is located above the second upper surface 10t.

[0022] The first imaging unit 12 is mounted on the end of the support column 11 so as to be able to image the second upper surface 10t from above. The first imaging unit 12 and the information processing device 20 are connected to each other so as to be able to communicate with each other. The first imaging unit 12 is controlled by the information processing device 20 and can image at least part or all of the first upper surface 10s and the second upper surface 10t of the mounting table 10.

[0023] The first imaging unit 12 is capable of generating images of the captured subject. For example, the first imaging unit 12 may be a camera. The first imaging unit 12 may continuously capture still images at any frame rate. The first imaging unit 12 may capture moving images. The first imaging unit 12 may have a function to generate images of only a specific space. Specifically, the first imaging unit 12 may be capable of generating images of only a space within a specified distance range. The first imaging unit 12 may be a depth camera.

[0024] The display unit 13 is mounted on, for example, the information processing device 20 and provides various information to the user. The display unit 13 may be any type of display. The display unit 13 and the information processing device 20 may be connected to each other. The display unit 13 is controlled by the information processing device 20 and displays images captured by the first imaging unit 12. The display unit 13 may have the functionality of a touchscreen for the input unit, which will be described later.

[0025] The second imaging unit 14 may be fixed to the ceiling above the mounting table 10. The second imaging unit 14 and the information processing device 20 are connected to each other so as to be able to communicate with each other. The second imaging unit 14 is capable of generating images of the captured subject. For example, the second imaging unit 14 may be a camera. The second imaging unit 14 is controlled by the information processing device 20 and can image the mounting table 10 and its surroundings. The images captured by the second imaging unit 14 may include the user in front of the mounting table 10 and other users lined up to use the information processing system 3. The second imaging unit 14 may be mounted so as to be able to image the mounting table 10 from a different angle than the first imaging unit 12. The second imaging unit may be mounted so as to be able to image the mounting table 10 from diagonally above. The second imaging unit 14 may continuously capture still images at any frame rate. The second imaging unit 14 may capture moving images. The second imaging unit may be capable of generating images of only any space. Specifically, the second imaging unit may be capable of generating images only within a specified distance range. The second imaging unit 14 may be a depth camera.

[0026] The measuring unit 15 is configured to measure the distance to an object. In this embodiment, the measuring unit 15 can measure the height of an object by measuring the distance to the object placed on the second upper surface 10t of the mounting base 10. The measuring unit 15 may be, for example, a depth sensor, a laser rangefinder, etc. In this embodiment, the measuring unit 15 is located on the second upper surface 10t and on the optical axis of the lens of the first imaging unit 12, but is not limited to this. The measuring unit 15 may be configured integrally with the first imaging unit 12.

[0027] The information processing device 20 will be explained using Figure 3. The information processing device 20 is configured, for example, by a computer and includes a communication unit 21, an input unit 22, a storage unit 23, and a control unit 24. Software for realizing the various functions described in this embodiment is executed in the information processing device 20.

[0028] The communication unit 21 includes at least one communication module that can connect to the network 2. The communication module may be, for example, a communication module that complies with standards such as wired LAN (Local Area Network) or wireless LAN. The communication unit 21 is connected to the network 2 via the wired LAN or wireless LAN by this communication module.

[0029] The communication unit 21 includes a communication module capable of communicating with the first imaging unit 12, the display unit 13, and the second imaging unit 14. The communication module is a communication module that conforms to the communication line standard. Communication by the communication unit 21 may be performed by at least one of wired or wireless means. The communication unit 21 functions as an acquisition unit that acquires an image of an object placed on the second upper surface 10t from at least one of the first imaging unit 12 and the second imaging unit 14.

[0030] The input unit 22 includes at least one input interface capable of receiving input from the user. The input interface is, for example, a touchscreen integrated with the display unit 13, physical keys, capacitive keys, a pointing device, or a microphone. In this embodiment, the input unit 22 is a touchscreen integrated with the display device 13 and acquires input from the user.

[0031] The memory unit 23 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), etc. ROM is, for example, EEPROM (Electrically Erasable Programmable Read Only Memory), etc. The storage unit 23 is the main It may function as a storage device, auxiliary storage device, or cache memory. The storage unit 23 stores programs and data used in the operation of the information processing device 20, and data obtained through the operation of the information processing device 20. The storage unit 23 may store images of objects sold in the store where the information processing system 3 is installed. The storage unit 23 may store a product information database. The product information database may include a correspondence table showing the relationship between the names and prices of objects sold in the store where the information processing system 3 is installed and information such as an ID to identify the object. This information may be stored in the server 4 and retrieved by the information processing system 3 as appropriate.

[0032] The control unit 24 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a CPU (Central Processing Unit) Alternatively, it may be a general-purpose processor such as a GPU (Graphics Processing Unit) or a dedicated processor specialized for a specific process. Dedicated circuits include, for example, FPGAs (Field-Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The control unit 24 is configured to control the communication unit 21, the input unit 22, and the storage unit 23. The control unit 24 executes processes related to the operation of the information processing device 20 while controlling each part of the information processing device 20.

[0033] <Specific examples of the operation of the information processing device 20> (Recognition processing) The process from when the user initiates checkout for an item until the object is recognized will be explained in detail below using Figures 4 to 9.

[0034] The user stands in front of the information processing system 3 with a container 40 containing the item they wish to purchase. The container 40 may be, for example, a shopping basket provided in a store for customers to place items in. The user operates the touchscreen of the display unit 13 to input a signal to the input unit 22 to start the checkout process. After that, the user places the container 40 on the second upper surface 10t of the display stand 10. Alternatively, the user may place the container 40 on the second upper surface 10t of the display stand 10 before inputting the signal to start the checkout process into the input unit 22.

[0035] Figure 4 shows an example of a container 40 containing an object placed on the second upper surface 10t. When the container 40 is placed on the second upper surface 10t, the control unit 24 controls the measuring unit 15 to obtain the height d1 from the second upper surface 10t to the top 40a of the container 40. Specifically, the measuring unit 15 measures the height d1 from the measuring unit 15 to the top 40a of the container 40. The control unit 24 may calculate the height d1 using an image of the container 40 taken from diagonally above by the second imaging unit 14. The control unit 24 may also obtain height information of the container 40 that has been previously stored in the storage unit 23. Note that the height d1 is a reference position for determining whether or not an object has been removed from the container 40, as will be described later. This reference position does not necessarily have to be the top of the container 40. The reference position can be any position at which the user can take an object out of the container 40 and determine that they intend to purchase it. For example, the reference position may be set to the height of the user's elbow using the image taken by the first imaging unit 12 of the user.

[0036] The control unit 24 acquires the first image 31 captured by the first imaging unit 12 or the second imaging unit 14. Figure 5 is an example of the first image 31 captured by the first imaging unit 12. The first image 31 includes images of a container 40 placed on the upper surface 10t by the user, and objects 41 contained in the container 40, namely the first object 41a, the second object 41b, the third object 41c, and the fourth object 41d. The control unit 24 may acquire the first image 31 captured by both the first imaging unit 12 and the second imaging unit 14. In this embodiment, the control unit 24 acquires the first image 31 by receiving the first image 31 captured by the first imaging unit 12 via the communication unit 21.

[0037] The control unit 24 performs a first recognition process, which is a category recognition process, on an object located at a distance from the second upper surface 10t from the uppermost part 40a. The category recognition process is a process for obtaining category information, which is a classification that separates objects. Categories include, for example, the shape of the object, the type of object, the type of packaging, and the method of selling the object. More specifically, an object can be classified into a category based on its shape, such as whether it is spherical, cubic, or elongated. An object can be classified into a category based on its type, such as whether it is a vegetable, fruit, confectionery, meat, or fresh fish. An object can be classified into a category based on its type of packaging, such as boxed, bagged, or unpackaged. An object can be classified into a category based on its method of selling, such as by weight, loose, or in bulk. The category to which an object belongs can be determined from an image containing the object by means of a DNN (Deep Neural Network) or pattern matching.

[0038] Figure 6 shows an example of the state of the mounting platform 10 when object category recognition processing is performed. When the user takes the first object 41a that they wish to purchase from the container 40, the first object 41a moves to a position away from the second upper surface 10t from the top 40a. The control unit 24 performs category recognition processing on the first object 41a when the distance from the second upper surface 10t to the first object 41a becomes greater than or equal to the height d1. The control unit 24 may be controlled not to perform category recognition processing when the container 40 is not placed on the second upper surface 10t. This prevents category recognition processing from being performed on objects captured by the first imaging unit 12 before the container 40 is placed on the second upper surface 10t. The control unit 24 may perform category recognition processing on the first object 41a when it detects that the first object 41a, which is at a position greater than or equal to the height d1 from the second upper surface 10t, is being held in the hand. By making the object held in the hand the target of category recognition processing, it is possible to prevent category recognition processing from being performed on objects that appear in the first imaging unit 12 while they are inside the container 40.

[0039] The control unit 24 decides, according to the category identified for the first object 41a, whether to perform a second recognition process, which is object recognition processing, on the first object 41a, or a third recognition process, which is symbol recognition processing, on the first symbol 42a assigned to the first object 41a. In this specification, the term "recognition processing" may be used to refer to any or all of the category recognition processing, object recognition processing, and symbol recognition processing.

[0040] Object recognition processing is the process of identifying information necessary to obtain the price of the first object 41a, for example, from the shape, color, and size of the object. Object recognition processing may include image recognition processing.

[0041] The symbol recognition process is a process that identifies the information necessary to obtain the price of the first object 41a by reading a symbol. The symbol includes, for example, one-dimensional codes and two-dimensional codes such as barcodes and two-dimensional barcodes attached to object 41.

[0042] In this embodiment, for example, a store sets prices for "items sold by weight," "meat," "fresh fish," and "prepared foods" according to their weight. When the control unit 24 identifies that the category of the first object 41a is one of "items sold by weight," "meat," "fresh fish," or "prepared foods," it performs symbol recognition processing on the first symbol 42a. "Items sold by weight" are items whose price is determined according to the amount purchased by the customer, such as nuts and alcoholic beverages. "Meat" includes beef, pork, chicken, and ground meat. "Fresh fish" includes tuna, salmon, and cod. "Prepared foods" include potato salad, namul, and pork cutlets. When the control unit 24 identifies that the category of the first object 41a is not one of "items sold by weight," "meat," "processed meat products," "fresh fish," or "prepared foods," it performs object recognition processing on the first object 41a. The type of object to be targeted for symbol recognition processing may be pre-configured in the information processing system according to the store's sales format.

[0043] Figure 7 is an example of a first image 31 that includes a first object 41a lifted by the user. For example, when the control unit 24 identifies the category of the first object 41a as "meat", it performs symbol recognition processing on the first symbol 42a. For example, when the control unit 24 identifies the category of the first object 41a as "confectionery", it performs object recognition processing on the image of the first object 41a.

[0044] Objects sold by weight, such as bulk goods, often have similar appearances because they are packaged in the same type of containers and packaging regardless of their weight. Therefore, accurately recognizing them and displaying their prices using object recognition processing is difficult. For such items, symbol recognition processing can be used to obtain weight information from the symbol, enabling accurate accounting. On the other hand, objects with specific packaging, such as "chocolate," often have distinctive appearances, making them easily distinguishable from other objects and allowing for accurate recognition by object recognition processing. Furthermore, since the symbol 42 assigned to each product is located in a different position, users have to spend time searching for the symbol 42. For such items, object recognition processing can provide convenience to the user.

[0045] The information processing device 20 performs category recognition processing, followed by symbol recognition processing and object recognition processing, using a single configuration including a control unit 24, but is not limited to this. For example, it may include a first recognition unit that performs category recognition processing, a second recognition unit that performs symbol recognition processing, and a third recognition unit that performs object recognition processing, and each of these processes may be executed.

[0046] The control unit 24 acquires information about the first object 41a through the determined symbol recognition process or object recognition process. The information about the first object 41a includes the name of the object, an ID that can identify the object, and other information necessary to obtain the price of the first object 41a. The control unit 24 uses the acquired ID, etc., to obtain the price information of the first object 41a. For example, control unit 2 4 performs symbol recognition processing on the first symbol 42a of the first object 41a, which has been identified as belonging to the "meat" category, and obtains an ID that can identify the object. The control unit 24 uses the product information database, etc., stored in the storage unit 23 to obtain the object name and price corresponding to the identified ID. For example, the control unit 24 performs object recognition processing on the first object 41a, which has been identified as belonging to the "confectionery" category, and obtains the name of the object and an ID that can identify the object. The control unit 24 uses the product information database, etc., stored in the storage unit 23 to obtain the object name and price corresponding to the identified ID. In addition, the control unit 24 may obtain the price of object 41a from the price tag attached to object 41a during the object recognition processing. The registered image of an object may be obtained from the registered images of products stored in the storage unit 23 using the object's name, ID, etc.

[0047] The control unit 24 displays the first name 43a, which is the object name of the acquired first object 41a, on the display unit 13. Figure 8 is an example of an image 30 with the first name 43a displayed. Image 30 includes a first region 13a for displaying the first image 31, a second region 13b for displaying the second image 32, and a third region 13c for displaying the boundary 33. In this embodiment, the first image 31 is displayed on the right side of image 30, and the second image 32 is displayed on the left side of image 30, but this is not limited to this. The first image 31 includes an image of the first object 41a captured by the first imaging unit 12, and a display of the first name 43a acquired by the control unit 24. In this embodiment, the first name 43a is displayed below the image of the first object 41a, but this is not limited to this, and it may be displayed above, to the left, or to the right of the image of the first object 41a. In addition, a line connecting the first name 43a and the image of the first object 41a may be displayed on the display unit 13. Furthermore, the first image 31 may include a display prompting the user to move the image of the first object 41a to the third region 13c. For example, the first image 31 may include a display of text such as "Please move the recognized product" or an arrow.

[0048] The user confirms the image of the first name 43a displayed on the display unit 13. If the first name 43a is correct and the user wishes to purchase the first object 41a, the user moves the image of the first object 41a to the third region 13c. In this embodiment, the user can move the image of the first object 41a to the third region 13c by moving the first object 41a, which is being captured by the first imaging unit 12, to the left relative to the second upper surface 10t (i.e., towards the first upper surface 10s). If the user does not wish to purchase the first object 41a, the user moves it in a direction other than the third region 13c. In this embodiment, the user can move the image of the first object 41a to a direction other than the third region 13c by moving the first object 41a from above the second upper surface 10t in at least one of the directions up, down, and right (i.e., a direction other than the first upper surface 10s) other than the left.

[0049] When the image of the first object 41a moves to the third region 13c, the control unit 24 adds the first object 41a to the purchase list as an item to be purchased. When the image of the first object 41a moves to the third region 13c, it may mean that the entire image of the first object 41a passes through the third region 13c, or that only a part of it (for example, 50%) passes through the third region 13c. The purchase list may be information temporarily stored in the storage unit 23. The control unit 24 also displays the contents of the purchase list in the second image 32. Figure 9 is an example of image 30 when the contents of the purchase list are displayed in the second image 32. The purchase list displayed in the second image 32 may include the first name 43a and the price information of the first object 41a. When the control unit 24 displays the first name 43a in the second image 32, it may also display the registered image 44 of the first object 41a. Alternatively, instead of the registered image 44, a reduced image of the first object 41a captured by the first imaging unit 12 may be displayed in the second image 32. If the first name 43a or price displayed in the purchase list is incorrect, it is possible that the object recognition process or symbol recognition process is incorrect. In this case, the user can correct the recognition result by the correction process described later.

[0050] The control unit 24 provides a predetermined notification when the first object 41a is added to the purchase list. The control unit may provide a predetermined notification when the image of the first object 41a moves to the third region 13c. The control unit may provide a predetermined notification when the image of the first object 41a moves to the second region 13b via the third region 13c. The control unit may provide a predetermined notification when the first name 43a is displayed on the second image 32. As a predetermined notification, the display form of the boundary 33 may be changed. As a change in the display form of the boundary 33, at least one of the following may be changed: the color, size, thickness, etc. As a predetermined notification, for example, a notification sound such as "beep" may be output. By providing a predetermined notification, the user can be sure that the first object 41a, which is the product they wish to purchase, has been added to the purchase list and can be sure that the purchase process is proceeding correctly.

[0051] The user checks the image of the first name 43a displayed on the display unit 13, and if the first name 43a is incorrect, that is, if the first object 41a is misidentified, the user inputs this information to the input unit 22. For example, the user touches the image labeled "Correction" included in the first image 31. The control unit 24 may also determine that the first object 41a is misidentified if the user does not move the image of the first object 41a to the third area 13c for a predetermined amount of time or longer.

[0052] When the control unit 24 receives input indicating that the first object 41a has been misrecognized, it displays at least one candidate for the correct object as the first object 41a on the display unit 13. The candidate for the correct object may be an object that is easily misrecognized as the first object 41a. Figure 10 shows an example of the candidate for the correct object displayed on the display unit 13 when input indicating misrecognition is received. The candidate for the correct object may include other objects that were deemed to have a high reliability in the recognition result when the control unit 24 performed object recognition processing on the first object 41a, or objects that are predefined to be easily misrecognized by each other. The user may select one of the candidate for the correct object displayed on the display unit 13. If the correct recognition result is not included, the user may input that fact to the input unit 22.

[0053] The user confirms the first name 43a displayed in the second image 32 and places the first object 41a on the first upper surface 10s. At this time, the first object 41a may be placed as is, or it may be placed in a container other than the container 40, such as a shopping bag or shopping basket placed on the upper surface 10t.

[0054] The user also performs the above operations on the second object 41b, the third object 41c, and the fourth object 41d. The information processing device 20 performs category recognition processing and object recognition processing / symbol recognition processing on the second object 41b, the third object 41c, and the fourth object 41d to recognize all the objects the user purchases and obtain price information.

[0055] (Correction process) The process of correcting the recognition result when the name of an object displayed in the purchase list is incorrect will be explained using Figures 11 and 12.

[0056] The user operates the touchscreen of the display unit 13 to correct the recognition result of an object included in the purchase list from the input unit 22. The control unit 24 can accept input for this operation. Figure 11 is an example of a second image 32 displaying the purchase list in which the second object 41b is registered. Here, the purchase list displays "cream bun" as the second name 43b, but the second object 41b is actually an "anpan" (sweet bean paste bun), so the second name 43b is incorrect. In other words, an incorrect result has been output in the object recognition process. The user makes an input to execute a correction process to correct the recognition result of the second object 41b. For example, the user may select the image of the second name 43b or the second object 41b in the purchase list displayed on the display unit 13, or press the button to correct the name, and start the correction process.

[0057] When the control unit 24 receives an operation to perform a correction process, it displays a third image 34 on the display unit 13 prompting the user to photograph the object again. Figure 12 is an example of the third image 34. For example, the control unit 24 may display the string 34a "We will recognize the product again. Please show the product you want to correct to the camera again." and an arrow 34b pointing to the first area 13a on the display unit 13 as the third image 34. Depending on the category of the object to be corrected, the control unit 24 may display the string 34a displayed as the third image 34 on the display unit 13. For example, when the category of the object to be corrected is at least one of "bulk goods," "meat," "fresh fish," or "prepared food," the control unit 24 may display the string 34a "Please show the barcode of the product you want to correct to the camera again." on the display unit 13. It may be displayed on 13. In this embodiment, the third image 34 is displayed superimposed on the second image 32, but is not limited to this. The user checks the third image 34 and moves the object to be corrected to the top of the container 40, thereby causing the first imaging unit 12 to image the second object 41b, which is the object for which correct information was not obtained, again. When the control unit 24 receives an operation to perform a correction process, it removes the information about the second object 41b to be corrected from the purchase list.

[0058] The control unit 24 performs category recognition processing and object recognition processing / symbol recognition processing again on the second object 41b that has moved to the top of the container 40. These recognition processes are as described above, so their description is omitted.

[0059] Furthermore, the control unit 24 may start correction processing when it detects, based on the image acquired by the first imaging unit 12, that the object 41 is moving from the first upper surface 10s to the second upper surface 10t. The control unit 24 may also start correction processing for the object when it detects, based on the image acquired by the first imaging unit 12, that the image of the object 41 is moving from an area outside the first area on the third area 13c side to the first area 13a. The user specifies any object included in the purchase list to replace the recognition result of the newly recognized object 41 by operating the touchscreen. The control unit 24 replaces the object in the purchase list specified on the touchscreen with the newly obtained recognition result by performing category recognition processing and object recognition processing / symbol recognition processing on object 41 again. That is, it deletes the object specified by the user in the purchase list and adds the information of the newly obtained object recognition result to the purchase list.

[0060] (Settlement process) The settlement process for items added to the purchase list will be explained using Figures 13 and 14.

[0061] Figure 13 is an example of an image 30 after the user has allowed the information processing system 3 to recognize all the objects 41 contained in the container 40. The first image 31 shows only the container 40. The second image 32 includes the names, prices, and images of the first object 41a, second object 41b, third object 41c, and fourth object 41d that were contained in the container 40 at the start of the transaction. If the user does not wish to purchase any additional objects other than the objects 41 included in the purchase list, they perform a settlement operation to settle the purchase of the objects included in the purchase list. The settlement operation may be an operation such as pressing a button to instruct the control unit 24 to settle the purchase of the objects.

[0062] When the control unit 24 receives a payment operation, it displays the fourth image 35 in the second area 13b. The fourth image 35 prompts the user to confirm whether all the items to be purchased are included in the purchase list. If the items to be purchased are not included in the purchase list, it is assumed that the user has forgotten to perform the recognition process for those items. Figure 14 may include text prompting the user to check for the presence of unrecognized items. For example, the fourth image 35 may include the text, "Are there any items that have not been registered?". Alternatively, the fourth image 35 may include text prompting the user to check for the presence of unrecognized items. For example, the fourth image 35 may include the text, "Have you checked under the cart?", or an illustration pointing to under the cart.

[0063] When the fourth image 35 is displayed, the user checks whether object 41 remains in container 40. If object 41 is not remaining in container 40, the user may proceed with the settlement process. Alternatively, if object 41 is still remaining in container 40, the user may perform the recognition process again. For example, if the user wishes to perform the recognition process again, they select the image labeled "still remaining" in the fourth image 35. For example, if the user wishes to proceed with the settlement process, they select the image labeled "not remaining" in the fourth image 35.

[0064] If the user selects an image that "still remains," the control unit 24 restarts the recognition process. The recognition process is omitted here as it overlaps with the above description.

[0065] If the user selects an image that is "not remaining," the control unit 24 displays the total price of all items included in the purchase list, stored in the storage unit 23, on the display unit 13. The user confirms the amount and pays it. Payment may be made by inserting the instructed amount into any payment machine (not shown), or by paying directly to a store clerk. Payment may be made not only with cash, but also by any other method, such as credit card or point card payment.

[0066] As described above, the information processing device 20 according to one embodiment of this disclosure can efficiently recognize only the objects that the user intends to purchase by recognizing the objects that the user has lifted to a predetermined height as objects that the user intends to purchase. Since objects that the user does not intend to purchase are not subject to recognition, they will not be mistakenly registered in the purchase list, thereby improving user convenience. Furthermore, by using the top 40a of the container 40 as the predetermined height, the user's intention to purchase can be determined with high accuracy. In addition, since the information processing device 20 recognizes objects that are close to the imaging unit, it can recognize objects with high accuracy without confusing them with other objects remaining in the container 40.

[0067] Furthermore, the information processing device 20 according to one embodiment of this disclosure can accurately recognize an object by identifying the category of the object and determining the object recognition process according to the identified category.

[0068] (System operation) Figure 15 is a flowchart showing the procedure of an information processing method according to one embodiment of the present disclosure. This information processing method includes steps S101 to S115. For example, when the control unit 24 receives an input from the input unit 22 instructing the start of accounting, it starts the processing of step S101.

[0069] The control unit 24 acquires information on the height d1 from the upper surface 10t of the mounting table 10 to the upper part 40a of the container 40 (step S101).

[0070] The control unit 24 receives the first image 31 from the first imaging unit 12 via the communication unit 21 and acquires the data of the first image 31 (step S102).

[0071] The control unit 24 performs object recognition processing on the object 41 that is included in the first image 31 and is located at a height higher than d1 from the upper surface 10t, and identifies the category of the object 40 (step S103).

[0072] The control unit 24 determines whether the category of the object 41 identified in step S103 is at least one of the following: "items sold by weight", "fresh meat", "processed meat products", "fresh fish", or "prepared foods" (step S104).

[0073] When the control unit 24 determines that the category of object 41 is at least one of "items sold by weight," "fresh meat," "processed meat products," "fresh fish," or "prepared foods" (step S104: YES), it proceeds to step S105.

[0074] When the control unit 24 determines that the category of object 41 is not one of the following (step S104: NO): "items sold by weight", "fresh meat", "processed meat products", "fresh fish", or "prepared foods", it proceeds to step S106.

[0075] The control unit 24 performs recognition processing on the symbol 42 assigned to the object 41 and obtains information about the object 41 (step S105).

[0076] The control unit 24 performs object recognition processing on the image of the object contained in the acquired first image 31 and obtains information about the object 41 (step S106).

[0077] The control unit 24 displays information such as the name, price, and registered image of the object 41 obtained by processing in at least one of steps S105 and S106 on the first image 31 (step S107).

[0078] The control unit 24 determines whether or not the object 41 has moved to the third region 13c (step S108).

[0079] When the control unit 24 detects that the object 41 has moved to the third region 13c (step S108: YES), it proceeds to step S108.

[0080] If the control unit 24 does not detect that the object 41 has moved to the third area 13c (step S108: NO), it returns to step S106. In other words, if the control unit 24 does not detect that the object 41 has moved to the third area 13c, it continues to display the acquired information such as the name, price, and registered image of the object 41 on the first image.

[0081] The control unit 24 temporarily stores information about the acquired object 41 in the storage unit 23 (step S109). The information temporarily stored in the storage unit 23 may include the name and price of the object 41.

[0082] The control unit 24 displays the acquired information of object 41 on the second image 32, and at this time, it highlights the boundary included in the third region 13c (step S110). The information of object 41 displayed on the second image 32 may include the name of object 41, price, registered image, etc.

[0083] The control unit 24 determines whether or not it has received an input to modify the information of object 41 (step S111).

[0084] When the control unit 24 receives input to modify the information of object 41 (step S111: YES), it returns to step S102.

[0085] If the control unit 24 does not receive an input to modify the information of object 41 (step S111: NO), it proceeds to step S112. Here, "when no input is received" may include cases where no input is received for a certain period of time, or when an input related to the processing in step S112, which will be described later, is received.

[0086] The control unit 24 determines whether or not it has received an input to settle the temporarily stored objects (step S112).

[0087] When the control unit 24 receives an input to settle the temporarily stored objects (step S112: YES), it proceeds to step S113.

[0088] If the control unit 24 does not receive an input to settle the temporarily stored object (step S112: NO), it returns to step S102. Here, "not receiving an input to settle" includes cases such as when a new object is recognized before receiving an input to settle.

[0089] The control unit 24 displays a third image 34 on the display unit 13 prompting the user to confirm the object 41 that they wish to purchase but which has not been recognized by the information processing device 20 (step S113).

[0090] The control unit 24 determines whether or not there are any unrecognized objects remaining (step S114).

[0091] When the control unit 24 determines that there are unrecognized objects remaining (step S114: YES), it returns to step S102. More specifically, when the control unit 24 receives an input indicating that there are objects remaining, it may determine that there are unrecognized objects remaining.

[0092] When the control unit 24 determines that there are no unrecognized objects remaining (step S114: NO), it proceeds to step S115. More specifically, when the control unit 24 receives an input indicating that there are no objects remaining, it may determine that there are no unrecognized objects remaining.

[0093] The control unit 24 displays an image on the display unit 13 that includes the total price of the objects 41 temporarily stored in the memory unit 23 (step S115).

[0094] The control unit 24 terminates the accounting process when it receives payment for an amount equal to or greater than the total price of the objects 41.

[0095] As described above, the information processing method according to one embodiment of this disclosure can efficiently recognize only objects that the user is highly likely to purchase by recognizing only those objects that the user is lifting. Products that the user is not interested in purchasing will not be recognized, thus improving user convenience. Furthermore, the information processing device 20 according to one embodiment of this disclosure can recognize objects with high accuracy because it recognizes objects that are located close to the imaging unit.

[0096] (Variation 1) Modification 1 is described below. Note that steps that overlap with the process described above are omitted.

[0097] In the embodiments described above, the second upper surface 10t was described as being formed of a material that does not transmit visible light. However, in Modification 1, the second upper surface 10t may be formed of a material that transmits visible light. For example, the second upper surface 10t may be formed of a material such as acrylic resin, polyvinyl chloride or polycarbonate, or glass. The second upper surface 10t may also transmit visible light by being formed in a mesh pattern.

[0098] In Modification 1, the container 140 may be formed of a material that transmits visible light, at least on its bottom surface. For example, the bottom surface of the container 140 may be made of a material such as acrylic resin, polyvinyl chloride or polycarbonate, or glass. The bottom surface of the container 140 may also transmit visible light by being formed in a mesh pattern.

[0099] In the modified example 1, the control unit 24 is configured to perform a first mode and a second mode, each capable of recognizing the presence of objects in different spaces. In the first mode, the presence of objects located in a space closer to a first distance d1 above the second upper surface 10t, or in a space below the second upper surface 10t, can be recognized. In other words, in the first mode, the presence of objects that have not been lifted to a predetermined first distance d1 from the second upper surface 10t can be recognized. Also, in the first mode, the presence of objects below the second upper surface 10t can be recognized. For objects that have been recognized, recognition processing such as category recognition processing, object recognition processing, and symbol recognition processing may be performed. On the other hand, in the second mode, only objects located in an area at a predetermined first distance d1 or more away from the second upper surface 10t are targeted for recognition processing. The control unit 24 executes the second mode when it recognizes that the container 40 has been placed on the second upper surface 10t. The control unit 24 performs object recognition processing using the method of this embodiment described above. When the recognition processing is completed and the system moves to settlement processing, it switches from the second mode to the first mode. When the control unit 24 executes the first mode and recognizes an object inside the container located on the second upper surface 10t, it may display a notification on the display unit 13 that an object remains inside the container. This allows the information processing device 20 to inform the user that there is an object that the user left in the container 140 and forgot to recognize. Furthermore, since the second upper surface 10t and the container 140 are made of a material that transmits visible light, the control unit 24 can detect objects located in the space below the second upper surface 10t. When the control unit 24 detects an object located in the space below the second upper surface 10t, it may display a notification on the display unit 13 that an object remains in the space below the second upper surface 10t. This allows the information processing device 20 to inform the user that there is an object that was not placed in the container 140 and remained in the space below the second upper surface 10t, and that the recognition process was forgotten.

[0100] The control unit 24 may stop the first mode and execute the second mode when the container 140 placed on the second upper surface 10t satisfies predetermined conditions. These predetermined conditions include when the number of objects 141 contained in the container 140 is greater than a predetermined number, or when the distance from the objects 141 contained in the container 140 to the first imaging unit 12 is shorter than a predetermined distance. If the number of objects purchased by the user is small, it is assumed that there are no objects hidden inside the container 140. In such cases, settlement processing can be performed on all objects 141 contained in the container 140 at once.

[0101] Furthermore, the control unit 24 may change the string displayed on the display unit 13 depending on whether it is executing the first mode or the second mode during the recognition process. For example, when executing the first mode, the control unit 24 may display the first image 131 with the string "Please arrange the items you wish to purchase." When executing the second mode, the control unit 24 may display the first image 131 with the string "Please pick up each item and scan them one by one."

[0102] Furthermore, the control unit 24 may execute the first mode when it recognizes that an object has been placed on the second upper surface 10t. The control unit 24 may also perform recognition processing on the object that is placed on the second upper surface 10t. If the user purchases only a few objects, they may not use the container 140. Even if the user does not use the container 140 and objects are placed directly on the second upper surface 10t, the control unit 24 can still perform recognition processing on those objects.

[0103] (Recognition process for the first modification) The recognition process performed by the recognition device of Modification 1 of this disclosure will be explained with reference to Figures 16 and 17.

[0104] Before accounting begins, the control unit 24 executes a first mode. By executing the first mode, the control unit 24 can recognize objects located within the imaging range of the first imaging unit 12.

[0105] The control unit 24 starts accounting processing when it detects an object on the second upper surface 10t based on the image captured by the first imaging unit 12. Furthermore, the control unit 24 may also start accounting processing when it receives an input instructing it to start accounting, even if it does not detect an object.

[0106] The control unit 24 determines the container 1 on the second upper surface 10t based on the image captured by the first imaging unit 12. When 40 is detected, the first mode is stopped, and the second mode is executed to start accounting processing. The recognition process during the execution of the second mode is omitted here as it overlaps with the description of the recognition process above.

[0107] When the first mode is being executed, the control unit 24 controls the object 141 placed on the upper surface 10t. Recognition processing is performed on objects 141a to 141d. Figure 17 is an example of an image 30 including objects 141a to 141d placed on the upper surface 10t. The control unit 24 performs the above-described recognition processing on each of objects 141a to 141d.

[0108] In this embodiment, symbol recognition processing or object recognition processing is performed for each of the first object 141a, second object 141b, third object 141c, and fourth object 141d. The control unit 24 displays the first name 143a, second name 143b, third name 143c, and fourth name 143d of the objects for which the recognition processing has been completed in the second image 132. At this time, the control unit 24 may also display the registered image 144 along with the name 143 in the second image 132. The control unit 24 may also display the object 141 and the name 143 in association in the image 130. For example, an image of a circled 1 may be superimposed on the image of the first object 141a included in the first image 131, and a circled 1 may be added to the beginning of the string of the first name 143a included in the second image 132.

[0109] (Settlement process for variation 1) When the control unit 24 receives an input to settle an object while the second mode is being executed, it stops the second mode and executes the first mode. When the control unit 24 receives an input to settle an object while the first mode is being executed, it may continue executing the first mode.

[0110] When the control unit 24 executes the first mode, it detects an object 141 inside the container 140, on the second upper surface 10t, or below the second upper surface 10t, and notifies the user that there are unregistered items remaining. Figure 18 is an example of a fifth image 136 that notifies the user that there are unregistered items remaining. When the control unit 24 detects the fifth object 141e, which is an unregistered item, it displays the fifth image 136, for example, "There are unregistered items remaining. Please check your shopping cart and the bottom of the cart." At this time, the control unit 24 may output sound simultaneously with displaying the fifth image 136. The control unit 24 may also highlight the detected fifth object 141e. For example, the control unit 24 may superimpose a colored image on the detected fifth object 141e, or display a bounding rectangle superimposed on the fifth object 141e.

[0111] In this way, the information processing device 20 can prevent user errors by changing the range of the area it recognizes according to the accounting processing status. Therefore, user convenience can be improved.

[0112] (Operation of the system in variation 1) Figure 19 is a flowchart showing the procedure of an information processing method according to Modification 1 of this disclosure. This information processing method includes steps S201 to S218. Note that descriptions that overlap with the descriptions of steps S101 to S114 described above are omitted.

[0113] The control unit 24 determines whether or not it has detected the container 140 (step S201).

[0114] When the control unit 24 detects the container 140 (step S201: YES), it proceeds to step S202.

[0115] When the control unit 24 does not detect the container 140 (step S201: NO), step Proceed to S204.

[0116] Step S202 is the same process as step S101 described above, so its description is omitted.

[0117] The control unit 24 executes the second mode (step S203). That is, the control unit 24 recognizes only objects that are located at a height d1 or more away from the second upper surface 10t.

[0118] The processes from step S204 to step S214 are the same as those described above from step S102 to step S112, so their description is omitted.

[0119] The control unit 24 executes the first mode (step S215). That is, the control unit 24 recognizes the object 141 on the second upper surface 10t.

[0120] The control unit 24 determines whether or not it has detected an object 141 on at least one of the container 140 and the second upper surface 10t (step S216).

[0121] When the control unit 24 detects an object 141 on at least one of the container 140 and the second upper surface 10t (step S216: YES), it proceeds to step S217.

[0122] If the control unit 24 does not detect the object 141 on either the container 140 or the second upper surface 10t (step S216: NO), it proceeds to step S218.

[0123] The control unit 24 notifies the user that there are unregistered items remaining (step S217). More specifically, it displays the fifth image 136 with the message, "There are unregistered items remaining. Please check your shopping cart and the items below the cart," or outputs an audio message. After that, it returns to step S204.

[0124] Step S218 is the same process as step S114 described above, so its description is omitted.

[0125] In this way, information processing methods can prevent user errors by changing the scope of the recognition area according to the accounting processing situation. Therefore, information processing methods can improve user convenience.

[0126] (Modification 2) Modification 2, another variation of this embodiment, will be described below with reference to Figure 20. In the embodiment described above, if category recognition processing is performed on object 41 and it is identified as "bulk-sold goods," "fresh meat," "processed meat products," "fresh fish," or "prepared food," then symbol recognition processing is performed. In Modification 2, if category recognition processing is performed on object 241 and it is identified as "vegetables," then it is checked whether it is a "bulk-sold item." Steps that overlap with the above-described process will be omitted.

[0127] In this modified example 2, the same processing is performed up to step S103 as described in this embodiment. That is, the control unit 24 performs category recognition processing on the object 241 included in the first image 231 and identifies the category. Details are omitted as they would overlap with the recognition processing described above.

[0128] When the control unit 24 identifies the category of the first object 241a as "vegetables," it performs object recognition processing on the first object 241a. Figure 20 shows the first object 241a being lifted by the user. This is an example of the first image 231 including the first object 241a. For example, if the first object 241a is an "onion", the control unit 24 identifies the category of the first object 241a as "fruit and vegetable" and performs object recognition processing on the first object 241a.

[0129] The control unit 24 performs object recognition processing on the first object 241a and determines whether or not the first object 241a is an object sold individually. More specifically, the control unit 24 identifies the first object 241a and uses the product information database stored in the storage unit 23 to confirm whether or not it is an object sold individually. The product information database is assumed to contain information on whether or not an object sold individually is available for sale in stores. The control unit 24 also identifies the range of the packaging of the first object 241a and the quantity of the first object 241a contained within the packaging. If this quantity is other than a predetermined quantity, the control unit 24 may determine that it is an item sold individually. Here, the predetermined quantity may be a predetermined quantity of the first object 241a that is packaged and sold in one package. When the first object 241a is determined to be an item sold individually, and multiple identical first objects 241a are detected in the first image 231, the display unit 13 displays a prompt to input the number of first objects 241a. Specifically, the first object 241a is an onion that is sold individually in the store. The user is trying to purchase multiple onions by putting them into one bag using the provided bag. Alternatively, multiple onions may be sold in the store already packaged in one bag. The control unit 24 recognizes the onions packaged in one bag in this way, and when it detects multiple onions in the bag, it displays an image on the display unit 13 prompting the user to input the number of onions. If the first object 241a is not an item sold individually, the first name 243a is displayed in the first image 231. This allows the information processing device 20 to display a prompt to the user to input the quantity of object 241 only when the user needs to input the quantity of object 241, thereby improving the usability of the information processing device 20.

[0130] Figure 21 shows an example of a sixth image 237 prompting input of the number of the first object 241a. The sixth image 237 is displayed including an image with numbers from 0 to 9. The sixth image 237 may also include the first name 243a, the first registration image 244a, the price of the first object 241a, etc. The control unit 24 accepts input of the number of the first object 241a when the image with numbers from 0 to 9 is selected. The control unit 24 can accept input indicating that the input of the number of the first object 241a has been completed. Input indicating that the input of the number has been completed may be an input such as pressing a button indicating that the input has been completed. When the control unit 24 accepts that the input of the number of the first object 241a has been completed, it temporarily stores the input number and the information of the first object 241a in the storage unit 23 as information of the object to be purchased. Furthermore, the control unit 24 displays the information of the first object 241a in the second image 232.

[0131] Figure 22 is an example of a second image 232 containing information on multiple first objects 241a. Image 232 may display multiple objects 241 as a single object. For example, if there are three "onions," which are first objects 241a to be purchased, the string "×3" representing the quantity may be displayed to the right of one first registration image 244a, or the string "×3" may be displayed after the first name 241a, "onion." Multiple first registration images 244a may also be superimposed to indicate the existence of multiple first objects 241a.

[0132] In this way, the information processing device 20 can reduce the burden on the user by providing a user interface that corresponds to the category of the object.

[0133] (System operation in modified example 2) Figure 23 is a flowchart showing the procedure of an information processing method according to Modification 2 of this disclosure. This information processing method includes steps S301 to S303 in addition to steps S101 to S115 described above.

[0134] The control unit 24 determines whether the category of the object 241 identified in step S104 is "fruits and vegetables", and whether the object 241 is an object that is sold individually in step S106 (step S301).

[0135] If the category of object 241 identified in step S104 is "fruits and vegetables" and object 241 is an object that is sold individually in step S106 (step S301: YES), the control unit 24 proceeds to step S302.

[0136] If the category of object 241 identified in step S104 is not "fruits and vegetables", or if object 241 is not an object sold individually in step S106 (step S301: NO), the control unit 24 proceeds to step S107.

[0137] The control unit 24 displays a sixth image 237 on the display unit 13 prompting the user to input the number of objects 241 that have been recognized (step S302).

[0138] The control unit 24 determines whether or not it has received input regarding the number of objects 241 (step S303).

[0139] When the control unit 24 receives input for the number of objects 241 (step S303: YES), it proceeds to step S109.

[0140] If the control unit 24 does not accept input for the number of objects 241 (step S303: NO), it returns to step S302. In other words, the control unit 24 continues to display the sixth image 237 until it accepts input for the number of objects 241.

[0141] In this way, the information processing method can reduce the user's burden by prompting the user and changing the next accounting operation they should perform depending on the category of the object.

[0142] (Variation 3) Modification 3, another modification of this embodiment, will be described below with reference to Figure 24. Modification 3 describes the recognition process when discount information is attached to an object. The recognition process of the object itself in Modification 3 is the same as in the embodiment described above. Steps that overlap with the process described above will be omitted.

[0143] If the object the user intends to purchase has discount information such as a discount sticker attached to it, in step S102 described in this embodiment, the first image 331 captured by the first imaging unit will include this information. The image of the first object 341a includes an image of discount information. When the control unit 24 recognizes that an object included in the first image 331 has discount information attached to it, it displays a discount image 345 representing the discount information on the display unit 13 in association with the object. Specifically, the first discount image 345 is displayed in the vicinity of the image of the first object 341a included in the first image 331, or superimposed on the image of the first object 331. The display format of the first discount image 345 is not limited to this. The patterns of discount stickers used in stores may be registered in the storage unit 23 in advance. The control unit 24 can detect the discount information included in the image of the first object 341a using known methods such as pattern matching.

[0144] Figure 24 is an example of the first image 331, which includes an image of the first object 341a with discount information attached. The image of the first object 341a includes a discount sticker, which is the discount information. When the control unit 24 recognizes from the image of the first object 341a that discount information is attached, it displays the first discount image 345a on the first image 331. The first discount image 345a is the discount This is an image that indicates it is an elephant, or an image that shows the content of the discount information. For example, when the control unit 24 recognizes discount information that says "20% off", it may display the first discount image 345a that says "20% off" in the upper right corner of the first image 331. Also, when the control unit 24 recognizes discount information that says "20 yen off", it may display the first discount image that says "20 yen off" in the upper right corner of the first image 331. The control unit 24 does not necessarily have to display the first discount image 345a in the upper right corner, and may display it in any position on the first image 331. The control unit 24 may display the price of the first object 341a to which the discount information has been applied, along with the image of the first object 341a, on the first image 331. For example, the price of the first object 341a, which is "pork scraps", is "376 yen". At this time, the control unit 24 may apply the discount information "20% off" to the price of the first object 341a and display the price "301 yen". The control unit 24 may display the price before the discount information is applied and the price after the discount information is applied to indicate that the discount information has been applied. The control unit 24 may display the price after the discount information is applied in a way that highlights or distinguishes it from the price before the discount information is applied. The control unit 24 may highlight or distinguish the display by changing the font size, font boldness, or font color.

[0145] Furthermore, when the control unit 24 recognizes discount information from the image of the first object 341a, it may output the discount information as audio. For example, when the control unit 24 recognizes discount information that says "20% off", it may output the audio "This is a 20% off item". Also, when the control unit 24 recognizes discount information that says "20 yen off", it may output the audio "It's 20 yen off". Furthermore, the control unit 24 may change the type of audio output depending on the type of discount information. For example, when the control unit 24 recognizes discount information that says "〇 discount", such as "10% off" or "20% off", it may output the audio "beep beep beep".

[0146] When the control unit 24 displays information about the first object 341a to which discount information has been added to the second image 332, it may also display the first discounted image 345a or information indicating that the object has been to which discount information has been added. When displaying the first discounted image 345a to the second image 332, the first discounted image 345a may be reduced in size. Figure 25 is an example of displaying a purchase list to which information about the first object 341a has been added to the second image 332. The second image 332 may include the first name 343a and the first registration image 344a as information about the first object 341a. The control unit 24 may also display the first discounted image 345a in the vicinity of the first registration image 344a. For example, the control unit 24 displays the first discounted image 345a reduced in size to the lower right of the first registration image 344a. The control unit 24 may also display the price of the first object 341a to which the discount information has been applied. Specifically, the control unit 24 may display the price to which the discount information has been applied. The control unit 24 may display the price before the discount information is applied and the price after the discount information is applied. The control unit 24 may display the price after the discount information is applied in a way that highlights or distinguishes it from the price before the discount information is applied. The control unit 24 may achieve this highlighting or distinguishing by changing the font size, font weight, or font color.

[0147] (Modification process for modified example 3) The process of correcting the name or discount information when the recognition process or discount information for an object added to the purchase list is incorrect is explained using Figures 26, 27, and 28.

[0148] Figure 26 is an example of a second image 332 displaying a purchase list with added information for the second object 341b. The control unit 24 can accept an operation to modify at least one of the recognition result and discount information of the second object 341b included in the second image 332. The process for modifying the object recognition result is the same as the process described above, so it is omitted here. For example, in Figure 26 The reduced second discount image 345b, displayed together with the second object 341b, indicates a "10 yen discount." However, the correct discount information attached to the second object 341b is "10% off." In this case, since a "10% discount" is correct on the price of the second object 341b, the user makes an input to perform a correction process to modify the second name 343b. To start the correction process, the user makes an input such as selecting the image of the second name 343b or the second object 341b in the display unit 13, or pressing the button to correct the name.

[0149] When the control unit 24 receives input to perform a correction process to modify the object recognition result and discount information, it displays a correction image 337 on the display unit 13 to prompt the user to input the items to be corrected. Figure 27 is an example of a correction image 337 that prompts the user to input the items to be corrected. The correction image 337 includes the text "Please select the items you wish to correct." The correction image 337 may include images containing the text "product" and "discount information," which are candidates for the items to be corrected.

[0150] When the control unit 24 receives input to correct the object recognition result, it displays a third image 334a on the display unit 13 prompting the user to photograph the object 341 again. The details are the same as those described above, so they are omitted here.

[0151] When the control unit 24 receives input to correct discount information, it displays a third image 334b ​​on the display unit prompting the user to photograph the second object 41b containing the discount information related to the said discount information again. Figure 28 is an example of a third image 334b ​​prompting the user to photograph the discount image 345b again. For example, the control unit 24 may display the text "We will recognize the product again. Please show the product you wish to correct to the camera again." or an arrow pointing to the first area 13a on the display unit 13 as the third image 334b ​​prompting the user to photograph again. In this embodiment, the third image 334b ​​is displayed superimposed on the second image 332, but is not limited to this. When the user confirms the third image 334b, they move the object 341 to be corrected to the top of the container 340, thereby causing the first imaging unit 12 to photograph the second object 341b, which is the object in which the discount information was incorrectly recognized, again.

[0152] The control unit 24 performs recognition processing again on the second discounted image 345b, which has moved to the top of the container 340. At this time, the control unit 24 may also perform recognition processing on the second object 341b. The processing after the second discounted image 345b is recognized again is the same as the processing described above, so it is omitted from the description.

[0153] The control unit 24 may, instead of having the user photograph the second object 41b that is the subject of the correction process again, have the user input the correct discount information. A list of discount information, which associates objects eligible for discounts with the details of the discounts for those objects, may be stored in the storage unit 23 in advance. The control unit 24 may determine the validity of the discount information input by the user using the discount information list. The control unit 24 may also have the user select the correct discount information using the discount information list and a set of options.

[0154] As described above, when the information processing device 20 recognizes a discount image, it displays on the captured image that the discount image has been recognized, allowing the user to easily recognize that they have recognized the discount information. Furthermore, when the information processing device 20 adds an object containing discount information to the purchase list, it displays on the purchase list that the object contains discount information, allowing the user to easily recognize that they have recognized the discount information.

[0155] (Operation of the system in modified example 3) Figure 29 is a flowchart showing the procedure of an information processing method according to Modification 3 of the present disclosure. This information processing method includes steps S401 to S415. Descriptions that overlap with those in Steps S101 to S114 will be omitted.

[0156] Steps S401 to S404 are the same processes as steps S101 to S104 described above, so their description is omitted.

[0157] The control unit 24 performs recognition processing on the symbol 342 assigned to the object 341 and the discount image 345 assigned to the object 341, and obtains information about the object 341 and discount information (step S405).

[0158] The control unit 24 performs object recognition processing on object 341 and recognition processing on the discount image 345 attached to object 341, and obtains information about object 341 and discount information.

[0159] Steps S407 to S410 are the same process as steps S107 to S110 described above, so their description is omitted.

[0160] The control unit 24 determines whether or not it has received an input to modify the information of object 341 or the discount information assigned to object 341 (step S411).

[0161] When the control unit 24 receives an input to modify the information of object 341 or the discount information assigned to object 341 (step S411: YES), it proceeds to step S412.

[0162] If the control unit 24 does not receive an input to modify the information of object 341 or the discount information assigned to object 341 (step S411: NO), it proceeds to step S413.

[0163] The control unit 24 receives the item to be modified from the information of object 341 or the discount information assigned to object 341 (step S412). When the item to be modified is received, the process proceeds to step S102.

[0164] Steps S413 to S415 are the same process as steps S112 to S114 described above, so their description is omitted.

[0165] As explained above, the information processing method displays on the captured image that a discount image has been recognized when the discount image is recognized, allowing the user to easily recognize that they have recognized the discount information. Furthermore, when an object containing discount information is added to the purchase list, the information processing method displays on the purchase list that the object contains discount information, allowing the user to easily recognize that they have recognized the discount information.

[0166] (Modification 4) Modification 4, another modification of this embodiment, will be described below with reference to Figure 30. In the embodiment described above, the case in which the control unit 24 displays the first image 31 captured by the first imaging unit 12 was described, but in Modification 4, the superimposed image 438 is superimposed on the first image 431 and displayed. Steps that overlap with the process described above will be omitted from the description.

[0167] In this modified example 4, when the control unit 24 receives an input instructing the start of accounting after the container 440 has been placed on the mounting table, it superimposes a superimposed image 438 onto the first image 431 captured by the first imaging unit and displays it on the display unit 13. The superimposed image is an image that transmits the image of the first image 431 with a first transparency α1. The superimposed image 438 may be an image large enough to cover at least the entire container 440 placed on the mounting table. In this modified example 4, the superimposed image 438 is a rectangular image large enough to cover the entire first image 431. In this modified example 4, the superimposed image 438 is black, but is not limited to this, and may be a single color such as white or gray, or two colors or more. The above gradient is acceptable.

[0168] Figure 30 shows an example of image 430 when the control unit 24 receives an operation to instruct the start of accounting. When the superimposed image 438 is superimposed on the first image 431, the visibility of the first image 431 is reduced. For example, if the superimposed image 438 is an image with black in it, the entire first image 431 becomes dark and difficult for the user to see.

[0169] The control unit 24 superimposes an explanatory image 439, which explains how to use the information processing system 3, onto the superimposed image 438. Since the explanatory image 439 is superimposed on the superimposed image 438, visibility is not reduced, and the user can clearly recognize it. For example, the explanatory image 439 includes an explanatory image 439a containing the text "Please lift the product so that the label is easy to see," and an explanatory image 439b containing an image of an object. The explanatory image 439 may be a series of still images or a moving image.

[0170] The control unit 24 starts the recognition process after the display of the explanatory image 439 has finished. When the control unit 24 starts the recognition process, it changes the transparency α of the superimposed image 438 to the second transparency α2, which is a higher value than the first transparency α1. Figure 31 is an example of image 430 when the recognition process has started. Since the superimposed image 438 has been changed from the first transparency α1 to the second transparency α2, the visibility of the first image 431 has improved.

[0171] The user takes an object of their choice from the objects 441 (objects 441a to 441c) contained in the container 440 and brings it close to the first imaging unit 12. The control unit 24 performs recognition processing on the object. The control unit 24 performs symbol recognition processing or object recognition processing on object 441 and, when it identifies the information necessary to obtain the price of the first object 441a, it changes the transparency of the superimposed image 438 that is superimposed on object 441. More specifically, it changes the second transparency α2 of the superimposed image 438 that is superimposed on object 441 to a third transparency α3, which is a higher value than the second transparency α2. Figure 32 is an example of image 430 when object 441 is recognized. When the control unit 24 recognizes the first object 441a, it changes the transparency of the portion of the superimposed image 438 that is superimposed on the image of the first object 441a from the second transparency α2 to the third transparency α3. At this time, the control unit 24 may change the transparency of the superimposed image 438, which is superimposed on the information of the first object 441, such as the first name 443b and the display prompting the movement of the first object 441, from the second transparency α2 to the third transparency α3. For example, the third transparency α3 may be 100% transparency.

[0172] When a user views the explanatory image 439 displayed on the display unit 13, the first image 431 can be considered unnecessary information. If the first image 431 is clearly displayed as the background of the explanatory image 439, the user may not be able to concentrate on viewing the explanatory image 439. In particular, if the explanatory image 439 includes an image of an object sold in a store, the visibility of the explanatory image 439 may decrease. In this modified example 4, the visibility of the first image 431 is reduced and the visibility of the explanatory image 439 is improved by superimposing the superimposed image 438 on the first image 431.

[0173] Furthermore, after viewing the explanatory image 439, the visibility of the first image 431 is improved by increasing the transparency of the superimposed image 438. In addition, when the information processing system 3 performs the recognition process for object 441, the transparency of the superimposed image 438 superimposed on the image of object 441 is further increased so that the user can clearly recognize that the recognition process for object 441 has been performed. The control unit 24 may also output a sound or voice message to inform the user that the recognition process for object 441 has been performed.

[0174] Furthermore, the means for changing the visibility of the first image 431 is not limited to using the superimposed image 438. For example, the control unit 24 may superimpose the superimposed image 438 onto the first image 431. Alternatively, by adjusting the brightness, lightness, saturation, and contrast of all or part of the first image 431, the same or similar effect as when using the superimposed image 438 can be achieved.

[0175] Furthermore, the control unit 24 may superimpose the superimposed image 438 only on images of objects in the space from the second upper surface 10t up to a predetermined height. Specifically, the control unit 24 may superimpose it on images of objects in the space from the second upper surface 10t up to a height d1. In other words, the superimposed image 438 does not need to be superimposed on images of objects in the space from the second upper surface 10t up to a height d1 or higher. In this case, an object 441 taken out of the container 440 by the user up to a height d1 or higher will not be affected by the superimposed image 438 and will be clearly visible. Instead of setting the transparency of the portion of the superimposed image 438 that overlaps with the image of the first object 441a to a third transparency α3, the superimposed image 438 may be displayed in a way that emphasizes the image of the first object 441a.

[0176] (Operation of the system in modified example 4) Figure 33 is a flowchart showing the procedure of an information processing method according to Modification 4 of this disclosure. This information processing method includes steps S501 to S517. Note that descriptions that overlap with the descriptions of steps S101 to S114 described above are omitted.

[0177] When the control unit 24 receives an input instructing the start of accounting, it acquires a first image 331 including the object 441 (step S501).

[0178] The control unit 24 displays the first image 331, the superimposed image 438 superimposed on the first image 331, and the explanatory image 439 superimposed on the superimposed image 438 in the first area 13a of the display unit 13 (step S502).

[0179] The control unit 24 changes the first transparency α1 of the superimposed image 438 displayed on the display unit 13 to the second transparency α2 (step S503).

[0180] Steps S504 to S509 are processed in the same way as steps S101 and S103 to S107 described above, so their description is omitted.

[0181] The control unit 24 changes the second transparency α2 of the superimposed image 438 superimposed on the recognized object 441 and name 443 to the third transparency α3 (step S510).

[0182] Steps S511 to S513 are the same processes as steps S108 to S110 described above, so their description is omitted.

[0183] The control unit 24 determines whether or not it has received an input to modify the information of object 441 (step S514).

[0184] When the control unit 24 receives input to modify the information of object 441 (step S514: YES), it proceeds to step S515.

[0185] If the control unit 24 does not receive an input to modify the information of object 441 (step S514: NO), it proceeds to step S516. Here, "when no input is received" may include cases where no input is received for a certain period of time, or when an input related to the processing in step S515, which will be described later, is received.

[0186] The control unit 24 receives the first image 431 from the first imaging unit 12 via the communication unit 21 and acquires the data of the first image 431 (step S515). After that, step Return to S505.

[0187] The control unit 24 determines whether or not it has received an input to settle the temporarily stored objects (step S516).

[0188] When the control unit 24 receives an input to settle the temporarily stored objects (step S516: YES), it proceeds to step S517.

[0189] If the control unit 24 does not receive an input to settle the temporarily stored object (step S516: NO), it returns to step S515. Here, "not receiving an input to settle" includes cases such as when a new object is recognized before receiving an input to settle.

[0190] The processes in steps S517 and S518 are the same as those in steps S113 and S114 described above, so their description is omitted.

[0191] (Variation 5) Modification 5, another modification of this embodiment, will be explained using Figure 34. Steps that overlap with the processes described above will be omitted. In this embodiment, the measurement unit 15 was described as being located above the second upper surface 10t, but in Modification 5, the measurement unit 515 is fixed to the support column 11 and is located at the same height as the first imaging unit 12. Also, in this embodiment, the control unit 24 was described as performing recognition processing on an object located at a height d1 higher than the height from the second upper surface 10t to the top 40a of the container 40, but in Modification 5, the control unit 24 performs recognition processing on an object located at a height d3 higher than a predetermined height from the second upper surface 10t.

[0192] In modified example 5, the measuring unit 515 is fixed to the support column 11 and is located at the same height as the first imaging unit 12. That is, it is configured integrally with the first imaging unit.

[0193] Figure 34 shows an example in which a container 40 containing an object is placed on the second upper surface 10t. The control unit 24 controls the measuring unit 515 to perform category recognition processing for objects located at a height d3 higher than a predetermined height from the second upper surface 10t. More specifically, the measuring unit 515 measures the distance D from the measuring unit 515 to the second upper surface 10t. The control unit 24 calculates the distance d4 from the measuring unit 515, i.e., the first imaging unit 12, from the difference between the measured distance D and the predetermined height d3. The control unit 24 performs category recognition processing for objects located within a distance d4 from the first imaging unit 12. Here, the predetermined height d3 includes, for example, the height of an adult male's waist when he is standing upright, the height of an adult male's elbow when he is standing upright, the height of an adult male's shoulder when he is standing upright, etc. The predetermined height d3 may be changed as appropriate based on the physical information of the user performing the accounting. For example, the distance from the second upper surface 10t to the user's waist height may be calculated using an image captured by the second imaging unit 14, or obtained from user information stored in the storage unit 23 beforehand.

[0194] Based on the above explanation, the information processing device 20 according to Modification 5 can improve user convenience by allowing the user to change the height threshold used to determine whether or not there is an intention to purchase.

[0195] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications, alterations, or combinations of embodiments of this disclosure are included within the scope of this disclosure.

[0196] This disclosure has been described based on this embodiment and its modifications. In this embodiment and its modifications, components with the same name have the same function unless otherwise specified. For example, container 40 and container 140 have the same function.

[0197] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, the first information processing device may swap the identifiers "First" and "Second" with those of the second information processing device. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers.

[0198] The input for selecting the image of an object and the recognition result of an object displayed in the display unit 13 in this disclosure is not limited to those described above, and the selection may be made for a predetermined period of time or longer. [Explanation of symbols]

[0199] 1. Settlement System 2 Network 3. Information Processing Systems 4 servers 10 Mounting platform 10s 1st top surface 10t 2nd top surface 11 Support pillar 12 First imaging unit 13 Display section 14. Second imaging unit 15 Measuring part 20 Information Processing Devices 21 Communications Department 22 Input section 23 Memory section 24 Control Unit 31 Image 1 40 containers 41 Object 41a 1st object 42 Symbols 42a First Symbol 43 Name 43a 1st name 44 Registered Images 44a First registered image 345 discount images 345a First discounted image

Claims

1. An acquisition unit that acquires a first image including an image of an object, A control unit that acquires information about the object based on the first image, The system comprises a first region for displaying a first image, a second region, and a display unit including a third region adjacent to the first region. The control unit causes the information of the object that has moved from the first region to the third region to be displayed in the second region. Information processing system.

2. The information processing system according to claim 1, wherein the third region is located between the first region and the preceding second region.

3. The control unit replaces the information of a specified object among the information of the object displayed in the second region with information of the object obtained based on an image of the object that has moved from the region outside the first region on the third region side to the first region. The information processing system according to claim 2.

4. A step of acquiring a first image containing an image of an object, A step of recognizing the object and acquiring information about the object, The process includes the step of displaying the information of the object in the second region when the image of the object moves from the first region where the first image is displayed to the third region located between the first region and the second region where the information of the object is displayed. Information processing methods.

5. An acquisition unit that acquires a first image including an image of an object, A control unit that acquires information about the object based on the first image, The system comprises a first region for displaying a first image, a second region, and a display unit including a third region adjacent to the first region. The control unit causes the information of the object that has moved from the first region to the third region to be displayed in the second region. Information processing device.

Images