Information processing system, information processing method, and information processing device

Abstract

An information processing system according to an embodiment of the present disclosure comprises a control unit which determines, according to a category of an object, whether to execute recognition processing for a symbol granted to the object in order to identify the object.

Description

Information processing system, information processing method, information processing device CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2023-028598, filed on February 27, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an information processing system, an information processing method, and an information processing device.

[0003] 2. Description of the Related Art Conventionally, information processing devices are known that recognize products from product images and determine whether the products are subject to barcode recognition processing or object recognition processing.

[0004] JP 2022-66264 A

[0005] An information processing system according to one embodiment of the present disclosure includes a control unit that determines whether or not to perform a recognition process of a symbol assigned to an object in order to identify the object, depending on the category of the object.

[0006] An information processing method according to one embodiment of the present disclosure includes the steps of: acquiring an image including an object; identifying a category of the object from the image; and determining, depending on the category, whether to perform a recognition process on a symbol assigned to the object or to perform an object recognition process on the image of the object.

[0007] An information processing device according to one embodiment of the present disclosure includes a control unit that determines, depending on the category of the object, whether to perform a recognition process of a symbol assigned to the object or to perform an object recognition process on an image of the object in order to identify the object.

[0008] 1 is a schematic configuration of a checkout system according to an embodiment of the present disclosure. FIG. 2 is a schematic configuration of the information processing system shown in FIG. 1. FIG. 3 is a block diagram of the information processing device shown in FIG. 2. FIG. 4 is a table on which a container is placed according to an embodiment of the present disclosure. FIG. 5 is a captured image acquired by an imaging unit. FIG. 6 is a view of the table when object category recognition processing is performed. FIG. 7 is an example of a captured image 31 including an object picked up by a user. FIG. 8 is an image displaying the name of the object. FIG. 9 is an image of a shopping list displaying the name of the object. FIG. 10 is an example of a display of a correct object candidate on the display unit 13 when an input indicating that an object has been incorrectly recognized is received. FIG. 11 is an example of an image displaying a shopping list in which an object is registered. FIG. 12 is an image prompting the user to take a photo of the object again. FIG. 13 is an image after all objects contained in a container have been recognized. FIG. 14 is an image prompting the user to check whether there are any unrecognized objects. FIG. 15 is a first flowchart illustrating the procedure of an information processing method according to an embodiment of the present disclosure. FIG. 16 is a second flowchart illustrating the procedure of an information processing method according to an embodiment of the present disclosure. FIG. 17 is an example of an information processing system when an object is placed on the top surface of the table. FIG. 18 is an example of an image including an object placed on the top surface of the table. 1 is a flowchart showing the procedure of an information processing method according to a first modification of the present disclosure. FIG. 2 is a flowchart showing the procedure of an information processing method according to a first modification of the present disclosure. FIG. 3 is a flowchart showing the procedure of an information processing method according to a second modification of the present disclosure. FIG. 4 is a flowchart showing the procedure of an information processing method according to a second modification of the present disclosure. FIG. 5 is a flowchart showing the procedure of an information processing method according to a third modification of the present disclosure. 10 is an example of an image displayed when the control unit receives an instruction to start a transaction.1 is an example of an image displayed on the display unit when recognition processing is started; FIG. 2 is an example of an image displayed on the display unit when an object is recognized; FIG. 3 is a first flowchart showing the procedure of an information processing method according to a fourth modification of the present disclosure; FIG. 4 is a second flowchart showing the procedure of an information processing method according to a fourth modification of the present disclosure; FIG. 5 is a view of a mounting table on which a container containing an object is placed.

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] (System Configuration) An overview of the settlement system 1 will be described using Figure 1. The settlement system 1 may be, for example, a POS (Point of Sales) system. The settlement system 1 includes at least one information processing system 3 and a server 4. In this embodiment, the settlement system 1 may include multiple information processing systems 3.

[0011] The information processing system 3 may be installed in any store, such as a convenience store, a supermarket, or a restaurant.

[0012] The information processing system 3 and the server 4 are connected via a network 2 and can communicate with each other. The network 2 may be any network including the Internet.

[0013] The information processing system 3 transmits information such as the total price of the objects purchased by the customer, information on each object purchased by the customer, and the price of the objects purchased by the customer as processing results to the server 4 via the network 2.

[0014] The server 4 receives the processing results of the information processing system 3 from the information processing system 3 via the network 2. The server 4 executes software for managing inventory status, etc., and manages the inventory status, etc., of the store where the information processing system 3 is installed based on the received processing results.

[0015] An overview of the information processing system 3 will be described with reference to FIG.

[0016] The information processing system 3 according to this embodiment is, for example, a cash register. "Cash register" is an abbreviation of "cash register" and is used, for example, to settle and pay for the amount of merchandise to be purchased. The information processing system 3 may be an unmanned cash register, a manned cash register, a self-service cash register, or a semi-self-service cash register.

[0017] 2 , the information processing system 3 includes a mounting table 10 and an information processing device 20. The information processing system 3 further includes a support column 11, a first imaging unit 12, a display unit 13, a second imaging unit 14, and a measurement unit 15. In this embodiment, the information processing device 20 is configured as a device separate from the first imaging unit 12, the display unit 13, and the measurement unit 15. The information processing device 20 may be configured integrally with at least one of the display unit 13, the first imaging unit 12, the measurement unit 15, the support column 11, the mounting table 10, and the second imaging unit 14.

[0018] The mounting table 10 includes at least a first top surface 10s and a second top surface 10t. The first top surface 10s and the second top surface 10t may be adjacent to each other. The first top surface 10s and the second top surface 10t are formed flat so that a user can place an object to be purchased or a container such as a shopping basket on them. In this embodiment, the first top surface 10s and the second top surface 10t are substantially rectangular, but are not limited thereto and may be any shape, such as a circular shape. The first top surface 10s and the second top surface 10t may have different heights. In this embodiment, the first top surface 10s is configured to be higher than the second top surface 10t.

[0019] A support column 11 is attached to the second upper surface 10t. The support column 11 is a columnar member made of any material. For example, one end of the support column 11 is attached to a side of the second upper surface 10t, and the other end is located above the second upper surface 10t.

[0020] The first imaging unit 12 is attached to the end of the support column 11 so as to be able to capture an image of the second upper surface 10t from above. The first imaging unit 12 and the information processing device 20 are connected to each other so as to be able to communicate with each other. The first imaging unit 12 is controlled by the information processing device 20 and can capture images of at least a part or all of the first upper surface 10s and the second upper surface 10t of the mounting table 10.

[0021] The first imaging unit 12 is capable of generating an image of the captured subject. As an example, the first imaging unit 12 may be a camera. The first imaging unit 12 may continuously capture still images at an arbitrary frame rate. The first imaging unit 12 may capture moving images. The first imaging unit 12 may have a function that enables generation of an image of only an arbitrary space. Specifically, the first imaging unit 12 may be capable of generating an image of only a space within a specified distance range. The first imaging unit 12 may be a depth camera.

[0022] The display unit 13 is placed on, for example, the information processing device 20 and provides various information to the user. The display unit 13 may be any display. The display unit 13 and the information processing device 20 may be connected to each other. The display unit 13 is controlled by the information processing device 20 and displays the image captured by the first imaging unit 12. The display unit 13 may have the function of a touch screen of the input unit, which will be described later.

[0023] The second imaging unit 14 may be fixed to the ceiling above the mounting platform 10. The second imaging unit 14 and the information processing device 20 are connected to each other so that they can communicate with each other. The second imaging unit 14 is capable of generating an image of an imaged subject. As an example, the second imaging unit 14 may be a camera. The second imaging unit 14 is controlled by the information processing device 20 and can capture images of the mounting platform 10 and its surroundings. The image captured by the second imaging unit 14 may include a user in front of the mounting platform 10 and other users lining up to use the information processing system 3. The second imaging unit 14 may be mounted so as to capture an image of the mounting platform 10 from an angle different from that of the first imaging unit 12. The second imaging unit may be mounted so as to capture an image of the mounting platform 10 from diagonally above. The second imaging unit 14 may continuously capture still images at any frame rate. The second imaging unit 14 may capture moving images. The second imaging unit may be capable of generating an image of only any space. Specifically, the second imaging unit 14 may be capable of generating an image of only the space within a specified distance range. The second imaging unit 14 may be a depth camera.

[0024] The measurement unit 15 is configured to be able to measure the distance to an object. In this embodiment, the measurement unit 15 is able to measure the height of an object by measuring the distance to the object placed on the second upper surface 10t of the mounting table 10. The measurement unit 15 may be, for example, a depth sensor, a laser rangefinder, or the like. In this embodiment, the measurement unit 15 is located on the second upper surface 10t and on the optical axis of the lens of the first image capture unit 12, but is not limited to this. The measurement unit 15 may be configured integrally with the first image capture unit 12.

[0025] An overview of the information processing device 20 will be described using Fig. 3. The information processing device 20 is configured by, for example, a computer, and includes a communication unit 21, an input unit 22, a storage unit 23, and a control unit 24. The information processing device 20 executes software for realizing the various functions described in this embodiment.

[0026] The communication unit 21 includes at least one communication module connectable to the network 2. The communication module may be, for example, a communication module compatible with standards such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 21 is connected to the network 2 via the wired LAN or wireless LAN by this communication module.

[0027] The communication unit 21 includes a communication module capable of communicating with the first imaging unit 12, the display unit 13, and the second imaging unit 14. The communication module is compatible with the standards of the communication line. Communication by the communication unit 21 may be performed by at least one of a wired and a wireless method. The communication unit 21 functions as an acquisition unit that acquires an image of an object placed on the second top surface 10t from at least one of the first imaging unit 12 and the second imaging unit 14.

[0028] The input unit 22 includes at least one input interface capable of receiving input from a user. The input interface is, for example, a touch screen, physical keys, capacitance keys, a pointing device, or a microphone that is provided integrally with the display unit 13. In this embodiment, the input unit 22 is a touch screen that is provided integrally with the display unit 13 and receives input from a user.

[0029] The storage unit 23 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory may be, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM may be, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM may be, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 23 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 23 stores programs and data used in the operation of the information processing device 20, as well as data obtained by the operation of the information processing device 20. The storage unit 23 may store images of objects sold in a store in which the information processing system 3 is installed. The storage unit 23 may also store a product information database. The product information database may include a correspondence table showing the correspondence between the names and prices of objects sold in the store where the information processing system 3 is installed and information such as an ID for identifying the object. This information may be stored in the server 4 and acquired by the information processing system 3 as needed.

[0030] The control unit 24 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 24 is configured to be able to control the communication unit 21, the input unit 22, and the storage unit 23. The control unit 24 executes processing related to the operation of the information processing device 20 while controlling each unit of the information processing device 20.

[0031] <Specific Example of Operation of Information Processing Device 20> (Recognition Process) The process from when a user starts a checkout for a product until object recognition is performed will be described in detail below with reference to FIGS. 4 to 9. FIG.

[0032] The user stands in front of the information processing system 3 holding a container 40 containing an object that is a product they wish to purchase. The container 40 may be, for example, a "basket" installed in a store for customers to place products they wish to purchase. The user operates the touch screen of the display unit 13 and inputs an instruction to start the transaction through the input unit 22. The user then places the container 40 on the second upper surface 10t of the platform 10. Alternatively, the user may place the container 40 on the second upper surface 10t of the platform 10 and then input an instruction to start the transaction through the input unit 22.

[0033] FIG. 4 illustrates an example in which a container 40 containing an object is placed on the second top surface 10t. When the container 40 is placed on the second top surface 10t, the control unit 24 controls the measurement unit 15 to acquire the height d1 from the second top surface 10t to the top 40a of the container 40. Specifically, the measurement unit 15 measures the height d1 from the measurement unit 15 to the top 40a of the container 40. The control unit 24 may calculate the height d1 using an image of the container 40 captured obliquely from above by the second imaging unit 14. The control unit 24 may also acquire height information of the container 40 stored in advance in the memory unit 23. Note that the height d1 is a reference position for determining whether or not an object has been removed from the container 40, as will be described later. This reference position does not necessarily have to be the top of the container 40, but may be any position at which a user can remove an object from the container 40 and determine that they have an intention to purchase the object. For example, the reference position may be set to the height of the user's elbow using an image captured by the first image capturing unit 12 of the user.

[0034] The control unit 24 acquires a first image 31 captured by the first imaging unit 12 or the second imaging unit 14. FIG. 5 shows an example of the first image 31 captured by the first imaging unit 12. The first image 31 includes images of a container 40 placed on the second upper surface 10t by the user and a first object 41a, a second object 41b, a third object 41c, and a fourth object 41d, which are objects 41 contained in the container 40. The control unit 24 may acquire the first image 31 captured by both the first imaging unit 12 and the second imaging unit 14. In this embodiment, the control unit 24 acquires the first image 31 by receiving the first image 31 captured by the first imaging unit 12 via the communication unit 21.

[0035] The control unit 24 performs a category recognition process, which is a first recognition process, on an object located farther from the second top surface 10t than the top portion 40a. The category recognition process is a process for acquiring category information, which is a classification for classifying objects. Examples of categories include the shape of the object, the type of object, the type of packaging, and the sales method of the object. More specifically, the shape of the object may be classified based on whether it is spherical, cubic, elongated, or the like. The type of object may be classified based on whether it is vegetable, fruit, confectionery, meat, or fresh fish. The type of packaging may be classified based on whether it is boxed, bagged, or unpackaged. The sales method of the object may be classified based on whether it is sold by weight, sold individually, or sold in bulk. The category to which an object belongs may be identified from an image containing the object using a deep neural network (DNN) or pattern matching, for example.

[0036] FIG. 6 illustrates an example of the state of the placement platform 10 when object category recognition processing is performed. When a user removes a first object 41a from the container 40, the first object 41a moves from the top 40a to a position away from the second top surface 10t. The control unit 24 performs category recognition processing on the first object 41a when the distance from the second top surface 10t to the first object 41a is equal to or greater than height d1. The control unit 24 may control the device not to perform category recognition processing when the container 40 is not placed on the second top surface 10t. This prevents the category recognition processing from being performed on an object captured by the first imaging unit 12 before the container 40 is placed on the second top surface 10t. The control unit 24 may perform category recognition processing on the first object 41a when it detects that the first object 41a, which is at a height d1 or greater from the second top surface 10t, is being held in the user's hand. By subjecting an object held in the hand to category recognition processing, it is possible to prevent category recognition processing from being performed on an object captured by the first imaging unit 12 while inside the container 40.

[0037] The control unit 24 determines whether to perform object recognition processing, which is a second recognition processing, on the first object 41 a, or whether to perform symbol recognition processing, which is a third recognition processing, on the first symbol 42 a assigned to the first object 41 a, depending on the category identified for the first object 41 a. Note that in this specification, the term "recognition processing" may refer to any one or all of the category recognition processing, object recognition processing, and symbol recognition processing.

[0038] The object recognition process is a process of identifying information necessary to acquire the price of the first object 41a from, for example, the shape, color, size, etc. of the object. The object recognition process may be a process including an image recognition process.

[0039] The symbol recognition process is a process of identifying information necessary to acquire the price of the first object 41 a by reading a symbol. The symbol includes, for example, a one-dimensional code or a two-dimensional code such as a barcode or a two-dimensional barcode attached to the object 41.

[0040] In this embodiment, for example, it is assumed that a store sets prices according to weight for "items sold by weight," "meat," "fresh fish," and "prepared dishes." When the control unit 24 determines that the category of the first object 41a is any of "items sold by weight," "meat," "fresh fish," and "prepared dishes," the control unit 24 performs a symbol recognition process on the first symbol 42a. "Items sold by weight" are items whose prices are determined according to the quantity purchased by a customer, such as nuts and alcoholic beverages. "Meat" is, for example, beef, pork, chicken, and ground meat. "Fresh fish" is, for example, tuna, salmon, and cod. "Prepared dishes" is, for example, potato salad, namul, and pork cutlet. When the control unit 24 determines that the category of the first object 41a is not any of "items sold by weight," "meat," "processed meat," "fresh fish," or "prepared dishes," the control unit 24 performs an object recognition process on the first object 41a. The type of object to be subjected to the symbol recognition process may be set in advance in the information processing system according to the sales format of the store.

[0041] 7 is an example of a first image 31 including a first object 41a being picked up by a user. For example, when the control unit 24 identifies the category of the first object 41a as "meat," the control unit 24 performs symbol recognition processing on the first symbol 42a. For example, when the control unit 24 identifies the category of the first object 41a as "confectionery," the control unit 24 performs object recognition processing on the image of the first object 41a.

[0042] Objects priced according to weight, such as bulk products, often have similar appearances because they are packaged in the same containers and packaging regardless of their weight. Therefore, accurately recognizing and pricing these items using object recognition processing is difficult. Therefore, for such items, symbol recognition processing can be performed to obtain weight information from symbol information, enabling accurate accounting. On the other hand, objects wrapped in a specific package, such as chocolate, often have distinctive appearances and are easily distinguishable from other objects, allowing accurate recognition using object recognition processing. Furthermore, the symbols 42 attached to each item are located in different positions, requiring users to search for the symbols 42. For such items, performing object recognition processing can provide convenience to users.

[0043] The information processing device 20 performs a category recognition process, and then executes a symbol recognition process and an object recognition process using a single component, the control unit 24. However, the information processing device 20 is not limited to this. For example, the information processing device 20 may include a first recognition unit that executes the category recognition process, a second recognition unit that executes the symbol recognition process, and a third recognition unit that executes the object recognition process, and executes each process.

[0044] The control unit 24 acquires information about the first object 41a through the determined symbol recognition process or object recognition process. The information about the first object 41a is information necessary for acquiring the price of the first object 41a, such as the name of the object and an ID that can identify the object. The control unit 24 acquires price information about the first object 41a using the acquired ID, etc. For example, the control unit 24 performs symbol recognition process on the first symbol 42a of the first object 41a identified as belonging to the "meat" category, and acquires an ID that can identify the object. The control unit 24 acquires the object name and price corresponding to the identified ID, using a product information database, etc. stored in the memory unit 23. For example, the control unit 24 performs object recognition process on the first object 41a identified as belonging to the "confectionery" category, and acquires the object name and ID that can identify the object. The control unit 24 acquires the object name and price corresponding to the identified ID, using a product information database, etc. stored in the memory unit 23. The control unit 24 may acquire the price of the first object 41 a from the price tag attached to the first object 41 a in the object recognition process. The control unit 24 may also acquire a registered image of the object from registered images of the product stored in the storage unit 23 using the name, ID, etc. of the object.

[0045] The control unit 24 displays a first name 43a, which is the object name of the acquired first object 41a, on the display unit 13. FIG. 8 shows an example of an image 30 displaying the first name 43a. The image 30 includes a first area 13a displaying a first image 31, a second area 13b displaying a second image 32, and a third area 13c displaying a boundary 33. In this embodiment, the first image 31 is displayed on the right side of the image 30, and the second image 32 is displayed on the left side of the image 30, but this is not limited thereto. The first image 31 includes an image of the first object 41a captured by the first imaging unit 12 and a display of the first name 43a acquired by the control unit 24. In this embodiment, the first name 43a is displayed below the image of the first object 41a, but this is not limited thereto and the first name 43a may be displayed above, to the left, or to the right of the image of the first object 41a. The control unit 24 may also display a line connecting the first name 43a and the image of the first object 41a on the display unit 13. Furthermore, the first image 31 may be displayed further including an indication that prompts the user to move the image of the first object 41 a to the third area 13 c. For example, the first image 31 may be displayed further including an indication of the text "Please move the recognized product" and an indication of an arrow.

[0046] The user checks the image of the first name 43a displayed on the display unit 13, and if the first name 43a is correct and the user wants to purchase the first object 41a, the user moves the image of the first object 41a to the third area 13c. In this embodiment, the user can move the image of the first object 41a to the third area 13c by moving the first object 41a captured by the first imaging unit 12 leftward relative to the second top surface 10t (i.e., toward the first top surface 10s). If the user does not want to purchase the first object 41a, the user moves the first object 41a in a direction other than the third area 13c. In this embodiment, the user can move the first object 41a from above the second top surface 10t in at least one direction other than the leftward direction (i.e., toward the first top surface 10s) to move the image of the first object 41a outside the third area 13c.

[0047] When the image of the first object 41a is moved to the third area 13c, the control unit 24 adds the first object 41a to a shopping list as a product to be purchased. The shopping list may be information temporarily stored in the storage unit. The control unit 24 also displays information about the contents of the shopping list on the second image 32. FIG. 9 is an example of an image 30 when the contents of the shopping list are displayed on the second image 32. The shopping list displayed on the second image 32 may include information about the first name 43a and the price of the first object 41a. When displaying the first name 43a on the second image 32, the control unit 24 may also display a registered image 44 of the first object 41a. Instead of the registered image 44, the control unit 24 may display a reduced image of the first object 41a captured by the first imaging unit 12 on the second image 32. If the first name 43a and price displayed on the shopping list are incorrect, it is possible that the object recognition process or the symbol recognition process is incorrect. In this case, the user can correct the recognition result by performing the correction process described below.

[0048] The control unit 24 may issue a predetermined notification when the first object 41a is added to the shopping list. The control unit 24 may issue a predetermined notification when the image of the first object 41a moves to the third area 13c. The control unit 24 may issue a predetermined notification when the image of the first object 41a moves to the second area 13b via the third area 13c. The control unit 24 may issue a predetermined notification when displaying the first name 43a on the second image 32. The control unit 24 may change the display form of the boundary 33 as the predetermined notification. The control unit 24 may change at least one of the color, size, thickness, etc. of the boundary 33 to change the display form of the boundary 33. The control unit 24 may output a notification sound, such as a "beep," as the predetermined notification. By receiving the predetermined notification, the user can recognize that the first object 41a, which is the product they want to purchase, has been added to the shopping list and can recognize that the purchase process is proceeding correctly.

[0049] The user checks the image of the first name 43a displayed on the display unit 13, and if the first name 43a is incorrect, that is, if the first object 41a has been erroneously recognized, the user inputs this fact into the input unit 22. For example, the user touches the image of "correction" included in the first image 31. Note that the control unit 24 may determine that the first object 41a has been erroneously recognized if the user does not move the image of the first object 41a to the third area 13c for a predetermined time or longer.

[0050] When the control unit 24 receives input that the first object 41a has been erroneously recognized, it causes the display unit 13 to display at least one candidate for the correct object as the first object 41a. The candidate for the correct object may be an object that is easily mistakenly recognized as the first object 41a. FIG. 10 shows an example of a candidate for the correct object displayed on the display unit 13 when input that the first object 41a has been erroneously recognized is received. The candidate for the correct object may include at least one of another object that is determined to have a high reliability in the recognition result when the control unit 24 performs object recognition processing on the first object 41a, and objects that have been predefined as being easily mistaken for each other. The user may select one of the candidates for the correct object displayed on the display unit 13. If the correct recognition result is not included, the user may input that fact into the input unit 22.

[0051] The user checks the first name 43a displayed on the second image 32 and places the first object 41a on the first upper surface 10s. At this time, the first object 41a may be placed as is, or may be placed in a container other than the container 40, such as a shopping bag or a shopping basket, placed on the second upper surface 10t.

[0052] The user also performs the above-described operation on the second object 41 b, the third object 41 c, and the fourth object 41 d. The information processing device 20 performs the category recognition process and the object recognition process / symbol recognition process on the second object 41 b, the third object 41 c, and the fourth object 41 d, thereby recognizing all of the objects that the user will purchase and acquiring price information.

[0053] (Correction Process) When the name of an object displayed on the purchase list is incorrect, the process of correcting the recognition result will be described with reference to FIGS. 11 and 12. FIG.

[0054] The user operates the touch screen of the display unit 13 to perform an operation to correct the recognition result of an object included in the shopping list from the input unit 22. The control unit 24 can accept the input of this operation. FIG. 11 is an example of a second image 32 displaying a shopping list in which a second object 41b is registered. Here, the shopping list displays "cream bun" as the second name 43b, but the second object 41b is actually "anpan" (sweet bean bun), so the second name 43b is incorrect. In other words, an incorrect result has been output in the object recognition process. The user performs an input to execute a correction process to correct the recognition result of the second object 41b. The user starts the correction process by, for example, selecting the image of the second name 43b or the second object 41b in the shopping list displayed on the display unit 13, pressing a button to correct the name, or the like.

[0055] When the control unit 24 receives an instruction to execute the correction process, the control unit 24 displays on the display unit 13 a third image 34 prompting the user to take another photograph of the object. FIG. 12 illustrates an example of the third image 34. For example, the control unit 24 may display on the display unit 13 a character string 34a that reads, "The product will be recognized again. Please photograph the product you wish to correct through the camera again." and an arrow 34b pointing toward the first area 13a as the third image 34. The control unit 24 may display on the display unit 13 the character string 34a to be displayed as the third image 34 depending on the category of the object for which the correction process is to be executed. For example, when the category of the object for which the correction process is to be executed is at least one of "products sold by weight," "meat," "fresh fish," and "prepared foods," the control unit 24 may display on the display unit 13 a character string 34a that reads, "Please photograph the barcode of the product you wish to correct through the camera again." In this embodiment, the third image 34 is displayed superimposed on the second image 32, but this is not limiting. The user checks the third image 34 and moves the object to be corrected to the top of the container 40, causing the first imaging unit 12 to again capture an image of the second object 41b, which is the object for which correct information was not obtained.

[0056] The control unit 24 again performs the category recognition process and the object recognition process / symbol recognition process on the second object 41b that has moved to the top of the container 40. These recognition processes are as described above, and therefore will not be described again.

[0057] (Payment Process) The payment process for an object added to the purchase list will be described with reference to FIGS.

[0058] 13 is an example of an image 30 after the user has caused the information processing system 3 to recognize all of the objects 41 contained in the container 40. In the first image 31, only the container 40 is displayed. In the second image 32, the names, prices, and images of the first object 41a, second object 41b, third object 41c, and fourth object 41d contained in the container 40 at the start of the checkout are included. If the user does not wish to purchase any new objects other than the object 41 included in the purchase list, the user performs a checkout operation to pay for the objects included in the purchase list. The checkout operation may be, for example, an operation of pressing a button to instruct the control unit 24 to check out the objects.

[0059] When the control unit 24 receives a checkout operation, it displays a fourth image 35 in the second area 13b. The fourth image 35 prompts the user to confirm whether all of the objects to be purchased are included in the purchase list. If the desired object is not included in the purchase list, it is assumed that the user has forgotten to perform a recognition process for that object. FIG. 14 may include text prompting the user to check for any unrecognized objects. For example, the fourth image 35 may include the text "Are there any products that have not been registered?" The fourth image 35 may also include text prompting the user to check for any unrecognized objects. For example, the fourth image 35 may include the text "Have you checked under the cart?" and an illustration pointing to the bottom of the cart.

[0060] When the fourth image 35 is displayed, the user checks whether any object 41 remains in the container 40. If there is no object 41 remaining in the container 40, the user may perform an operation to proceed with the checkout process. Furthermore, if there is an object 41 remaining in the container 40, the user may perform an operation to perform the recognition process again. For example, if the user wishes to perform the recognition process again, the user selects the image that reads "still remaining" included in the fourth image 35. For example, if the user wishes to proceed with the checkout process, the user selects the image that reads "nothing remaining" included in the fourth image 35.

[0061] If the user selects an image that is "still remaining," the control unit 24 starts the recognition process again. The recognition process overlaps with the above-described content, so a description thereof will be omitted.

[0062] When the user selects the image "No items left," the control unit 24 causes the display unit 13 to display the total price of all the items included in the shopping list stored in the memory unit 23. The user confirms the total price and pays it. The payment may be made by inserting the specified amount into an optional payment machine (not shown), or by paying directly to a store clerk. Payment may be made not only by cash, but also by any method, such as by credit card or points on a point card.

[0063] As described above, the information processing device 20 according to an embodiment of the present disclosure can efficiently recognize only objects that the user intends to purchase by recognizing an object lifted by the user to a predetermined height as an object that the user intends to purchase. Objects that the user does not intend to purchase are not recognized, so they are not mistakenly registered on the purchase list, improving user convenience. Furthermore, by using the top 40a of the container 40 as the predetermined height, the user's intention to purchase can be accurately determined. Furthermore, because the information processing device 20 recognizes an object located close to the imaging unit, it can accurately recognize the object without confusing it with other objects remaining in the container 40.

[0064] Furthermore, the information processing device 20 according to an embodiment of the present disclosure can accurately recognize an object by identifying the category of the object and determining the object recognition process according to the identified category.

[0065] 15A and 15B are flowcharts illustrating the steps of an information processing method according to an embodiment of the present disclosure. This information processing method includes steps S101 to S115. For example, when control unit 24 receives an input via input unit 22 instructing the start of a transaction, control unit 24 starts the process of step S101.

[0066] The control unit 24 acquires information on the height d1 from the second upper surface 10t of the mounting table 10 to the upper portion 40a of the container 40 (step S101).

[0067] The control unit 24 receives the first image 31 from the first imaging unit 12 via the communication unit 21, thereby acquiring data of the first image 31 (step S102).

[0068] The control unit 24 performs an object recognition process on the object 41 that is included in the first image 31 and is located at a position higher than the height d1 from the second top surface 10t, and identifies the category of the object 41 (step S103).

[0069] The control unit 24 determines whether the category of the object 41 identified in step S103 is at least one of "items sold by weight," "meat," "processed meat," "fresh fish," and "prepared food" (step S104).

[0070] When the control unit 24 determines that the category of the object 41 is at least one of "items sold by weight," "meat," "processed meat," "fresh fish," and "prepared food" (step S104: YES), the control unit 24 proceeds to step S105.

[0071] When the control unit 24 determines that the category of the object 41 is not any of "bulk sales," "meat," "processed meat," "fresh fish," or "prepared food" (step S104: NO), the control unit 24 proceeds to step S106.

[0072] The control unit 24 executes a recognition process on the symbol 42 attached to the object 41, and acquires information about the object 41 (step S105).

[0073] The control unit 24 executes an object recognition process on the image of the object included in the acquired first image 31, and acquires information about the object 41 (step S106).

[0074] The control unit 24 displays information such as the name, price, and registered image of the object 41 acquired by at least one of the processes in step S105 and step S106 on the first image 31 (step S107).

[0075] The control unit 24 determines whether the object 41 has moved to the third region 13c (step S108).

[0076] When the control unit 24 detects that the object 41 has moved to the third area 13c (step S108: YES), the control unit 24 proceeds to step S109.

[0077] When the control unit 24 does not detect that the object 41 has moved to the third area 13c (step S108: NO), the control unit 24 returns to step S107. That is, when the control unit 24 does not detect that the object 41 has moved to the third area 13c, the control unit 24 continues to display the acquired information of the object 41, such as the name, price, and registered image, on the first image.

[0078] The control unit 24 temporarily stores the acquired information about the object 41 in the storage unit 23 (step S109). The information temporarily stored in the storage unit 23 may include the name, price, etc. of the object 41.

[0079] The control unit 24 displays the acquired information about the object 41 in the second image 32, emphasizing the boundary 33 included in the third region 13c (step S110). The information about the object 41 displayed in the second image 32 may include the name, price, registered image, etc. of the object 41.

[0080] The control unit 24 determines whether or not an input to modify the information of the object 41 has been received (step S111).

[0081] When the control unit 24 receives an input to correct the information of the object 41 (step S111: YES), the control unit 24 returns to step S102.

[0082] When the control unit 24 has not received an input to modify the information of the object 41 (step S111: NO), the process proceeds to step S112. Here, when no input has been received may include when no input has been received for a certain period of time, when an input related to the processing of step S112 described later has been received, etc.

[0083] The control unit 24 determines whether or not an input to settle the temporarily stored object has been received (step S112).

[0084] When the control unit 24 receives an input to settle the temporarily stored object (step S112: YES), the control unit 24 proceeds to step S113.

[0085] When the control unit 24 does not receive an input to settle the temporarily stored object (step S112: NO), the process returns to step S102. Here, when the control unit 24 does not receive an input to settle the object includes when a new object is recognized before receiving an input to settle the object.

[0086] The control unit 24 causes the display unit 13 to display a third image 34 that prompts the user to confirm an object 41 that the user wants to purchase but that has not been recognized by the information processing device 20 (step S113).

[0087] The control unit 24 determines whether or not there are any objects remaining that have not been recognized (step S114).

[0088] When the control unit 24 determines that an unrecognized object remains (step S114: YES), the control unit 24 returns to step S102. More specifically, when the control unit 24 receives an input indicating that an object remains, the control unit 24 may determine that an unrecognized object remains.

[0089] When the control unit 24 determines that no unrecognized objects remain (step S114: NO), the control unit 24 proceeds to step S115. More specifically, when the control unit 24 receives an input indicating that no objects remain, the control unit 24 may determine that no unrecognized objects remain.

[0090] The control unit 24 causes the display unit 13 to display an image including the total price of the objects 41 temporarily stored in the storage unit 23 (step S115).

[0091] When the control unit 24 receives payment for an amount equal to or greater than the total price of the objects 41, the control unit 24 ends the transaction process.

[0092] As described above, the information processing method according to an embodiment of the present disclosure can efficiently recognize only objects that the user is highly interested in purchasing by recognizing only objects that are held up by the user. Since products that the user has no intention of purchasing are not recognized, user convenience is improved. Furthermore, the information processing device 20 according to an embodiment of the present disclosure can accurately recognize objects because it recognizes objects that are located close to the imaging unit.

[0093] (Modification 1) Modification 1 will be described below. Note that the description of steps that overlap with the above-described process will be omitted.

[0094] In the above-described embodiment, the second upper surface 10t is formed of a material that does not transmit visible light, but in Modification 1, the second upper surface 10t may be formed of a material that transmits visible light. For example, the second upper surface 10t may be formed of a material such as an acrylic resin, a resin such as polyvinyl chloride or polycarbonate, or glass. The second upper surface 10t may be formed in a mesh shape to transmit visible light.

[0095] In Modification 1, at least the bottom surface of the container 140 may be made of a material that transmits visible light. For example, the bottom surface of the container 140 may be made of at least one material selected from the group consisting of acrylic resin, resins such as polyvinyl chloride and polycarbonate, and glass. The bottom surface of the container 140 may be formed in a mesh shape to transmit visible light.

[0096] In the first modification, the control unit 24 is configured to be able to execute a first mode and a second mode, each capable of recognizing the presence of an object in a different space. In the first mode, the control unit 24 can recognize the presence of an object located in a space closer than the first distance d1 in the upward direction from the second upper surface 10t or in a space below the second upper surface 10t. In other words, in the first mode, the control unit 24 can recognize the presence of an object that is not elevated above the second upper surface 10t by the predetermined first distance d1. In addition, in the first mode, the control unit 24 can recognize the presence of an object located below the second upper surface 10t. In the first mode, recognition processes such as category recognition, object recognition, and symbol recognition may be performed on an object whose presence is recognized. On the other hand, in the second mode, only objects located in an area at least the predetermined first distance d1 from the second upper surface 10t are subject to the recognition process. The control unit 24 executes the second mode when it recognizes that a container 40 has been placed on the second upper surface 10t. The control unit 24 performs object recognition processing using the method of the present embodiment described above. When the recognition process is completed and the process transitions to the checkout process, the control unit 24 switches from the second mode to the first mode. When the control unit 24 executes the first mode and recognizes an object inside a container located on the second top surface 10t, the control unit 24 may cause the display unit 13 to display a notification that an object remains inside the container. This allows the information processing device 20 to notify the user that there is an object that the user left in the container 140 and for which recognition processing has been forgotten. Furthermore, because the second top surface 10t and the container 140 are formed of a material that transmits visible light, the control unit 24 can detect objects located in the space below the second top surface 10t. When the control unit 24 detects an object located in the space below the second top surface 10t, the control unit 24 may cause the display unit 13 to display a notification that an object remains in the space below the second top surface 10t. This allows the information processing device 20 to notify the user that there is an object that was left in the space below the second top surface 10t but not placed in the container 140 and for which recognition processing has been forgotten.

[0097] The control unit 24 may stop the first mode and execute the second mode when the container 140 placed on the second upper surface 10t satisfies a predetermined condition. The predetermined condition may be when the number of objects 141 contained in the container 140 is greater than a predetermined number, or when the distance from the objects 141 contained in the container 140 to the first imaging unit 12 is shorter than a predetermined distance. When the number of objects purchased by the user is small, it is considered that there are no objects hidden within the container 140. In such a case, the control unit 24 can perform a checkout process for all the objects 141 contained in the container 140 at once.

[0098] Furthermore, in the recognition process, the control unit 24 may change the character string displayed on the display unit 13 depending on whether the first mode is being executed or the second mode is being executed. For example, when executing the first mode, the control unit 24 may cause the first image 131 to be displayed with the character string "Please arrange the products you wish to purchase." When executing the second mode, the control unit 24 may cause the first image 131 to be displayed with the character string "Please pick up the products one by one and scan them."

[0099] The control unit 24 may execute the first mode when it recognizes that an object has been placed on the second top surface 10t. The control unit 24 may also execute a recognition process on the object placed on the second top surface 10t. If the user has only a few objects to purchase, the user may not use the container 140. Even if the user has not used the container 140 and an object has been placed directly on the second top surface 10t, the control unit 24 can execute a recognition process on the object.

[0100] (Recognition Process of Modification 1) The steps of the recognition process executed by the recognition device of Modification 1 of the present disclosure will be described with reference to FIGS. 16 and 17 .

[0101] Before the start of the transaction, the control unit 24 executes the first mode. By executing the first mode, the control unit 24 can recognize objects located within the imaging range of the first imaging unit 12.

[0102] The control unit 24 starts the transaction process when it detects an object on the second top surface 10t in the image captured by the first imaging unit 12. Furthermore, the control unit 24 may start the transaction process when it receives an input instructing the start of the transaction even if it does not detect an object.

[0103] When the control unit 24 detects a container 140 on the second top surface 10t in the image captured by the first imaging unit 12, the control unit 24 stops the first mode and starts the transaction process by executing the second mode. The description of the recognition process when the second mode is executed overlaps with the description of the recognition process described above, so it will not be repeated here.

[0104] When the first mode is being executed, the control unit 24 performs a recognition process on the objects 141 (objects 141a to 141d) placed on the second upper surface 10t. Fig. 17 shows an example of an image 30 including the objects 141a to 141d placed on the second upper surface 10t. The control unit 24 performs the above-described recognition process on each of the objects 141a to 141d.

[0105] In this embodiment, the control unit 24 executes the symbol recognition process or the object recognition process for each of the first object 141a, the second object 141b, the third object 141c, and the fourth object 141d. The control unit 24 displays the first name 143a, the second name 143b, the third name 143c, and the fourth name 143d of the objects for which the recognition process has been completed in the second image 132. At this time, the control unit 24 may display the registered image 144 in the second image 132 together with the name 143. The control unit 24 may also display the object 141 and the name 143 in association with each other in the image 130. For example, the control unit 24 may superimpose an image of a circled number 1 on the image of the first object 141a included in the first image 131, and display the character string of the first name 143a included in the second image 132 with a circled number 1 at the beginning.

[0106] (Settlement Process of Modification Example 1) When the control unit 24 receives an input to settle an object while the second mode is being executed, the control unit 24 stops the second mode and executes the first mode. When the control unit 24 receives an input to settle an object while the first mode is being executed, the control unit 24 may continue executing the first mode.

[0107] By executing the first mode, the control unit 24 notifies the user that an unregistered product remains when it detects an object 141 inside the container 140, on the second top surface 10t, or below the second top surface 10t. FIG. 18 shows an example of a fifth image 136 notifying the user that an unregistered product remains. When the control unit 24 detects a fifth object 141e, which is an unregistered product, it displays the fifth image 136, for example, stating, "There are unregistered products remaining. Please check your shopping cart or the area below the cart." At this time, the control unit 24 may output a sound simultaneously with displaying the fifth image 136. The control unit 24 may also highlight the detected fifth object 141e. For example, the control unit 24 may superimpose a colored image on the detected fifth object 141e or display a circumscribing rectangle superimposed on the fifth object 141e.

[0108] In this way, the information processing device 20 can reduce operational errors by changing the range of the area to be recognized depending on the status of the transaction, thereby improving user convenience.

[0109] 19A and 19B are flowcharts showing the steps of an information processing method according to Modification 1 of the present disclosure. This information processing method includes steps S201 to S218. Note that descriptions that overlap with the descriptions of steps S101 to S114 described above will be omitted.

[0110] The control unit 24 determines whether or not the container 140 is detected (step S201).

[0111] When the control unit 24 detects the container 140 (step S201: YES), the process proceeds to step S202.

[0112] When the control unit 24 does not detect the container 140 (step S201: NO), the control unit 24 proceeds to step S204.

[0113] Step S202 is the same process as step S101 described above, and therefore description thereof will be omitted.

[0114] The control unit 24 executes the second mode (step S203), that is, the control unit 24 recognizes only objects located at a distance equal to or greater than the height d1 from the second upper surface 10t.

[0115] The processing from step S204 to step S214 is the same as the processing from step S102 to step S112 described above, and therefore the description thereof will be omitted.

[0116] The control unit 24 executes the first mode (step S215), that is, the control unit 24 recognizes the object 141 on the second upper surface 10t.

[0117] The control unit 24 determines whether or not the object 141 is detected on at least one of the container 140 and the second upper surface 10t (step S216).

[0118] When the control unit 24 detects the object 141 on at least one of the container 140 and the second upper surface 10t (step S216: YES), the control unit 24 proceeds to step S217.

[0119] When the control unit 24 does not detect the object 141 on at least one of the container 140 and the second upper surface 10t (step S216: NO), the control unit 24 proceeds to step S218.

[0120] The control unit 24 notifies the user that there are still unregistered products (step S217). More specifically, the control unit 24 displays a fifth image 136 or outputs a voice message saying, "There are still unregistered products. Please check your shopping cart or the area below the cart." Then, the process returns to step S204.

[0121] Step S218 is the same process as step S114 described above, and therefore description thereof will be omitted.

[0122] In this way, the information processing method can reduce operational errors by changing the range of the recognition area depending on the transaction situation, thereby improving user convenience.

[0123] (Variation 2) Variation 2, which is another variation of this embodiment, will be described below with reference to FIG. 20. In the above-described embodiment, it was explained that when category recognition processing is performed on object 41 and the object is identified as a "product sold by weight," "meat," "processed meat," "fresh fish," or "prepared food," symbol recognition processing is performed. In variation 2, when category recognition processing is performed on object 241 and the object is identified as a "vegetable," it is confirmed whether the object is a "product sold in bulk." Note that a description of steps that overlap with the above-described processing will be omitted.

[0124] In this modified example 2, the same processing is performed up to step S103 as described in the present embodiment. That is, the control unit 24 performs category recognition processing on the object 241 included in the first image 231 to identify the category. Details overlap with the above-described recognition processing, so description will be omitted.

[0125] When the control unit 24 identifies the category of the first object 241a as "vegetables," it performs object recognition processing on the first object 241a. Fig. 20 is an example of a first image 231 including the first object 241a being held up by the user. For example, the first object 241a is an "onion," and the control unit 24 identifies the category of the first object 241a as "fruit and vegetables," and performs object recognition processing on the first object 241a.

[0126] The control unit 24 performs object recognition processing on the first object 241a and determines whether the first object 241a is an object sold individually. More specifically, the control unit 24 identifies the first object 241a and uses a product information database or the like stored in the storage unit 23 to confirm whether the first object 241a is an object sold individually. Objects sold individually are objects that are not assigned a symbol and are sold individually. The product information database is assumed to store information indicating whether or not an object sold individually is available. The control unit 24 may also identify the extent of the packaging of the first object 241a and the number of first objects 241a contained in the packaging. If the number is other than a predetermined number, the control unit 24 may determine that the first object 241a is a product sold individually. Here, the predetermined number may be a predetermined number of first objects 241a that are packaged and sold in one package. When the control unit 24 determines that the first object 241a is a product sold individually and detects multiple identical first objects 241a in the first image 231, the control unit 24 causes the display unit 13 to display an image prompting the user to input the number of first objects 241a. Specifically, the first object 241a is an "onion" that is sold individually in the sales area of ​​a store. The user is trying to purchase multiple "onions" by putting them in a bag provided in the store. Alternatively, multiple "onions" may be sold in advance in a single bag. The control unit 24 recognizes "onions" placed in a single bag in this way, and when the control unit 24 detects multiple "onions" in the bag, the control unit 24 causes the display unit 13 to display an image prompting the user to input the number of "onions." When the control unit 24 determines that the first object 241a is not a product sold individually, the control unit 24 causes the first name 243a to be displayed in the first image 231. This allows the information processing device 20 to display a message prompting the user to input the number of objects 241 only when the user needs to input the number of objects 241, thereby improving the convenience of the information processing device 20.

[0127] FIG. 21 illustrates an example of a sixth image 237 that prompts the user to input the number of first objects 241a. The sixth image 237 is displayed including an image with numbers 0 to 9 written on it. The sixth image 237 may be displayed including a first name 243a, a first registered image 244a, the price of the first object 241a, and the like. The control unit 24 accepts input of the number of first objects 241a by selecting an image with numbers 0 to 9 written on it. The control unit 24 may accept input indicating that input of the number of first objects 241a has been completed. The input indicating that input of the number has been completed may be, for example, an input indicating that input has been completed by pressing a button. When the control unit 24 accepts that input of the number of first objects 241a has been completed, the control unit 24 temporarily stores the entered number and information about the first object 241a in the memory unit 23 as information about the object to be purchased. Furthermore, the control unit 24 displays information about the first object 241a in the second image 232.

[0128] 22 is an example of a second image 232 including information on a plurality of first objects 241a. The second image 232 may display a plurality of objects 241 as a single entity. For example, if there are three "onions" that are the first objects 241a to be purchased, a character string indicating the number, such as "x3," may be displayed to the right of one first registered image 244a, or the character string "x3" may be displayed following "onion" in the first name 243a. The presence of a plurality of first objects 241a may be indicated by displaying a plurality of first registered images 244a in a stacked manner.

[0129] In this way, the information processing device 20 can reduce the burden on the user by providing a user interface according to the category of the object.

[0130] 23 is a flowchart showing the procedure of an information processing method according to Modification 2 of the present disclosure. This information processing method includes steps S301 to S303 in addition to the above-described steps S101 to S115.

[0131] The control unit 24 determines whether the category of the object 241 identified in step S104 is "fruit and vegetables" and whether the object 241 is an object sold individually in step S106 (step S301).

[0132] If the category of the object 241 identified in step S104 is "fruit and vegetables" and if the object 241 is an object sold individually in step S106 (step S301: YES), the control unit 24 proceeds to step S302.

[0133] If the category of the object 241 identified in step S104 is not "fruit and vegetables" or if the object 241 is not an object sold individually in step S106 (step S301: NO), the control unit 24 proceeds to step S107.

[0134] The control unit 24 causes the display unit 13 to display a sixth image 237 that prompts the user to input the number of objects 241 that have been subjected to the object recognition process (step S302).

[0135] The control unit 24 determines whether or not the input of the number of objects 241 has been received (step S303).

[0136] When the control unit 24 receives the input of the number of objects 241 (step S303: YES), the control unit 24 proceeds to step S109.

[0137] When the control unit 24 has not received the input of the number of objects 241 (step S303: NO), the control unit 24 returns to step S302. That is, the control unit 24 continues to display the sixth image 237 until the input of the number of objects 241 is received.

[0138] In this way, the information processing method can reduce the burden on the user by prompting the user to change the next checkout operation to be performed depending on the object category.

[0139] (Variation 3) Variation 3, which is another variation of this embodiment, will be described below with reference to FIG. 24. Variation 3 describes the recognition process when discount information is attached to an object. The recognition process of the object itself in Variation 3 is the same as in this embodiment described above. Note that a description of steps that overlap with the process described above will be omitted.

[0140] If the object the user is attempting to purchase has discount information, such as a discount sticker, attached to it, the image of the first object 341a included in the first image 331 captured by the first imaging unit in step S102 described in this embodiment includes an image of the discount information. When the control unit 24 recognizes that the object included in the first image 331 has discount information attached to it, it causes the display unit 13 to display a discount image 345 representing the discount information in association with the object. Specifically, it causes the first discount image 345a to be displayed near the image of the first object 341a included in the first image 331 or superimposed on the image of the first object 341a. Note that the display form of the first discount image 345a is not limited to this. Note that the pattern of the discount sticker used in the store may be registered in advance in the storage unit 23. The control unit 24 can detect the discount information included in the image of the first object 341a using a known method such as pattern matching.

[0141] FIG. 24 illustrates an example of a first image 331 including an image of a first object 341a to which discount information is attached. The image of the first object 341a includes a discount sticker that displays the discount information. When the control unit 24 recognizes that discount information is attached from the image of the first object 341a, it displays a first discount image 345a on the first image 331. The first discount image 345a is an image indicating that a discount is available or an image showing the content of the discount information. For example, when the control unit 24 recognizes discount information that reads "20% OFF," it may display the first discount image 345a that reads "20% OFF" in the upper right corner of the first image 331. When the control unit 24 recognizes discount information that reads "20 yen off," it may display the first discount image 345a that reads "20 yen off" in the upper right corner of the first image 331. The control unit 24 does not necessarily have to display the first discount image 345a in the upper right corner; it may display it anywhere on the first image 331. The control unit 24 may display the price of the first object 341a to which discount information has been applied, along with an image of the first object 341a, in the first image 331. For example, the price of the first object 341a, which is a "cut pork," is "376 yen." In this case, the control unit 24 may apply discount information such as "20% off" to the price of the first object 341a and display the price as "301 yen." To indicate that the discount information has been applied, the control unit 24 may display the price before and after the discount information has been applied. The control unit 24 may display the price after the discount information has been applied in an emphasized or differentiated manner from the price before the discount information has been applied. The control unit 24 may display the price after the discount information has been applied in an emphasized or differentiated manner by changing the font size, font thickness, font color, etc.

[0142] When displaying information about the first object 341a to which discount information has been assigned in the second image 332, the control unit 24 may also display a first discount image 345a or information indicating that the object is to which discount information has been assigned. When displaying the first discount image 345a in the second image 332, the first discount image 345a may be reduced in size. FIG. 25 illustrates an example of a case in which a shopping list to which information about the first object 341a has been added is displayed in the second image 332. The second image 332 may include a first name 343a and a first registered image 344a as information about the first object 341a. The control unit 24 may also display the first discount image 345a near the first registered image 344a. For example, the control unit 24 may reduce the first discount image 345a and display it in the lower right corner of the first registered image 344a. The control unit 24 may also display the price of the first object 341a to which the discount information has been applied. Specifically, the control unit 24 may display the price to which the discount information has been applied. The control unit 24 may display the price before the discount information has been applied and the price after the discount information has been applied. The control unit 24 may display the price after the discount information has been applied in an emphasized or distinguished manner from the price before the discount information has been applied. The control unit 24 may display the emphasized or distinguished price by changing the size, thickness, or color of the characters, for example.

[0143] (Correction Process of Modification 3) When the result of the recognition process for an object added to the purchase list or the discount information is incorrect, the process of correcting the name or discount information will be described with reference to FIGS. 26, 27, and 28. FIG.

[0144] FIG. 26 is an example of a second image 332 displaying a shopping list to which information about a second object 341b has been added. The control unit 24 may accept an operation to modify at least one of the recognition result and discount information for the second object 341b included in the second image 332. The process for modifying the object recognition result is similar to the process described above, and therefore will not be described here. For example, in FIG. 26, a reduced second discount image 345b displayed together with the second object 341b indicates "10 yen off." However, the correct discount information attached to the second object 341b is "10% off." In this case, since "10% off" is correct for the price of the second object 341b, the user performs an input to perform a modification process to modify the second name 343b. To start the modification process, the user may, for example, select an image of the second name 343b or the second object 341b on the display unit 13, press a button to modify the name, or perform other input.

[0145] When the control unit 24 receives an input to execute a correction process for correcting the object recognition result and the discount information, the control unit 24 displays a correction image 337 on the display unit 13, which prompts the user to input the items to be corrected. FIG. 27 is an example of the correction image 337 that prompts the user to input the items to be corrected. The correction image 337 includes a character string that reads, "Please select the item you want to correct." The correction image 337 may include an image that includes the character strings "product" and "discount information," which are candidates for the items to be corrected.

[0146] When the control unit 24 receives an input to correct the object recognition result, it causes the display unit 13 to display a third image 334a that prompts the user to take another photograph of the object 341. The details are the same as those of the above-described process, and therefore will not be described again.

[0147] When the control unit 24 receives an input to correct the discount information, it displays a third image 334b ​​on the display unit 13, prompting the user to re-photograph the second object 41b containing discount information related to the discount information. FIG. 28 shows an example of a third image 334b ​​prompting the user to re-photograph the second discount image 345b. For example, the control unit 24 may display on the display unit 13 a text string that reads, "The product will be re-recognized. Please re-capture the product you wish to correct." along with an arrow pointing toward the first area 13a, as the third image 334b ​​prompting the user to re-photograph. In this embodiment, the third image 334b ​​is displayed superimposed on the second image 332, but this is not limited thereto. When the user confirms the third image 334b, the user moves the object 341 to be corrected to the top of the container 340, thereby causing the first imaging unit 12 to re-image the second object 341b, the object whose discount information was incorrectly recognized.

[0148] The control unit 24 performs the recognition process again on the second discount image 345b that has moved to the upper part of the container 340. At this time, the control unit 24 may also perform the recognition process on the second object 341b. The process after recognizing the second discount image 345b again is the same as the process described above, and therefore will not be described again.

[0149] Note that the control unit 24 may have the user input correct discount information instead of having the user re-photograph the second object 41b that is the target of the correction process. The control unit 24 may store in advance in the storage unit 23 a discount information list that associates objects eligible for discounts with the details of the discounts for those objects. The control unit 24 may use the discount information list to determine the validity of the discount information input by the user. The control unit 24 may use the discount information list to have the user select correct discount information using options.

[0150] As described above, when the information processing device 20 recognizes a discount image, the information processing device 20 displays in the captured image that the discount image has been recognized, allowing the user to easily recognize that the discount information has been recognized. Furthermore, when the information processing device 20 adds an object containing discount information to a shopping list, the information processing device 20 displays in the shopping list that the object contains discount information, allowing the user to easily recognize that the discount information has been recognized.

[0151] 29A and 29B are flowcharts showing the steps of an information processing method according to Modification 3 of the present disclosure. This information processing method includes steps S401 to S415. Note that descriptions that overlap with the descriptions of steps S101 to S114 described above will be omitted.

[0152] Steps S401 to S404 are the same as steps S101 to S104 described above, and therefore description thereof will be omitted.

[0153] The control unit 24 executes a recognition process on the symbol 342 attached to the object 341 and the discount image 345 attached to the object 341, and acquires information about the object 341 and discount information (step S405).

[0154] The control unit 24 executes an object recognition process on the object 341 and a recognition process on the discount image 345 attached to the object 341, and acquires information about the object 341 and discount information.

[0155] Steps S407 to S410 are the same as steps S107 to S110 described above, and therefore description thereof will be omitted.

[0156] The control unit 24 determines whether or not an input for correcting the information on the object 341 or the discount information assigned to the object 341 has been received (step S411).

[0157] When the control unit 24 receives an input to modify the information on the object 341 or the discount information assigned to the object 341 (step S411: YES), the control unit 24 proceeds to step S412.

[0158] When the control unit 24 does not receive an input to modify the information on the object 341 or the discount information assigned to the object 341 (step S411: NO), the control unit 24 proceeds to step S413.

[0159] The control unit 24 accepts an item to be corrected from the information on the object 341 or the discount information given to the object 341 (step S412). When the item to be corrected is accepted, the process proceeds to step S102.

[0160] Steps S413 to S415 are the same as steps S112 to S114 described above, and therefore description thereof will be omitted.

[0161] As described above, the information processing method, when a discount image is recognized, displays in the captured image that the discount image has been recognized, allowing the user to easily recognize that the discount information has been recognized. Furthermore, when an object including discount information is added to a shopping list, the information processing method, when a discount image is recognized, displays in the shopping list that the object includes discount information, allowing the user to easily recognize that the discount information has been recognized.

[0162] (Variation 4) Variation 4, which is another variation of this embodiment, will be described below with reference to Fig. 30. In the above-described embodiment, the control unit 24 displays the first image 31 captured by the first imaging unit 12. However, in Variation 4, a superimposed image 438 is displayed superimposed on the first image 431. Note that a description of steps that overlap with the above-described process will be omitted.

[0163] In the fourth modification, when the container 440 is placed on the placing platform and an input instructing the start of checkout is received, the control unit 24 superimposes a superimposed image 438 on the first image 431 captured by the first imaging unit and displays the superimposed image 438 on the display unit 13. The superimposed image 438 is an image in which the image of the first image 431 is made transparent at a first transparency α1. The superimposed image 438 may be an image large enough to cover at least the entire container 440 placed on the placing platform. In the fourth modification, the superimposed image 438 is a rectangular image large enough to cover the entire first image 431. In the fourth modification, the superimposed image 438 is black, but is not limited to this and may be a single color such as white or gray, or a gradation of two or more colors, etc.

[0164] 30 is an example of image 430 displayed when control unit 24 receives an instruction to start a transaction. Superimposing superimposed image 438 on first image 431 reduces the visibility of first image 431. For example, if superimposed image 438 is a black image, first image 431 becomes dark overall, making it difficult for the user to see.

[0165] The control unit 24 superimposes an explanatory image 439 that explains how to use the information processing system 3 on the superimposed image 438. Because the explanatory image 439 is superimposed on the superimposed image 438, its visibility is not reduced and the user can clearly recognize it. For example, the explanatory image 439 includes an explanatory image 439a that includes the character string "Please lift the product so that the label can be easily seen," and an explanatory image 439b that includes an image of an object. The explanatory image 439 may be a plurality of still images or a moving image.

[0166] The control unit 24 starts the recognition process after the display of the explanation image 439 has finished. When starting the recognition process, the control unit 24 changes the transparency α of the superimposed image 438 to a second transparency α2, which is a value higher than the first transparency α1. Fig. 31 shows an example of the image 430 when the recognition process has started. Because the superimposed image 438 has been changed from the first transparency α1 to the second transparency α2, the visibility of the first image 431 has improved.

[0167] The user takes out any one of the objects 441 (objects 441a to 441c) contained in the container 440 and brings it close to the first imaging unit 12. The control unit 24 executes a recognition process on the object. The control unit 24 executes a symbol recognition process or an object recognition process on the object 441, and when the control unit 24 identifies information necessary to acquire the price of the first object 441a, changes the transparency of the superimposed image 438 superimposed on the object 441. More specifically, the control unit 24 changes the second transparency α2 of the superimposed image 438 superimposed on the object 441 to a third transparency α3, which is higher than the second transparency α2. FIG. 32 shows an example of an image 430 when the object 441 is recognized. When the control unit 24 recognizes the first object 441a, the control unit 24 changes the transparency of the portion of the superimposed image 438 superimposed on the image of the first object 441a from the second transparency α2 to the third transparency α3. At this time, the control unit 24 may change the transparency of the superimposed image 438, which is superimposed on the first name 443b, which is information about the first object 441a, and the display prompting the first object 441a to move, from the second transparency α2 to a third transparency α3. For example, the third transparency α3 may be 100% transparency.

[0168] When a user views explanatory image 439 displayed on display unit 13, first image 431 can be considered unnecessary information. If first image 431 is clearly displayed as the background of explanatory image 439, the user may not be able to concentrate on viewing explanatory image 439. In particular, if explanatory image 439 includes an image of an object sold in a store, the visibility of explanatory image 439 may be reduced. In this fourth modification, superimposed image 438 is superimposed on first image 431, thereby reducing the visibility of first image 431 and improving the visibility of explanatory image 439.

[0169] Furthermore, after viewing of the commentary image 439 is completed, the transparency of the superimposed image 438 is increased to improve the visibility of the first image 431. Furthermore, when the information processing system 3 executes the recognition process of the object 441, the transparency of the superimposed image 438 superimposed on the image of the object 441 is further increased, so that the user can clearly recognize that the recognition process of the object 441 has been executed. Note that the control unit 24 may output a sound or a voice message to notify the user that the recognition process of the object 441 has been executed.

[0170] It should be noted that the means for changing the visibility of first image 431 is not limited to using superimposed image 438. For example, instead of superimposing superimposed image 438 on first image 431, control unit 24 can adjust the luminance, brightness, saturation, and contrast of all or part of first image 431 to achieve the same or similar effect as when superimposed image 438 is used.

[0171] Furthermore, the control unit 24 may superimpose the superimposed image 438 only on images of objects present in a space ranging from the second top surface 10t to a predetermined height. Specifically, the control unit 24 may superimpose the superimposed image 438 on images of objects present in a space ranging from the second top surface 10t to a height d1. In other words, the superimposed image 438 does not need to be superimposed on images of objects present in a space at a height d1 or greater from the second top surface 10t. In this case, the object 441 removed by the user from the container 440 to a height d1 or greater is not affected by the superimposed image 438 and can be clearly seen. Instead of setting the transparency of the portion of the superimposed image 438 that overlaps with the image of the first object 441a to the third transparency α3, the superimposed image 438 may be displayed to emphasize the image of the first object 441a.

[0172] 33A and 33B are flowcharts showing the steps of an information processing method according to Variation 4 of the present disclosure. This information processing method includes steps S501 to S517. Note that descriptions that overlap with the descriptions of steps S101 to S114 described above will be omitted.

[0173] When the control unit 24 receives an input instructing the start of a transaction, the control unit 24 acquires a first image 331 including the object 441 (step S501).

[0174] The control unit 24 causes the first area 13a of the display unit 13 to display the first image 331, the superimposed image 438 superimposed on the first image 331, and the commentary image 439 superimposed on the superimposed image 438 (step S502).

[0175] The control unit 24 changes the first transparency α1 of the superimposed image 438 displayed on the display unit 13 to the second transparency α2 (step S503).

[0176] Steps S504 to S509 are similar to the processes in step S101 and steps S103 to S107 described above, and therefore descriptions thereof will be omitted.

[0177] The control unit 24 changes the second transparency α2 of the superimposed image 438 superimposed on the recognized object 441 and the name 443 to the third transparency α3 (step S510).

[0178] Steps S511 to S513 are the same as steps S108 to S110 described above, and therefore description thereof will be omitted.

[0179] The control unit 24 determines whether or not an input to modify the information of the object 441 has been received (step S514).

[0180] When the control unit 24 receives an input to correct the information of the object 441 (step S514: YES), the control unit 24 proceeds to step S515.

[0181] When the control unit 24 has not received an input to modify the information of the object 441 (step S514: NO), the process proceeds to step S516. Here, when no input has been received may include when the input has not been received for a certain period of time, when an input related to the processing of step S515 described later has been received, and the like.

[0182] The control unit 24 receives the first image 431 from the first imaging unit 12 via the communication unit 21, thereby acquiring data of the first image 431 (step S515). Then, the process returns to step S505.

[0183] The control unit 24 determines whether or not an input to settle the temporarily stored object has been received (step S516).

[0184] When the control unit 24 receives an input to settle the temporarily stored object (step S516: YES), the control unit 24 proceeds to step S517.

[0185] When the control unit 24 does not receive an input to settle the temporarily stored object (step S516: NO), the process returns to step S515. Here, when the control unit 24 does not receive an input to settle the object includes when a new object is recognized before receiving an input to settle the object.

[0186] The processes in steps S517 and S518 are the same as those in steps S113 and S114 described above, and therefore will not be described here.

[0187] (Variation 5) Variation 5, which is another variation of this embodiment, will be described using FIG. 34 . Note that a description of steps that overlap with the processes described above will be omitted. In this embodiment, the measurement unit 15 is located above the second top surface 10t. However, in Variation 5, the measurement unit 515 is fixed to the support column 11 and is located at the same height as the first imaging unit 12. Also, in this embodiment, the control unit 24 performs recognition processing on an object located at a position higher than the height d1 from the second top surface 10t to the top 40a of the container 40. However, in Variation 5, the control unit 24 performs recognition processing on an object located at a position higher than a predetermined height d3 from the second top surface 10t.

[0188] In the fifth modification, the measuring unit 515 is fixed to the support column 11 and is located at the same height as the first imaging unit 12. That is, the measuring unit 515 is configured integrally with the first imaging unit.

[0189] FIG. 34 shows an example in which a container 40 containing an object is placed on the second top surface 10t. The control unit 24 controls the measurement unit 515 to perform category recognition processing on an object located at a height higher than a predetermined height d3 from the second top surface 10t. More specifically, the measurement unit 515 measures the distance D from the measurement unit 515 to the second top surface 10t. The control unit 24 calculates the distance d4 from the measurement unit 515, i.e., the first image capture unit 12, based on the difference between the measured distance D and the predetermined height d3. The control unit 24 performs category recognition processing on an object located within the distance d4 from the first image capture unit 12. Here, the predetermined height d3 includes, for example, the waist height of an adult male when standing upright, the elbow height of an adult male when standing upright, and the shoulder height of an adult male when standing upright. The predetermined height d3 may be changed as appropriate based on the physical information of the user performing the transaction. For example, the specified height d3 may be calculated by using an image captured by the second imaging unit 14 to measure the distance from the second upper surface 10t to the height of the user's waist, or may be obtained from user information previously stored in the memory unit 23.

[0190] As explained above, the information processing device 20 according to the fifth modification example can improve user convenience by allowing the user to change the height threshold value used to determine whether or not there is an intention to purchase.

[0191] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications, alterations, or combinations of the embodiments of the present disclosure are included in the scope of the present disclosure.

[0192] The present disclosure has been described based on the present embodiment and a modified example of the present embodiment. In the present embodiment and the modified example, components with the same name have the same function unless otherwise specified. For example, container 40 and container 140 have the same function.

[0193] In the present disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in the present disclosure can exchange numbers in the configuration. For example, a first information processing device can exchange identifiers "first" and "second" with a second information processing device. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by symbols. The descriptions of identifiers such as "first" and "second" in the present disclosure should not be used solely to interpret the order of the configurations or to justify the existence of identifiers with smaller numbers.

[0194] In the present disclosure, the input for selecting the image of an object and the object recognition result to be displayed on the display unit 13 is not limited to the above, and the selection may be made for a predetermined period of time or more.

[0195] In one embodiment, (1) the information processing system includes a control unit that determines, depending on a category of an object, whether to execute a symbol recognition process for identifying the object using a symbol assigned to the object.

[0196] (2) In the information processing system of (1) above, the control unit determines whether to perform an object recognition process to identify an object based on the shape, color, or size of the object, depending on the category of the object.

[0197] (3) In the information processing system according to (1) or (2), the control unit identifies a category of the object based on an image of the object.

[0198] (4) In the information processing system according to any one of (1) to (3), the categories are classified according to the type of packaging of the object.

[0199] (5) In the information processing system according to any one of (1) to (3), the categories are classified according to the sales method of the object.

[0200] (6) In the information processing system according to any one of (1) to (5), the control unit performs the symbol recognition process when the category of the object is a commodity sold by weight.

[0201] (7) In the information processing system of (1) to (5) above, when the category of the object is fruit or vegetables, the control unit executes the object recognition process, and when it recognizes multiple identical objects and recognizes that the objects are sold individually, it displays an image on the display unit prompting the user to input the number of the objects.

[0202] (8) In the information processing system according to any one of (1) to (6), the control unit performs the symbol recognition process when the category of the object is any one of meat, processed meat, fresh fish, or prepared food.

[0203] (9) The information processing system of (1) to (8) above has a mounting section for mounting a container, and a first imaging section for acquiring an image of the container placed on the mounting section and containing a plurality of the objects, and the control section identifies the category of the object removed from the container based on the image captured by the first imaging section.

[0204] In one embodiment, (10) an information processing method includes the steps of: acquiring an image including an object; identifying a category of the object based on the image; and determining, depending on the category, whether to perform a symbol recognition process to identify the object using a symbol assigned to the object, or to perform an object recognition process to identify the object based on the shape, color, or size of the object.

[0205] In one embodiment, (11) the information processing device includes a control unit that determines, depending on the category of the object, whether to perform a symbol recognition process that identifies the object using a symbol assigned to the object, or to perform an object recognition process that identifies the object based on the shape, color, or size of the object.

[0206] REFERENCE SIGNS LIST 1 settlement system 2 network 3 information processing system 4 server 10 placing platform 10s first upper surface 10t second upper surface 11 support column 12 first imaging unit 13 display unit 14 second imaging unit 15 measuring unit 20 information processing device 21 communication unit 22 input unit 23 memory unit 24 control unit 31 first image 40 container 41 object 41a first object 42 symbol 42a first symbol 43 name 43a first name 44 registered image 44a first registered image 345 discount image 345a first discount image

Claims

1. a control unit that determines whether to execute a symbol recognition process for identifying an object using a symbol assigned to the object according to the category of the object; Information processing system.

2. a control unit that determines whether to execute an object recognition process that identifies an object based on the shape, color, or size of the object according to the object category; The information processing system according to claim 1 .

3. the control unit identifies a category of the object based on an image of the object.

3. The information processing system according to claim 1.

4. The information processing system according to claim 1 , wherein the categories are classified according to the type of packaging of the object.

5. The information processing system according to claim 1 , wherein the categories are classified according to a method of selling the object.

6. The control unit performs the symbol recognition process when the category of the object is a commodity sold by weight.

3. The information processing system according to claim 1 or 2.

7. When the category of the object is fruit and vegetables, the control unit executes the object recognition process, and when the control unit recognizes a plurality of identical objects and recognizes that the objects are sold individually, causes the control unit to display an image on the display unit that prompts the user to input the number of the objects. The information processing system according to claim 2 .

8. The control unit performs the symbol recognition process when the category of the object is any one of meat, processed meat, fresh fish, or prepared food.

3. The information processing system according to claim 1 or 2.

9. a placement section on which a container is placed; a first imaging unit that acquires an image of a container that contains a plurality of the objects and is placed on the placement unit; the control unit identifies a category of the object taken out of the container based on the image captured by the first imaging unit.

3. The information processing system according to claim 1 or 2.

10. acquiring an image including the object; identifying a category of the object based on the image; and determining whether to perform a symbol recognition process for identifying the object using a symbol assigned to the object, or to perform an object recognition process for identifying the object based on the shape, color, or size of the object, depending on the category. Information processing methods.

11. a control unit that determines, depending on a category of an object, whether to perform a symbol recognition process that identifies the object using a symbol assigned to the object, or to perform an object recognition process that identifies the object based on the shape, color, or size of the object; Information processing device.