Information processing system, information processing method, and information processing program

Abstract

To provide an information processing system, an information processing method, and an information processing program that are capable of easily registering the attribute of a user who uses a store and purchased merchandise information of the user.SOLUTION: An information processing system includes: a merchandise acquisition processing unit that obtains purchased merchandise information including information of a purchased merchandise of a user; an image acquisition processing unit that obtains a captured image obtained by capturing an image of the user; an extraction processing unit that extracts a user image of the user from the captured image that is obtained by the image acquisition processing unit; a determination processing unit that determines an attribute of the user based on the user image that is extracted by the extraction processing unit; and a registration processing unit that registers attribute information indicating the attribute of the user that is determined by the determination processing unit and the purchased merchandise information that is obtained by the merchandise acquisition processing unit, in association with each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program for registering attribute information of a user. [Background technology]

[0002] In restaurants, retail stores, and other stores, attributes such as the age and gender of users (customers) are acquired from images of the users (customers) captured by cameras in the facility, and the acquired attributes are used for marketing. For example, a system has been proposed in which cameras placed at the entrances and aisles of a store continuously capture images of customers moving around the store, and the customer's age, gender, and other attributes are determined from the captured images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4778532 Summary of the Invention [Problem to be solved by the invention]

[0004] In store marketing, it is desirable to understand the relationship between the attributes of a user and the products purchased by the user. However, with conventional technology, it is difficult to determine the relationship between the attributes of a user and the products purchased by the user. In addition, a method in which a store clerk registers the attributes of a user in association with the products purchased by the user places a heavy burden on the store clerk, and there is a problem that the accuracy is low because the determination is based on the store clerk's subjective judgment.

[0005] An object of the present invention is to provide an information processing system, an information processing method, and an information processing program that can easily register attributes of users who use a store and information on products purchased by the users. [Means for solving the problem]

[0006] The information processing system of the present invention is a system comprising: a product acquisition processing unit that acquires purchased product information corresponding to information of a user's purchased product; an image acquisition processing unit that acquires an image of the user; an extraction processing unit that extracts a user image of the user from the image acquired by the image acquisition processing unit; a determination processing unit that determines attributes of the user based on the user image extracted by the extraction processing unit; and a registration processing unit that associates and registers attribute information indicating the attributes of the user determined by the determination processing unit with the purchased product information acquired by the product acquisition processing unit.

[0007] The information processing method of the present invention is a method in which one or more processors execute the following steps: a product acquisition step of acquiring purchased product information corresponding to information of a user's purchased product; an image acquisition step of acquiring an image of the user; an extraction step of extracting a user image of the user from the image acquired in the image acquisition step; a determination step of determining attributes of the user based on the user image extracted by the extraction step; and a registration step of associating and registering attribute information indicating the user's attributes determined in the determination step with the purchased product information acquired in the product acquisition step.

[0008] The information processing program of the present invention is a program for causing one or more processors to execute the following steps: a product acquisition step for acquiring purchased product information corresponding to information of a product purchased by a user; an image acquisition step for acquiring an image of the user; an extraction step for extracting a user image of the user from the image acquired in the image acquisition step; a determination step for determining attributes of the user based on the user image extracted by the extraction step; and a registration step for associating and registering attribute information indicating the user's attributes determined in the determination step with the purchased product information acquired in the product acquisition step. Effect of the Invention

[0009] According to the present invention, there are provided an information processing system, an information processing method, and an information processing program that can easily register attributes of users who use a store and information on products purchased by the users. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of a transaction system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of an image analysis information table used in the accounting system according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram showing an example of an attribute information table used in the transaction system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of transaction information used in the transaction system according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing an example of purchased product information used in the accounting system according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing an example of user information used in the transaction system according to an embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 13]FIG. 13 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 15] FIG. 15 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 16] FIG. 16 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 17] FIG. 17 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 18] FIG. 18 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 19] FIG. 19 is a diagram showing an example of an accounting screen displayed on an accounting terminal according to an embodiment of the present invention. [Figure 20] FIG. 20 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 21] FIG. 21 is a diagram showing an example of an accounting screen displayed on an accounting terminal according to an embodiment of the present invention. [Figure 22] FIG. 22 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. [Figure 23] FIG. 23 is a flowchart showing an example of the procedure of a transaction process executed at a transaction terminal according to an embodiment of the present invention. [Figure 24] FIG. 24 is a flowchart showing an example of the procedure of a transaction process executed at a transaction terminal according to an embodiment of the present invention. [Diagram 25] FIG. 25 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings to facilitate understanding of the present invention. Note that the following embodiment is an example of the present invention and does not limit the technical scope of the present invention.

[0012] [Accounting System 100] 1, an accounting system 100 according to an embodiment of the present invention includes an accounting terminal 1, a camera 2, and a database DB. The accounting terminal 1, the camera 2, and the database DB can communicate with each other via a communication network N1, such as a LAN, the Internet, a WAN, or a public telephone line.

[0013] The camera 2 is a digital camera that captures an image of a subject and outputs it as digital image data. The camera 2 has a communication function that transmits the digital image data to the accounting terminal 1 through the communication network N1. There may be one camera 2 or multiple cameras. Specifically, the camera 2 is installed in a store and captures an image of a predetermined area including users (customers) of the store. For example, the camera 2 is installed at a cash register in the store and captures an image of a predetermined area inside the store from the cash register. The camera 2 captures an image of the predetermined area at a predetermined frame rate and transmits the image data to the accounting terminal 1. The predetermined area is, for example, an area in which an image of a user performing a transaction can be captured. The predetermined area may also be, for example, the entire area of ​​the store. In the present invention, it is also possible to use a camera already installed in the store as the camera 2. The camera 2 may also be mounted on the accounting terminal 1.

[0014] The accounting terminal 1 is a device (e.g., a POS terminal) that is installed in a store and performs accounting for items purchased by a user. The accounting terminal 1 may perform accounting by accepting operations from a store clerk or by accepting operations from the user. The accounting terminal 1 also determines the attributes (age, gender, etc.) of the user performing the accounting based on an image of a specified area captured by the camera 2, and registers attribute information indicating the user's attributes in association with purchased item information regarding the items purchased by the user. The store may be, for example, a restaurant or a retail store. In the following embodiment, a restaurant will be described as an example of a store.

[0015] In this embodiment, the accounting terminal 1 alone corresponds to the information processing system according to the present invention, but the information processing system according to the present invention may be configured to include the accounting terminal 1, the camera 2, and the database DB. Also, the accounting terminal 1, the camera 2, and the database DB may be integrated into one device.

[0016] [Accounting terminal 1] As shown in FIG. 1, accounting terminal 1 is an information processing device (e.g., a POS terminal) equipped with a control unit 11, a memory unit 12, an operation display unit 13, and a communication unit 14. Note that accounting terminal 1 is not limited to a single computer, but may be a computer system in which multiple computers work in cooperation. Furthermore, various processes executed by accounting terminal 1 may be executed in a distributed manner by one or multiple processors. Accounting terminal 1 is an example of an information processing system of the present invention.

[0017] The communication unit 14 is a communication interface that connects the accounting terminal 1 to the communication network N1 via a wired or wireless connection, and performs data communication in accordance with a specified communication protocol with one or more external devices such as a camera 2 via the communication network N1.

[0018] The operation display unit 13 is a user interface that includes a display unit, such as a liquid crystal display or an organic EL display, that displays various information, and an operation unit, such as a mouse, a keyboard, or a touch panel, that accepts operations.

[0019] The storage unit 12 is a non-volatile storage unit including a semiconductor memory, a hard disk drive (HDD), or a solid state drive (SSD) for storing various information. Specifically, the storage unit 12 stores an image analysis information table D1 for registering the analysis results of the captured image captured by the camera 2, an attribute information table D2 for registering the attributes of the user (customer) extracted from the captured image, and the like. FIG. 2 is a diagram showing an example of the image analysis information table D1, and FIG. 3 is a diagram showing an example of the attribute information table D2. The captured image includes an image of one or more users making a payment at the cash register.

[0020] As shown in FIG. 2, the image analysis information table D1 registers information such as a user number, face area (horizontal), face area (vertical), gender, age, and display flag for each face number set in the face image of the user extracted from the captured image. The user number is user identification information set in response to an operation by a store clerk to select a user image included in the captured image. The face area (horizontal and vertical) is information indicating the position and range of the face image of the user included in the captured image. The gender is the gender of the user determined by the control unit 11 based on the user image included in the captured image. The age is the age of the user determined by the control unit 11 based on the user image included in the captured image. The display flag is information indicating whether or not the attributes of the user have been displayed on the accounting screen (see FIG. 7, etc.) displayed on the operation display unit 13 of the accounting terminal 1. When the control unit 11 acquires the captured image, it analyzes the captured image (image analysis) and registers each piece of information in the image analysis information table D1.

[0021] As shown in Fig. 3, in the attribute information table D2, information such as gender, age, generation, and correction flag is registered for each user number. Each time the store clerk selects a user image in the captured image (see Fig. 9, etc.), the control unit 11 registers the gender, age, and generation in association with the user number corresponding to the order of the selection operation. The correction flag is information indicating whether or not an operation (correction operation of the present invention) (see Fig. 13, etc.) is being performed (correction is being performed) to correct at least one of the registered gender and age.

[0022] The control unit 11 registers each piece of information in an image analysis information table D1 and an attribute information table D2 based on the captured image.

[0023] In another embodiment, one or both of the image analysis information table D1 and the attribute information table D2 may be stored in another server accessible via the communication network N1 from the accounting terminal 1. In this case, the control unit 11 of the accounting terminal 1 may obtain various information from the server as necessary and execute various processes such as the accounting process described below (see FIGS. 23 and 24).

[0024] The control unit 11 also executes transaction processing based on various information stored in database DB. Database DB stores data such as transaction information D3, purchased product information D4, and user information D5. Fig. 4 is a diagram showing an example of transaction information D3, Fig. 5 is a diagram showing an example of purchased product information D4, and Fig. 6 is a diagram showing an example of user information D5.

[0025] As shown in Figure 4, transaction information D3 includes information such as the transaction number, number of items purchased, and number of people in a group for each transaction. The transaction number is identification information assigned to each transaction. The number of items purchased is the total number of items included in the transaction. The number of people in a group is the total number of people such as families and group customers, for example the number of people occupying a table in a restaurant.

[0026] 5, the purchased product information D4 includes information such as the transaction number and purchased product (ordered product) for each transaction. In the case of a group that includes multiple users, multiple purchased products are associated with and registered for the same transaction number, for example.

[0027] The transaction information D3 and purchased product information D4 may be stored in the database DB based on information entered into an order terminal (not shown), for example.

[0028] 6, user information D5 includes information such as gender, age, and generation for each transaction. The control unit 11 registers the user attributes registered in attribute information table D2 in association with the corresponding transaction number in user information D5.

[0029] The storage unit 12 also stores control programs such as an accounting program (corresponding to an information processing program of the present invention) for causing the control unit 11 to execute accounting processing (see FIGS. 23 and 24) described below. For example, the accounting program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, and is read by a reading device (not shown) such as a CD drive or DVD drive provided in the accounting terminal 1 and stored in the storage unit 12.

[0030] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 11 controls the accounting terminal 1 by executing various control programs stored in advance in the ROM or the storage unit 12 by the CPU.

[0031] Specifically, as shown in Fig. 1, control unit 11 includes various processing units such as product acquisition processing unit 111, image acquisition processing unit 112, extraction processing unit 113, determination processing unit 114, registration processing unit 115, display processing unit 116, and reception processing unit 117. Control unit 11 functions as the various processing units by executing various processes in accordance with the accounting program using the CPU. Some or all of the processing units included in control unit 11 may be configured with electronic circuits. The accounting program may be a program for causing multiple processors to function as the various processing units.

[0032] The product acquisition processing unit 111 acquires purchased product information corresponding to the information of the purchased product of the user. Specifically, the product acquisition processing unit 111 references database DB to acquire transaction information D3 and purchased product information D4 corresponding to the user. Transaction information D3 (see FIG. 5) and purchased product information D4 (see FIG. 6) are examples of the purchased product information. For example, at a restaurant, when a user finishes their meal and pays at the register, the user hands the receipt to the cashier, and the cashier reads the information code on the receipt using a reading device or the like. As a result, the product acquisition processing unit 111 acquires purchased product information of the user based on the transaction number (see FIG. 5 and FIG. 6) corresponding to the information code. The transaction number is an example of purchased product information of the present invention. The product acquisition processing unit 111 is also an example of a product acquisition processing unit of the present invention.

[0033] The image acquisition processing unit 112 acquires an image of a predetermined area including a user in front of a cash register of a store, captured by the camera 2. The image acquisition processing unit 112 is an example of the image acquisition processing unit of the present invention.

[0034] The extraction processing unit 113 extracts a user image of a user from the captured image acquired by the image acquisition processing unit 112. Specifically, the extraction processing unit 113 extracts a facial image of the user from the captured image. If the captured image includes multiple users, the extraction processing unit 113 extracts a facial image for each user. When the extraction processing unit 113 extracts a facial image, it registers information on the facial area of ​​the facial image in image analysis information table D1 (see FIG. 2) in association with a face number. The extraction processing unit 113 is an example of an extraction processing unit of the present invention.

[0035] The determination processing unit 114 determines the attributes of the user based on the user image extracted by the extraction processing unit 113. For example, the determination processing unit 114 estimates the gender and age of the user from the face image using a trained model generated by performing machine learning. When the captured image includes multiple users, the determination processing unit determines the gender and age of each user. After determining the attributes (gender and age) of the user, the determination processing unit 114 registers the gender and age information in the image analysis information table D1 (see FIG. 2) in association with the face number. The determination processing unit 114 is an example of a determination processing unit of the present invention.

[0036] The registration processing unit 115 associates and registers attribute information indicating the attributes of the user determined by the determination processing unit 114 with the purchased product information acquired by the product acquisition processing unit 111. For example, the registration processing unit 115 stores in the database DB user information D5 (see FIG. 6) in which the account number acquired by the product acquisition processing unit 111 based on the information code of the slip is associated with information on the user's gender and age. The registration processing unit 115 also registers in the user information D5 an age group corresponding to the age determined by the determination processing unit 114. The registration processing unit 115 is an example of a registration processing unit of the present invention.

[0037] The display processing unit 116 displays the captured image acquired by the image acquisition processing unit 112 on the operation display unit 13, and also displays attribute information of the user on the operation display unit 13 in correspondence with the user image of the user extracted from the captured image.

[0038] FIG. 7 shows an example of an accounting screen displayed on the operation display unit 13. The accounting screen displays information on all items that can be ordered at a restaurant, a captured image P1 of the users (here, a group of three) making the payment, order information on the items ordered (purchased products) of the group, and an attribute editing field P2 for displaying and editing the attributes of each user. As shown in FIG. 8, the display processing unit 116 displays the captured image P1 on the accounting screen, and also displays the gender and age (attribute information) of each user determined from the captured image P1 near the user image (e.g., face image). This allows the store clerk to know the gender and age of the users on the accounting screen. The display processing unit 116 is an example of a display processing unit of the present invention.

[0039] The reception processing unit 117 accepts a selection operation to select the user image in the captured image P1 of the checkout screen displayed on the operation display unit 13. For example, as shown in Fig. 9, the store clerk touches a user image (face image) of one user (e.g., the user on the far right) included in the captured image P1. This causes the reception processing unit 117 to accept the touch operation of the store clerk.

[0040] When the reception processing unit 117 receives the touch operation (see FIG. 9), the registration processing unit 115 registers "1" in the image analysis information table D1 as the user number associated with the gender and age of the user of the touched user image. The display processing unit 116 also displays the user number "1" near the face image of the captured image P1 on the checkout screen (see FIG. 10), and displays the gender and generation (age) in the field corresponding to the number "1" in the attribute editing field P2. In the example shown in FIG. 9 and FIG. 10, the display processing unit 116 displays "male" and "40-49 (41)" in association with the number "1" in the attribute editing field P2. The image of the user number "1" displayed on the checkout screen is an example of a sequence image of the present invention. Note that the sequence image of the present invention is not limited to numbers, and may be letters or symbols such as alphabets.

[0041] When the display processing unit 116 displays the user's attributes in the attribute editing field P2, the registration processing unit 115 registers "ON (already displayed)" in the display flag corresponding to the user number "1" in the image analysis information table D1 (see FIG. 2). The registration processing unit 115 also registers the user's information (here, "male", "41", and "40-49") in the gender, age, and generation corresponding to the user number "1" in the attribute information table D2 (see FIG. 3).

[0042] After the attributes of one user included in the captured image P1 are registered, the store clerk then touches the user image (face image) of another user (e.g., the user on the left side) included in the captured image P1, as shown in Fig. 11. The reception processing unit 117 accepts the touch operation of the store clerk.

[0043] When the reception processing unit 117 receives the touch operation (see FIG. 11), the registration processing unit 115 registers "2" in the image analysis information table D1 as the user number associated with the gender and age corresponding to the user of the touched user image. In addition, the display processing unit 116 displays the user number "2" near the face image of the captured image P1 on the checkout screen (see FIG. 11), and displays the gender and generation (age) in the field corresponding to the number "2" in the attribute editing field P2. In the example shown in FIG. 11, the display processing unit 116 registers "female" and "30-39 (38)" in association with the number "2" in the attribute editing field P2.

[0044] When the display processing unit 116 displays the user's attributes in the attribute editing field P2, the registration processing unit 115 registers "ON (already displayed)" in the display flag corresponding to the user number "2" in the image analysis information table D1 (see FIG. 2). The registration processing unit 115 also registers the user's information (here, "female", "38", and "30-39") in the gender, age, and generation corresponding to the user number "2" in the attribute information table D2 (see FIG. 3).

[0045] 12, the store clerk touches the user image (face image) of one of the remaining users (for example, the user in the center) included in the captured image P1. The reception processing unit 117 accepts the touch operation by the store clerk.

[0046] When the reception processing unit 117 receives the touch operation (see FIG. 12), the registration processing unit 115 registers "3" in the image analysis information table D1 as the user number associated with the gender and age corresponding to the user of the touched user image. In addition, the display processing unit 116 displays the user number "3" near the face image of the captured image P1 on the checkout screen (see FIG. 12), and displays the gender and generation (age) in the field corresponding to the number "3" in the attribute editing field P2. In the example shown in FIG. 12, the display processing unit 116 registers "female" and "10-19(11)" in association with the number "3" in the attribute editing field P2.

[0047] When the display processing unit 116 displays the user's attributes in the attribute editing field P2, the registration processing unit 115 registers "ON (already displayed)" in the display flag corresponding to the user number "3" in the image analysis information table D1 (see FIG. 2). The registration processing unit 115 also registers the user's information (here, "female," "11," and "10-19") in the gender, age, and generation corresponding to the user number "3" in the attribute information table D2 (see FIG. 3).

[0048] In this manner, the attributes of the user (gender and age) included in the captured image are associated with the purchased product information of the user and registered. In the above example, "Fried chicken set meal 700 yen", "Pork cutlet bowl 600 yen", and "Ramen 550 yen" are associated with user number "1" "Male 40-49 (41)", user number "2" "Female 30-39 (38)", and user number "3" "Female 10-19 (11)". When the registration operation is completed, the store clerk touches the "OK" button on the accounting screen shown in FIG. 12. When the store clerk touches the "OK" button, the control unit 11 executes the accounting process for the purchased products.

[0049] Here, the store clerk can modify the gender and age of the user registered in the attribute editing field P2 on the checkout screen. The reception processing unit 117 accepts a modification operation to modify the attribute information on the checkout screen. Furthermore, the display processing unit 116 displays the modified attribute information when the reception processing unit 117 accepts the modification operation. For example, as shown in FIG. 13, when the store clerk wants to modify at least one of the gender and age of the user number "1", the store clerk touches the "Modify" button of the user number "1". As a result, the reception processing unit 117 accepts the modification operation. Furthermore, the registration processing unit 115 sets the modification flag of the user number "1" in the attribute information table D2 (see FIG. 3) to "ON". Furthermore, the display processing unit 116 changes the color of the area of ​​the touched "Modify" button (see FIG. 14). After that, when the store clerk touches "male" of the user number "1" (see FIG. 14), the display processing unit 116 modifies "male" to "female" and displays it. Moreover, the registration processing unit 115 corrects the "male" of the user number "1" in the image analysis information table D1 and the attribute information table D2 to "female" and registers the corrected information.

[0050] Furthermore, when the store clerk touches the "Edit" button for user number "1" and then touches "40-49(41)" (see FIG. 15), the display processing unit 116 displays a numeric keypad for inputting (selecting) numbers, as shown in FIG. 16. When the store clerk inputs "50" using the numeric keypad and touches the "OK" button (see FIG. 17), the display processing unit 116 corrects "40-49(41)" to "50-59(50)" and displays it, as shown in FIG. 18. Furthermore, the registration processing unit 115 corrects "40-49(41)" for user number "1" in the image analysis information table D1 and the attribute information table D2 to "50-59(50)" and registers it.

[0051] As described above, the registration processing unit 115 registers the user's attribute information in the image analysis information table D1 and the attribute information table D2. The registration processing unit 115 also registers the attribute information in the transaction information D3 (see FIG. 4). That is, the registration processing unit 115 registers the user's gender and age in association with the purchased product information. The display processing unit 116 also displays the user number information, the attribute information, and the purchased product information in association with each other on the transaction screen on which the captured image P1 is displayed.

[0052] In the above-described embodiment, the determination processing unit 114 determines the attributes (gender and age) of the user from the user image included in the captured image, and the registration processing unit 115 registers the user number in association with the attribute information corresponding to the user image selected by the user (store clerk). In another embodiment, the registration processing unit 115 may automatically register the user number in association with the attribute information corresponding to each user image included in the captured image, without relying on the selection operation of the store clerk.

[0053] In addition, in the other embodiment, when the determination processing unit 114 cannot determine the attribute of the user, the registration processing unit 115 may register the attribute based on the operation of the store clerk. For example, as shown in FIG. 19, when the determination processing unit 114 cannot determine (estimate) the gender and age of the user of the rightmost user image included in the captured image P1, the gender and age information is not displayed on the user image. In this case, when the store clerk touches the user image, the display processing unit 116 displays the user number "4" near the face image of the user image, and changes the color of the "Modify" button area of ​​the user number "4" (see FIG. 20). In addition, the display processing unit 116 displays the provisional gender ("male") and age ("00-09(00)") in the attribute editing field P2 (see FIG. 20). In addition, when the store clerk touches the gender and age to input the gender and age, the display processing unit 116 corrects the gender and age. Furthermore, the registration processing unit 115 registers the gender and age for the user number "4" in the image analysis information table D1 and the attribute information table D2.

[0054] Also, a user who does not belong to the group may appear in the captured image P1. For example, as shown in FIG. 21, the leftmost user image included in the captured image P1 is an image of a user who does not belong to the same group. In this case, the store clerk can perform an operation to delete the information of the user. Specifically, the store clerk touches the user number "2" of the user image at the left end of the captured image P1 on the checkout screen shown in FIG. 21. When the store clerk touches the user number "2", the display processing unit 116 deletes the user number "2", and the registration processing unit 115 deletes the information corresponding to the user number "2" in the image analysis information table D1 and the attribute information table D2. Also, as shown in FIG. 22, the display processing unit 116 raises the numbers "3" and "4" of other user images in the captured image P1 to the numbers "2" and "3", respectively, and raises the attribute information of the numbers "3" and "4" in the attribute editing field P2 to the numbers "2" and "3". Furthermore, the registration processing unit 115 increments the attribute information of the user numbers "3" and "4" in the image analysis information table D1 and the attribute information table D2 to the user numbers "2" and "3."

[0055] In this way, the reception processing unit 117 accepts an operation to delete the user number displayed in association with the user image on the accounting screen. The display processing unit 116 then deletes the user number information and the attribute information that were displayed in association with each other in the attribute editing field P2 on the accounting screen.

[0056] [Accounting] The following describes the accounting process executed by the control unit 11 of the accounting terminal 1. For example, the control unit 11 starts the accounting process by starting execution of the accounting program by acquiring an image of a predetermined area in front of a cash register captured by the camera 2. Note that when one or more cameras 2 capture images of the respective predetermined areas of multiple cash registers, the control unit 11 may execute the accounting process individually in parallel for each acquired image.

[0057] The present invention can be understood as an accounting method for executing one or more steps included in the accounting process. The accounting method is one example of an information processing method of the present invention.

[0058] One or more steps included in the accounting process described here may be omitted as appropriate. The steps in the accounting process may be executed in a different order as long as the same action and effect is achieved. Although the example described here is one in which the control unit 11 executes each step in the accounting process, in other embodiments, the steps in the accounting process may be executed by multiple processors in a distributed manner.

[0059] 23 is a flowchart showing an example of the procedure for the attribute determination process included in the transaction process. During the transaction process, the control unit 11 first executes an attribute determination process to determine the attributes of the user based on the captured image.

[0060] In step S1, the control unit 11 acquires purchased product information corresponding to information about products purchased by the user. Specifically, the control unit 11 references database DB and acquires transaction information D3 (see FIG. 4) and purchased product information D4 (see FIG. 5) corresponding to the user. For example, the control unit 11 acquires the transaction number from the slip, and acquires the transaction information D3 and purchased product information D4 associated with the transaction number. Step S1 is an example of a product acquisition step of the present invention.

[0061] Next, in step S2, the control unit 11 acquires an image P1 of a predetermined area including a user in front of the cash register of the store, captured by the camera 2. The control unit 11 displays the acquired image P1 on the checkout screen (see FIG. 7). Step S2 is an example of an image acquisition step of the present invention.

[0062] Next, in step S3, the control unit 11 sets the face number in the image analysis information table D1 (see FIG. 2) to "0" and clears (initializes) the gender, age, and displayed flag.

[0063] Next, in step S4, the control unit 11 starts scanning (scanning process) of the acquired captured image P1. For example, the control unit 11 scans the captured image P1 from the right side in the lateral direction.

[0064] Next, in step S5, the control unit 11 determines whether or not a face image is detected from the captured image P1. If the control unit 11 detects the face image from the captured image P1 (S5: Yes), the process proceeds to step S6. On the other hand, if the control unit 11 does not detect the face image from the captured image P1 (S5: No), the process proceeds to step S10.

[0065] In step S6, the control unit 11 extracts the face image from the captured image P1 and registers the face number "1" and the face area (horizontal and vertical directions) in the image analysis information table D1 (see FIG. 2). Steps S5 and S6 are an example of an extraction step of the present invention.

[0066] Next, in step S7, the control unit 11 determines (estimates) the sex and age based on the extracted face image. Step S7 is an example of the determination step of the present invention.

[0067] Next, in step S8, the control unit 11 registers the determined gender and age in association with the face number "1" in the image analysis information table D1. Here, the control unit 11 registers "male" and "41" in association with the face number "1".

[0068] Next, in step S9, the control unit 11 displays the determined gender and age on the captured image P1 on the checkout screen (see FIG. 8). Note that for convenience, the genders and ages of three users are displayed in FIG. 8, but in step S9, only the rightmost user, "Male 41," is displayed.

[0069] Next, in step S10, the control unit 11 determines whether scanning has been completed up to the end (here, the left end) of the captured image P1. If scanning is completed (S10: Yes), the control unit 11 ends the attribute determination process. On the other hand, if scanning is not completed (S10: No), the control unit 11 proceeds to step S101 and continues scanning.

[0070] When the control unit 11 continues scanning the captured image P1 and detects the next face image (S5: Yes), it executes the above-mentioned processes of steps S6 to S9. In this way, the control unit 11 repeats the above-mentioned processes until no face image is detected from the captured image P1. When the control unit 11 ends the attribute determination process, it executes the attribute registration process shown below. Figure 8 shows a state where the control unit 11 has finished the attribute registration process for the captured image P1.

[0071] 24 is a flow chart showing an example of the procedure for the transaction process. After the attribute determination process (see FIG. 23) is completed, the control unit 11 executes an attribute registration process for registering the attributes of the user in response to operations by the user (store clerk), an attribute editing process for modifying the registered attributes, and a transaction process.

[0072] In step S11, the control unit 11 sets the user number to "0" and sets the displayed flag and the correction flag to "OFF" in the image analysis information table D1 and the attribute information table D2.

[0073] Next, in step S12, the control unit 11 determines whether or not a touch input by a store clerk has been detected on the checkout screen (see FIG. 8). If the control unit 11 detects the touch input (S12: Yes), the process proceeds to step S13. The control unit 11 waits until the touch input is detected (S12: No).

[0074] In step S13, the control unit 11 determines whether the position of the touch input is a face image in the captured image P1. If the position of the touch input is the face image (S13: Yes), the process proceeds to step S14. If the position of the touch input is not the face image (S13: No), the process proceeds to step S21.

[0075] In step S14, the control unit 11 determines whether the attributes (gender and age) registered in the image analysis information table D1 in association with the face number corresponding to the face image have been displayed in the attribute editing field P2 (see FIG. 9) of the checkout screen. If the attributes have not been displayed in the attribute editing field P2 (S14: No), the process proceeds to step S15. On the other hand, if the attributes have been displayed in the attribute editing field P2 (S14: Yes), the process returns to step S12.

[0076] In step S15, the control unit 11 registers "1" in the user number corresponding to the attribute in the image analysis information table D1 (see FIG. 2).

[0077] Next, in step S16, the control unit 11 displays the user number "1" near the face image (see FIG. 10).

[0078] Next, in step S17, the control unit 11 displays the gender, age, and generation in the field corresponding to the user number "1" in the attribute editing field P2 (see FIG. 9) (see FIG. 10).

[0079] Next, in step S18, the control unit 11 sets the displayed flag in the image analysis information table D1 to "ON."

[0080] Next, in step S19, the control unit 11 registers the gender, age, and generation in association with the user number "1" in the attribute information table D2 (see FIG. 3). After that, the control unit 11 returns to step S12 and repeats the above-mentioned process. The store clerk touches all the user images included in the captured image P1 to display the attribute information of each user in the attribute editing field P2.

[0081] If the position of the touch input is not the face image in step S13 (S13: No), the process proceeds to step S21, where the control unit 11 determines whether the position of the touch input is the position of the correction button in the attribute editing field P2 (see FIG. 13). If the position of the touch input is the position of the correction button (S21: Yes), the process proceeds to step S22. If the position of the touch input is not the position of the correction button (S21: No), the process proceeds to step S31.

[0082] In step S22, the control unit 11 accepts an operation from the store clerk to correct an attribute in the attribute editing field P2, and executes a correction process to correct the attribute.

[0083] For example, as shown in Fig. 13, when a store clerk touches the correction button for user number "1," the control unit 11 sets the correction flag for user number "1" in attribute information table D2 to "ON," and changes the color of the area of ​​the touched correction button on the checkout screen (see Fig. 14). The control unit 11 may also light up or blink the correction button.

[0084] When the correction flag is turned ON and the store clerk touches "male" of user number "1" in attribute editing field P2 (see FIG. 14), the control unit 11 corrects "male" to "female" and displays it. The control unit 11 also corrects "male" of user number "1" in image analysis information table D1 and attribute information table D2 to "female" and registers it.

[0085] Furthermore, when the correction flag is turned ON and the store clerk touches "40-49(41)" of user number "1" in the attribute editing field P2 (see FIG. 15), the control unit 11 displays a numeric keypad for inputting (selecting) numbers, as shown in FIG. 16. When the store clerk inputs "50" using the numeric keypad and touches the OK button (see FIG. 17), the control unit 11 corrects "40-49(41)" to "50-59(50)" and displays it, as shown in FIG. 18. Furthermore, the control unit 11 corrects "40-49(41)" of user number "1" in the image analysis information table D1 and the attribute information table D2 to "50-59(50)" and registers it.

[0086] When the clerk has completed editing the attribute editing field P2, the control unit 11 sets the correction flag to OFF and returns the color of the correction button to its original color.

[0087] If the position of the touch input is not the position of the correction button in step S13 (S21: No), the process proceeds to step S31, where the control unit 11 determines whether the position of the touch input is the position of the OK button (see FIG. 12). If the position of the touch input is the position of the OK button (S31: Yes), the process proceeds to step S32. If the position of the touch input is not the position of the OK button (S31: No), the process ends.

[0088] In step S32, the control unit 11 registers the attribute information in the user information D5 (see FIG. 6). For example, the control unit 11 registers the gender and age information of the user in association with the accounting number obtained based on the information code of the user's slip in the user information D5. Step S32 is an example of a registration step of the present invention.

[0089] In the final step S33, the control unit 11 executes the accounting process for the purchased items. The control unit 11 executes the accounting process for the purchased items by the user in the manner described above.

[0090] In the above-mentioned transaction process, the control unit 11 determines the user's attributes (gender and age) from the user image included in the captured image P1, and registers the user number in association with the attribute information corresponding to the user image selected by the store clerk. In another embodiment, the control unit 11 may automatically register the user number in association with the attribute information corresponding to each user image included in the captured image P1, without relying on the selection operation of the store clerk.

[0091] In the other embodiment, the control unit 11 may register the attribute based on the operation of the store clerk when the attribute of the user cannot be determined. For example, as shown in FIG. 19, when the control unit 11 cannot determine the gender and age of the user of the rightmost user image included in the captured image, the gender and age information is not displayed on the user image. In this case, when the store clerk touches the user image, the control unit 11 displays the user number "4" near the face image of the user image and changes the color of the correction button area of ​​the user number "4" (see FIG. 20). When the store clerk touches the gender and age to input the gender and age, the control unit 11 corrects the gender and age. The control unit 11 also registers the gender and age to the user number "4" in the image analysis information table D1 and the attribute information table D2.

[0092] 21, when the store clerk touches the user number "2" of the user image at the left edge of the captured image P1, the control unit 11 deletes the user number "2" of the user image and deletes the information corresponding to the user number "2" in the image analysis information table D1 and the attribute information table D2. Also, as shown in FIG. 22, the control unit 11 increments the numbers "3" and "4" of other user images in the captured image P1 to the numbers "2" and "3", respectively, and increments the attribute information of the numbers "3" and "4" in the attribute editing field P2 to the numbers "2" and "3". Also, the control unit 11 increments the attribute information of the user numbers "3" and "4" in the image analysis information table D1 and the attribute information table D2 to the user numbers "2" and "3".

[0093] As described above, the accounting system 100 according to this embodiment acquires purchased product information including information on products purchased by a user, acquires a captured image P1 of the user, and extracts a user image of the user from the captured image P1. The accounting system 100 also determines the attributes (gender and age) of the user based on the extracted user image, and associates and registers attribute information indicating the determined attributes of the user with the acquired purchased product information. This makes it possible to easily register the attributes of users who use a store and the purchased product information of the users. In addition, it is not necessary for a store clerk to register the attribute information of the users. This reduces the workload of the store clerk and allows accurate attribute information to be registered. This improves the accuracy of the store's market research and marketing.

[0094] The accounting system 100 of the present invention is not limited to the above-described embodiment. The accounting system 100 may have the following forms.

[0095] As another embodiment, for example, as shown in Fig. 25, when the control unit 11 is unable to determine attributes such as the gender and age of a user from the user image (face image) included in the captured image P1, it may display "?" near the face image of the user image. In this case, for example, a store clerk touches the face image with "?" added on the checkout screen shown in Fig. 25 to input "age". This allows the control unit 11 to register and display the age corresponding to the user. [Explanation of symbols]

[0096] 1: Cashier terminal 2: Camera 100: Accounting system 111: Product acquisition processing unit 112: Image acquisition processing unit 113: Extraction processing unit 114: Judgment processing unit 115: Registration processing unit 116: Display processing unit 117: Reception department D1: Image analysis information table D2: Attribute information table D3: Accounting information D4: Purchase product information D5: User information P1: Captured image P2: Attribute editing field

Claims

1. a product acquisition processing unit that acquires purchased product information corresponding to information on a product purchased by a user; an image acquisition processing unit that acquires an image of the user; an extraction processing unit that extracts a user image of the user from the captured image acquired by the image acquisition processing unit; a determination processing unit that determines an attribute of the user based on the user image extracted by the extraction processing unit; a registration processing unit that associates and registers attribute information indicating the attributes of the user determined by the determination processing unit and the purchased product information acquired by the product acquisition processing unit; a display processing unit that displays the captured image acquired by the image acquisition processing unit on a display unit, and that displays the attribute information of the user on the display unit in association with the user image of the user extracted from the captured image; a reception processing unit that receives a selection operation for selecting the user image from the captured image displayed on the display unit; Equipped with The display processing unit displays an order image indicating an order of the selection operation in association with the selected user image.

2. The display processing unit displays the attribute information in the vicinity of the user image. The information processing system according to claim 1 .

3. The registration processing unit registers the attribute information and the purchased product information in association with each other.

3. The information processing system according to claim 1 or 2.

4. the display processing unit causes the display unit on which the captured image is displayed to display the order information, the attribute information, and the purchased product information in association with each other. The information processing system according to claim 3 .

5. the reception processing unit receives a correction operation for correcting the attribute information on the display unit, the display processing unit displays the corrected attribute information when the reception processing unit receives the correction operation.

5. The information processing system according to claim 4.

6. the reception processing unit receives an operation to delete the sequence image displayed in association with the user image on the display unit, the display processing unit deletes the order information and the attribute information that are displayed in association with each other on the display unit.

5. The information processing system according to claim 4.

7. The extraction processing unit extracts user images of each of a plurality of users from the captured image, The determination processing unit determines an attribute of the user for each of the user images. The information processing system according to any one of claims 1 to 6.

8. One or more processors A product acquisition step of acquiring purchased product information corresponding to information on a product purchased by a user; an image acquisition step of acquiring a captured image of the user; an extraction step of extracting a user image of the user from the captured image acquired in the image acquisition step; a determination step of determining attributes of the user based on the user image extracted by the extraction step; a registration step of registering attribute information indicating the attributes of the user determined in the determination step and the purchased product information acquired in the product acquisition step in association with each other; a display step of displaying the captured image acquired in the image acquisition step on a display unit, and displaying the attribute information of the user on the display unit in association with the user image of the user extracted from the captured image; a receiving step of receiving a selection operation for selecting the user image from the captured image displayed on the display unit; Run In the display step, an order image indicating an order of the selection operation is displayed in association with the selected user image.

9. A product acquisition step of acquiring purchased product information corresponding to information on a product purchased by a user; an image acquisition step of acquiring a captured image of the user; an extraction step of extracting a user image of the user from the captured image acquired in the image acquisition step; a determination step of determining attributes of the user based on the user image extracted by the extraction step; a registration step of registering attribute information indicating the attributes of the user determined in the determination step and the purchased product information acquired in the product acquisition step in association with each other; a display step of displaying the captured image acquired in the image acquisition step on a display unit, and displaying the attribute information of the user on the display unit in association with the user image of the user extracted from the captured image; a receiving step of receiving a selection operation for selecting the user image from the captured image displayed on the display unit; on one or more processors, An information processing program that displays, in the display step, an order image indicating an order of the selection operation in association with the selected user image.

Images