Information processing system, information processing device, and control program therefor

Abstract

To provide an information processing system, an information processor, and a program thereof that can reduce the stress caused by a user's own registration operation of purchased items.SOLUTION: An information processor 10 includes a first acquisition unit 111, a second acquisition unit 112, and a switching unit 113. The first acquisition unit 111 acquires user information of a user who performs a registration operation of purchased items. The second acquisition unit 112 acquires time related to the registration operation of the purchased items based on the user information acquired by the first acquisition unit 111. The switching unit 113 switches a registration operation method of the purchased items when time required for the registration operation of the purchased items performed by the user elapses the time related to the registration operation of the purchased items obtained by the second acquisition unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing system, an information processing device, and a control program therefor. [Background technology]

[0002] In recent years, self-service point-of-sale (POS) terminals have been gaining attention in supermarkets and other retail stores from the perspectives of reducing labor costs and preventing infectious diseases. Self-service POS terminals are fully self-service payment terminals that allow customers to perform all operations themselves, from registering purchased items to paying. Purchase registration methods include, for example, a method using so-called barcode recognition, in which a product is identified by reading a barcode representing a unique product code attached to the purchased item, and a method using so-called object recognition, in which a product is recognized from a captured image of the product's exterior. Users unfamiliar with these registration methods may find the registration process confusing, resulting in a long time required for purchase registration and significant stress. Therefore, there is a need for technology that can reduce the stress caused by users registering purchased items themselves. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-59208 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the embodiments of the present invention aim to solve is to provide an information processing system, an information processing device, and a control program therefor that can reduce the stress that users experience when registering purchased products themselves. [Means for solving the problem]

[0005] In one embodiment, the information processing device includes a first acquisition unit, a second acquisition unit, and a switching unit. The first acquisition unit acquires user information of a user performing a purchase product registration operation. The second acquisition unit acquires a time related to the purchase product registration operation based on the user information acquired by the first acquisition unit. The switching unit switches the purchase product registration operation method when the time required for the purchase product registration operation performed by the user exceeds the time related to the purchase product registration operation acquired by the second acquisition unit. The user information includes facial features of the user, and the first acquisition unit acquires the user information of the user from a captured image including the user's face. The user information further includes the user's age, and the second acquisition unit acquires a second average registration time, which is the average purchase registration time required for a user to perform a purchase product registration operation, based on the user's facial features.If the second average registration time cannot be acquired, the second acquisition unit acquires a first average registration time, which is the average purchase registration time required for a user of that age group to perform a purchase product registration operation, based on the user's age. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating the external configuration of the self-service POS terminal. [Figure 3] FIG. 3 is a block diagram showing the main circuit configuration of the self-service POS terminal. [Figure 4] FIG. 4 is a schematic diagram showing an example of an era table. [Figure 5] FIG. 5 is a block diagram showing the main circuit configuration of the store server. [Figure 6] FIG. 6 is a schematic diagram showing the data structure of a member data record stored in the member database. [Figure 7] FIG. 7 is a schematic diagram showing an example of purchase registration data. [Figure 8] FIG. 8 is a flowchart showing a control procedure of the main parts of the processor in the self-service POS terminal. [Figure 9] FIG. 9 is a flowchart showing a control procedure of the main parts of the processor in the self-service POS terminal. [Figure 10] FIG. 10 is a flowchart showing a control procedure of the main parts of the processor in the self-service POS terminal. [Figure 11]FIG. 11 is a flowchart showing a control procedure of the main parts of the processor in the self-service POS terminal. [Figure 12] FIG. 12 is a flowchart showing a control procedure of the main parts of the processor in the self-service POS terminal. [Figure 13] FIG. 13 is a flowchart showing a control procedure of the main parts of the processor in the store server. [Figure 14] FIG. 14 is a flowchart showing a control procedure of the main parts of the processor in the store server. [Figure 15] FIG. 15 is a flowchart showing a control procedure of the main parts of the processor in the store server. [Figure 16] FIG. 16 is a schematic diagram showing an example of the first purchase registration screen. [Figure 17] FIG. 17 is a schematic diagram showing an example of the switching screen. [Figure 18] FIG. 18 is a schematic diagram showing an example of the second purchase registration screen. [Figure 19] FIG. 19 is a flowchart showing a control procedure of a main part of a processor in a self-checkout POS terminal in the second embodiment. [Figure 20] FIG. 20 is a flowchart showing a control procedure of a main part of a processor in a self-checkout POS terminal in the second embodiment. [Figure 21] FIG. 21 is a flowchart showing a control procedure of a main part of a processor in a self-checkout POS terminal in the second embodiment. [Figure 22] FIG. 22 is a flowchart showing a control procedure of a main part of a processor in a self-checkout POS terminal in the second embodiment. [Figure 23] FIG. 23 is a flowchart showing a control procedure of a main part of the processor in the store server in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Each embodiment will be described below with reference to the drawings.

[0008] (First embodiment) 1 is a block diagram showing a schematic configuration of an information processing system S. The information processing system S includes a plurality of self-service POS terminals 10 and a store server 60. The self-service POS terminals 10 and the store server 60 are connected to a communication network 1 such as a LAN (Local Area Network). The LAN may be wired or wireless.

[0009] The self-service POS terminal 10 is a fully self-service payment terminal that allows a customer to perform operations from registering the products they wish to purchase (i.e., purchased products) to paying for them. The self-service POS terminal 10 is an example of an information processing device.

[0010] The store server 60 is a computer that manages information relating to the store in which the information processing system S is installed and executes various information processes relating to the store.

[0011] FIG. 2 is a diagram illustrating the external configuration of the self-checkout POS terminal 10. As shown in FIG. The self-service POS terminal 10 includes a main body 30 installed on the floor and a bagging stand 40 installed beside the main body 30. A touch panel 31 is attached to the top of the main body 30. The touch panel 31 is composed of a display and a touch sensor. The display is a device for displaying various screens to a customer, who is a user operating the self-service POS terminal 10. The touch sensor is a device for detecting touch inputs by a customer to the screen. A camera 311 is built into the touch panel 31. The camera 311 functions as a still or video capture device. The camera 311 is installed in a position that allows it to capture an image of a customer standing in front of the self-service POS terminal 10 and facing the main body 30, the bagging stand 40, the basket stand 50, etc. The camera 311 captures, for example, an image of a customer registering purchased items. Note that the camera 311 may be external to the touch panel 31. The camera 311 functions as an imaging unit that captures an image of a user, i.e., a customer, performing the purchase item registration operation.

[0012] The main body 30 has a basket stand 50 in the center of the side opposite the side where the bagging stand 40 is installed. The basket stand 50 is for customers coming from the sales floor to place baskets containing purchased items. Customers work by standing on the front side of the main body 30 in Figure 2 so that they can see the screen of the touch panel 31. Therefore, from the customer's perspective, the basket stand 50 is on the right side of the main body 30 and the bagging stand 40 is on the left side. In the following explanation, the side where the customer stands is referred to as the front of the main body 30, the side where the bagging stand 40 is installed is referred to as the left side of the main body 30, and the side where the basket stand 50 is installed is referred to as the right side of the main body 30.

[0013] The main body 30 has, on its front surface, a reading window 32 for the scanner 17 (see FIG. 3), a card insertion slot 33, a receipt issuing slot 34, a coin insertion slot 35, a coin dispensing slot 36, a bill insertion slot 37, and a bill dispensing slot 38. In addition, a communication cable 51 extends from the right side of the main body 30 to the outside, and a reader / writer 52 for electronic money media is connected to the tip of this communication cable 51. The reader / writer 52 is placed on a stand 53 provided on the upper right side of the main body 30.

[0014] A display pole 54 is attached to the top surface of the main body 30. The display pole 54 has a light-emitting unit 55 at its tip that selectively emits light, for example, in blue or red. The display pole 54 displays the status of the self-service POS terminal 10, such as standby, operating, calling, or error, depending on the color of light emitted by the light-emitting unit 55.

[0015] The bagging table 40 has a structure in which a bag holder 42 is attached to the top of a housing 41. The bag holder 42 has a pair of holding arms 43, which hold plastic bags provided by the store or shopping bags brought by customers, so-called "my bags," etc.

[0016] FIG. 3 is a block diagram showing the main circuit configuration of the self-checkout POS terminal 10. As shown in FIG. 3, the self-checkout POS terminal 10 includes a processor 11, a main memory 12, an auxiliary storage device 13, a clock 14, a communication interface 15, an automatic change dispenser 16, a scanner 17, a touch panel 31, a printer 18, a card reader 19, a reader / writer 52, a light-emitting unit 55, a timer / counter 20, a human presence sensor 21, and a system transmission path 22. The system transmission path 22 includes an address bus, a data bus, a control signal line, and the like. The system transmission path 22 connects the processor 11 to each of the other components directly or via a signal input / output circuit, and transmits data signals exchanged between them. The processor 11, the main memory 12, and the auxiliary storage device 13 connected via the system transmission path 22 constitute a computer of the self-checkout POS terminal 10.

[0017] The processor 11 corresponds to the central part of the computer. The processor 11 controls each part as a control unit to realize various functions of the self-service POS terminal 10 in accordance with an operating system or application programs stored in the main memory 12 or the auxiliary storage device 13. In this embodiment, the processor 11 is described as executing various processes as a control unit. The processor 11 is, for example, a CPU (Central Processing Unit).

[0018] The main memory 12 corresponds to the main storage portion of the computer. The main memory 12 includes a nonvolatile memory area and a volatile memory area. The main memory 12 stores an operating system or application programs in the nonvolatile memory area. The main memory 12 may store data required for the processor 11 to execute processes for controlling each part in either the nonvolatile or volatile memory area. The main memory 12 uses the volatile memory area as a work area where data is rewritten by the processor 11 as appropriate. The nonvolatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0019] The auxiliary storage device 13 corresponds to the auxiliary storage portion of the computer. For example, the auxiliary storage device 13 may be an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive). The auxiliary storage device 13 stores data used by the processor 11 when performing various processes, data created by the processes in the processor 11, etc. The auxiliary storage device 13 may also store the application programs described above.

[0020] The clock 14 functions as a time information source for the self-checkout POS terminal 10. The processor 11 keeps track of the current date and time (date and time) based on the time information kept by the clock 14.

[0021] The communication interface 15 performs data communication with the store server 60 or other self-service POS terminals 10 connected via the communication network 1 in accordance with a preset communication protocol.

[0022] The automatic change dispenser 16 includes a coin unit and a banknote unit. The coin unit sorts coins inserted into the coin insertion slot 35 one by one, identifies the denomination, and stores them in a safe by denomination. The coin unit, for example, based on change data, removes coins of the appropriate denomination from the safe and dispenses them to the coin dispensing outlet 36. The banknote unit sorts banknotes inserted into the banknote insertion slot 37 one by one, identifies the denomination, and stores them in the safe by denomination. The banknote unit, for example, based on change data, removes banknotes of the appropriate denomination from the safe and dispenses them to the banknote dispensing outlet 38.

[0023] The scanner 17 captures an image of the purchased item held over the reading window 32.

[0024] The printer 18 prints receipt data showing details of the transaction on receipt paper. The receipt paper with the printed receipt data is ejected from the receipt issuing port 34, cut by a cutter, and issued as a receipt.

[0025] Card reader 19 reads card data recorded on a card medium such as a credit card, a point card, etc. Card reader 19 pulls the card medium inserted into card insertion slot 33 into main body 30, reads the card data, and then ejects the card from card insertion slot 33.

[0026] The reader / writer 52 reads and rewrites electronic money recorded on an electronic money medium. The electronic money medium is, for example, a contactless IC card. The electronic money medium may also be an electronic device such as a smartphone or a tablet terminal.

[0027] The timer counter 20 counts up the timer value T at regular intervals, for example, every time the clock 14 measures one second. The human presence sensor 21 detects the presence of a person, such as a customer, positioned opposite the self-service POS terminal 10. The human presence sensor 21 may be a well-known sensor such as an infrared type.

[0028] The self-service POS terminal 10 configured as described above uses part of the volatile memory area of ​​the main memory 12 as areas for the era table 121 (see FIG. 4), face image memory 122, product image memory 123, average registration time memory 124, and actual registration time memory 125. Note that the storage destinations for the era table 121, face image memory 122, product image memory 123, average registration time memory 124, and actual registration time memory 125 are not limited to the main memory 12. The era table 121, face image memory 122, product image memory 123, average registration time memory 124, and actual registration time memory 125 may also be stored in the auxiliary storage device 13.

[0029] FIG. 4 is a schematic diagram showing an example of the generation table 121. As shown in FIG. 4, the generation table 121 associates and describes the first average registration time. The generation is categorized, for example, into teens (10 to 19 years old), twenties (20 to 29 years old), etc., based on the customer's age. The generation may be set arbitrarily. The first average registration time is the average purchase registration time required for customers of the associated generation to perform a purchase registration operation. The first average registration time is, for example, an average purchase registration time based on the average registration time for the first item and the average registration time for the second and subsequent items. The average registration time for the first item is, for example, the average purchase registration time required from the time when the first purchase registration screen is displayed on the touch panel 31 of the self-service POS terminal 10 to the time when object recognition or barcode recognition of the purchased item is successful. The first purchase registration screen will be described later. Object recognition refers to recognizing purchased items through object recognition processing (general object recognition processing) from an image capturing the appearance of a purchased item held over the reading window 32 of the scanner 17. Object recognition can also be referred to as a first registration operation method for recognizing and registering an image of a purchased item. Barcode recognition refers to recognizing purchased items by capturing an image of the purchased item held over the reading window 32 and then reading the barcode attached to the purchased item. Each item sold in a store is assigned a code symbol in a barcode system representing its product code. The product code is product identification information assigned to each item to identify it individually. The product code is an example of identification information for purchased items. Barcode recognition can also be referred to as a second registration operation method for recognizing and registering the identification information of purchased items. The average registration time for the second and subsequent points is, for example, the average purchase registration time required from the time when object recognition or barcode recognition of the previous purchased item is successful to the time when object recognition or barcode recognition of the next purchased item is successful. The average registration time for the first point and the average registration time for the second point and thereafter are calculated by, for example, analyzing past purchase registration times by customer age. The first average registration time may be set taking into consideration, for example, that customers of an age group known as the elderly or children are often unfamiliar with purchase registration operations.The first average registration time is an example of the time related to the registration operation of the purchased product. Note that the era table 121 is not limited to the data of the above items.

[0030] Returning to the explanation of Figure 3. The face image memory 122 stores a frame image (captured image) including the customer's face captured by the camera 311. The product image memory 123 stores a frame image including a purchased product captured by the scanner 17. The product image memory 123 stores a frame image used for object recognition processing. The frame images stored in the product image memory 123 are used, for example, when performing learning processing related to the object recognition processing. The average registration time memory 124 stores a first average registration time or a second average registration time. The second average registration time will be described later. The actual registration time memory 125 stores the actual registration time. The actual registration time is the purchase registration time required for the customer to actually perform the purchase registration operation. In the case of the first purchased product, the actual registration time is, for example, the purchase registration time required from the time the first purchase registration screen is displayed on the touch panel 31 of the self-service POS terminal 10 to the time object recognition or barcode recognition of the purchased product is successful. In the case of the second or subsequent purchased item, the actual registration time is, for example, the purchase registration time required from the point at which object recognition or barcode recognition of the previous purchased item is successful to the point at which object recognition or barcode recognition of the next purchased item is successful. Note that the actual registration time may be measured, for example, as the time during which purchase registration is being performed. The timing for measuring the actual registration time is not limited to the above and may be designed or set arbitrarily.

[0031] In the self-service POS terminal 10 configured as described above, part of the storage area of ​​the auxiliary storage device 13 is used as an area for the age dictionary file 131. The age dictionary file 131 saves age data records that store typical attribute data related to human faces corresponding to age groups. The age data records include items such as age group and facial attribute data. The facial attribute data is a quantified feature of facial attributes. Examples of facial attributes include hairstyle, skin condition, eyes, wrinkles around the mouth, presence or absence of beard, presence or absence of makeup, etc. The storage destination of the age dictionary file 131 is not limited to the auxiliary storage device 13. The age dictionary file 131 may be stored in part of the volatile memory area of ​​the main memory 12, or may be stored in a storage device external to the self-service POS terminal 10.

[0032] The processor 11 has functions as a first acquisition unit 111, a second acquisition unit 112, and a switching unit 113. The first acquisition unit 111 has a function of acquiring user information of a user who performs an operation to register a purchased product. The first acquisition unit 111 can be rephrased as a first acquisition means.

[0033] The second acquisition unit 112 has a function of acquiring the time related to the registration operation of the purchased product based on the user information acquired by the first acquisition unit 111. The second acquisition unit 112 can be rephrased as a second acquisition means.

[0034] The switching unit 113 has a function of switching the method of registering the purchased product when the time required for the user to register the purchased product exceeds the time related to the registration operation of the purchased product acquired by the second acquisition unit 112. The switching unit 113 can be referred to as a switching means.

[0035] Fig. 5 is a block diagram showing the main circuit configuration of the store server 60. As shown in Fig. 5, the store server 60 includes a processor 61, a main memory 62, an auxiliary storage device 63, a communication interface 64, and a system transmission path 65. The system transmission path 65 includes an address bus, a data bus, a control signal line, etc. The store server 60 connects the processor 61, the main memory 62, the auxiliary storage device 63, and the communication interface 64 to the system transmission path 65. In the store server 60, the processor 61, the main memory 62, the auxiliary storage device 63, and the system transmission path 65 connecting them form a computer.

[0036] The processor 61 corresponds to the central part of the computer. The processor 61 controls each part as a control unit to realize various functions of the store server 60 in accordance with an operating system or application programs stored in the main memory 62 or the auxiliary storage device 63. In this embodiment, the processor 61 is described as executing various processes as a control unit. The processor 61 is, for example, a CPU.

[0037] The main memory 62 corresponds to the main storage portion of the computer. The main memory 62 includes a nonvolatile memory area and a volatile memory area. The main memory 62 stores an operating system or application programs in the nonvolatile memory area. The main memory 62 may store data required for the processor 61 to execute processes for controlling each part in either the nonvolatile or volatile memory area. The main memory 62 uses the volatile memory area as a work area where data is rewritten by the processor 61 as appropriate. The nonvolatile memory area is, for example, ROM. The volatile memory area is, for example, RAM.

[0038] The auxiliary storage device 63 corresponds to the auxiliary storage portion of the computer. For example, an EEPROM, HDD, or SSD can be the auxiliary storage device 63. The auxiliary storage device 63 stores data used by the processor 61 when performing various processes, or data created by the processes in the processor 61. The auxiliary storage device 63 may also store the application programs described above.

[0039] The communication interface 64 performs data communication with the self-service POS terminal 10 connected via the communication network 1 in accordance with a preset communication protocol.

[0040] In the store server 60 configured as above, part of the memory area of ​​the auxiliary storage device 63 is used as the area for the member database 631, the product database 632, and the product learning dictionary file 633. The storage destination of the member database 631, the product database 632, and the product learning dictionary file 633 is not limited to the auxiliary storage device 63. The member database 631, the product database 632, and the product learning dictionary file 633 may be stored in part of the volatile memory area of ​​the main memory 62, or may be stored in a storage device external to the store server 60.

[0041] The member database 631 stores a member data record 6311 (see FIG. 6) that describes member data for each member.

[0042] FIG. 6 is a schematic diagram showing the data structure of a member data record 6311 stored in the member database 631. As shown in FIG. 6, the member data record 6311 includes items such as a member ID, name, gender, email address, facial image, facial feature values, a second average registration time, and purchase registration data 6312 (see FIG. 7). The member ID is member identification information assigned to each member to identify the member individually. The facial image is a pre-captured image of the member's face. The facial feature values ​​are quantified features indicating the relative positions of characteristic facial features (e.g., eyes, nose, mouth, etc.) based on the member's facial image. The second average registration time is the average purchase registration time required for each member to register a purchased item. The second average registration time may be, for example, the average value of the actual registration times described in the purchase registration data 6312, or a value obtained by adding a small margin to the average value. The second average registration time is calculated when a predetermined number or more actual registration times are described in the purchase registration data 6312. The predetermined number is, for example, 100, 200, etc. The predetermined number can be set arbitrarily. The second average registration time may be updated as appropriate. If the actual registration time described in the purchase registration data 6312 does not meet the predetermined number, a NULL value is set for the second average registration time. The second average registration time is an example of the time related to the registration operation of the purchased product. Note that the member data record 6311 is not limited to the data of the above-mentioned items.

[0043] FIG. 7 is a schematic diagram showing an example of purchase registration data 6312. As shown in FIG. 7, purchase registration data 6312 consists of, in numerical order, the transaction date and time, the product code of the product purchased by the user, the product name, price, actual registration time, etc. The transaction date and time will be described later. In the case of a registration operation using object recognition, the actual registration time is recorded as the purchase registration time required for the customer to actually perform the registration operation using object recognition. In the case of a registration operation using barcode recognition, the actual registration time is recorded as the purchase registration time required for the customer to actually perform the registration operation using barcode recognition. Note that purchase registration data 6312 is not limited to the data of the items described above.

[0044] The product database 632 stores product data records that record data on products sold in the store. The product data records include items such as product code, product name, price, and first product feature. The first product feature is a quantification of the standard appearance feature of a product, such as the product shape, surface color, pattern, and unevenness, for each product. Note that the product data records are not limited to the data of the above items.

[0045] The product learning dictionary file 633 stores learning data records that record data related to purchased products captured by the scanner 17. The product learning dictionary file 633 is used when performing learning processing to improve the recognition rate of objects in object recognition processing. The learning data record includes items such as product code, product image, and second product feature. The product image is a frame image containing the purchased product captured by the scanner 17. The second product feature is a quantified version of the feature of the product read from the product image, such as the shape, surface color, pattern, and unevenness of the product. For example, if the feature cannot be read from the product image, a NULL value is set for the second product feature. Note that the learning data record is not limited to the data of the above-mentioned items.

[0046] 8 to 12 are flowcharts showing the control procedure of the main part of the processor 11 (control unit) in the self-service POS terminal 10. FIGS. 13 to 15 are flowcharts showing the control procedure of the main part of the processor 61 (control unit) in the store server 60. Below, using these flowcharts, we will explain the main operations of the information processing system S when a customer uses the self-service POS terminal 10 to perform the process from registering purchased items to paying for them by themselves. Note that the operations explained below are just an example. There are no particular limitations on the procedures as long as similar results are obtained.

[0047] First, a customer who comes to the checkout area to pay stands in front of an empty self-service POS terminal 10 and places a basket containing purchased items on the basket stand 50. Meanwhile, when a customer stands in front of the self-service POS terminal 10, for example, using the human presence sensor 21, the self-service POS terminal 10 detects the customer and starts information processing according to the procedure shown in the flowchart of FIG.

[0048] The processor 11 outputs an imaging-on signal to the camera 311 as ACT1. This imaging-on signal causes the camera 311 to capture an image of the face of a customer standing in front of the self-service POS terminal 10. The processor 11 captures a frame image including the customer's face captured by the camera 311 as ACT2. The processor 11 stores the frame image including the customer's face in the face image memory 122 of the main memory 12 as ACT3.

[0049] The processor 11 performs face recognition processing on the frame image as ACT4. As the face recognition processing, the processor 11 calculates, for example, facial features of the customer included in the frame image. The processor 11 acquires the facial features of the customer calculated by the function of the first acquisition unit 111. The facial features of the customer are an example of user information. Such face recognition processing technology is well known, so a detailed description will be omitted.

[0050] The processor 11 controls the communication interface 15 to transmit a face authentication command as ACT5 to the store server 60. This control causes the communication interface 15 to transmit the face authentication command. The face authentication command includes the feature amount acquired in the face recognition process of ACT4.

[0051] Upon receiving the face authentication command, the processor 61 of the store server 60 starts a command reception process according to the procedure shown in the flowchart of FIG.

[0052] The processor 61 detects features from the face recognition command in ACT 81. In ACT 82, the processor 61 checks whether a member data record 6311 including facial features that match the features detected in the processing of ACT 81 exists in the member database 631. At this time, if the similarity between the features detected in the processing of ACT 81 and the facial features is equal to or greater than a predetermined value, the processor 61 determines that the features match.

[0053] If the corresponding member data record 6311 does not exist in the member database 631, the processor 61 determines NO in ACT82 and proceeds to ACT83. If the member data record 6311 does not exist in the member database 631, this means, for example, that the customer is a non-member. In ACT83, the processor 61 controls the communication interface 64 to send a non-membership command to the self-service POS terminal 10. This control causes the non-membership command to be sent from the communication interface 64. The non-membership command is received by the self-service POS terminal 10 via the communication network 1.

[0054] If the corresponding member data record 6311 exists in the member database 631, the processor 61 determines YES in ACT82 and proceeds to ACT 84. The processor 61 acquires the member ID included in the member data record 6311 in ACT84.

[0055] The processor 61 checks whether the second average registration time is stored in the corresponding member data record 6311 as ACT85.

[0056] If the second average registration time is stored, the processor 61 determines YES in ACT 85 and proceeds to ACT 86. The processor 61 acquires the second average registration time in ACT 86. The processor 61 then proceeds to ACT 87.

[0057] If the second average registration time is not stored, that is, if a NULL value is set for the second average registration time, the processor 61 determines NO in ACT85, skips the processing of ACT86, and proceeds to ACT87.

[0058] The processor 61 controls the communication interface 64 to send a member command to the self-POS terminal 10 as ACT87. This control causes the member command to be sent from the communication interface 64. The member command is received by the self-POS terminal 10 via the communication network 1. If the second average registration time is acquired in the processing of ACT86, the member command includes the member ID and the second average registration time. If the second average registration time is not stored in the member data record 6311, the member command includes the member ID.

[0059] In this way, the store server 60 that has received the face authentication command transmits a non-member or member command to the self-service POS terminal 10 that sent the face authentication command in ACT 83 or ACT 87. After completing transmission of either command, the processor 61 ends the face authentication command reception process.

[0060] Returning to the explanation of Figure 8. The processor 11 of the self-service POS terminal 10, which controlled the transmission of the face authentication command in ACT5, waits for a membership command from the store server 60 in ACT6.

[0061] If no membership command is received from the store server 60, the processor 11 determines NO in ACT6 and proceeds to ACT7.

[0062] The processor 11 checks whether or not a non-member command has been received in ACT 7. If a non-member command has not been received, the processor 11 determines NO in ACT 7 and returns to the processing of ACT 6. That is, the processor 11 returns to a standby state for ACT 6 and ACT 7.

[0063] When the processor 11 receives a member command from the store server 60 while in the standby state of ACT6 and ACT7, the processor 11 determines YES in ACT6 and proceeds to ACT8. The processor 11 stores the member ID included in the member command in the main memory 12 as ACT8.

[0064] In ACT 9, processor 11 checks whether the member command includes the second average registration time. If the member command includes the second average registration time, processor 11 determines YES in ACT 9 and proceeds to ACT 12. In ACT 12, processor 11 stores the second average registration time in average registration time memory 124 using the function of second acquisition unit 112.

[0065] If the member command does not include the second average registration time, the processor 11 determines NO in ACT9 and proceeds to ACT10.

[0066] If a non-membership command is received from the store server 60 during the standby state of ACT6 and ACT7, the processor 11 determines YES in ACT7 and proceeds to ACT 10. In this way, the processor 11 proceeds to ACT10 if the received member command does not include the second average registration time or if a non-membership command is received.

[0067] The processor 11 performs age estimation processing on the frame image including the customer's face stored in the face image memory 122 as ACT10. In the age estimation process, the processor 11 extracts, for example, facial attributes of customers included in a frame image. The processor 11 digitizes the extracted attributes and extracts from the age dictionary file 131 an age data record including facial attribute data that is most similar to the digitized attributes. The processor 11 acquires an age associated with the facial attribute data using the function of the first acquisition unit 111. The age is an example of user information. Such age estimation process technology is well known, so a detailed description thereof will be omitted.

[0068] In ACT 11, processor 11 refers to the era table 121 and obtains the first average registration time associated with the era obtained in the era estimation process in ACT 10. That is, in the case of a customer for whom the second average registration time is not stored in the member data record 6311 or a customer who is not a member, processor 11 estimates the era of the customer and obtains the first average registration time. Processor 11 then proceeds to ACT 12. In this case, processor 11 stores the first average registration time in average registration time memory 124 using the function of second acquisition unit 112 in ACT 12.

[0069] As ACT13, processor 11 causes touch panel 31 to display a registration start screen. The registration start screen is a screen that allows a customer to select, for example, whether to use a plastic bag provided in the store or to use a reusable bag. The registration start screen displays an image of a plastic bag button that allows a customer to specify whether to use a plastic bag, and an image of a my bag button that allows a customer to specify whether to use a reusable bag. After checking the registration start screen, if a customer wants to use a plastic bag, the customer holds the plastic bag in holding arm 43 and touches the plastic bag button. If a customer wants to use a reusable bag, the customer holds the reusable bag in holding arm 43 and touches the my bag button.

[0070] The processor 11 waits for the plastic bag button or the my bag button to be touched in ACT 14. If the plastic bag button or the my bag button is touched, the processor 11 determines YES in ACT 14 and proceeds to ACT 21 in FIG.

[0071] In ACT21, the processor 11 causes the touch panel 31 to display the first purchase registration screen 100 (see FIG. 16).

[0072] FIG. 16 is a schematic diagram illustrating an example of a first purchase registration screen 100. As shown in FIG. 16, the first purchase registration screen 100 displays a details field 101 for displaying details of the registered items, a total field 102 for displaying the total number of items and the total price of the registered items, and an image of a checkout button 103 for instructing the customer to checkout. The details field 101 includes the product name, price, number of items, and purchase price. The number of items is the number of items identified by the product code, and the purchase price is the product obtained by multiplying the price and number of items. The first purchase registration screen 100 displays a message urging the customer to hold the purchased item over the glass surface, i.e., the reading window 32 of the scanner 17, along with a guide image 104 guiding the customer to hold the purchased item over the reading window 32. In other words, the guide image 104 is an image that guides the customer through the registration operation method using object recognition. Note that the content and image of the text data displayed in FIG. 16 are merely examples.

[0073] After checking the first purchase registration screen 100, the customer performs a registration operation using object recognition. That is, the customer takes out each item of purchase from the basket placed on the basket stand 50. Then, the customer holds the purchased item over the reading window 32.

[0074] Returning to the explanation of Figure 9. In ACT22, the processor 11 resets the timer value T of the timer counter 20 to "0." In ACT23, the processor 11 starts time measurement by the timer counter 20.

[0075] The processor 11 checks whether the timer value T has passed the time stored in the average registration time memory 124 as ACT24. Specifically, if the first average registration time is stored in the average registration time memory 124, the processor 11 checks whether the timer value T has passed the first average registration time. If the second average registration time is stored in the average registration time memory 124, the processor 11 checks whether the timer value T has passed the second average registration time.

[0076] If the timer value T has passed the time stored in the average registration time memory 124, the processor 11 determines YES in ACT24 and proceeds to ACT51 in Fig. 11. The processing in ACT51 will be described later.

[0077] If the timer value T has not reached the time stored in the average registration time memory 124, the processor 11 determines NO in ACT 24 and proceeds to ACT 25. In ACT 25, the processor 11 waits for the scanner 17 to capture an image of the purchased product.

[0078] If the purchased product is not photographed, the processor 11 determines NO in ACT25 and proceeds to ACT26.

[0079] Processor 11 checks in ACT26 whether a checkout has been instructed. As described above, an image of checkout button 103 is displayed in part of first purchase registration screen 100. After completing purchase registration, the customer touches checkout button 103. When checkout button 103 is touched, processor 11 recognizes that a checkout has been instructed. If a checkout has not been instructed, processor 11 determines NO in ACT26 and returns to processing ACT25. That is, processor 11 returns to a standby state for ACT25 and ACT26.

[0080] When the purchased product is photographed in the standby state of ACT25 and ACT26, the processor 11 determines YES in ACT25 and proceeds to ACT27.

[0081] In ACT 27, the processor 11 captures a frame image including the purchased product captured by the scanner 17. In ACT 28, the processor 11 stores the frame image including the purchased product in the product image memory 123.

[0082] In ACT 29, processor 11 performs object recognition processing on the frame image. As part of the object recognition processing, processor 11, for example, detects whether or not a skin-colored area is present in the frame image. If a skin-colored area is detected, i.e., if the reflection of a customer's hand is detected, processor 11 extracts a contour line or the like from the binarized image of the frame image. Using this, processor 11 attempts to extract the contour of the purchased item that is assumed to be held by the customer's hand. Once the contour of the purchased item is extracted, processor 11 reads features of the purchased item, such as its shape, surface color, pattern, and unevenness, from the image within the contour, i.e., second product features. If processor 11 is able to read the second product features, processor 11 considers the purchased item to have been recognized. In other words, processor 11 considers object recognition to have been successful. If processor 11 is unable to read the second product features, processor 11 considers the purchased item to have been unrecognized. In other words, processor 11 considers object recognition to have been unsuccessful. The case where the second product feature cannot be read means, for example, when the outline of the purchased product cannot be extracted because the customer's hand holding the purchased product is too large in the image. Such technology for recognizing objects contained in images is well known, so a detailed description will be omitted.

[0083] The processor 11 checks whether or not the object recognition was successful in ACT 30. If the object recognition failed, the processor 11 determines NO in ACT 30 and returns to the processing of ACT 24. The processor 11 then repeats the processing of ACT 24 to ACT 30 in the same manner as described above.

[0084] If the object recognition is successful, the processor 11 determines YES in ACT 30 and proceeds to ACT 31. In ACT 31, the processor 11 stops the timer counter 20 from timing.

[0085] The processor 11 stores the timer value T measured by the timer counter 20 as ACT32 in the actual registration time memory 125.

[0086] In ACT33, the processor 11 resets the timer value T of the timer counter 20 to "0." In ACT34, the processor 11 starts time measurement by the timer counter 20.

[0087] The processor 11 controls the communication interface 15 to send a first product command to the store server 60 as ACT41 in FIG. 10. This control causes the communication interface 15 to send the first product command. The first product command includes image data of a frame image including the purchased product stored in the product image memory 123 in the processing of ACT28, the second product feature amount read in the object recognition processing of ACT29, the timer value T stored in the actual registration time memory 125 in the processing of ACT32, and the transaction date and time. The transaction date and time is the date and time measured by the clock 14 at the start of transmission of the first product command. If a member ID is stored in the main memory 12, the first product command includes image data of the frame image, the second product feature amount, the timer value T, the transaction date and time, and the member ID.

[0088] Upon receiving the first product command, the processor 61 of the store server 60 starts a command reception process according to the procedure shown in the flowchart of FIG.

[0089] In ACT91, the processor 61 detects a frame image including the purchased product, the second product feature, the timer value T, and the transaction date and time from the first product command. In ACT92, the processor 61 extracts from the product database 632 a product data record including the first product feature that is most similar to the second product feature detected in the processing of ACT91.

[0090] In ACT 93, processor 61 checks whether a member ID has been detected from the first product command. If a member ID has been detected, processor 61 determines YES in ACT 93 and proceeds to ACT 94. In ACT 94, processor 61 extracts member data record 6311 including the member ID from member database 631.

[0091] In ACT95, the processor 61 writes data into the purchase registration data 6312 included in the member data record 6311. Specifically, the processor 61 writes into the purchase registration data 6312 the transaction date and time detected in the processing of ACT91, the timer value T detected in the processing of ACT91 as the actual registration time, and the product code, product name, and price included in the product data record extracted in the processing of ACT92, in association with each other. The processor 61 then proceeds to ACT96.

[0092] If the member ID is not detected, the processor 61 determines NO in ACT93, skips the processes in ACT94 and ACT95, and proceeds to ACT96.

[0093] The processor 61 adds the learning data record to the product learning dictionary file 633 in ACT96. Specifically, the processor 61 associates the frame image detected as the product image in the processing of ACT91, the second product feature detected in the processing of ACT91, and the product code included in the product data record extracted in the processing of ACT92, and stores them in the product learning dictionary file 633.

[0094] The processor 61 controls the communication interface 24 to send a first acquisition command to the self-service POS terminal 10 as ACT97. This control causes the first acquisition command to be sent from the communication interface 24. The first acquisition command is received by the self-service POS terminal 10 via the communication network 1. The first acquisition command includes the product data record extracted in the processing of ACT92.

[0095] Returning to the explanation of Figure 10. The processor 11 of the self-service POS terminal 10, which controlled the transmission of the first product command in ACT41, waits for a first obtain command from the store server 60 in ACT42. When the first obtain command is received from the store server 60, the processor 11 determines YES in ACT42 and proceeds to ACT43.

[0096] In ACT43, processor 11 updates first purchase registration screen 100 based on the product data record included in the first acquisition command. That is, processor 11 updates first purchase registration screen 100 so that the product name, price, number of items, and purchase amount of the product registered for purchase are added, and the number of items and purchase amount are added to the total number of items and total amount. Then, processor 11 returns to ACT24 in Fig. 9. That is, processor 11 checks whether timer value T, which is the time required to register the next purchased product, has passed the time stored in average registration time memory 124.

[0097] When processor 11 is in the standby state in ACT 25 and ACT 26 and is instructed to make a payment, processor 11 determines YES in ACT 26 and proceeds to ACT 35. In ACT 35, processor 11 displays a payment screen on touch panel 31. The payment screen displays, for example, the total price of the products registered for purchase.

[0098] Processor 11 executes payment processing as ACT 36. Payment processing is well known, so a detailed description will be omitted. Processor 11 controls printer 18 as ACT 37 to issue a receipt from receipt issuing port 34. The customer receives the receipt issued from receipt issuing port 34 and leaves the store with the shopping bag or personal bag removed from holding arm 43. With this, processor 11 completes the information processing of the procedures shown in the flowcharts of Figures 8 to 10.

[0099] Now, processor 11 determines YES in ACT24 when timer value T has elapsed the time stored in average registration time memory 124. Processor 11 switches from object recognition to barcode recognition when the time required for the purchase product registration operation performed by the user, i.e., timer value T, via the function of switching unit 113, has elapsed the time related to the purchase product registration operation acquired by second acquisition unit 112, i.e., the time stored in average registration time memory 124. Then, processor 11 proceeds to ACT51 in FIG.

[0100] The processor 11 displays the switching screen 200 (see FIG. 17) on the touch panel 31 in ACT51.

[0101] Fig. 17 is a schematic diagram showing an example of a switching screen 200. As shown in Fig. 17, a message is displayed on the switching screen 200 to notify the customer that the registration operation method will be switched. This switching screen 200 disappears after a certain period of time. Note that the content of the text data displayed in Fig. 17 is an example.

[0102] Returning to the explanation of FIG. The processor 11 causes the touch panel 31 to display the second purchase registration screen 300 (see FIG. 18) in ACT52.

[0103] FIG. 18 is a schematic diagram showing an example of a second purchase registration screen 300. As shown in FIG. 18, the second purchase registration screen 300 displays a details field 101 for displaying details of the registered items, a total field 102 for displaying the total number of registered items and the total price, and an image of a checkout button 103 for instructing the customer to checkout. The second purchase registration screen 300 displays a message urging the customer to hold the barcode of the purchased item over the glass surface, i.e., the reading window 32 of the scanner 17, as well as a guide image 301 that guides the customer to hold the barcode of the purchased item over the reading window 32. In other words, the guide image 301 is an image that guides the customer through the registration operation method using barcode recognition. Note that the content and image of the text data displayed in FIG. 18 are merely examples.

[0104] After the switching screen 200 is displayed, the customer checks the second purchase registration screen 300 and performs a registration operation by barcode recognition. That is, the customer holds the barcode of the purchased product over the reading window 32.

[0105] Returning to the explanation of FIG. In ACT53, the processor 11 waits for the scanner 17 to capture an image of the purchased product with a barcode attached thereto.

[0106] If the purchased product is not photographed, the processor 11 determines NO in ACT53 and proceeds to ACT54.

[0107] The processor 11 checks whether a payment has been instructed in ACT 54. If a payment has not been instructed, the processor 11 determines NO in ACT 54 and returns to the processing of ACT 53. That is, the processor 11 returns to the standby state for ACT 53 and ACT 54.

[0108] When the purchased product is imaged in the standby state of ACT53 and ACT54, the processor 11 determines YES in ACT53 and proceeds to ACT 55. The processor 11 captures a frame image including the purchased product imaged by the scanner 17 in ACT55.

[0109] Processor 11 performs barcode recognition processing on the frame image in ACT 56. In this barcode recognition processing, processor 11, for example, analyzes the frame image to detect the barcode attached to the purchased product. Processor 11 reads binary code information, i.e., the product code, from the barcode. If the product code can be read, processor 11 considers the purchased product to have been recognized. That is, processor 11 considers barcode recognition to have been successful. If the product code cannot be read, processor 11 considers the purchased product to have been unrecognized. That is, processor 11 considers barcode recognition to have failed. Examples of cases where the product code cannot be read include when the barcode is distorted because the purchased product to which the barcode is attached is dented, or when the barcode cannot be detected because a discount sticker or the like is attached to the barcode portion of the purchased product. Such image analysis barcode recognition technology is well known, so a detailed description will be omitted.

[0110] The processor 11 checks whether the barcode recognition was successful in ACT 57. If the barcode recognition failed, the processor 11 determines NO in ACT 57 and returns to the processing in ACT 53. The processor 11 then repeats the processing from ACT 53 to ACT 57 in the same manner as described above.

[0111] If the barcode recognition is successful, the processor 11 determines YES in ACT 57 and proceeds to ACT 58. In ACT 58, the processor 11 stops the time measurement by the timer counter 20. In ACT 59, the processor 11 stores the timer value T measured by the timer counter 20 in the actual registration time memory 125.

[0112] In ACT60, the processor 11 resets the timer value T of the timer counter 20 to "0." In ACT61, the processor 11 starts time measurement by the timer counter 20.

[0113] The processor 11 controls the communication interface 15 to send a second product command to the store server 60 as ACT 71 in FIG. 12. This control causes the communication interface 15 to send the second product command. The second product command includes the product code read in the barcode recognition process of ACT 56, the timer value T stored in the actual registration time memory 125 in the process of ACT 59, and the transaction date and time. The transaction date and time is the date and time measured by the clock 14 at the start of transmission of the second product command. If a member ID is stored in the main memory 12, the second product command includes the product code, the timer value T, the transaction date and time, and the member ID. Furthermore, if a frame image including the purchased product is stored in the product image memory 123, the second product command also includes the frame image. The case where a frame image including a purchased product is stored in the product image memory 123 means that when object recognition is not successful within the first average registration time or the second average registration time stored in the average registration time memory 124, a frame image including a purchased product for which object recognition has failed is stored in the product image memory 123.

[0114] Upon receiving the second product command, the processor 61 of the store server 60 starts a command reception process according to the procedure shown in the flowchart of FIG.

[0115] In ACT101, the processor 61 detects the product code, timer value T, and transaction date and time from the second product command. In ACT102, the processor 61 extracts from the product database 632 a product data record that includes the product code detected in the processing of ACT101.

[0116] In ACT 103, the processor 61 checks whether a member ID has been detected from the second product command. If a member ID has been detected, the processor 61 determines YES in ACT 103 and proceeds to ACT 104. In ACT 104, the processor 61 extracts a member data record 6311 including the member ID from the member database 631.

[0117] In ACT105, the processor 61 writes data into the purchase registration data 6312 included in the member data record 6311. Specifically, the processor 61 writes into the purchase registration data 6312 the product code and transaction date and time detected in the processing of ACT101, the timer value T detected in the processing of ACT101 as the actual registration time, and the product name and price included in the product data record extracted in the processing of ACT102, in association with each other. The processor 61 then proceeds to ACT106.

[0118] If the member ID is not detected, the processor 61 determines NO in ACT103, skips the processing of ACT104 and ACT105, and proceeds to ACT106.

[0119] The processor 61 checks whether or not a frame image including the purchased product has been detected from the second product command in ACT 106. If a frame image has been detected, the processor 61 determines YES in ACT 106 and proceeds to ACT 107.

[0120] In ACT107, the processor 61 adds a learning data record to the product learning dictionary file 633. Specifically, the processor 61 associates the product code with the frame image detected in the processing of ACT106 as the product image and stores them. Note that a NULL value is set to the second product feature. Then, the processor 61 proceeds to ACT108.

[0121] If a frame image is not detected, the processor 61 determines NO in ACT106, skips the processing of ACT107, and proceeds to ACT108.

[0122] The processor 61 controls the communication interface 64 to send a second obtainment command to the self-service POS terminal 10 in ACT108. This control causes the second obtainment command to be sent from the communication interface 64. The second obtainment command is received by the self-service POS terminal 10 via the communication network 1. The second obtainment command includes the product data record extracted in the processing of ACT102.

[0123] Returning to the explanation of FIG. The processor 11 of the self-service POS terminal 10, which controlled the transmission of the second product command in ACT71, waits for a second obtain command from the store server 60 in ACT72. When the second obtain command is received from the store server 60, the processor 11 determines YES in ACT72 and proceeds to ACT73.

[0124] The processor 11 updates the second purchase registration screen 300 based on the product data record included in the second acquisition command as ACT73.

[0125] In ACT74, the processor 11 checks whether a frame image including the purchased product is stored in the product image memory 123. If a frame image is stored in the product image memory 123, the result of the determination in ACT74 is YES, and the process proceeds to ACT75.

[0126] In ACT 75, the processor 11 clears the product image memory 123. Then, the processor 11 returns to ACT 53 in Fig. 11. That is, when the registration operation method is switched from object recognition to barcode recognition, subsequent registration operations for purchased products will be performed using barcode recognition.

[0127] When a payment instruction is received while processor 11 is in the standby state in ACT 53 and ACT 54, processor 11 determines YES in ACT 54 and proceeds to ACT 35 in Fig. 9. Processor 11 then executes the processing in ACT 35 to ACT 37 in the same manner as described above. With this, processor 11 completes the information processing of the procedures shown in the flowcharts of Figs. 8 to 12.

[0128] Therefore, in this information processing system S, when the camera 311 captures an image of a customer standing in front of the self-service POS terminal 10, facial recognition processing is performed on the frame image. When a facial authentication command including features acquired by the facial recognition processing is sent from the self-service POS terminal 10 to the store server 60, the store server 60 checks whether a member data record 6311 including facial features matching the facial recognition command exists in the member database 631 and, if so, whether a second average registration time is stored, and then sends a member or non-member command. When the self-service POS terminal 10 receives a non-member command or a member command not including the second average registration time, an age estimation process is performed, and a first average registration time is acquired, which is the average time required for customers of the estimated age to complete the registration operation. When the self-service POS terminal 10 receives a member command including the second average registration time, a second average registration time is acquired, which is the average time required for the customer to complete the registration operation. A registration start screen is then displayed on the touch panel 31, followed by the first purchase registration screen 100. The customer then performs the registration operation using object recognition. If object recognition is not successful within the first or second average registration time stored in average registration time memory 124, the registration operation method switches from object recognition to barcode recognition. That is, switching screen 200 is displayed on touch panel 31, and second purchase registration screen 300 is displayed. The customer then performs the registration operation using barcode recognition. The object recognition registration operation requires only holding the outer appearance of the purchased product over the reading window 32 of the scanner 17, which can result in a faster purchase registration time than the barcode recognition registration operation, which requires holding the barcode of the purchased product over the reading window 32. Even if the object recognition registration operation does not work properly because the customer's hand holding the purchased product is too visible, the registration operation method switches to barcode recognition, so the customer does not have to spend a long time struggling with the object recognition registration operation. This reduces the stress that customers experience when registering purchased products themselves. In addition, the registration operation method switches based on the customer's own average purchase registration time (second average registration time) or the average purchase registration time of customers of an estimated age group (first average registration time). This allows the self-service POS terminal 10 to register purchased products according to the customer's needs.

[0129] (Second embodiment) Next, a second embodiment will be described with reference to FIGS.

[0130] The second embodiment differs from the first embodiment in that a second purchase registration screen 300 is displayed after a registration start screen is displayed. The second embodiment also differs from the first embodiment in that if barcode recognition is not successful within the first or second average registration time stored in the average registration time memory 124, a switching screen 200 is displayed, followed by the first purchase registration screen 100. Note that in the drawings and the following description of the second embodiment, the same elements as those in the first embodiment are denoted by the same reference numerals. Therefore, the description of these same elements may be omitted. Incidentally, Figures 1 to 8, 13, 14, and 16 to 18 used in the description of the first embodiment are also common to the second embodiment, and therefore will not be repeated here.

[0131] 19 to 22 are flowcharts showing the main control procedures executed by the processor 11 (control unit) of the self-service POS terminal 10 according to a control program in the second embodiment. FIG. 23 is a flowchart showing the main control procedures executed by the processor 61 (control unit) of the store server 60 according to a control program in the second embodiment. 19, 20, 21, 22, and 23 correspond to FIGS. 9, 10, 11, 12, and 15, respectively, described in the first embodiment. Therefore, the same process steps as those in the first embodiment are denoted by the same reference numerals. The operation contents described below are merely examples. The operation procedures and contents are not particularly limited as long as similar results can be obtained.

[0132] As can be seen by comparing Figure 19 with Figure 9, in the second embodiment, when the plastic bag button or the my bag button on the purchase start screen is touched, the processor 11 displays the second purchase registration screen 300 on the touch panel 31 as ACT111.

[0133] In the second embodiment, the processing of ACT23 is followed by the processing of ACT112 to ACT116. In ACT112, the processor 11 checks whether the timer value T has passed the first average registration time or the second average registration time stored in the average registration time memory 124. If the timer value T has not passed the time stored in the average registration time memory 124, the processor 11 determines NO in ACT112 and proceeds to ACT113.

[0134] The processing in ACT113 to ACT116 is the same as the processing in ACT53 and ACT55 to ACT57 in the first embodiment. That is, in ACT113, the processor 11 waits for the scanner 17 to capture an image of the purchased product bearing a barcode or for an instruction to check out. When the image of the purchased product is captured while the processor 11 is in the standby state in ACT113 and ACT26, the processor 11 determines YES in ACT113 and proceeds to ACT114. In ACT114, the processor 11 captures a frame image including the purchased product captured by the scanner 17. In ACT115, the processor 11 performs barcode recognition processing on the frame image. In ACT116, the processor 11 checks whether the barcode recognition was successful. If the barcode recognition failed, the processor 11 determines NO in ACT116 and returns to the processing in ACT112. The processor 11 then repeats the processing in ACT112 to ACT116 and ACT26 in the same manner as described above. If the barcode recognition is successful, the processor 11 determines YES in ACT 116 and proceeds to ACT 30. The processor 11 performs the processes of ACT 30 to ACT 34 in the same manner as in the first embodiment.

[0135] In the second embodiment, the processing of ACT34 is followed by the processing of ACT121 to ACT123 in FIG. The processor 11 controls the communication interface 15 to send a second product command to the store server 60 in ACT121. This control causes the communication interface 15 to send the second product command. The second product command includes the product code read in the barcode recognition process in ACT115, the timer value T stored in the actual registration time memory 125 in the process of ACT32, and the transaction date and time. The transaction date and time is the date and time measured by the clock 14 at the start of transmission of the second product command. If a member ID is stored in the main memory 12, the second product command includes the product code, the timer value T, the transaction date and time, and the member ID.

[0136] Upon receiving the second product command, the processor 61 of the store server 60 starts a command reception process according to the procedure shown in the flowchart of FIG. 23 and 15, the second embodiment differs from the first embodiment in that the processor 61 of the store server 60 omits the processes of ACT106 and ACT107. In the process of ACT107, the learning data record added to the product learning dictionary file 633 is the frame image used in the object recognition process. Therefore, the frame image used in the barcode recognition process is not stored in the product image memory 123, and the frame image is not included in the second product command.

[0137] Returning to the explanation of Figure 20. The processor 11 of the self-service POS terminal 10, which controlled the transmission of the second product command in ACT121, waits for a second obtain command from the store server 60 in ACT122. When the second obtain command is received from the store server 60, the processor 11 determines YES in ACT122 and proceeds to ACT123.

[0138] In ACT 123, the processor 11 updates the second purchase registration screen 300 based on the product data record included in the second acquisition command. Then, the processor 11 returns to ACT 112 in Fig. 19. That is, the processor 11 checks whether the timer value T, which is the time required to register the next purchased product, has exceeded the time stored in the average registration time memory 124.

[0139] Processor 11 determines YES in ACT 112 when timer value T has elapsed the time stored in average registration time memory 124. Processor 11 switches from barcode recognition to object recognition when the time required for the purchase product registration operation performed by the user, i.e., timer value T, using the function of switching unit 113, has elapsed the time related to the purchase product registration operation acquired by second acquisition unit 112, i.e., the time stored in average registration time memory 124. Then, processor 11 proceeds to ACT 51 in FIG. 21.

[0140] As can be seen from comparing FIG. 21 with FIG. 11, in the second embodiment, the processor 11 performs the processes of ACT131 to ACT136 after the process of ACT51. The processor 11 displays the first purchase registration screen 100 on the touch panel 31 in ACT131.

[0141] The processing in ACT132 to ACT136 is the same as the processing in ACT25 and ACT27 to ACT30 in the first embodiment. That is, in ACT132, the processor 11 waits for the scanner 17 to capture an image of the purchased item or for a checkout instruction to be issued. When the purchased item is captured while the processor 11 is in the standby state in ACT132 and ACT54, the processor 11 determines YES in ACT132 and proceeds to ACT133. In ACT133, the processor 11 captures a frame image including the purchased item captured by the scanner 17. In ACT134, the processor 11 stores the frame image including the purchased item in the item image memory 123. In ACT135, the processor 11 performs object recognition processing on the frame image. In ACT136, the processor 11 checks whether the object recognition was successful. If the object recognition failed, the processor 11 determines NO in ACT136 and returns to the processing in ACT132. The processor 11 then repeats the processes of ACT 132 to ACT 136 and ACT 54 in the same manner as described above. If the object recognition is successful, the processor 11 determines YES in ACT 136 and proceeds to ACT 58. The processor 11 performs the processes of ACT 58 to ACT 61 in the same manner as in the first embodiment.

[0142] In the second embodiment, after the processing in ACT61, the process proceeds to ACT141 in FIG. 22 and 12, the second embodiment differs from the first embodiment in that the processor 11 omits the processes of ACT74 and ACT75. Each time a frame image including a purchased product captured by the scanner 17 used in the object recognition process is captured, it is stored in the product image memory 123. A first product command including the frame image is then transmitted to the store server 60. Therefore, the processor 11 does not need to check whether a frame image is stored in the product image memory 123 and clear the product image memory 123.

[0143] In ACT141, the processor 11 controls the communication interface 15 to send a first product command to the store server 60. This control causes the communication interface 15 to send the first product command. In ACT142, the processor 11 waits for a first obtain command from the store server 60. When the first obtain command is received from the store server 60, the processor 11 determines YES in ACT142 and proceeds to ACT143. In ACT143, the processor 11 updates the first purchase registration screen 100 based on the product data record included in the first obtain command. Then, the processor 11 returns to ACT132 in FIG. 21. In other words, when the registration operation method is switched from barcode recognition to object recognition, the registration operation of the purchased product thereafter will be performed using object recognition.

[0144] As described above, in the second embodiment, first, the second purchase registration screen 300 is displayed. The customer performs the registration operation using barcode recognition. If the barcode recognition is not successful within the first average registration time or the second average registration time stored in the average registration time memory 124, the registration operation method switches from barcode recognition to object recognition. That is, the switching screen 200 is displayed on the touch panel 31, and the first purchase registration screen 100 is displayed. The customer then performs the registration operation using object recognition. Barcode recognition identifies purchased items from product codes represented by barcodes, which can be more reliable than object recognition, which recognizes purchased items from the features of an image of the purchased item. Even if the registration operation using barcode recognition does not work properly because the barcode is distorted due to a dent in the purchased item, the registration operation method switches to object recognition, so customers do not have to spend a long time and effort performing the registration operation using barcode recognition. This reduces the stress that customers may feel when registering purchased items themselves. Furthermore, as in the first embodiment, the registration operation method switches based on the first or second average registration time, allowing customers to register purchased items using the self-service POS terminal 10 according to their needs.

[0145] Although the first and second embodiments have been described above, the embodiments are not limited to these.

[0146] In the above embodiment, each product sold in a store is provided with a code symbol in a barcode system representing the product code. For example, the code symbol may be a two-dimensional data code system.

[0147] In the above embodiment, the barcode recognition is exemplified as recognizing the purchased product by capturing an image of the purchased product held over the reading window 32 of the scanner 17 and then reading the barcode attached to the purchased product. For example, the barcode recognition may be performed by optically reading the barcode using the reflection of laser light to recognize the purchased product.

[0148] In the embodiment described above, the generation table 121 describes the generation and the first average registration time in association with each other. The generation table 121 may describe, for example, the generation, the gender, and the first average registration time in association with each other. In this case, the first average registration time may be calculated by analyzing the past purchase registration times based on, for example, a combination of the customer's generation and gender.

[0149] In the above embodiment, the product data record includes items such as a product code, a product name, a price, and a first product feature. For example, the product data record may further include an image of the product. The image of the product may be, for example, an image of a product having a standard appearance.

[0150] In the above embodiment, the first average registration time is exemplified as the average purchase registration time required for customers of the associated age group to perform a purchase registration operation, without distinguishing between registration operations using object recognition and registration operations using barcode recognition. For example, the first average registration time may be divided into a first average registration time related to object recognition and a first average registration time related to barcode recognition. The first average registration time related to object recognition is the average purchase registration time required for customers of the associated age group to perform a purchase registration operation using object recognition. The first average registration time related to barcode recognition is the average purchase registration time required for customers of the associated age group to perform a purchase registration operation using barcode recognition. In this case, the first average registration time related to object recognition may be used when a registration operation using object recognition is performed, and the first average registration time related to barcode recognition may be used when a registration operation using barcode recognition is performed.

[0151] In the above embodiment, the actual registration time is described in the purchase registration data 6312 as the purchase registration time actually required by the customer to perform the registration operation using object recognition in the case of a registration operation using object recognition, and as the purchase registration time is described in the purchase registration data 6312 as the purchase registration time actually required by the customer to perform the registration operation using barcode recognition in the case of a registration operation using barcode recognition. For example, the purchase registration data 6312 may further include a flag item for identifying whether the registration operation is based on object recognition or barcode recognition. In this case, for example, the first product command and the second product command may include data for identifying whether the registration operation is based on object recognition or barcode recognition. For example, if the processor 61 of the store server 60 detects a product code from the command received from the self-service POS terminal 10, it may determine that the registration operation is based on barcode recognition, and if it does not detect a product code, it may determine that the registration operation is based on object recognition and set a flag in the purchase registration data 6312. For example, the actual registration time item included in purchase registration data 6312 may be divided into an item for actual registration time related to object recognition and an item for actual registration time related to barcode recognition. The actual registration time related to object recognition is the purchase registration time required for a customer to actually perform a purchase registration operation using object recognition. The actual registration time related to barcode recognition is the purchase registration time required for a customer to actually perform a purchase registration operation using barcode recognition.

[0152] In the above embodiment, the member data record 6311 includes a second average registration time. For example, the member data record 6311 may include a second average registration time related to object recognition and a second average registration time related to barcode recognition. The second average registration time related to object recognition may be, for example, the average value of the actual registration times described in the item "actual registration time related to object recognition" in the purchase registration data 6312 described above. If the actual registration time related to object recognition does not satisfy the predetermined number, a NULL value is set in the second average registration time related to object recognition. The second average registration time related to barcode recognition may be, for example, the average value of the actual registration times described in the item "actual registration time related to barcode recognition" in the purchase registration data 6312 described above. If the actual registration time related to barcode recognition does not satisfy the predetermined number, a NULL value is set in the second average registration time related to barcode recognition. The predetermined number may be different or the same for object recognition and barcode recognition, for example.

[0153] In the above embodiment, the registration start screen is exemplified as a screen that allows a customer to select whether to use a shopping bag provided by the store or to use a reusable bag. For example, the registration start screen may be a screen that accepts the selection of a payment method. In this case, when the selection of the payment method is accepted, the processor 11 determines YES in ACT 14 of FIG. 8 and proceeds to ACT 21 of FIG. 9.

[0154] In the above embodiment, the processor 11 performs face recognition processing on a frame image including a customer's face captured by the camera 311. For example, the processor 61 of the store server 60 or a processor of another server may perform the face recognition processing. In this case, the processor 11 may transmit a command including image data of the frame image to the store server 60 or the other server.

[0155] In the above embodiment, the switching screen 200 is displayed on the touch panel 31. For example, a message may be displayed as a pop-up image on the first purchase registration screen 100 or the second purchase registration screen 300 to notify the customer that the registration operation method will be switched. The pop-up image disappears after a certain period of time. For example, a voice message may be output to inform the customer that the registration operation method will be switched.

[0156] In the above embodiment, when a checkout instruction is given, the processor 11 displays a checkout screen on the touch panel 31. For example, if a checkout instruction is given even though there are no products registered for purchase, the processor 11 may display an error screen on the touch panel 31. The error screen may display, for example, a message informing the customer that there are no products registered for purchase, along with an image of a "Yes" button to indicate that the customer has confirmed the message.

[0157] In the above embodiment, the self-service POS terminal 10 has been described as one aspect of the information processing device. For example, a terminal having the functions of the self-service POS terminal 10 and further having the functions of the store server 60 may be used as the information processing device. Furthermore, the self-service POS terminal 10 as the information processing device may execute part of the functions of the store server 60.

[0158] In the above embodiment, the member database 631 and the product database 632 are stored in the auxiliary storage device 63 of the store server 60. The member database 631 and the product database 632 may be stored in, for example, the main memory 12 or the auxiliary storage device 13 of the self-checkout POS terminal 10. In this case, the processor 11 does not need to send the face authentication command, the first product command, and the second product command to the store server 60. After executing, for example, a payment process, the processor 11 may send a command to the store server 60 including, for example, a learning data record for one transaction (product code, product image, second product feature amount, etc.). This learning data record may further include, for example, a timer value T.

[0159] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims. The following is a summary of the scope of claims as originally filed in this application. [C1] a first acquisition unit that acquires user information of a user who performs a purchase product registration operation; a second acquisition unit that acquires a time period related to the registration operation of the purchased product based on the user information acquired by the first acquisition unit; a switching unit that switches the registration operation method for the purchased product when the time required for the purchase product registration operation performed by the user exceeds the time related to the purchase product registration operation acquired by the second acquisition unit; An information processing device comprising: [C2] the switching unit switches from a first registration operation method of recognizing and registering an image of the purchased product to a second registration operation method of recognizing and registering identification information of the purchased product when the time required for the registration operation of the purchased product performed by the user has exceeded the time related to the registration operation of the purchased product acquired by the second acquisition unit; [C1] The information processing device according to the present invention. [C3] the switching unit switches from the second registration operation method of recognizing and registering the identification information of the purchased product to the first registration operation method of recognizing and registering an image of the purchased product when the time required for the registration operation of the purchased product performed by the user has exceeded the time related to the registration operation of the purchased product acquired by the second acquisition unit; [C1] The information processing device according to the present invention. [C4] The user information is the age of the user. [C1] The information processing device according to the present invention. [C5] an imaging unit that captures an image of a user performing a purchase product registration operation; an information processing device including a first acquisition unit that acquires user information of the user captured by the imaging unit; a second acquisition unit that acquires a time related to the registration operation of the purchased item based on the user information acquired by the first acquisition unit; and a switching unit that switches the registration operation method for the purchased item when the time required for the registration operation of the purchased item performed by the user exceeds the time related to the registration operation of the purchased item acquired by the second acquisition unit. [C6] The computer of the information processing device, a first acquiring means for acquiring user information of a user who performs a purchase product registration operation; a second acquiring means for acquiring a time period related to the registration operation of the purchased product based on the user information acquired by the first acquiring means; and a switching means for switching the method of registering the purchased product when the time required for the user to register the purchased product has exceeded the time related to the registration operation of the purchased product acquired by the second acquisition means; A control program that functions as a [Explanation of symbols]

[0160] 1...communication network, 10...self-service POS terminal, 11, 61...processor, 12, 62...main memory, 13, 63...auxiliary storage device, 14...clock, 15, 64...communication interface, 16...automatic change dispenser, 17...scanner, 18...printer, 19...card reader, 20...timer counter, 21...human presence sensor, 22, 65...system transmission path, 30...main body, 31...touch panel, 32...reading window, 33...card insertion port, 34...receipt issuing port, 35...coin insertion port, 36...coin dispensing port, 37...banknote insertion port, 38...banknote dispensing port, 40...bagging table, 41...housing, 42...bag holder, 43...holding arm, 50...basket stand, 51...communication cable, 52...reader / writer, 53...stand, 54...display pole, 55...light-emitting unit, 60...store server, 121...age table, 122...face image memory, 123...product image memory, 124...average registration time memory, 125...actual registration time memory, 631...membership database, 311...camera, 632...product database, 633...product learning dictionary file, 6311...membership data record, 6312...purchase registration data, S...information processing system, 111...first acquisition unit, 112...second acquisition unit, 113...switching unit.

Claims

1. a first acquisition unit that acquires user information of a user who performs a purchase product registration operation; a second acquisition unit that acquires a time period related to the registration operation of the purchased product based on the user information acquired by the first acquisition unit; a switching unit that switches the registration operation method for the purchased product when a time required for the purchase product registration operation performed by the user exceeds a time related to the purchase product registration operation acquired by the second acquisition unit; Equipped with the user information includes facial features of the user; The first acquisition unit acquires the user information of the user from a captured image including a captured face of the user. The user information further includes an age group of the user; The second acquisition unit acquires a second average registration time, which is the average purchase registration time required for the user to perform the registration operation for the purchased product, based on the facial features of the user, and if the second average registration time cannot be acquired, acquires a first average registration time, which is the average purchase registration time required for users of the user's age group to perform the registration operation for the purchased product, based on the user's age group.

2. the switching unit switches from a first registration operation method of recognizing and registering an image of the purchased product to a second registration operation method of recognizing and registering identification information of the purchased product when a time required for the registration operation of the purchased product performed by the user has exceeded a time related to the registration operation of the purchased product acquired by the second acquisition unit; 2. The information processing device according to claim 1.

3. the switching unit switches from the second registration operation method of recognizing and registering the identification information of the purchased product to the first registration operation method of recognizing and registering an image of the purchased product when the time required for the registration operation of the purchased product performed by the user has exceeded the time related to the registration operation of the purchased product acquired by the second acquisition unit; 2. The information processing device according to claim 1.

4. an imaging unit that captures an image of a user performing a purchase product registration operation; an information processing device including a first acquisition unit that acquires user information of the user captured by the imaging unit; a second acquisition unit that acquires a time related to the purchase product registration operation based on the user information acquired by the first acquisition unit; and a switching unit that switches a purchase product registration operation method when a time required for the purchase product registration operation performed by the user exceeds the time related to the purchase product registration operation acquired by the second acquisition unit, the user information includes facial features of the user; an information processing system, wherein the first acquisition unit acquires the user information of the user from a captured image including a face of the user captured by the imaging unit, The user information further includes an age group of the user; The second acquisition unit acquires a second average registration time, which is the average purchase registration time required for the user to perform the registration operation for the purchased product, based on the facial features of the user, and if the second average registration time cannot be acquired, acquires a first average registration time, which is the average purchase registration time required for users of the user's age group to perform the registration operation for the purchased product, based on the user's age group.

5. The computer of the information processing device, a first acquiring means for acquiring user information of a user who performs a purchase product registration operation; a second acquiring means for acquiring a time period related to the registration operation of the purchased product based on the user information acquired by the first acquiring means; and a switching means for switching the method of registering the purchased product when the time required for the user to register the purchased product has exceeded the time related to the registration operation of the purchased product acquired by the second acquisition means; It functions as the user information includes facial features of the user; the first acquisition means is a control program that acquires the user information of the user from a captured image including a captured face of the user, The user information further includes an age group of the user; The second acquisition means is a control program that acquires a second average registration time, which is the average purchase registration time required for the user to perform the registration operation for the purchased product, based on the facial features of the user, and if the second average registration time cannot be acquired, acquires a first average registration time, which is the average purchase registration time required for users of the user's age group to perform the registration operation for the purchased product, based on the user's age group.

Images