Product sales data processing device and program

Abstract

The present invention provides a product sales data processing device and program that can immediately notify store staff when a customer needs assistance. [Solution] The self-service POS terminal (product sales data processing device) includes a product registration processing unit that reads the identification information of the product the customer wishes to purchase and registers the product through the customer's own operation; a registration time measurement unit that measures the registration time required for each product by the customer; a customer photography unit (photography unit) that photographs the customer's body while the customer is registering the product; an anxiety level estimation unit (estimation unit) that estimates the degree to which the customer is feeling anxious based on the customer's behavior detected from the video captured by the customer photography unit; and an assistance request unit that requests assistance from a store employee's terminal (portable terminal) if it is determined that the customer needs assistance based on the product registration time and the degree to which the customer is feeling anxious.

Description

Technical Field

[0001] Embodiments of the present invention relate to a product sales data processing apparatus and a program.

Background Art

[0002] There is known a system for collecting the attributes of customers who have purchased products by extracting the characteristics of the customers from images of the customers taken by cameras installed in POS terminals (for example, Patent Document 1). On the other hand, in recent years, self-POS terminals that enable customers to perform operations from product registration to settlement by themselves have become widespread.

[0003] Customers who are not used to operating self-POS terminals often operate while feeling anxious, but Patent Document 1 does not disclose or suggest estimating whether a customer is feeling anxious from an image of the customer taken by a camera.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide a product sales data processing apparatus and a program that can immediately inform a store clerk that a customer needs assistance.

Means for Solving the Problems

[0005] The product sales data processing apparatus according to the embodiment includes a product registration processing unit, a registration time measurement unit, a photographing unit, a behavior estimation unit, and an assistance request unit. The product registration processing unit reads identification information of a product that a customer wishes to purchase by the customer's own operation and registers the product. The registration time measurement unit measures the registration time required by the customer for product registration for each product. The photographing unit photographs the customer's body while the product is being registered. The behavior estimation unit estimates the degree to which the customer is feeling anxious from the behavior of the customer detected from the video taken by the photographing unit. The assistance request unit requests assistance to a mobile terminal held by a store clerk on the condition that it is determined that assistance is required for the customer based on the registration time and the degree to which the customer is feeling anxious. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a block diagram showing an example of a schematic configuration of a self-service POS system according to an embodiment. [Figure 2] Figure 2 is a hardware block diagram showing an example of the hardware configuration of a self-POS terminal in the self-POS system of the embodiment. [Figure 3] Figure 3 shows an example of the data structure of a customer state estimation model stored in an image processing device. [Figure 4] Figure 4 is a hardware block diagram showing an example of the hardware configuration of a store clerk terminal in a self-service POS system. [Figure 5] Figure 5 illustrates an example of how a self-service POS terminal determines whether a customer needs assistance. [Figure 6] Figure 6 is a functional block diagram showing an example of the functional configuration of the self-POS terminal and the employee terminal included in the self-POS system of this embodiment. [Figure 7] Figure 7 is a flowchart showing an example of the processing flow performed by the self-service POS system of the embodiment. [Modes for carrying out the invention]

[0007] The embodiments will be described in detail below with reference to the drawings. However, the invention is not limited to the embodiments described below.

[0008] Hereinafter, an embodiment of the self-service POS system 10 described in this disclosure will be explained with reference to the drawings.

[0009] (Outline configuration of a self-service POS system) The schematic configuration of the self-service POS system 10 of the embodiment will be explained using Figure 1. Figure 1 is a block diagram showing an example of the schematic configuration of the self-service POS system of the embodiment.

[0010] The self-service POS system 10 is installed in stores and other locations, and allows customers to register the products they wish to purchase and to make payments for the registered products, all through their own operations.

[0011] The self-service POS system 10 has a configuration in which a store server 11, self-service POS terminals 12, and a wireless router 37 are connected to each other via a network 38. In addition, employee terminals 40, which are carried by store employees, are wirelessly connected to the network 38 via the wireless router 37. Generally, the self-service POS system 10 includes multiple self-service POS terminals 12 and multiple employee terminals 40.

[0012] The store server 11 monitors the operation of each part of the self-service POS system 10. The store server 11 also manages sales data at the store.

[0013] The self-service POS terminal 12 allows customers to register the items they wish to purchase and to make payments for those items. The self-service POS terminal 12 comprises a POS terminal 20, an image processing device 30, and a camera 36. The POS terminal 20 and the image processing device 30 are connected via a network 38. The camera 36 is also connected to the image processing device 30.

[0014] The POS terminal 20 performs product registration processing to register products and payment processing to pay for the registered products. The POS terminal 20 also measures the time required to register each product and outputs it to the image processing device 30.

[0015] The image processing device 30 estimates the degree to which a customer is feeling anxious from the video footage captured by the camera 36 of the customer registering products at the POS terminal 20. Furthermore, if the image processing device 30 determines that the customer registering products at the POS terminal 20 needs assistance, it requests assistance from the store clerk terminal 40.

[0016] The camera 36 photographs the customers who are registering products at the POS terminal 20. The camera 36 is installed at a position where it can photograph the expressions of the customers, the movements of both hands, and the movements of the body. That is, the camera 36 is installed so that the entire customer is within the photographing range.

[0017] The wireless router 37 is a device that wirelessly connects the store employee terminal 40 held by a store employee to the self-POS terminal 12 and the store server 11 via the network 38.

[0018] The network 38 communicably connects each part constituting the self-POS system 10. The network 38 is, for example, a wired LAN (Local Area Network). Note that the network 38 may be constituted by a wireless LAN.

[0019] The store employee terminal 40 is held by a store employee and acquires information related to the situation of the store from the store server 11, the self-POS terminal 12, and another store employee terminal 40. Further, the store employee terminal 40 outputs information from the store employee to the store server 11 and another store employee terminal 40. Further, the store employee terminal 40 acquires a request for assistance for a customer who is registering products at the POS terminal 20 from the image processing device 30 and notifies the store employee. The store employee terminal 40 is, for example, a tablet terminal or a smartphone. Note that the store employee terminal 40 is an example of the portable terminal in the present disclosure.

[0020] (Hardware Configuration of Self-POS Terminal) Using FIG. 2, the hardware configuration of the self-POS terminal 12 will be described. FIG. 2 is a hardware block diagram showing an example of the hardware configuration of the self-POS terminal included in the self-POS system of the embodiment.

[0021] The POS terminal 20 included in the self-POS terminal 12 has a configuration in which a control unit 21, a storage unit 22, a peripheral device controller 24, and a communication controller 25 are connected to each other by an internal bus 23.

[0022] The control unit 21 controls the overall operation of the self-POS terminal 12. The control unit 21 includes a CPU (Central Processing Unit) 211, a ROM (Read Only Memory) 212, and a RAM (Random Access Memory) 213. The CPU 211 is connected to the ROM 212 and the RAM 213 via internal buses such as an address bus and a data bus. The CPU 211 expands various programs stored in the ROM 212 and the storage unit 22 into the RAM 213. The CPU 211 controls the operation of the self-POS terminal 12 by operating according to the various programs expanded in the RAM 213. That is, the control unit 21 has the configuration of a general computer.

[0023] The storage unit 22 is a storage device such as a HDD (Hard Disk Drive) or a SSD (Solid State Drive). Also, the storage unit 22 may be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 22 stores a control program 221, a product master 222, a product registration file 223, and a settlement file 224.

[0024] The control program 221 is a program that controls the overall operation of the self-POS terminal 12. The control program 221 may be provided in a state stored in the storage unit 22, or may be recorded on a computer-readable non-temporary recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD in an installable or executable file format and provided. Also, the control program 221 may be stored on a computer connected to a network and provided by downloading it via the network. Furthermore, the control program 221 may be provided or distributed via a network such as the Internet.

[0025] The product master 222 is a master file that stores product information such as the product name and price of the products to be sold. The product master 222 stores the product name, unit price, additional information, etc., associated with a product code that uniquely identifies each product. Since the contents of the product master are updated as needed, the self-service POS terminal 12 always updates the contents of the product master 222 to the latest state by obtaining the latest product master from the store server 11 via the communication controller 25.

[0026] The product registration file 223 is a file that stores registered product information (e.g., product name, quantity, unit price, etc.) and the total purchase amount, associated with a transaction code that identifies a single transaction.

[0027] Payment file 224 is a file that stores information such as the date and time of payment, payment method, and payment details.

[0028] The control unit 21 is connected via the peripheral device controller 24 to various input / output devices, namely the display device 241, the operation device 242, the scanner 243, the payment terminal 244, the change dispenser 245, and the receipt printer 246.

[0029] The display device 241 displays various information related to product registration and payment. The display device 241 is, for example, an LCD monitor or an OLED monitor.

[0030] The operating device 242 detects customer actions on various buttons displayed on the display device 241. The operating device 242 is, for example, a touch panel stacked on the display device 241.

[0031] Scanner 243 reads a code symbol (e.g., a barcode) attached to a product to obtain product identification information (e.g., a product code). Scanner 243 has a built-in camera and reads the code symbol attached to a product by capturing an image of the product held up to the scanner's reading window. Scanner 243 may also perform so-called object recognition, recognizing the product from an image of the product's appearance.

[0032] The payment terminal 244 is a terminal device that reads credit cards, electronic money cards, payment code information displayed on the customer's mobile device, etc., when a customer makes a cashless payment, and performs payment based on the contents stored in the product registration file 223.

[0033] The change dispenser 245 is a device that accepts payments in banknotes and coins when a customer pays with cash, and dispenses banknotes and coins as change.

[0034] The receipt printer 246 prints a receipt that shows the name, quantity, price, and payment result of the items purchased by the customer.

[0035] Furthermore, the control unit 21 communicates with the store server 11 and the image processing device 30 via the communication controller 25. The POS terminal 20 obtains the latest product master from the store server 11. The POS terminal 20 also outputs a product registration file 223 and a payment file 224 to the store server 11, associated with identification information that identifies the POS terminal 20, in order to manage sales.

[0036] The image processing device 30 in the self-service POS terminal 12 has a configuration in which a control unit 31, a storage unit 32, and a communication controller 34 are connected to each other by an internal bus 33.

[0037] The control unit 31 controls the overall operation of the image processing device 30. The control unit 31 comprises a CPU 311, a ROM 312, and a RAM 313. The CPU 311 connects to the ROM 312 and the RAM 313 via internal buses such as an address bus and a data bus. The CPU 311 loads various programs stored in the ROM 312 and the memory unit 32 into the RAM 313. The CPU 311 controls the operation of the image processing device 30 by operating according to the various programs loaded into the RAM 313. In other words, the control unit 31 has the configuration of a typical computer.

[0038] The memory unit 32 is a storage device such as an HDD or SSD. Alternatively, the memory unit 32 may be a non-volatile memory such as flash memory that retains stored information even when the power is turned off. The memory unit 32 stores the control program 321, the customer state estimation model 322, and the video data 323.

[0039] The control program 321 is a program that controls the overall operation of the image processing device 30. The control program 321 may be provided stored in the storage unit 32, or it may be provided as an installable or executable file recorded on a computer-readable non-temporary recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD. The control program 321 may also be provided by storing it on a computer connected to a network and downloading it via the network. Furthermore, the control program 321 may be provided or distributed via a network such as the Internet.

[0040] The customer state estimation model 322 is a model that estimates the degree of anxiety of a customer registering products at the POS terminal 20. The customer state estimation model 322 estimates the degree of anxiety a customer is feeling from images or videos of the customer registering products at the POS terminal 20, which are captured by the camera 36. More details about the customer state estimation model 322 will be described later (see Figure 3). Note that the customer state estimation model 322 is an example of an estimation model in this disclosure.

[0041] The video data 323 is footage of a customer registering products at the POS terminal 20, captured by camera 36. The video data 323 is input into the customer state estimation model 322 and used to estimate the customer's level of anxiety.

[0042] Furthermore, the control unit 31 communicates with the POS terminal 20 and the employee terminal 40 via the communication controller 34. The image processing device 30 obtains from the POS terminal 20 the time it took for a customer to register one item. Also, if the image processing device 30 determines that the customer is feeling anxious and needs assistance, it requests assistance from the employee terminal 40.

[0043] The camera 36 of the self-service POS terminal 12 is connected to the internal bus 33 of the image processing device 30.

[0044] In this embodiment, the image processing device 30 is configured separately from the POS terminal 20. This is to prevent the image processing device 30, which processes large amounts of video data of customers, from affecting the POS terminal 20's ability to process product sales data. In other words, the image processing device 30 in this embodiment acts as a so-called edge device and, by coordinating with the POS terminal 20 via the network 38, estimates the level of anxiety of customers registering products on the POS terminal 20. It should be noted that the POS terminal 20 may also have a configuration that incorporates the functions of the image processing device 30.

[0045] (Structure of the customer state estimation model) The structure of the customer state estimation model 322 will be explained using Figure 3. Figure 3 shows an example of the data structure of the customer state estimation model stored in the image processing device.

[0046] The customer state estimation model 322 is a machine learning model generated by pre-training on the relationship between video footage of multiple customers registering products at the POS terminal 20 and the degree of anxiety the customers felt at that time. The degree of anxiety felt by the customers is, for example, the result of the customers themselves evaluating their anxiety level on a scale from 1 (not feeling any anxiety) to 5 (feeling very anxious). The form of the customer state estimation model 322 is not limited, but here we will explain an example using a model described with a multi-layered neural network.

[0047] The image processing device 30 captures an image I(t) of a customer registering products at the POS terminal 20. Image I(t) is a so-called moving image, captured, for example, 30 times per second, where t represents the time of capture.

[0048] The image processing device 30 calculates multiple (in this case, n) feature quantities Na1, Na2, ..., Naan from the image I(t). Each feature quantity is calculated from the image I(t) and may relate to the customer's facial expression, the movement of the customer's hands, or the movement of the customer's body. While the type of feature quantity to be calculated is not specified, it is desirable to calculate features that are thought to reflect the customer's level of anxiety. The calculated n feature quantities are input to the input layer La of the customer state estimation model 322.

[0049] The image processing device 30 calculates m intermediate evaluation values ​​Nb1, Nb2, ..., Nbm by assigning weights to the n calculated feature quantities Na1, Na2, ..., Nan and adding them together. The calculated m intermediate evaluation values ​​are input to the intermediate layer Lb of the customer state estimation model 322.

[0050] More specifically, the intermediate evaluation value Nb1 shown in equation (1) is calculated by adding the features Na1, Na2, ..., Naan, each weighted by w11, w12, ..., w1n (w11+w12+...+w1n=1), respectively.

[0051] Nb1=w11*Na1+w12*Na2+…+w1n*Nan…(1)

[0052] Similarly, by assigning weights w21, w22, ..., w2n (w21+w22+...+w2n=1) to the features Na1, Na2, ..., Nan respectively and adding them together, the intermediate evaluation value Nb2 shown in equation (2) is calculated.

[0053] Nb2=w21*Na1+w22*Na2+…+w2n*Nan…(2)

[0054] The image processing device 30 then adds weighted intermediate evaluation values ​​Nb1, Nb2, ..., Nbm input to the intermediate layer Lb, thereby outputting the final evaluation value Nc to the output layer Lc.

[0055] The image processing device 30 performs tuning, which involves adjusting the weights used when calculating the intermediate evaluation values ​​Nb1, Nb2, ..., Nbm and the final evaluation value Nc, so that the correlation between the final evaluation value Nc and the degree of anxiety felt by the customer is as high as possible. In this way, the image processing device 30 generates the customer state estimation model 322.

[0056] The final evaluation value Nc output by the customer state estimation model 322 generated in this way represents the customer's level of anxiety D (hereinafter also referred to as the degree of anxiety D).

[0057] Note that the customer state estimation model 322 described here is just one example, and the image processing device 30 may generate a different form of customer state estimation model.

[0058] (Hardware configuration of the employee terminal) The hardware configuration of the employee terminal 40 will be explained using Figure 4. Figure 4 is a hardware block diagram showing an example of the hardware configuration of an employee terminal included in a self-service POS system.

[0059] The employee terminal 40 has a configuration in which a control unit 41, a storage unit 42, a peripheral device controller 44, and a communication controller 45 are connected to each other by an internal bus 43.

[0060] The control unit 41 controls the overall operation of the employee terminal 40. The control unit 41 comprises a CPU 411, a ROM 412, and a RAM 413. The CPU 411 connects to the ROM 412 and the RAM 413 via internal buses such as an address bus and a data bus. The CPU 411 loads various programs stored in the ROM 412 and the memory unit 42 into the RAM 413. The CPU 411 controls the operation of the employee terminal 40 by operating according to the various programs loaded into the RAM 413. In other words, the control unit 41 has the configuration of a typical computer.

[0061] The storage unit 42 is a storage device such as an HDD or SSD. Alternatively, the storage unit 42 may be a non-volatile memory such as flash memory that retains stored information even when the power is turned off. The storage unit 42 stores the control program 421.

[0062] The control program 421 is a program that controls the overall operation of the employee terminal 40. The control program 421 may be provided stored in the memory unit 42, or it may be provided as an installable or executable file recorded on a computer-readable non-temporary recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD. The control program 421 may also be provided by storing it on a computer connected to a network and downloading it via the network. Furthermore, the control program 421 may be provided or distributed via a network such as the Internet.

[0063] The control unit 41 is connected to the display device 441, the operation device 442, and the speaker 443 via the peripheral device controller 44.

[0064] The display device 441 displays various information related to product registration and payment. The display device 441 is, for example, an LCD monitor or an OLED monitor.

[0065] The operating device 442 detects customer actions on various buttons displayed on the display device 441. The operating device 442 is, for example, a touch panel stacked on the display device 441.

[0066] Speaker 443 outputs sound and voice.

[0067] Furthermore, the control unit 41 communicates with the image processing device 30 and the store server 11 via the communication controller 45. The employee terminal 40 receives assistance requests from the image processing device 30 if a customer who is registering products at the POS terminal 20 is feeling uneasy. The employee terminal 40 also receives various communications from the store server 11 regarding store management, etc. The employee terminal 40 may also make voice calls and send and receive messages with other employee terminals 40.

[0068] (Method for determining whether a customer needs assistance) Figure 5 illustrates how the self-service POS terminal 12 (image processing device 30) determines whether a customer registering a product requires assistance. Figure 5 is a diagram illustrating an example of how the self-service POS terminal determines whether a customer requires assistance.

[0069] The image processing device 30 determines whether the customer who has registered the product needs assistance, according to the determination map M shown in Figure 5.

[0070] The image processing device 30 obtains the registration time required for a customer to register one product from the POS terminal 20. The registration time required to register one product is the time from when the scanner 243 of the POS terminal 20 reads the product code of the product to when it reads the product code of the next product. For the first product to be registered, it is the time from when the registration start button is pressed to start product registration until the product code of the first product is read.

[0071] Generally, the longer it takes to register a single item, the more likely it is that the customer is having difficulty registering the item. However, in this embodiment, the registration time required to register a single item includes the time it takes to place the item, whose product code has been read, into a shopping bag or basket, etc., so it also includes the possibility that the customer is having difficulty with aspects other than the usability of the self-service POS terminal 12. Therefore, the image processing device 30 in this embodiment estimates whether the customer needs assistance by evaluating together the registration time required for the customer to register a single item and the aforementioned level of anxiety D (see Figure 3), which is estimated from the customer's behavior during product registration.

[0072] The following describes how the image processing device 30 determines whether the customer needs assistance.

[0073] The image processing device 30 obtains the time T (registration time) required to register one product from the POS terminal 20. The image processing device 30 then determines whether the time T is less than or equal to a threshold. The threshold for time T referred to here is an example of a first threshold in this disclosure.

[0074] Furthermore, within the time T required to register one product, the image processing device 30 calculates the customer's anxiety level D from the customer's image I(t) captured by the camera 36 using the customer state estimation model 322. The image processing device 30 then determines whether the anxiety level D is below a threshold or above a threshold. The threshold for the anxiety level D referred to here is an example of a second threshold in this disclosure.

[0075] In other words, the image processing device 30 determines which of the following four states applies to the customer's product registration operation, based on the combination of the time T required to register one product and the customer's level of anxiety D.

[0076] If the time T required to register one product is less than a threshold (first threshold) and the customer's level of anxiety D is less than a threshold (second threshold), that is, if it corresponds to region Ra in the judgment map M, the image processing device 30 determines that customer assistance is not required.

[0077] Furthermore, if the time T required to register one product is greater than or equal to a threshold (first threshold), and the customer's level of anxiety D is less than a threshold (second threshold), that is, if it corresponds to region Rb in the judgment map M, the image processing device 30 determines that customer assistance is not required. For example, if the customer is taking time to place the registered product into their shopping bag or basket, this corresponds to region Rb.

[0078] Furthermore, if the time T required to register one product is less than the threshold (first threshold), and the customer's level of anxiety D is greater than or equal to the threshold (second threshold), that is, if it falls under region Rc in the judgment map M, the image processing device 30 transmits the customer's video (image I(t)) along with identification information that identifies the self-POS terminal 12 to the store clerk terminal 40 to ask the clerk whether assistance is needed. If the clerk determines that assistance is needed, the clerk rushes to the self-POS terminal 12 and asks, "Is there anything I can help you with?" and provides assistance to the customer as needed. If it falls under region Rc, product registration is performed without problems, but the customer's behavior (facial expressions and movements) suggests anxiety. In such cases, it cannot be said that assistance from a store clerk is necessarily needed, so it is desirable to have the clerk review the customer's video.

[0079] Furthermore, if the time T required to register one product is greater than or equal to a threshold (first threshold), and the customer's level of anxiety D is greater than or equal to a threshold (second threshold), that is, if it falls under region Rd in the judgment map M, the image processing device 30 notifies the employee terminal 40 of identification information that identifies the self-POS terminal 12, along with information that assistance will be provided to the customer. The employee holding the employee terminal 40 who receives the notification rushes to the corresponding self-POS terminal 12 and asks, "Is there anything I can help you with?" and provides assistance to the customer as needed.

[0080] (Functional configuration of self-service POS terminals and employee terminals) Using Figure 6, the functional configuration of the self-POS terminal 12 and the employee terminal 40 provided in the self-POS system 10 will be explained. Figure 6 is a functional block diagram showing an example of the functional configuration of the self-POS terminal and the employee terminal provided in the self-POS system of this embodiment.

[0081] (POS terminal function configuration) First, let's explain the functional configuration of the POS terminal 20.

[0082] The control unit 21 of the POS terminal 20 implements the product registration processing unit 51, the registration time measurement unit 52, and the payment processing unit 53 shown in Figure 6 as functional units by deploying the control program 221 to the RAM 213 and operating it. Note that all or part of these functional units may be implemented by dedicated hardware.

[0083] The product registration processing unit 51 reads the identification information (product code) of the product that the customer wishes to purchase, based on the customer's own operation, and registers the product. The product code is assumed to be registered, for example, as a code symbol (e.g., a barcode) attached to the product.

[0084] The registration time measurement unit 52 measures the registration time required for a customer to register a product, for each product. Specifically, the registration time measurement unit 52 uses a timer in the CPU 211 to measure the time from when the product registration processing unit 51 reads a product code until it reads the product code of the next product. The registration time measurement unit 52 also outputs the measured registration time to the image processing device 30.

[0085] The payment processing unit 53 performs payment processing for the goods registered by the product registration processing unit 51. The payment processing unit 53 supports both cash and cashless payments.

[0086] (Functional configuration of image processing equipment) Next, the functional configuration of the image processing device 30 will be described.

[0087] The control unit 31 of the image processing device 30 deploys the control program 321 to the RAM 313 and operates it, thereby realizing the customer image capture unit 61, the anxiety level estimation unit 62, and the assistance request unit 63 shown in Figure 6 as functional units. Note that all or part of these functional units may be implemented using dedicated hardware.

[0088] The customer photography unit 61 uses camera 36 to photograph the customer's body while they are registering their products. The customer photography unit 61 is an example of a photography unit in this disclosure.

[0089] The anxiety level estimation unit 62 estimates the degree to which a customer is feeling anxious based on the customer's behavior detected from the video (image I(t)) captured by the customer filming unit 61. Note that the anxiety level estimation unit 62 is an example of an estimation unit in this disclosure.

[0090] The assistance request unit 63, based on the time required to register the products obtained from the POS terminal 20 and the degree to which the customer is feeling anxious, determines that the customer needs assistance and requests assistance from the employee terminal 40 held by the employee.

[0091] (Functional configuration of the store clerk's terminal) Next, I will explain the functional configuration of the employee terminal 40.

[0092] The control unit 41 of the employee terminal 40 implements the assistance request acquisition unit 71 and the notification unit 72 shown in Figure 6 as functional units by deploying the control program 421 to the RAM 413 and operating it. Note that all or part of these functional units may be implemented with dedicated hardware.

[0093] The assistance request acquisition unit 71 acquires from the image processing device 30 an assistance request to the customer and identification information that identifies the POS terminal 20 that the customer is operating.

[0094] The notification unit 72, on the condition that the assistance request acquisition unit 71 has acquired an assistance request from a customer, notifies the employee holding the employee terminal 40 that an assistance request has been received from a customer and provides information identifying the POS terminal 20 that the customer is operating. The method of notification is not limited, but the notification may be displayed on the display device 441 of the employee terminal 40, or an audio or voice message indicating that an assistance request has been received may be output from the speaker 443 of the employee terminal 40.

[0095] (Process flow performed by the self-service POS system) Figure 7 illustrates the processing flow of the self-service POS system 10. Figure 7 is a flowchart showing an example of the processing flow of the self-service POS system in this embodiment.

[0096] (Process flow performed by the POS terminal) First, let's explain the processing flow performed by the POS terminal 20.

[0097] The product registration processing unit 51 determines whether product registration has started (step S11). If it is determined that product registration has started (step S11: Yes), the process proceeds to step S12. On the other hand, if it is not determined that product registration has started (step S11: No), step S11 is repeated. The product registration processing unit 51 determines that product registration has started, for example, when the operating device 242 of the POS terminal 20 presses the registration start button (not shown) displayed on the display device 241.

[0098] In step S11, if it is determined that product registration has started, the registration time measurement unit 52 outputs information indicating that product registration has started to the image processing device 30 (step S12).

[0099] Next, the registration time measurement unit 52 starts measuring the registration time (step S13).

[0100] The product registration processing unit 51 determines whether the product has been registered (step S14). If it is determined that the product has been registered (step S14: Yes), the process proceeds to step S15. On the other hand, if it is determined that the product has not been registered (step S14: No), step S14 is repeated. The product registration processing unit 51 determines that the product has been registered when the scanner 243 of the POS terminal 20 reads product identification information, such as a product code.

[0101] In step S14, if it is determined that the product has been registered, the registration time measurement unit 52 outputs the registration time at that time to the image processing device 30 (step S15).

[0102] Next, the registration time measurement unit 52 resets the registration time (step S16).

[0103] The product registration processing unit 51 determines whether there is an instruction to complete product registration (step S17). If it is determined that there is an instruction to complete product registration (step S17: Yes), the process proceeds to step S18. On the other hand, if it is not determined that there is an instruction to complete product registration (step S17: No), the process returns to step S13 and begins measuring the time required for the next product registration.

[0104] In step S17, if it is determined that there is an instruction to complete product registration, the product registration processing unit 51 outputs information to the image processing unit 30 indicating that product registration has been completed (step S18).

[0105] Next, the payment processing unit 53 performs payment processing related to the registered product details (step S19). After that, the POS terminal 20 terminates the process.

[0106] (Processing flow performed by the image processing device) Next, we will explain the processing flow performed by the image processing device 30.

[0107] The customer scanning unit 61 determines whether it has received information from the POS terminal 20 that product registration has started (step S21). If it is determined that it has received information that product registration has started (step S21: Yes), it proceeds to step S22. On the other hand, if it is not determined that it has received information that product registration has started (step S21: No), it repeats step S21.

[0108] In step S21, if it is determined that information indicating that product registration has started has been obtained, the customer photography unit 61 photographs the customer with the camera 36 (step S22).

[0109] The anxiety level estimation unit 62 estimates the customer's anxiety level D by inputting the customer image I(t) taken by the customer photography unit 61 into the customer state estimation model 322 (step S23).

[0110] The anxiety level estimation unit 62 determines whether it has obtained the registration time required for the customer to record one product from the POS terminal 20 (step S24). If it is determined that the registration time has been obtained (step S24: Yes), the process proceeds to step S25. On the other hand, if it is determined that the recording time has not been obtained (step S24: No), the process returns to step S22 and the customer's recording continues.

[0111] In step S24, if it is determined that the registration time has been obtained, the assistance request unit 63 determines whether the customer needs assistance based on the customer's anxiety level D and the product registration time, using the determination map M (see Figure 5) (step S26). If it is determined that the customer needs assistance (step S26: Yes), the process proceeds to step S27. On the other hand, if it is determined that the customer does not need assistance (step S26: No), the process proceeds to step S28. The assistance request unit 63 determines that the customer needs assistance if it is determined that the customer's condition belongs to area Rd in the determination map M in Figure 5.

[0112] In step S26, if it is determined that the customer needs assistance, the assistance request unit 63 requests assistance from the store clerk terminal 40 (step S27). At this time, the assistance request unit 63 also outputs identification information that identifies the POS terminal 20 that the customer is operating.

[0113] Subsequently, the customer photography unit 61 finishes photographing the customer (step S29). Then, the image processing device 30 finishes processing.

[0114] Furthermore, if it is determined in step S26 that the customer does not require assistance, the assistance request unit 63 determines whether it has obtained information from the POS terminal 20 that product registration has been completed (step S28). If it is determined that it has obtained information that product registration has been completed (step S28: Yes), the process proceeds to step S29. On the other hand, if it is not determined that it has obtained information that product registration has been completed (step S28: No), the process returns to step S22 and continues taking photos of the customer.

[0115] Although not shown in Figure 7, the assistance request unit 63 may, in step S26, output an image I(t) of the customer during product registration to the image processing device 30 if it is determined that the customer's condition belongs to region Rc in the determination map M of Figure 5, that is, if the time T required to register one product is less than the threshold (first threshold) and the customer's level of anxiety D is equal to or greater than the threshold (second threshold).

[0116] (Processing flow performed by the store clerk's terminal) Next, we will explain the processing flow performed by the employee terminal 40.

[0117] The assistance request acquisition unit 71 determines whether it has acquired an assistance request for the customer from the image processing device 30 (step S31). If it is determined that an assistance request for the customer has been acquired (step S31: Yes), the process proceeds to step S32. On the other hand, if it is not determined that an assistance request for the customer has been acquired (step S31: No), step S31 is repeated.

[0118] In step S31, if it is determined that a request for assistance has been received from a customer, the notification unit 72 notifies the employee holding the employee terminal 40 that a request for assistance from a customer has been received, and provides identification information to identify the POS terminal 20 that the customer is operating (step S32). The employee terminal 40 then terminates processing.

[0119] Furthermore, customer assistance requests are output to multiple employee terminals 40. If an employee who receives an assistance request is near the POS terminal 20 the customer is using, they will go to assist the customer. At that time, it is desirable to notify other employees' employee terminals 40 that they will be handling the assistance request.

[0120] Furthermore, when the image processing device 30 requests assistance from the store clerk terminal 40, it may use the image I(t) of the customer captured by the camera 36 to further train the customer state estimation model 322.

[0121] (Effects of the embodiment) As described above, the self-service POS terminal 12 (product sales data processing device) of this embodiment includes a product registration processing unit 51 that reads the identification information of the product the customer wishes to purchase and registers the product through the customer's own operation, a registration time measurement unit 52 that measures the time T (registration time) required for the customer to register the product for each product, a customer photography unit 61 (photography unit) that photographs the customer's body while the product is being registered, an anxiety level estimation unit 62 (estimation unit) that estimates the degree to which the customer is feeling anxious based on the customer's behavior detected from the video captured by the customer photography unit 61, and an assistance request unit 63 that requests assistance from the store clerk's terminal 40 (portable terminal) if it is determined that the customer needs assistance based on the time T (registration time) and the degree to which the customer is feeling anxious.Therefore, the store clerk can be immediately notified that the customer needs assistance.

[0122] Furthermore, in the self-service POS terminal 12 (product sales data processing device) of this embodiment, the anxiety level estimation unit 62 (estimation unit) estimates the degree of anxiety a customer feels based on a customer state estimation model 322 (estimation model) that has learned the relationship between multiple feature quantities extracted from video footage of multiple customers when registering products and the degree of anxiety felt by multiple customers when registering products. Therefore, the degree of anxiety a customer feels can be estimated accurately with simple processing.

[0123] Furthermore, in the self-service POS terminal 12 (product sales data processing device) of this embodiment, the customer's behavior detected from the video captured by the customer camera unit 61 (camera unit) includes at least the customer's facial expressions and gestures. Therefore, since information that easily indicates anxiety can be obtained, the customer's level of anxiety D can be estimated with high accuracy.

[0124] Furthermore, in the self-service POS terminal 12 (product sales data processing device) of this embodiment, the assistance request unit 63 requests assistance from the store clerk terminal 40 (mobile terminal) on the condition that the time T (registration time) for each product exceeds a first threshold and the degree of anxiety felt while registering the product exceeds a second threshold. Therefore, it is possible to immediately determine that assistance is needed and to request assistance from a store clerk.

[0125] Furthermore, in the self-service POS terminal 12 (product sales data processing device) of this embodiment, the assistance request unit 63 transmits the video captured by the customer recording unit 61 (recording unit) to the store clerk terminal 40 (mobile terminal) on the condition that the time T (registration time) for each product does not exceed a first threshold, and the degree of anxiety felt while registering the product exceeds a second threshold. Therefore, if it is difficult to determine whether assistance is needed, the store clerk can be made to decide whether the customer needs assistance.

[0126] In this embodiment, the image processing device 30 estimates the level of anxiety D of a customer registering products at the POS terminal 20. However, the image processing device 30 may also be configured to estimate the level of anxiety D of a customer while they are performing the payment procedure that follows product registration, and to request assistance from the store clerk terminal 40 if assistance from a store clerk is needed.

[0127] Although embodiments of the present invention have been described above, these embodiments are illustrative and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0128] 10 Self-service POS systems 11 Store Servers 12 Self-service POS terminal (product sales data processing device) 20 POS terminals 30 Image Processing Device 36 Cameras 40. Staff terminal (mobile terminal) 51 Product Registration Processing Unit 52 Registration time measurement unit 53 Payment Processing Unit 61. Customer Photography Department (Photography Department) 62. Anxiety level estimation unit (estimation unit) 63. Aid Request Department 71 Assistance request acquisition department 72 Hochi Department 322 Customer State Estimation Model (Estimation Model) D. Anxiety level (degree of anxiety) I(t) Image La Input Layer Lb middle layer Lc output layer M Judgment Map T Time (Registration Time) [Prior art documents] [Patent Documents]

[0129] [Patent Document 1] Japanese Patent Publication No. 2020-204942

Claims

1. A product registration processing unit reads the identification information of the product that the customer wishes to purchase through the customer's own operation and registers the product, A registration time measurement unit measures the registration time required for each product for the customer to register the product, A camera unit that photographs the customer's body while the aforementioned product is being registered, An estimation unit estimates the degree to which the customer is feeling anxious based on the customer's behavior detected from the video captured by the aforementioned camera unit, Based on the registration time and the degree to which the customer is feeling anxious, the assistance request unit, on the condition that it is determined that the customer needs assistance, requests assistance via a mobile terminal held by the store clerk, A product sales data processing device equipped with the following features.

2. The estimation unit, Based on an estimation model that has previously learned the relationship between multiple features extracted from videos of multiple customers registering a product and the degree to which those customers felt anxious when registering the product, the degree to which the customer felt anxious is estimated. Product sales data processing device according to claim 1.

3. The aforementioned behavior includes, at least, the customer's facial expression and the customer's gestures. Product sales data processing device according to claim 2.

4. The aforementioned assistance request unit is, The mobile device is used to request assistance if the registration time for each product exceeds a first threshold, and the degree of anxiety experienced while registering the product exceeds a second threshold. Product sales data processing device according to claim 1.

5. The aforementioned assistance request unit is, The video captured by the camera unit is transmitted to the mobile terminal, provided that the registration time for each product does not exceed a first threshold, and the degree of anxiety felt while registering the product exceeds a second threshold. Product sales data processing device according to claim 1.

6. A computer that controls a product sales data processing device, A product registration processing unit reads the identification information of the product that the customer wishes to purchase through the customer's own operation and registers the product, A registration time measurement unit measures the registration time required for each product for the customer to register the product, A camera unit that photographs the customer's body while the aforementioned product is being registered, An estimation unit estimates the degree to which the customer is feeling anxious based on the customer's behavior detected from the video captured by the aforementioned camera unit, Based on the registration time and the degree to which the customer is feeling anxious, if it is determined that the customer requires assistance, the system will operate as an assistance request unit that requests assistance from the mobile terminal held by the store employee. program.

Images