Product sales data processing device, program, and product sales data processing system

Abstract

To provide a commodity sales data processing device, a program and a commodity sales data processing system, capable of supporting selection and registration of a commodity when a visually impaired person goes shopping.SOLUTION: A commodity sales data processing device according to an embodiment includes audio input means, shopping list generation means, commodity image acquisition means and registration means. The audio input means receives audio input. The shopping list generation means generates a shopping list on the basis of the audio input. The commodity image acquisition means acquires a commodity image obtained by capturing a commodity in a shop, from a walking aid stick. When commodity information corresponding to the commodity image is included in the shopping list, the registration means registers the commodity corresponding to the commodity information as a settlement object.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention relate to a product sales data processing device, a program, and a product sales data processing system.

Background Art

[0002] Conventionally, in stores such as supermarkets, technologies for realizing POS (Point Of Sale) processing such as product purchases using mobile terminals such as smartphones and tablet terminals are known. In such technologies, customers can select the products they purchase at the storefront, register the products themselves on the mobile terminal, and perform settlement.

[0003] By the way, when a customer is visually impaired, it may be difficult to select desired products in the store by themselves and register the products using a mobile terminal.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide a product sales data processing device, a program, and a product sales data processing system that can assist visually impaired people in selecting and registering products when shopping.

Means for Solving the Problems

[0005] The product sales data processing device according to the embodiment includes a voice input means, a shopping list generation means, a product image acquisition means, and a registration means. The voice input means receives voice input. The shopping list generation means generates a shopping list based on the voice input. The product image acquisition means acquires a product image of a product photographed in the store from a walking aid cane. The registration means registers a product corresponding to the product information as a settlement target when the product information corresponding to the product image is included in the shopping list.

Brief Description of the Drawings

[0006] [Figure 1] Figure 1 shows an example of a schematic configuration of a product sales data processing system according to an embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of a mobile terminal according to this embodiment. [Figure 3] Figure 3 shows an example of the software configuration of a mobile terminal according to this embodiment. [Figure 4] Figure 4 shows an example of the hardware configuration of a smart white cane according to an embodiment. [Figure 5] Figure 5 shows an example of the software configuration of a smart white cane according to an embodiment. [Figure 6] Figure 6 shows an example of the hardware configuration of a store server according to the embodiment. [Figure 7] Figure 7 shows an example of the configuration of a product image database according to this embodiment. [Figure 8] Figure 8 shows an example of the configuration of a product information database according to this embodiment. [Figure 9] Figure 9 shows an example of the software configuration of a store server according to this embodiment. [Figure 10-1] Figure 10-1 is a sequence diagram showing an example of the processing flow performed by the product sales data processing system according to the embodiment. [Figure 10-2] Figure 10-2 is a continuation of the sequence diagram in Figure 10-1. [Modes for carrying out the invention]

[0007] (Embodiment) Figure 1 shows an example of the schematic configuration of the product sales data processing system 1 according to this embodiment. The product sales data processing system 1 of this embodiment includes a mobile terminal 2, a smart white cane 3, a store server 4, and a wireless access point 5. However, the configuration of the product sales data processing system 1 is not limited to this, and it is sufficient if it includes at least the mobile terminal 2 and the smart white cane 3.

[0008] The product sales data processing system 1 is used, for example, in stores such as supermarkets or convenience stores. In this embodiment, the user is a visually impaired person, but the user may also be a caregiver (a sighted person) assisting the visually impaired person. The user enters the store carrying a mobile terminal 2 and a smart white cane 3. Also, since it is difficult for a visually impaired person to operate the mobile terminal 2, they may enter the store with the mobile terminal 2 stored in a bag or clothing pocket. The product sales data processing system 1 assists such users when they shop in the store. If the user is a caregiver assisting a visually impaired person, the caregiver may carry the mobile terminal 2 and the visually impaired person may carry the smart white cane 3. In this embodiment, the term "user" may refer to either the visually impaired person or the caregiver, or it may refer to both the visually impaired person and the caregiver.

[0009] The store server 4 is, for example, a server device installed in a store such as a supermarket or convenience store. However, the store server 4 does not necessarily have to be physically installed within the store; for example, it may be installed on a cloud server.

[0010] The store server 4 and the wireless access point 5 are connected via network Na.

[0011] Network Na can use the Internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., individually or in appropriate combinations.

[0012] The wireless access point 5 is a device that can connect to the mobile terminal 2 via the wireless network Nb. The wireless network Nb is a wireless LAN implemented by, for example, Bluetooth® or Wi-Fi®. The wireless access point 5 is installed in the store and establishes communication with the mobile terminal 2 within a connectable distance by periodically transmitting a beacon. In this embodiment, the wireless access point 5 is installed in the store and connects to communicate with the mobile terminal 2 carried by a user who enters the store.

[0013] Although only one wireless access point 5 is shown in Figure 1, multiple wireless access points 5 may actually be installed within the store. By each of the multiple wireless access points 5 communicating with the mobile terminal 2, the location of the mobile terminal 2 within the store can be determined. Alternatively, the location of the mobile terminal 2 may be determined using a gyroscope (not shown) provided in the mobile terminal 2 or a gyroscope 307 (see Figure 4) provided in the smart white cane 3.

[0014] Mobile terminal 2 is, for example, a smartphone or tablet device, and is an information processing device that the user can carry. Mobile terminal 2 is an example of a product sales data processing device in this embodiment. Mobile terminal 2 may be owned by the user or provided by the store.

[0015] Mobile terminal 2 connects to wireless access point 5 via wireless network Nb. Therefore, when mobile terminal 2 is within the communication range of wireless access point 5, it can communicate with store server 4 via wireless network Nb, wireless access point 5, and network Na.

[0016] The Smart White Cane 3 is a white cane equipped with information processing functions to assist visually impaired users in walking. The Smart White Cane 3 is an example of a walking aid cane in this embodiment.

[0017] The mobile terminal 2 and the smart white cane 3 are connected by a wireless network Nc such as Bluetooth (registered trademark).

[0018] The product sales data processing system 1 may further include a terminal that can be carried by a store clerk. For example, such a terminal can notify the store clerk of notifications from the store server 4.

[0019] Next, the hardware configuration of the mobile terminal 2 will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the mobile terminal 2 according to the present embodiment.

[0020] As shown in FIG. 2, the mobile terminal 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a first communication unit 204, a second communication unit 205, a display unit 206, an operation unit 207, a microphone 208, a speaker 209, and a storage unit 210.

[0021] The CPU 201 is an example of a processor and comprehensively controls the operation of the mobile terminal 2. The ROM 202 stores various programs. The RAM 203 is the main memory device of the mobile terminal 2 and functions as a workspace where programs and various data are expanded.

[0022] The CPU 201, the ROM 202, and the RAM 203 are connected via a bus and constitute a control unit 2 of the computer configuration. In the control unit 2, various processes are executed by the CPU 201 operating according to the programs stored in the ROM 202 and the storage unit 210 and expanded in the RAM 203. Such a control unit 2 is connected to each unit (the first communication unit 204, the second communication unit 205, the display unit 206, the operation unit 207, the microphone 208, the speaker 209, and the storage unit 210) via a bus.

[0023] The first communication unit 204 is a wireless communication interface that can connect to wireless networks Nb and Nc. The first communication unit 204 communicates with external devices such as the store server 4 and the smart white cane 3 via the wireless networks Nb and Nc. In this embodiment, the mobile terminal 2 uses the first communication unit 204 to perform processes such as acquiring product images and product information, notifying the user, receiving the result of selecting a payment method, and registering the products to be paid for, via the wireless networks Nb, Nc or ​​the wireless access point 5.

[0024] The second communication unit 205 is a communication interface that connects to a mobile communication system such as 4G or 5G. The second communication unit 205 connects to the internet via the mobile communication system.

[0025] The display unit 206 is an example of a display device installed in the mobile terminal 2, and is, for example, an LCD (Liquid Crystal Display) or a touch panel display, which displays various information according to the control of the control unit 20.

[0026] The operation unit 207 has various operation buttons and outputs key information to the control unit 20 according to the operator's actions. The operation unit 207 includes a touch panel provided on the display surface of the display unit 206.

[0027] Microphone 208 is an example of an audio input device that accepts user voice input. Microphone 208 is an example of an audio input means in this embodiment.

[0028] Speaker 209 is an example of an audio output device and outputs audio according to the control of the control unit 20.

[0029] The storage unit 210 is a non-volatile storage device that maintains its contents even when the power is cut off, and is composed of, for example, an SSD (Solid State Drive) or flash memory. The storage unit 210 stores various programs executed by the CPU 201 and various setting information related to the operation of the mobile terminal 2. For example, the storage unit 210 stores programs 211, such as application programs for performing product registration and payment processing on the mobile terminal 2, which are provided from the store server 4 or a cloud server located on the internet. In this embodiment, the application program for performing product registration and payment processing on the mobile terminal 2 is called a POS (Point of Sale) app.

[0030] Note that the hardware configuration of the mobile terminal 2 is not limited to the example shown in Figure 2. For example, the mobile terminal 2 may also be equipped with various other functions such as a camera and a motor for vibration.

[0031] Next, the software configuration of the mobile terminal 2 will be described. Figure 3 is a diagram showing an example of the software configuration of the mobile terminal 2 according to this embodiment. Specifically, Figure 3 illustrates various functions realized by the POS application installed on the mobile terminal 2. The program 211 executed on the mobile terminal 2 of this embodiment has a module configuration including the shopping list generation unit 21, route generation unit 22, verification processing unit 23, registration processing unit 24, and payment processing unit 25 shown in Figure 3. The CPU 201 reads the program 211 from the storage unit 210 and loads each of the above units (shopping list generation unit 21, route generation unit 22, verification processing unit 23, registration processing unit 24, and payment processing unit 25) onto the RAM 203. As a result, each of the above units is generated on the RAM 203. Note that the functions of the mobile terminal 2 are not limited to those shown in Figure 3.

[0032] The shopping list generation unit 21 generates a shopping list based on the user's desired product names, which are input via voice from the microphone 208. The shopping list generation unit 21 is an example of a shopping list generation means in this embodiment.

[0033] A shopping list is a list of product names or codes of items that the user wishes to purchase and that are in stock at the store.

[0034] More specifically, the shopping list generation unit 21 queries the store server 4 for inventory information using the user's desired product name, which is input via voice from the microphone 208. If the store has inventory of a product that matches the user's desired product name, the shopping list generation unit 21 registers that product name in the shopping list. The shopping list generation unit 21 stores the generated shopping list in, for example, the storage unit 210. The shopping list generation unit 21 may list product names, or it may obtain product codes corresponding to product names from the store server 4 and list them. For example, JAN (Japanese Article Number) codes can be used as product codes. In this embodiment, the shopping list is a list of product names. Hereinafter, products corresponding to product names included in the shopping list will also be referred to as products included in the shopping list.

[0035] Furthermore, if the store does not have in stock the product name that matches the user's desired name, the shopping list generation unit 21 will output a voice message via the speaker 209 recommending online shopping. In addition to generating a shopping list before the user enters the store and begins shopping, the shopping list generation unit 21 can also add product names to the shopping list and remove product names from the shopping list while the user is shopping inside the store.

[0036] The route generation unit 22 generates a route that passes in front of the shelves where the products corresponding to the product information included in the shopping list are located, based on the shopping list generated by the shopping list generation unit 21 and the store's map information. More specifically, the route generation unit 22 obtains store information, including map information indicating the location of the shelves where the products are located within the store, from the store server 4 via the first communication unit 204. The route generation unit 22 then generates a shopping route that goes from the store entrance, through the shelves where the products corresponding to the product names included in the shopping list are located, to the store exit. The route generation unit 22 is an example of a map information acquisition means in this embodiment.

[0037] A shopping route is the path within the store, starting from the entrance, passing through shelves where the products corresponding to the names on the shopping list are located, and ending at the store exit. If a single product is located in multiple sections of the store, the shopping route only needs to pass through at least one of those sections.

[0038] Furthermore, if a product name is added to the shopping list, the route generation unit 22 modifies the shopping route so that it passes in front of the shelf where the product corresponding to the added product name is located. In addition, if a product name that is no longer needed is deleted from the shopping list, the route generation unit 22 modifies the shopping route so that it does not pass in front of the shelf where the product corresponding to the deleted product name is located. The route generation unit 22 is an example of a route generation means in this embodiment.

[0039] The matching processing unit 23 compares the items included in the shopping list with the items identified from the images of the items (hereinafter also referred to as product images). More specifically, when the matching processing unit 23 acquires the product image captured by the camera of the smart white cane 3 via the first communication unit 204, it transmits the product image to the store server 4 via the first communication unit 204. The matching processing unit 23 then obtains product information from the store server 4 that allows for the identification of the product corresponding to the product image.

[0040] The product image is either an image of the product in the store, or an image of a symbol attached to the product (such as a barcode or QR code), and in this embodiment, it is a still image of the product's appearance taken from at least one direction. The product image is captured by the camera 309 of the smart white cane 3. However, the form of the product image is not limited to this, and it may also be a video or a set of images of the product taken from multiple directions.

[0041] Product information includes, for example, the product name and / or product code. In this embodiment, the product name is described as an example of product information. The matching processing unit 23 determines that a product is included in the shopping list if the product name obtained from the store server 4 is included in the shopping list. The matching processing unit 23 also determines that a product is not included in the shopping list if the product name obtained from the store server 4 is not included in the shopping list. In this case, the matching processing unit 23 requests the store server 4 to notify a store employee. By notifying the store employee, if the store server 4 mistakenly picks up a product that is not on the shopping list, the employee who receives the notification can go to the user and handle the return. The matching processing unit 23 is an example of the product image acquisition means, transmission means, and product information acquisition means in this embodiment.

[0042] The registration processing unit 24 registers the product corresponding to the product information as a payment target if the product information corresponding to the product image is included in the shopping list. More specifically, the registration processing unit 24 registers the product corresponding to the product name as a payment target if the product name obtained from the store server is included in the shopping list. In this embodiment, if the product name obtained from the store server is not included in the shopping list, the registration processing unit 24 does not register the product corresponding to the product name as a payment target. The registration processing unit 24 is an example of a registration means in this embodiment. In this embodiment, the functions of the matching processing unit 23 and the registration processing unit 24 have been described separately, but the functions of the matching processing unit 23 may also be functions of the registration processing unit 24.

[0043] Furthermore, the registration processing unit 24 notifies the user if the product information corresponding to the product image is not included in the shopping list. More specifically, if the product name obtained from the store server is not included in the shopping list, the registration processing unit 24 notifies the user via the first communication unit 204 to the smart white cane 3 that the product does not correspond to the product name in the shopping list. For this reason, the registration processing unit 24 is an example of a notification means in this embodiment.

[0044] The payment processing unit 25 works in conjunction with the store server 4 to process payments for items registered as payment targets by the registration processing unit 24. Payment processing begins when the user instructs to start payment via voice input to the smart white cane 3 or mobile terminal 2, or when the user leaves the store.

[0045] When the user instructs the payment to begin, the payment processing unit 25 accepts the user's selection of a payment method based on their operation of the smart white cane 3. For example, the number of times the smart white cane 3 is struck against the floor may be pre-associated with a payment method and registered in the storage unit 210. As a result, after the user is instructed to begin payment, within a predetermined time, the payment method is specified based on the user's actions: once striking the smart white cane 3 corresponds to credit card payment, twice striking it corresponds to electronic money payment, three times striking it corresponds to code payment, and four times striking it corresponds to cryptocurrency payment.

[0046] For visually impaired individuals, identifying different types of cash (banknotes and coins) is not easy, making cash payments inconvenient. Therefore, the payment method used by the payment processing unit 25 is basically cashless. However, if a visually impaired person is accompanied by a caregiver, the caregiver can have a barcode displayed on their mobile terminal 2 read by a cashier (not shown), and the cashier can then make the payment using the cashier's change dispenser. The cashier is an example of a product sales data processing device in this disclosure.

[0047] Furthermore, if a user leaves the store without paying for items registered as eligible for payment, the payment processing unit 25 will pay for the registered items using a predetermined method. The predetermined method is, for example, the payment method registered by the user when setting up the POS application. The predetermined payment method is stored in the storage unit 210. When the first communication unit 204 of the mobile terminal 2 receives an exit signal from the wireless access point 5, the payment processing unit 25 notifies the user that payment has started and then performs payment using the predetermined method. The exit signal is transmitted from the wireless access point 5 installed near the store exit, and the first communication unit 204 of the mobile terminal 2 receives the exit signal from the wireless access point 5 when the mobile terminal 2 enters the communication area of ​​the wireless access point 5 installed near the store exit. The notification of payment start is, for example, made by voice output via the smart white cane 3, but may also be made by voice output from the speaker 209 of the mobile terminal 2. The payment processing unit 25 is an example of the selection acceptance means and payment means in this embodiment.

[0048] Next, the hardware configuration of the smart white cane 3 will be described. Figure 4 is a diagram showing an example of the hardware configuration of the smart white cane 3 according to this embodiment.

[0049] As shown in Figure 4, the smart white cane 3 includes a processor such as a CPU 301, a ROM 302, a RAM 303, a communication unit 304, a motor 305, an acceleration sensor 306, a gyroscope 307, an ultrasonic sensor 308, a camera 309, a speaker 310, a microphone 311, and a storage unit 312.

[0050] The CPU 301, ROM 302, and RAM 303 are connected via a bus to form a control unit 30 of the computer configuration. In the control unit 30, the CPU 301 operates according to programs stored in the ROM 302 and memory unit 312 and loaded into the RAM 303, thereby executing various processes. This control unit 30 is connected via a bus to each component (communication unit 304, motor 305, acceleration sensor 306, gyroscope 307, ultrasonic sensor 308, camera 309, speaker 310, microphone 311, and memory unit 312). The control unit 30 may be configured, for example, as a single-board computer.

[0051] The communication unit 304 is a wireless communication interface that can connect to the wireless network Nc. The communication unit 304 communicates with the mobile terminal 2 via the wireless network Nc and sends and receives various types of information. For example, the communication unit 304 is an example of an acquisition means for obtaining the shopping route within a store from the mobile terminal 2. The communication unit 304 is also an example of a transmission means for sending product images captured by the camera 309 and the results of the user's payment method selection to the mobile terminal 2.

[0052] The motor 305 is a drive unit used for the vibrator function and vibrates the smart white cane 3 under the control of the control unit 30.

[0053] The acceleration sensor 306 measures the acceleration caused by the movement of the smart white cane 3. The gyroscope 307 measures the angular acceleration of the smart white cane 3 to detect its rotation and changes in orientation. The acceleration sensor 306 and the gyroscope 307 are examples of sensors capable of detecting the user's action of tapping the floor with the smart white cane 3 in this embodiment.

[0054] The ultrasonic sensor 308 measures the distance to surrounding obstacles by transmitting and receiving ultrasonic waves.

[0055] Camera 309 is a digital camera capable of photographing surrounding objects through user operation, and has an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary MOS). Camera 309 is an example of a photographic means capable of photographing products inside a store.

[0056] The speaker 310 is an example of an audio output device and outputs sound according to the control of the control unit 30.

[0057] Microphone 311 is an example of an audio input device that accepts user voice input.

[0058] The memory unit 312 is a non-volatile storage device composed of, for example, an SSD or flash memory. The memory unit 312 stores various programs executed by the CPU 301 and various setting information related to the operation of the smart white cane 3. For example, the memory unit 312 stores a program 313 that performs processing to support the user's shopping in cooperation with the mobile terminal 2.

[0059] Note that the hardware configuration of the smart white cane 3 is not limited to the example shown in Figure 4. For example, the smart white cane 3 may also include a module for reading RFID (Radio Frequency Identification), a barcode reader, or various sensors.

[0060] Next, the software configuration of the smart white cane 3 will be described. Figure 5 shows an example of the software configuration of the smart white cane 3 according to this embodiment. The program 313 executed in the smart white cane 3 of this embodiment has a module configuration that includes the voice input receiving unit 31, guidance unit 32, product image transmission unit 33, selection operation receiving unit 34, and notification unit 35 shown in Figure 5. The CPU 301 reads the program 313 from the storage unit 312 and loads each of the above units (voice input receiving unit 31, guidance unit 32, product image transmission unit 33, selection operation receiving unit 34, and notification unit 35) onto the RAM 303. As a result, each of the above units is generated on the RAM 303. Note that the functions of the smart white cane 3 are not limited to those shown in Figure 5.

[0061] The voice input receiving unit 31 receives voice input from the user via the microphone 311.

[0062] The guidance unit 32 guides the user to the shelf where the items on the shopping list are located, based on the shopping route acquired from the mobile terminal 2. The guidance unit 32 may, for example, inform the user of the direction of travel by controlling the speaker 310 to output a voice message, or by controlling the motor 305 to inform the user of the direction of travel by vibration. The guidance unit 32 is an example of the guidance means of the smart white cane 3 in this embodiment.

[0063] The guidance unit 32 may obtain the user's current location within the store from the mobile terminal 2 and change the guidance content according to the user's movement along the shopping route. The user's current location within the store is determined, for example, by communication between the wireless access point 5 and the mobile terminal 2.

[0064] Furthermore, each shelf in the store may be marked with a QR code (registered trademark) or barcode or other symbol that can identify it. In this case, the guidance unit 32 may determine the user's current location from the QR code (registered trademark) or the like on the store shelf, which is captured by the camera 309 of the smart white cane 3, and guide the user in the direction of travel based on the determined location.

[0065] The product image transmission unit 33 transmits the product image captured by the camera 309 to the mobile terminal 2 via the communication unit 304. The product image transmission unit 33 is an example of the image transmission means for the smart white cane 3 in this embodiment. The communication unit 304 may also be an example of the image transmission means for the smart white cane 3.

[0066] The selection operation reception unit 34 receives an operation by the user to select a payment method. More specifically, after the start of payment is instructed, if the acceleration sensor 306 and gyroscope 307 detect an action of tapping the floor surface with the smart white cane 3 within a predetermined time, the selection operation reception unit 34 receives a selection of a payment method corresponding to the number of taps. The selection operation reception unit 34 transmits the received payment method to the mobile terminal 2 via the communication unit 304. Note that the association between the user's actions and the payment method may be made on the mobile terminal 2 side or on the smart white cane 3 side. For example, the selection operation reception unit 34 may transmit the number of taps to the mobile terminal 2, or it may transmit the payment method corresponding to that number to the mobile terminal 2.

[0067] The notification unit 35 notifies the user by controlling the speaker 310 to output an audio message. The notification unit 35 and the speaker 310 are examples of notification means of the smart white cane 3 in this embodiment.

[0068] Next, the hardware configuration of the store server 4 will be described. Figure 6 shows an example of the hardware configuration of the store server 4 according to this embodiment.

[0069] As shown in Figure 6, the store server 4 includes a processor such as a CPU 401, a ROM 402, a RAM 403, a communication unit 404, a display unit 405, an operation unit 406, and a storage unit 407.

[0070] The CPU 401, ROM 402, and RAM 403 are connected via a bus and constitute the control unit 40 of the computer configuration. In the control unit 40, the CPU 401 operates according to programs stored in the ROM 402 and memory unit 407 and loaded into the RAM 403, thereby executing various processes. This control unit 40 is connected via a bus to each unit (communication unit 404, display unit 405, operation unit 406, and memory unit 407).

[0071] The communication unit 404 is a wireless communication interface that can connect to network Na. The communication unit 404 communicates with the mobile terminal 2 via network Na, wireless access point 5, and wireless network Nb, and sends and receives various types of information.

[0072] The display unit 405 is an example of a display device, such as an LCD or a touch panel display, and displays various information according to the control of the control unit 40.

[0073] The operation unit 406 is, for example, an operating device such as a keyboard or mouse. The display unit 405 and the operation unit 406 may be configured as a touch panel display. Furthermore, the display unit 405 and the operation unit 406 may be devices connected to the store server 4 via a network Na or the like.

[0074] The storage unit 407 is a non-volatile storage device composed of, for example, an SSD or flash memory. The storage unit 407 stores various programs executed by the CPU 401 and various setting information related to the operation of the store server 4. For example, the storage unit 407 stores a program 408 that performs product registration and payment processing in cooperation with the mobile terminal 2. The storage unit 407 also stores store information 409, inventory information 410, a product image database (DB) 411, and a product information database 412.

[0075] Store information 409 includes, at a minimum, map information indicating the location of the shelves on which the products are placed within the store.

[0076] Inventory information 410 is information about the inventory of products in a store. For example, inventory information 410 is information that associates a product name or product code with the number of products in stock at the store. Inventory information 410 may include not only the inventory of the store in question, but also the number of products in stock at online shops and the number of products in nearby stores.

[0077] Figure 7 shows an example of the configuration of the product image database 411 according to this embodiment. As shown in Figure 7, the product image database 411 is a database in which product images, product names, and product codes are associated.

[0078] Figure 8 shows an example of the configuration of the product information database 412 according to this embodiment. As shown in Figure 8, the product information database 412 is a database in which product codes, product names, and unit prices are associated. The product information database 412 may also include further information about products.

[0079] The configuration of the product image database 411 and the product information database 412 is not limited to the examples shown in Figures 7 and 8; for example, only one of the product name or product code may be registered.

[0080] Next, the software configuration of the store server 4 will be described. Figure 9 shows an example of the software configuration of the store server 4 according to this embodiment. The program 408 executed by the store server 4 in this embodiment has a module configuration that includes the product image matching unit 41, registration processing unit 42, payment processing unit 43, staff calling unit 44, store information provision unit 45, and inventory information provision unit 46 shown in Figure 9. The CPU 401 reads the program 408 from the storage unit 407 and loads each of the above units (product image matching unit 41, registration processing unit 42, payment processing unit 43, staff calling unit 44, store information provision unit 45, and inventory information provision unit 46) onto the RAM 403. As a result, each of the above units is generated on the RAM 403. Note that the functions of the store server 4 are not limited to those shown in Figure 9.

[0081] The product image matching unit 41 compares the product image obtained from the mobile terminal 2 via the communication unit 404 with the product image registered in the product image database 411. For example, the product image matching unit 41 may calculate product feature quantities (product shape, representative color, etc.) extracted from the product image, or calculate the similarity of the product images and match the product based on the feature quantities or similarity, or it may decode the symbols extracted from the product image into a product code and match the product based on the product code. When the product image matching unit 41 identifies a product image from the product image database 411 that matches the product image obtained from the mobile terminal 2, it transmits the product name associated with that product image to the mobile terminal 2 via the communication unit 404. The product image matching unit 41 may transmit the product code instead of the product name, or it may transmit both the product name and the product code. The technology for matching product images is not particularly limited, and known image processing technologies may be employed.

[0082] The registration processing unit 42 receives an instruction from the mobile terminal 2 to register a product to be paid for and registers the product. The registration processing unit 42, for example, searches the product information database 412 for the unit price corresponding to the product to be paid for and stores the product name and unit price in association with each other in the storage unit 407. The registration processing unit 42 may also store the product code in association with the unit price instead of the product name, or it may store both the product name and product code in association with the unit price.

[0083] The payment processing unit 43 receives notification from the mobile terminal 2 that payment processing has begun and executes the payment processing for the registered items to be paid for. The payment processing unit 43 pays for the items using either the payment method selected by the user from the mobile terminal 2 or the default payment method used when the user leaves the store.

[0084] The staff calling unit 44 notifies the staff member of the request when a request for staff is received from the mobile terminal 2. The method of calling is not particularly limited, but for example, the staff calling unit 44 may send a signal to the staff member's terminal indicating the user's location within the store and instruct the staff member to go to the user.

[0085] When the store information provision unit 45 receives an inquiry about store information from the mobile terminal 2, it reads the store information 409 from the storage unit 407 and transmits it to the mobile terminal 2 via the communication unit 404.

[0086] When the inventory information provision unit 46 receives an inquiry about product inventory information from the mobile terminal 2, it identifies the number of items in stock from the inventory information 410 in the storage unit 407 and transmits it to the mobile terminal 2 via the communication unit 404.

[0087] Next, we will explain the processing flow executed by the product sales data processing system 1 configured as described above.

[0088] Figures 10-1 and 10-2 are sequence diagrams showing an example of the processing flow performed by the product sales data processing system 1 according to this embodiment.

[0089] When a user carrying a mobile terminal 2 and a smart white cane 3 enters a store, the first communication unit 204 of the mobile terminal 2 receives an entry signal transmitted from the wireless access point 5 (S1). In this case, the POS application on the mobile terminal 2 is launched (S2).

[0090] Then, the route generation unit 22 of the mobile terminal 2 queries the store server 4 via the first communication unit 204 for store information, including map information indicating the location of the shelves on which the products are placed within the store (S3).

[0091] The store information provision unit 45 of the store server 4 receives an inquiry from the mobile terminal 2, reads the store information 409 from the storage unit 407, and transmits it to the mobile terminal 2 via the communication unit 404 (S4).

[0092] Here, the user who has entered the store voice-inputs the name of the product they wish to purchase into the mobile terminal 2. The microphone 208 of the mobile terminal 2 receives this voice input (S5). Note that in Figure 10-1, the mobile terminal 2 receives the user's voice input, but the microphone 311 of the smart white cane 3 may also receive the voice input, transmit the user's voice input from the communication unit 304 of the smart white cane 3 to the mobile terminal 2, and receive it at the first communication unit 204 of the mobile terminal 2 (S5).

[0093] The shopping list generation unit 21 of the mobile terminal 2 queries the store server 4 for inventory information using the desired product name of the user, which is input by voice from the microphone 208 (S6).

[0094] When the store server 4 receives an inquiry about product inventory information from the mobile terminal 2, the inventory information unit 46 of the store server 4 transmits the inventory information of the product to the mobile terminal 2 via the communication unit 404 (S7).

[0095] Then, if the shopping list generation unit 21 of the mobile terminal 2 does not have the product matching the user's desired name in stock at the store, it outputs a voice message via the speaker 209 recommending online shopping (S8). In Figure 10-1, the speaker 209 of the mobile terminal 2 outputs the online shopping message, but the speaker 310 of the smart white cane 3 may also output the same message. Furthermore, the online shopping message is not mandatory, and the mobile terminal 2 or smart white cane 3 may simply notify the user that the product is not in stock at the store.

[0096] Next, the shopping list generation unit 21 of the mobile terminal 2 generates a shopping list of products that are in stock at the store if one or more of the voice-inputted product names are in stock (S9).

[0097] Then, the route generation unit 22 of the mobile terminal 2 generates a shopping route within the store based on the shopping list and store information obtained from the store server 4 (S10).

[0098] The route generation unit 22 of the mobile terminal 2 transmits the generated shopping route to the smart white cane 3 via the first communication unit 204 (S11).

[0099] Then, the guidance unit 32 of the smart white cane 3 controls the speaker 310 or the vibration motor 305 to start route guidance based on the shopping route (S12). The guidance unit 32 of the smart white cane 3 guides the user to the shelf where the items registered in the shopping list are placed. The guidance unit 32 may also output information indicating the location of the items, such as which shelf the items registered in the shopping list are located on, as an audio message from the speaker 310.

[0100] Then, guided by the smart white cane 3, the user arrives at the product shelf, picks up the product, and takes a picture of it with the smart white cane 3's camera 309. Once the product is photographed by the smart white cane 3's camera 309 (S13), the smart white cane 3's product image transmission unit 33 transmits the product image to the mobile terminal 2 via the communication unit 304 (S14). The user may then place the photographed product in, for example, a shopping basket.

[0101] While shopping, the user walks using the smart white cane 3 and may also be holding a shopping basket. Therefore, both hands may be occupied, making it difficult to photograph products with the mobile device 2. However, in this embodiment, the camera 309 of the smart white cane 3 can capture product images, allowing the user to photograph products without having to take the mobile device 2 out of their pocket or bag each time.

[0102] When the matching processing unit 23 of the mobile terminal 2 acquires a product image captured by the camera of the smart white cane 3 via the first communication unit 204, it sends the product image to the store server 4 via the first communication unit 204 to inquire about the product name (S15).

[0103] Then, the product image matching unit 41 of the store server 4 searches the product image database 411 using the product image acquired from the mobile terminal 2 via the communication unit 404 (S16).

[0104] When the product image matching unit 41 identifies a product image from the product image database 411 that matches the product image acquired from the mobile terminal 2, it transmits the product name associated with that product image to the mobile terminal 2 via the communication unit 404 (S17).

[0105] The matching processing unit 23 of the mobile terminal 2 compares the product name obtained from the store server 4 with the product name included in the shopping list (S18).

[0106] If the product name obtained from the store server 4 matches the product name in the shopping list, the registration processing unit 24 instructs the store server 4 to register the product name as a payment target (S19). The registration processing unit 42 of the store server 4 receives an instruction from the mobile terminal 2 to register a product to be paid for and executes a registration process to register the product as a payment target (S20).

[0107] When the registration processing unit 42 of the store server 4 registers the product specified by the mobile terminal 2 as a payment target, it sends a notification of registration completion to the mobile terminal 2 via the communication unit 404 (S21).

[0108] Then, when the matching processing unit 23 of the mobile terminal 2 receives a registration completion notification from the store server 4, it transmits the registration completion notification and the registered product name to the smart white cane 3 via the first communication unit 204 (S22).

[0109] The notification unit 35 of the smart white cane 3 controls the speaker 310 to output a voice message notifying the user of the product name of the product registered for payment and that the product has been registered for payment (S23). The content of the voice message may be, for example, "This is xx (product name). It will be registered as a product for purchase," but is not limited to this. Through this voice message, a visually impaired user can recognize that the product photographed in S13 matches the product entered by voice in S5, and that the product has been registered for payment.

[0110] In response to this, if the product name obtained from the store server 4 does not match the product name in the shopping list, the registration processing unit 24 notifies the smart white cane 3 via the first communication unit 204 that the product does not match the product name in the shopping list (S24).

[0111] Then, the notification unit 35 of the smart white cane 3 controls the speaker 310 to output a voice message notifying the user of the product name and that the product is not on the shopping list (S25). The content of the voice message may be, for example, "This is yy (product name). It is not registered on the shopping list," but is not limited to this.

[0112] Furthermore, if the product name obtained from the store server 4 does not match the product name in the shopping list, the matching processing unit 23 of the mobile terminal 2 requests the store server 4 to notify the store staff (S26).

[0113] The staff calling unit 44 of the store server 4 notifies the staff of the call when a request for staff is received from the mobile terminal 2 (S27). The staff calling unit 44 may, for example, notify the staff that the user has picked up the wrong item and ask them to help the user find the correct item. Alternatively, the staff calling unit 44 may notify the staff of a message instructing them to confirm that the item the user mistakenly picked up has been returned to the shelf in order to reduce the occurrence of incorrect transactions.

[0114] Furthermore, during route guidance by the smart white cane 3, the user can add desired items to the shopping list. For example, if the user voice-inputs the name of an additional item into the microphone 208 of the mobile terminal 2 during route guidance (S28), the shopping list generation unit 21 of the mobile terminal 2 queries the store server 4 for inventory information using the name of the item voice-input from the microphone 208 (S29). From the inventory inquiry process in S29 to the online shopping guidance process in S31, the process is the same as S6 to S8 described above. Note that although Figure 10-2 shows the user voice-inputting the items to be added to the shopping list into the mobile terminal 2, the user may also voice-input the items to be added into the smart white cane 3, and the voice input may be transmitted from the communication unit 304 of the smart white cane 3 to the mobile terminal 2 and received by the first communication unit 204 of the mobile terminal 2 (S28).

[0115] Alternatively, the user may remove unwanted items from their shopping list while route guidance is being provided by the smart white cane 3. For example, during route guidance, the user can voice-input the name of the item to be removed into the microphone 208 of the mobile terminal 2 or the microphone 311 of the smart white cane 3 (S28).

[0116] Then, the route generation unit 22 of the mobile terminal 2 updates the shopping list by adding items that are in stock from the added items (S32). Alternatively, the route generation unit 22 of the mobile terminal 2 updates the shopping list based on the deleted items (S32).

[0117] The route generation unit 22 of the mobile terminal 2 regenerates the shopping route based on the updated shopping list (S33).

[0118] Then, the route generation unit 22 of the mobile terminal 2 transmits the regenerated shopping route to the smart white cane 3 via the first communication unit 204 (S34).

[0119] Then, the guidance unit 32 of the smart white cane 3 starts route guidance based on the regenerated shopping route (S35).

[0120] The processes from S13 to S35 are executed repeatedly after the start of route guidance in S12 until payment processing begins due to the user leaving the store or giving a payment instruction. Once payment processing begins, route guidance by Smart White Cane 3 ends.

[0121] Here, we will explain the payment process when a user attempts to leave the store without paying for their purchases. As the user approaches the store exit, the first communication unit 204 of the mobile terminal 2 carried by the user receives an exit signal from the wireless access point 5 located near the exit (S36).

[0122] The payment processing unit 25 of the mobile terminal 2 notifies the smart white cane 3 to start payment when the first communication unit 204 of the mobile terminal 2 receives an exit signal from the wireless access point 5 (S37).

[0123] In this case, the guidance unit 32 of the smart white cane 3 terminates the route guidance. Then, the notification unit 35 of the smart white cane 3 controls the speaker 310 to output an audio message notifying that payment has started (S38).

[0124] The payment processing unit 25 of the mobile terminal 2 works in conjunction with the store server 4 to initiate payment processing using the default payment method registered in advance by the user (S39).

[0125] The payment processing unit 43 of the store server 4 receives notification from the mobile terminal 2 that payment processing has started and executes the payment processing using the default payment method (S40). The payment processing unit 43 sends the payment result to the mobile terminal 2 (S41).

[0126] Next, we will explain the case where payment is initiated by the user. If the user gives a voice command to start payment into the microphone 311 of the smart white cane 3 while route guidance is being provided by the smart white cane 3 (S42), the guidance unit 32 of the smart white cane 3 will terminate the route guidance.

[0127] The selection operation reception unit 34 of the smart white cane 3 then receives an operation from the user to select a payment method (S43). For example, the selection operation reception unit 34 receives the operation to select credit card payment if the user taps the smart white cane 3 once, electronic money payment if the user taps it twice, code payment if the user taps it three times, and cryptocurrency payment if the user taps it four times. Note that the selection method is not limited to tapping the smart white cane 3; for example, voice input to the smart white cane 3 may also be used.

[0128] Then, the selection operation reception unit 34 of the smart white cane 3 transmits the payment method selected by the user to the mobile terminal 2 via the communication unit 304 (S44).

[0129] The payment processing unit 25 of the mobile terminal 2 works in conjunction with the store server 4 to initiate payment processing using the payment method selected by the user (S45).

[0130] The payment processing unit 43 of the store server 4 receives notification from the mobile terminal 2 that payment processing has started and executes the payment processing according to the payment method selected by the user (S46). The payment processing unit 43 sends the payment result to the mobile terminal 2 (S47). At this point, the processing in this sequence diagram ends.

[0131] Thus, according to the mobile terminal 2 of this embodiment, a shopping list is generated based on the user's voice information, and if product information corresponding to a product image acquired from the smart white cane 3 held by the user is included in the shopping list, the corresponding product is registered as the item to be paid for, thereby supporting the selection and registration of products when a visually impaired user makes a purchase.

[0132] Furthermore, the mobile terminal 2 of this embodiment notifies the user if the product information corresponding to the product image obtained from the store server is not included in the shopping list. Therefore, with the mobile terminal 2 of this embodiment, if a visually impaired user picks up the wrong product, the user can be quickly notified of the error. This reduces the occurrence of situations where the user purchases products they do not want.

[0133] Furthermore, the mobile terminal 2 of this embodiment generates a shopping route based on the shopping list and store map information. If product information is added by the user via voice information during shopping, the mobile terminal 2 of this embodiment adds the product information to the shopping list and changes the shopping route so that it passes in front of the product shelf where the product corresponding to the added product information is located. Therefore, the mobile terminal 2 of this embodiment can assist the user even if the number of products the user wants to buy increases during shopping.

[0134] Furthermore, in this embodiment, if product information is deleted by the user's voice during shopping, the mobile terminal 2 will remove the product information from the shopping list and change the shopping route so that the user does not pass in front of the shelf where the product corresponding to the deleted product information is located. Therefore, with the mobile terminal 2 of this embodiment, it is possible to assist the user even if they end up with products they no longer need while shopping.

[0135] Furthermore, the mobile terminal 2 of this embodiment accepts the user's selection of a payment method based on their operation of the smart white cane 3. Therefore, the mobile terminal 2 of this embodiment makes it easy for the user to select their desired payment method.

[0136] Furthermore, in this embodiment, if a user leaves a store without paying for items registered for payment, the mobile terminal 2 will process the payment for those items using a predetermined method. Therefore, with the mobile terminal 2 of this embodiment, it is possible to reduce the occurrence of missed payments even if a user forgets to pay and tries to leave the store. In addition, even if a malicious user tries to leave the store without paying, the payment will be forcibly processed using a predetermined method, thus preventing shoplifting.

[0137] Furthermore, the product sales data processing system 1 of this embodiment includes a mobile terminal 2 and a smart white cane 3. The smart white cane 3 guides the user based on the shopping route generated by the mobile terminal 2 and transmits product images captured by the smart white cane 3's camera 309 to the mobile terminal 2. If the product images obtained from the smart white cane 3 are included in the shopping list, the mobile terminal 2 registers the product as a payment target. Therefore, the product sales data processing system 1 of this embodiment can assist visually impaired users in selecting and registering products when shopping.

[0138] In the above embodiment, the microphone 208 of the mobile terminal 2 was used as an example of a voice input means. However, the smart white cane 3 may receive voice input of a product list from the user, the communication unit 304 of the smart white cane 3 may transmit the voice input to the mobile terminal 2, and the first communication unit 204 of the mobile terminal 2 may receive the voice input. In this case, the first communication unit 204, which is the wireless communication interface of the mobile terminal 2, becomes a voice input means that receives the voice-inputted product list from the smart white cane 3. If this configuration is adopted, the first communication unit 204 becomes an example of a voice input means. Alternatively, either the microphone 208 or the first communication unit 204 of the mobile terminal 2 may be used as the first voice input means and the other as the second voice input means.

[0139] Furthermore, as described above, the mobile terminal 2 uses the first communication unit 204 to perform processes such as acquiring product images and product information via the wireless network Nb, Nc or ​​wireless access point 5, notifying the user, receiving the result of selecting a payment method, and registering the product to be paid for. For this reason, the first communication unit 204 may be an example of the product image acquisition means, product information acquisition means, transmission means, registration means, notification means, map acquisition means, and selection acceptance means in this embodiment.

[0140] In the above-described embodiment, a method for assisting a user's shopping through the functions of the mobile terminal 2 and the smart white cane 3 was explained. However, the division of functions between the mobile terminal 2 and the smart white cane 3 is not limited to the above example. The functions described as functions of the mobile terminal 2 in the above-described embodiment may be provided by the smart white cane 3, or the functions described as functions of the smart white cane 3 may be provided by the mobile terminal 2.

[0141] In the embodiment described above, the system was configured to identify the product picked up by the user using a product image captured by the camera 309 of the smart white cane 3. However, other methods besides product images may be used. For example, instead of a product image, a barcode, QR code (registered trademark), RFID tag, etc., read by the smart white cane 3 may be used.

[0142] Furthermore, in the above-described embodiment, the mobile terminal 2 communicated with the smart white cane 3 and the store server 4 to send and receive information, but the smart white cane 3 and the store server 4 may also send and receive information directly via wireless communication.

[0143] Furthermore, while the above-described embodiment shows an example where the mobile terminal 2 and the store server 4 notify the store clerk if the product corresponding to the product image is not included in the shopping list, the response when a user picks up a product that is not included in the shopping list is not limited to this. For example, a user may use a shopping cart equipped with two baskets, one basket for items to be paid for and the other basket for items to be returned. If the smart white cane 3 notifies the user that the photographed product is not included in the shopping list, the user places the product in the basket for items to be returned. The basket in which the user has placed the product is recognized by a camera or the like installed in the cart.

[0144] Furthermore, in the above-described embodiment, if the product corresponding to the product image does not match the shopping list, i.e., if the user picks up the wrong product, the store server 4 notifies the user to call a store employee and request a return. However, the method of return is not limited to this. For example, the route generation unit 22 of the mobile terminal 2 may generate a return route for returning the product to the store shelf if the product corresponding to the product image does not match the shopping list. The return route is a route that passes in front of the shelf where the product that does not match the shopping list was placed. Also, if a return occurs during shopping, the return route may be incorporated as part of the shopping route.

[0145] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0146] (Note) (1) A means for obtaining the shopping route within a store from a mobile device based on the user's desired shopping list, A guidance means that guides the user to the shelf where the items included in the shopping list are placed, based on the shopping route, A photographic means capable of photographing products inside the store, A transmission means for transmitting product images captured by the aforementioned imaging means to an external device (mobile terminal or store server), If the external device notifies the user that the product depicted in the product image does not correspond to the shopping list, the notification means notifies the user that the product does not correspond to the shopping list. A walking aid cane equipped with [features / equipment]. (2) When the notification means receives notification from the external device that the product depicted in the product image corresponds to the shopping list, it notifies the user that the product corresponds to the shopping list and the name of the product. The walking aid described in (1) above. (3) It is further equipped with a sensor capable of detecting the user's movement when they tap the floor with a walking aid cane, The transmission means transmits a payment method to the mobile terminal that corresponds to the number of times the sensor detects the action of tapping the floor surface with the walking cane within a predetermined time after payment has been instructed. A walking aid cane as described in (1) or (2) above. [Explanation of Symbols]

[0147] 1. Product Sales Data Processing System 2 Mobile devices 3. Smart White Cane 4 Store Servers 5 Wireless access points 20 Control Unit 21 Shopping List Generation Section 22 Route generation unit 23. Matching Processing Unit 24 Registration Processing Unit 25 Payment Processing Unit 30 Control Unit 31 Voice Input Reception Unit 32 Information Department 33 Product Image Transmission Section 34 Selection operation reception unit 35 Notification Department 40 Control Unit 41 Product Image Matching Section 42 Registration Processing Unit 43 Payment Processing Unit 44. Staff Calling Department 45 Store Information Department 46 Inventory Information Department 204 First Communications Department 205 Second Communications Department 206 Display section 207 Operation section 208 Microphones 209 speakers 210 Storage section 211 Programs 304 Communications Department 305 Motor 306 Accelerometer 307 Gyroscope 308 Ultrasonic Sensor 309 Camera 310 speakers 311 Microphone 312 Storage section 313 Programs 404 Communications Department 405 Display section 406 Operation section 407 Storage section 408 Programs 409 Store Information 410 Inventory Information 411 Product Image Database 412 Product Information Database Na network Nb, NC Wireless Network [Prior art documents] [Patent Documents]

[0148] [Patent Document 1] Japanese Patent Publication No. 2015-007846

Claims

1. A product sales data processing device configured as a mobile terminal, A voice input method that accepts voice input, A shopping list generation means that generates a shopping list based on the aforementioned voice input, A product image acquisition method that obtains product images of products in a store from a walking aid cane, If product information corresponding to the aforementioned product image is included in the shopping list, a registration means for registering the product corresponding to said product information as a payment target, A product sales data processing device equipped with the following features.

2. The system further includes a notification mechanism for notifying the user if the product information corresponding to the product image is not included in the shopping list. Product sales data processing device according to claim 1.

3. A map acquisition means that acquires map information indicating the location of product shelves on which products are placed within the store, The system further comprises a route generation means that generates a shopping route that passes in front of shelves on which products corresponding to the product information included in the shopping list are placed, based on the shopping list and the map information. The aforementioned voice input means accepts the addition or deletion of product information via voice, The shopping list generation means adds or removes the product information added or removed by the voice input means to the shopping list. The route generation means modifies the shopping route so that, when product information is added to the shopping list, it passes in front of the shelf where the product corresponding to the added product information is located, or modifies the shopping route so that, when product information is deleted from the shopping list, it does not pass in front of the shelf where the product corresponding to the deleted product information is located. Product sales data processing device according to claim 1.

4. The system further includes a selection acceptance means for receiving the user's selection of a payment method based on the operation of the walking aid cane. Product sales data processing device according to claim 1.

5. If a user leaves the store without paying for the items registered as eligible for payment, the system further provides a payment method for paying for those items using a predetermined method. Product sales data processing device according to any one of claims 1 to 4.

6. A computer mounted on a user-portable information processing device, A voice input step that accepts voice input, A shopping list generation step that generates a shopping list based on the aforementioned voice input, The product image acquisition step involves obtaining product images of items in the store from a walking aid cane, and If product information corresponding to the aforementioned product image is included in the shopping list, a registration step is performed to register the product corresponding to that product information as a payment target. A program to execute.

7. A product sales data processing device configured as a mobile terminal, Including walking aids, The aforementioned product sales data processing device is A voice input method that accepts voice input, A shopping list generation means that generates a shopping list based on the aforementioned voice input, Image acquisition means for acquiring product images of products in the store from the walking aid cane, If product information corresponding to the aforementioned product image is included in the shopping list, the system includes a registration means for registering the product corresponding to the said product information as a payment target, The aforementioned walking aid cane is A guidance means for guiding the user to a shelf where products corresponding to the product information included in the shopping list are placed, A photographic means capable of photographing products inside the store, The system includes an image transmission means for transmitting the captured product image to the product sales data processing device, Product sales data processing system.

Images