Commodity sales data processing device, program and commodity sales data processing system

Abstract

To provide a commodity sales data processing device, a program and a commodity sales data processing system, capable of supporting selection and registration of a commodity when a visually impaired person goes shopping.SOLUTION: A commodity sales data processing device according to an embodiment includes audio input means, shopping list generation means, commodity image acquisition means and registration means. The audio input means receives audio input. The shopping list generation means generates a shopping list on the basis of the audio input. The commodity image acquisition means acquires a commodity image obtained by capturing a commodity in a shop, from a walking aid stick. When commodity information corresponding to the commodity image is included in the shopping list, the registration means registers the commodity corresponding to the commodity information as a settlement object.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to a merchandise sales data processing device, a program, and a merchandise sales data processing system. [Background technology]

[0002] Conventionally, in stores such as supermarkets, there is known a technology that realizes POS (Point Of Sale) processing such as purchasing goods by using a mobile terminal such as a smartphone or a tablet terminal. With such a technology, a customer can select the goods he or she wants to purchase at the store, register the goods on the mobile terminal himself or herself, and make a payment.

[0003] However, if a customer is visually impaired, it may be difficult for him or her to select a desired product in a store and register the product using a mobile terminal. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a product sales data processing device, program, and product sales data processing system capable of assisting visually impaired people in selecting and registering products when shopping. [Means for solving the problem]

[0005] A product sales data processing device according to an embodiment includes a voice input means, a shopping list generation means, a product image acquisition means, and a registration means. The voice input means accepts voice input. The shopping list generation means generates a shopping list based on the voice input. The product image acquisition means acquires product images of products in a store from a walking aid cane. If product information corresponding to the product image is included in the shopping list, the registration means registers the product corresponding to the product information as a payment target. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a product sales data processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the mobile terminal according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a software configuration of the mobile terminal according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the hardware configuration of a smart white cane according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the software configuration of the smart white cane according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a store server according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a configuration of a product image database according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a configuration of a product information database according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a software configuration of a store server according to the embodiment. [Figure 10-1] FIG. 10-1 is a sequence diagram illustrating an example of a flow of processing executed in the product sales data processing system according to the embodiment. [Figure 10-2] Figure 10-2 is a continuation of the sequence diagram in Figure 10-1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] (Embodiment) 1 is a diagram showing an example of a schematic configuration of a product sales data processing system 1 according to this embodiment. The product sales data processing system 1 of this embodiment includes a mobile terminal 2, a smart white cane 3, a store server 4, and a wireless access point 5. Note that the configuration of the product sales data processing system 1 is not limited to this, and it is sufficient that the system includes at least the mobile terminal 2 and the smart white cane 3.

[0008] The product sales data processing system 1 is used in a store such as a supermarket or a convenience store. In this embodiment, the user is a visually impaired person, but the user may also be an assistant (healthy person) who assists the visually impaired person. The user enters the store carrying the mobile terminal 2 and the smart white cane 3. In addition, since the visually impaired person has difficulty operating the mobile terminal 2, the user may enter the store with the mobile terminal 2 stored in a bag or a pocket of clothes. The product sales data processing system 1 assists such a user when shopping in the store. In addition, if the user is an assistant who assists the visually impaired person, the assistant may carry the mobile terminal 2 and the visually impaired person may carry the smart white cane 3. In this embodiment, the user may refer to either the visually impaired person or the assistant, or may refer to both the visually impaired person and the assistant.

[0009] The store server 4 is, for example, a server device installed in a store such as a supermarket or a convenience store. Note that the store server 4 does not necessarily have to be physically installed in the store, and may be installed in, for example, a cloud server.

[0010] The store server 4 and the wireless access point 5 are connected via a network Na.

[0011] The network Na may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination.

[0012] The wireless access point 5 is a terminal that can be connected to the mobile terminal 2 via the wireless network Nb. The wireless network Nb is, for example, a wireless LAN realized by Bluetooth (registered trademark) or Wi-Fi (registered trademark). The wireless access point 5 is installed in a store and periodically transmits a Beacon to establish communication with a mobile terminal 2 within a connectable distance range. In this embodiment, the wireless access point 5 is installed in a store and is communicatively connected to a mobile terminal 2 carried by a user who enters the store.

[0013] 1 shows only one wireless access point 5, but in reality, multiple wireless access points 5 may be installed in the store. The location of the mobile terminal 2 in the store can be identified by each of the multiple wireless access points 5 communicating with the mobile terminal 2. Alternatively, the location of the mobile terminal 2 may be identified using a gyroscope (not shown) provided in the mobile terminal 2 or a gyroscope 307 (see FIG. 4) provided in the smart white cane 3.

[0014] The mobile terminal 2 is, for example, a smartphone or a tablet terminal, and is an information processing device that can be carried by a user. The mobile terminal 2 is an example of a product sales data processing device in this embodiment. The mobile terminal 2 may be owned by the user or provided by the store.

[0015] The mobile terminal 2 connects to the wireless access point 5 via the wireless network Nb. Therefore, when the mobile terminal 2 is located within the communication range of the wireless access point 5, the mobile terminal 2 can communicate with the store server 4 via the wireless network Nb, the wireless access point 5, and the network Na.

[0016] The smart white cane 3 is a white cane equipped with an information processing function, and assists a visually impaired user in walking. The smart white cane 3 is an example of a walking assistance cane in this embodiment.

[0017] The mobile terminal 2 and the smart white cane 3 are connected via a wireless network Nc such as Bluetooth (registered trademark).

[0018] The product sales data processing system 1 may further include a terminal that can be carried by a store clerk. The terminal can notify the store clerk of notifications from the store server 4, for example.

[0019] Next, a description will be given of the hardware configuration of the mobile terminal 2. Fig. 2 is a diagram showing an example of the hardware configuration of the mobile terminal 2 according to this embodiment.

[0020] As shown in FIG. 2, the mobile terminal 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a first communication unit 204, a second communication unit 205, a display unit 206, an operation unit 207, a microphone 208, a speaker 209, and a memory unit 210.

[0021] The CPU 201 is an example of a processor, and controls the overall operation of the mobile terminal 2. The ROM 202 stores various programs. The RAM 203 is a main storage device of the mobile terminal 2, and functions as a workspace on which programs and various data are developed.

[0022] The CPU 201, ROM 202, and RAM 203 are connected via a bus to constitute a computer-configured control unit 20. In the control unit 20, the CPU 201 operates according to a program stored in the ROM 202 or storage unit 210 and loaded in the RAM 203, thereby executing various processes. Such a control unit 20 is connected to each unit (a first communication unit 204, a second communication unit 205, a display unit 206, an operation unit 207, a microphone 208, a speaker 209, and a storage unit 210) via the bus.

[0023] The first communication unit 204 is a wireless communication interface connectable to the wireless network Nb and the wireless network Nc. The first communication unit 204 communicates with external devices such as the store server 4 and the smart white cane 3 via the wireless networks Nb and Nc. In this embodiment, the mobile terminal 2 executes processes such as obtaining product images and product information (to be described later) via the wireless networks Nb and Nc or ​​the wireless access point 5, notifying the user, accepting the selection result of the payment method, and registering the product to be paid for, using the first communication unit 204.

[0024] The second communication unit 205 is a communication interface that connects to a mobile communication system such as 4G or 5G. The second communication unit 205 connects to the Internet via the mobile communication system.

[0025] The display unit 206 is an example of a display device provided in the mobile terminal 2, and is, for example, an LCD (Liquid Crystal Display) or a touch panel display.

[0026] The operation unit 207 has various operation buttons, and outputs key information according to an operation by an operator to the control unit 20. The operation unit 207 includes a touch panel provided on the display surface of the display unit 206.

[0027] The microphone 208 is an example of a voice input device that receives voice input from the user. The microphone 208 is an example of a voice input unit in this embodiment.

[0028] The speaker 209 is an example of an audio output device, and outputs audio under the control of the control unit 20.

[0029] The storage unit 210 is a non-volatile storage device that is configured with, for example, an SSD (Solid State Drive) or a flash memory, and maintains the stored contents even when the power is cut off. The storage unit 210 stores various programs executed by the CPU 201 and various setting information related to the operation of the mobile terminal 2. For example, the storage unit 210 stores a program 211 such as an application program for executing product registration processing and payment processing on the mobile terminal 2, which is provided from the store server 4 or a cloud server existing on the Internet. In this embodiment, the application program for executing product registration processing, payment processing, etc. on the mobile terminal 2 is called a POS (Point of Sale) application.

[0030] The hardware configuration of the mobile terminal 2 is not limited to the example shown in Fig. 2. For example, the mobile terminal 2 may further include various functions such as a camera and a motor for a vibration function.

[0031] Next, the software configuration of the mobile terminal 2 will be described. FIG. 3 is a diagram showing an example of the software configuration of the mobile terminal 2 according to this embodiment. Specifically, FIG. 3 illustrates various functions realized by a POS application installed in the mobile terminal 2. The program 211 executed by the mobile terminal 2 of this embodiment has a modular configuration including a shopping list generating unit 21, a route generating unit 22, a matching processing unit 23, a registration processing unit 24, and a payment processing unit 25 shown in FIG. 3. The CPU 201 reads the program 211 from the storage unit 210 and loads the above units (shopping list generating unit 21, route generating unit 22, matching processing unit 23, registration processing unit 24, and payment processing unit 25) onto the RAM 203. As a result, the above units are generated on the RAM 203. Note that the functions of the mobile terminal 2 are not limited to those shown in FIG. 3.

[0032] The shopping list generating unit 21 generates a shopping list based on the name of a product desired by the user, which is voice-input from the microphone 208. The shopping list generating unit 21 is an example of a shopping list generating means in this embodiment.

[0033] The shopping list is a list of product names or codes of products that the user wishes to purchase and that are in stock at the store.

[0034] More specifically, the shopping list generating unit 21 queries the store server 4 for inventory information using the user's desired product name, which is voice-input from the microphone 208. If the store has an inventory of a product matching the user's desired product name, the shopping list generating unit 21 registers the product name in the shopping list. The shopping list generating unit 21 stores the generated shopping list, for example, in the storage unit 210. The shopping list generating unit 21 may create a list of product names, or may obtain product codes corresponding to the product names from the store server 4 and create a list. The product codes may be, for example, JAN (Japanese Article Number) codes. In this embodiment, the shopping list is a list of product names. Hereinafter, products corresponding to the product names included in the shopping list are also referred to as products included in the shopping list.

[0035] Furthermore, if the store does not have in stock a product that matches the product name desired by the user, the shopping list generation unit 21 outputs a voice message recommending online shopping via the speaker 209. Note that the shopping list generation unit 21 can not only generate a shopping list before the user enters the store and starts shopping, but can also add product names to the shopping list and delete product names from the shopping list while the user is shopping in the store.

[0036] Based on the shopping list generated by the shopping list generation unit 21 and map information of the store, the route generation unit 22 generates a route that passes in front of the product shelves on which products corresponding to the product information included in the shopping list are placed. More specifically, the route generation unit 22 acquires store information including map information indicating the positions of the product shelves on which the products are placed in the store from the store server 4 via the first communication unit 204. Then, the route generation unit 22 generates a shopping route that runs from the entrance of the store to the exit of the store, passing through the shelves on which the products corresponding to the product names included in the shopping list are placed. The route generation unit 22 is an example of a map information acquisition means in this embodiment.

[0037] The shopping route is a route within the store that goes from the store entrance to the store exit via shelves on which products corresponding to the product names included in the shopping list are placed. If one product is placed in multiple sales areas within the store, the shopping route only needs to pass through at least one of the multiple sales areas.

[0038] In addition, when a product name is added to the shopping list, the route generating unit 22 changes the shopping route so that the shopping route passes in front of the product shelf on which the product corresponding to the added product name is placed. Furthermore, when a product name that is no longer needed is deleted from the shopping list, the route generating unit 22 changes the shopping route so that the shopping route does not pass in front of the product shelf on which the product corresponding to the deleted product name is placed. The route generating unit 22 is an example of a route generating means in this embodiment.

[0039] The matching processor 23 matches products included in the shopping list with products identified from images of the products (hereinafter also referred to as product images). More specifically, when the matching processor 23 acquires a product image captured by the camera of the smart white cane 3 via the first communication unit 204, the matching processor 23 transmits the product image to the store server 4 via the first communication unit 204. Then, the matching processor 23 acquires product information from the store server 4 that can identify the product corresponding to the product image.

[0040] The product image is an image of a product in a store, or an image of a symbol (such as a barcode or QR code (registered trademark)) attached to the product, and in this embodiment is a still image of the exterior of the product captured from at least one direction. The product image is captured by the camera 309 of the smart white cane 3. Note that the form of the product image is not limited to this, and it may be a video or a set of images of the product captured from multiple directions.

[0041] The product information is, for example, a product name and / or a product code. In this embodiment, the product name will be described as an example of the product information. If the product name acquired from the store server 4 is included in the shopping list, the matching processing unit 23 determines that the product corresponds to the shopping list. Also, if the product name acquired from the store server 4 is not included in the shopping list, the matching processing unit 23 determines that the product does not correspond to the shopping list. In this case, the matching processing unit 23 requests the store server 4 to notify the store clerk. By the store server 4 notifying the store clerk, if the user mistakenly picks up a product that is not on the shopping list, the store clerk who receives the notification can go to the user and handle the return. The matching processing unit 23 is an example of a product image acquisition means, a transmission means, and a product information acquisition means in this embodiment.

[0042] When product information corresponding to a product image is included in the shopping list, the registration processing unit 24 registers the product corresponding to the product information as a payment target. More specifically, when a product name acquired from the store server is included in the shopping list, the registration processing unit 24 registers the product corresponding to the product name as a payment target. Also, in this embodiment, when a product name acquired from the store server is not included in the shopping list, the registration processing unit 24 does not register the product corresponding to the product name as a payment target. The registration processing unit 24 is an example of a registration means in this embodiment. Note that, in this embodiment, the functions of the matching processing unit 23 and the registration processing unit 24 have been described separately, but the function of the matching processing unit 23 may be the function of the registration processing unit 24.

[0043] Furthermore, the registration processing unit 24 notifies the user when the product information corresponding to the product image is not included in the shopping list. More specifically, when the product name acquired from the store server is not included in the shopping list, the registration processing unit 24 notifies the user by notifying the smart white cane 3 via the first communication unit 204 that the product does not correspond to the product name in the shopping list. For this reason, the registration processing unit 24 is an example of a notification means in this embodiment.

[0044] The payment processing unit 25 cooperates with the store server 4 to perform payment processing for products registered as payment targets by the registration processing unit 24. The payment processing is started when the user issues an instruction to start payment by voice input to the smart white cane 3 or the mobile terminal 2, or when the user leaves the store.

[0045] When the user instructs the start of payment, the payment processing unit 25 accepts the selection of the payment method from the user's operation of the smart white cane 3. For example, the number of times the smart white cane 3 is struck on the floor may be associated with the payment method in advance and registered in the storage unit 210. As a result, after the user instructs the start of payment, within a predetermined time, the payment method is designated by the user's action, such as a single strike of the smart white cane 3 for credit card payment, a double strike for electronic money payment, a triple strike for code payment, and a quadruple strike for virtual currency payment.

[0046] For visually impaired people, it is not easy to distinguish between types of cash (types of banknotes and coins) and cash payment is inconvenient, so the payment method using the payment processing unit 25 is basically cashless payment. However, if a caregiver accompanies the visually impaired person, the caregiver can have an accounting machine (not shown) read the barcode displayed on the mobile terminal 2 and make a cash payment using the change dispenser provided on the accounting machine. The accounting machine is an example of a product sales data processing device in this disclosure.

[0047] In addition, when the user leaves the store without making payment for the product registered as the payment target in the store, the payment processing unit 25 makes payment for the product registered as the payment target by a default method. The default method is, for example, the payment method registered by the user when setting up the POS application. The default payment method is stored in the storage unit 210. When the first communication unit 204 of the mobile terminal 2 receives a store exit signal from the wireless access point 5, the payment processing unit 25 notifies the user of the start of the payment and makes the payment by the default method. The store exit signal is transmitted from the wireless access point 5 installed near the exit of the store, and the first communication unit 204 of the mobile terminal 2 receives the store exit signal from the wireless access point 5 when the mobile terminal 2 enters the communication area of ​​the wireless access point 5 installed near the exit of the store. The notification of the start of the payment is made by, for example, a voice output via the smart white cane 3, but may also be a voice output from the speaker 209 of the mobile terminal 2. The payment processing unit 25 is an example of a selection receiving means and a payment means in this embodiment.

[0048] Next, there will be described a hardware configuration of the smart white cane 3. Fig. 4 is a diagram showing an example of a hardware configuration of the smart white cane 3 according to this embodiment.

[0049] As shown in FIG. 4, the smart white cane 3 includes a processor such as a CPU 301, a ROM 302, a RAM 303, a communication unit 304, a motor 305, an acceleration sensor 306, a gyroscope 307, an ultrasonic sensor 308, a camera 309, a speaker 310, a microphone 311, and a memory unit 312.

[0050] The CPU 301, ROM 302, and RAM 303 are connected via a bus to constitute a computer-configured control unit 30. In the control unit 30, the CPU 301 operates according to a program stored in the ROM 302 or the storage unit 312 and loaded in the RAM 303, thereby executing various processes. The control unit 30 is connected to each unit (a communication unit 304, a motor 305, an acceleration sensor 306, a gyroscope 307, an ultrasonic sensor 308, a camera 309, a speaker 310, a microphone 311, and a storage unit 312) via the bus. The control unit 30 may be configured as, for example, a single board computer.

[0051] The communication unit 304 is a wireless communication interface connectable to the wireless network Nc. The communication unit 304 communicates with the mobile terminal 2 via the wireless network Nc to transmit and receive various information. For example, the communication unit 304 is an example of an acquisition means that acquires a shopping route in a store from the mobile terminal 2. The communication unit 304 is also an example of a transmission means that transmits product images captured by the camera 309 and the results of the user's selection of a payment method to the mobile terminal 2.

[0052] The motor 305 is a driving device used for the vibrator function, and vibrates the smart white cane 3 under the control of the control unit 30.

[0053] The acceleration sensor 306 measures the acceleration caused by the movement of the smart white cane 3. Furthermore, the gyroscope 307 detects the rotation and change in orientation of the smart white cane 3 by measuring the angular acceleration of the smart white cane 3. The acceleration sensor 306 and the gyroscope 307 are examples of sensors capable of detecting the movement of the user poking the smart white cane 3 into the floor surface in this embodiment.

[0054] The ultrasonic sensor 308 measures the distance to surrounding obstacles by transmitting and receiving ultrasonic waves.

[0055] The camera 309 can capture images of surrounding objects by user operation, and is, for example, a digital camera having an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary MOS). The camera 309 is an example of an image capture means capable of capturing images of products in a store.

[0056] The speaker 310 is an example of an audio output device, and outputs audio under the control of the control unit 30.

[0057] The microphone 311 is an example of a voice input device, and accepts voice input from the user.

[0058] The storage unit 312 is a non-volatile storage device configured, for example, by an SSD, a flash memory, etc. The storage unit 312 stores various programs executed by the CPU 301 and various setting information related to the operation of the smart white cane 3. For example, the storage unit 312 stores a program 313 that cooperates with the portable terminal 2 to execute a process for supporting the user in shopping.

[0059] The hardware configuration of the smart white cane 3 is not limited to the example shown in Fig. 4. For example, the smart white cane 3 may further include a module for reading RFID (Radio Frequency Identification), a barcode reader, or various sensors.

[0060] Next, the software configuration of the smart white cane 3 will be described. FIG. 5 is a diagram showing an example of the software configuration of the smart white cane 3 according to this embodiment. The program 313 executed by the smart white cane 3 of this embodiment has a modular configuration including a voice input reception unit 31, a guidance unit 32, a product image transmission unit 33, a selection operation reception unit 34, and a notification unit 35, as shown in FIG. 5. The CPU 301 reads out the program 313 from the storage unit 312, and loads the above-mentioned units (the voice input reception unit 31, the guidance unit 32, the product image transmission unit 33, the selection operation reception unit 34, and the notification unit 35) onto the RAM 303. As a result, the above-mentioned units are generated on the RAM 303. Note that the functions of the smart white cane 3 are not limited to those shown in FIG. 5.

[0061] The voice input receiving unit 31 receives a voice input from the user via a microphone 311 .

[0062] The guidance unit 32 guides the user to a shelf on which the products included in the shopping list are placed, based on the shopping route acquired from the mobile terminal 2. The guidance unit 32 may, for example, control the speaker 310 to output a voice message to inform the user of the traveling direction, or may control the motor 305 to inform the user of the traveling direction by vibration. The guidance unit 32 is an example of the guidance means of the smart white cane 3 in this embodiment.

[0063] The guidance unit 32 may acquire the user's current location in the store from the mobile terminal 2 and change the guidance content according to the user's movement status on the shopping route. The user's current location in the store is identified by, for example, communication between the wireless access point 5 and the mobile terminal 2.

[0064] Furthermore, the shelves in the store may be provided with symbols such as QR codes (registered trademark) or bar codes that can identify each shelf. In this case, the guidance unit 32 may identify the user's current location from the QR codes (registered trademark) or the like on the shelves in the store photographed by the camera 309 of the smart white cane 3, and guide the user in the direction to travel based on the identified location.

[0065] The product image transmission unit 33 transmits the product image captured by the camera 309 to the mobile terminal 2 via the communication unit 304. The product image transmission unit 33 is an example of the image transmission means of the smart white cane 3 in this embodiment. The communication unit 304 may also be an example of the image transmission means of the smart white cane 3.

[0066] The selection operation acceptance unit 34 accepts an operation of selecting a payment method by the user. More specifically, when the acceleration sensor 306 and the gyroscope 307 detect a motion of poking the floor surface with the smart white cane 3 within a predetermined time after an instruction to start payment is given, the selection operation acceptance unit 34 accepts the selection of a payment method according to the number of times of the poking motion. The selection operation acceptance unit 34 transmits the accepted payment method to the mobile terminal 2 via the communication unit 304. Note that the user's motion and the payment method may be associated on the mobile terminal 2 side or on the smart white cane 3 side. For example, the selection operation acceptance unit 34 may transmit the number of poking motions to the mobile terminal 2, or may transmit the payment method corresponding to the number of poking motions to the mobile terminal 2.

[0067] The notification unit 35 notifies the user by outputting a voice message by controlling the speaker 310. The notification unit 35 and the speaker 310 are an example of the notification means of the smart white cane 3 in this embodiment.

[0068] Next, a description will be given of the hardware configuration of the store server 4. Fig. 6 is a diagram showing an example of the hardware configuration of the store server 4 according to this embodiment.

[0069] As shown in FIG. 6, the store server 4 includes a processor such as a CPU 401, a ROM 402, a RAM 403, a communication unit 404, a display unit 405, an operation unit 406, and a storage unit 407.

[0070] The CPU 401, ROM 402, and RAM 403 are connected via a bus to constitute a computer-configured control unit 40. In the control unit 40, the CPU 401 operates according to a program stored in the ROM 402 or storage unit 407 and loaded into the RAM 403, thereby executing various processes. The control unit 40 is connected to each unit (a communication unit 404, a display unit 405, an operation unit 406, and a storage unit 407) via the bus.

[0071] The communication unit 404 is a wireless communication interface that can be connected to the network Na. The communication unit 404 communicates with the mobile terminal 2 via the network Na, the wireless access point 5, and the wireless network Nb, and transmits and receives various information.

[0072] The display unit 405 is an example of a display device, such as an LCD or a touch panel display, and displays various information according to the control of the control unit 40.

[0073] The operation unit 406 is an operation device such as a keyboard or a mouse. The display unit 405 and the operation unit 406 may be configured as a touch panel display. The display unit 405 and the operation unit 406 may be devices connected to the store server 4 via a network Na or the like.

[0074] The storage unit 407 is a non-volatile storage device constituted by, for example, an SSD, a flash memory, or the like. The storage unit 407 stores various programs executed by the CPU 401 and various setting information related to the operation of the store server 4. For example, the storage unit 407 stores a program 408 that executes product registration processing and payment processing in cooperation with the mobile terminal 2. The storage unit 407 also stores store information 409, inventory information 410, a product image database (DB) 411, and a product information database 412.

[0075] The store information 409 includes at least map information indicating the positions of the shelves on which the products are placed within the store.

[0076] Inventory information 410 is information about the inventory of products in a store. For example, inventory information 410 is information in which product names or product codes are associated with the number of products in stock in a store. Note that inventory information 410 may include information about the inventory of products in not only the store but also the number of products in stock in online shops and in nearby stores.

[0077] Fig. 7 is a diagram showing an example of the configuration of the product image database 411 according to this embodiment. As shown in Fig. 7, the product image database 411 is a database in which product images, product names, and product codes are associated with each other.

[0078] Fig. 8 is a diagram showing an example of the configuration of product information database 412 according to this embodiment. As shown in Fig. 8, product information database 412 is a database in which product codes, product names, and unit prices are associated with each other. Furthermore, product information database 412 may further include information related to products.

[0079] The configurations of the product image database 411 and the product information database 412 are not limited to the examples shown in Figures 7 and 8, and for example, only either the product name or the product code may be registered.

[0080] Next, the software configuration of the store server 4 will be described. FIG. 9 is a diagram showing an example of the software configuration of the store server 4 according to this embodiment. The program 408 executed by the store server 4 according to this embodiment has a modular configuration including a product image matching unit 41, a registration processing unit 42, a payment processing unit 43, a store clerk calling unit 44, a store information providing unit 45, and an inventory information providing unit 46 shown in FIG. 9. The CPU 401 reads the program 408 from the storage unit 407 and loads the above units (the product image matching unit 41, the registration processing unit 42, the payment processing unit 43, the store clerk calling unit 44, the store information providing unit 45, and the inventory information providing unit 46) onto the RAM 403. As a result, the above units are generated onto the RAM 403. Note that the functions of the store server 4 are not limited to those shown in FIG. 9.

[0081] The product image matching unit 41 matches the product image acquired from the mobile terminal 2 via the communication unit 404 with the product image registered in the product image database 411. For example, the product image matching unit 41 may calculate the feature amount of the product (shape of the product, representative color of the product, etc.) extracted from the product image or calculate the similarity of the product image and match the product based on the feature amount or similarity, or may match the product based on the product code by decoding the symbol extracted from the product image into a product code. When the product image matching unit 41 specifies a product image that matches the product image acquired from the mobile terminal 2 from the product image database 411, it transmits the product name associated with the product image to the mobile terminal 2 via the communication unit 404. The product image matching unit 41 may transmit the product code instead of the product name, or may transmit both the product name and the product code. The technology for matching the product image is not particularly limited, and a known image processing technology may be adopted.

[0082] The registration processing unit 42 receives an instruction to register a product for which payment is to be made from the mobile terminal 2 and registers the product. For example, the registration processing unit 42 searches for a unit price corresponding to the product for which payment is to be made from the product information database 412, and stores the product name and unit price of the product for which payment is to be made in association with each other in the storage unit 407. The registration processing unit 42 may store the product code instead of the product name in association with the unit price, or may store both the product name and product code in association with the unit price.

[0083] The payment processing unit 43 executes the payment processing for the products registered as payment targets upon receiving a notification of the start of the payment processing from the mobile terminal 2. The payment processing unit 43 pays for the products using either the payment method selected by the user from the mobile terminal 2 or the default payment method adopted when the user leaves the store.

[0084] The store clerk calling unit 44 notifies the store clerk of the call when a call to a store clerk is requested from the mobile terminal 2. The method of calling is not particularly limited, but for example, the store clerk calling unit 44 may send a signal notifying the store clerk of the user's location in the store to a terminal carried by the store clerk and instruct the store clerk to go to the user.

[0085] When the shop information providing unit 45 receives an inquiry about shop information from the mobile terminal 2 , the shop information providing unit 45 reads out the shop information 409 from the storage unit 407 and transmits it to the mobile terminal 2 via the communication unit 404 .

[0086] When the inventory information providing unit 46 receives an inquiry about inventory information of a product from the mobile terminal 2, it identifies the inventory quantity of the product from the inventory information 410 in the storage unit 407 and transmits it to the mobile terminal 2 via the communication unit 404.

[0087] Next, a flow of processes executed in the product sales data processing system 1 configured as above will be described.

[0088] 10-1 and 10-2 are sequence diagrams showing an example of the flow of processing executed in the product sales data processing system 1 according to this embodiment.

[0089] When a user carrying a mobile terminal 2 and a smart white cane 3 enters a store, the first communication unit 204 of the mobile terminal 2 acquires a store entry signal transmitted from a wireless access point 5 (S1). In this case, a POS application of the mobile terminal 2 is started (S2).

[0090] Then, the route generation unit 22 of the mobile terminal 2 inquires of the store information including map information indicating the position of the product shelf on which the product is placed in the store from the store server 4 via the first communication unit 204 (S3).

[0091] Upon receiving the inquiry from the mobile terminal 2, the store information providing unit 45 of the store server 4 reads out the store information 409 from the storage unit 407 and transmits it to the mobile terminal 2 via the communication unit 404 (S4).

[0092] Here, the user who has entered the store inputs the name of the product they wish to purchase at the store by voice into the mobile terminal 2. The microphone 208 of the mobile terminal 2 accepts the voice input (S5). Note that, although in FIG. 10-1 the mobile terminal 2 accepts the user's voice input, the microphone 311 of the smart white cane 3 may accept the voice input, and the user's voice input may be transmitted from the communication unit 304 of the smart white cane 3 to the mobile terminal 2, and accepted by the first communication unit 204 of the mobile terminal 2 (S5).

[0093] The shopping list generating unit 21 of the mobile terminal 2 inquires of the store server 4 about stock information using the name of the product desired by the user, which is input by voice through the microphone 208 (S6).

[0094] When the inventory information providing unit 46 of the store server 4 receives an inquiry about inventory information of a product from the mobile terminal 2, the inventory information providing unit 46 transmits the inventory information of the product to the mobile terminal 2 via the communication unit 404 (S7).

[0095] If the store does not have a product in stock that matches the product name desired by the user, the shopping list generating unit 21 of the mobile terminal 2 outputs audio guidance recommending online shopping via the speaker 209 (S8). Note that, although the speaker 209 of the mobile terminal 2 outputs the online shopping guidance in FIG. 10-1, the speaker 310 of the smart white cane 3 may output the guidance. Also, the online shopping guidance is not essential, and the mobile terminal 2 or smart white cane 3 may simply notify the user that the store does not have the product in stock.

[0096] Next, if one or more of the voice-input product names are in stock, the shopping list generation unit 21 of the mobile terminal 2 generates a shopping list of products that are in stock at the store (S9).

[0097] Then, the route generation unit 22 of the mobile terminal 2 generates a shopping route within the store based on the shopping list and the store information acquired from the store server 4 (S10).

[0098] The route generation unit 22 of the mobile terminal 2 transmits the generated shopping route to the smart white cane 3 via the first communication unit 204 (S11).

[0099] The guidance unit 32 of the smart white cane 3 then controls the speaker 310 or the vibration motor 305 to start route guidance based on the shopping route (S12). The guidance unit 32 of the smart white cane 3 guides the user to the shelf on which the products registered in the shopping list are placed. The guidance unit 32 may also output information indicating the location of the products registered in the shopping list, such as the shelf level on which the products are located, as a voice message from the speaker 310.

[0100] Then, when the user arrives at a product shelf guided by the smart white cane 3, he or she picks up the product and photographs the product with the camera 309 of the smart white cane 3. When the product is photographed by the camera 309 of the smart white cane 3 (S13), the product image transmission unit 33 of the smart white cane 3 transmits the product image to the mobile terminal 2 via the communication unit 304 (S14). The user may place the photographed product, for example, in a basket for storing products.

[0101] When shopping, the user walks while pushing the smart white cane 3, and may also hold a basket to put products in. This may make it difficult to photograph products with the mobile terminal 2 because both hands are full, but in this embodiment, the camera 309 of the smart white cane 3 can take product images, so the user can take photos of products without having to take the mobile terminal 2 out of their pocket or bag each time.

[0102] When the matching processing unit 23 of the mobile terminal 2 acquires a product image captured by the camera of the smart white cane 3 via the first communication unit 204, it transmits the product image via the first communication unit 204 to the store server 4 and inquires about the product name (S15).

[0103] Then, the commodity image matching unit 41 of the store server 4 searches the commodity image database 411 with the commodity image acquired from the mobile terminal 2 via the communication unit 404 (S16).

[0104] When the product image matching unit 41 identifies a product image from the product image database 411 that matches the product image acquired from the mobile terminal 2, the product image matching unit 41 transmits the product name associated with the product image to the mobile terminal 2 via the communication unit 404 (S17).

[0105] The verification processing unit 23 of the mobile terminal 2 verifies the product name acquired from the store server 4 with the product names included in the shopping list (S18).

[0106] If the product name obtained from the store server 4 corresponds to a product name in the shopping list, the registration processing unit 24 instructs the store server 4 to register the product name as a payment target (S19). The registration processing unit 42 of the store server 4 receives an instruction to register the product to be paid for from the mobile terminal 2 and executes a registration process to register the product as a payment target (S20).

[0107] When the registration processing unit 42 of the store server 4 registers the product specified by the mobile terminal 2 as a payment target, the registration processing unit 42 transmits a registration completion notification to the mobile terminal 2 via the communication unit 404 (S21).

[0108] Then, when the verification processing unit 23 of the portable terminal 2 receives a notification of the completion of registration from the store server 4, it transmits a notification of the completion of registration and the registered product name to the smart white cane 3 via the first communication unit 204 (S22).

[0109] The notification unit 35 of the smart white cane 3 controls the speaker 310 to output a voice message notifying the user of the name of the product registered as a payment target and that the product has been registered as a payment target (S23). The content of the voice message is, for example, "This is xx (product name). This product will be registered as a purchase target," but the content is not limited to this. The voice message allows the visually impaired user to recognize that the product photographed in S13 matches the product input by voice in S5, and that the product has been registered as a payment target.

[0110] On the other hand, if the product name obtained from the store server 4 does not correspond to the product name on the shopping list, the registration processing unit 24 notifies the smart white cane 3 via the first communication unit 204 that the product does not correspond to the product name on the shopping list (S24).

[0111] Then, the notification unit 35 of the smart white cane 3 controls the speaker 310 to output the product name and a voice message informing the user that the product is not on the shopping list (S25). The content of the voice message is, for example, "This is yy (product name). It is not registered on the shopping list," but the content is not limited to this.

[0112] Moreover, if the product name acquired from the store server 4 does not correspond to the product name in the shopping list, the verification processing unit 23 of the mobile terminal 2 requests the store server 4 to notify the store clerk (S26).

[0113] When a call for a store clerk is requested from the mobile terminal 2, the store clerk call unit 44 of the store server 4 notifies the store clerk of the call (S27). For example, the store clerk call unit 44 may notify the store clerk of a message to assist the user in finding the correct item because the user picked up the wrong item. In addition, the store clerk call unit 44 may notify the store clerk of a message to instruct the user to confirm that the item picked up by the user in error is returned to the shelf in order to reduce the occurrence of incorrect accounting.

[0114] Also, during route guidance using the smart white cane 3, the user can add a desired product to the shopping list. For example, when the user voice-inputs the name of an additional product into the microphone 208 of the mobile terminal 2 during route guidance (S28), the shopping list generating unit 21 of the mobile terminal 2 inquires of the store server 4 about stock information using the product name voice-input from the microphone 208 (S29). The process from the stock inquiry in S29 to the process of online shopping guidance in S31 is similar to S6 to S8 described above. Note that, although it has been described in FIG. 10-2 that the user voice-inputs the product to be added to the shopping list into the mobile terminal 2, the user may voice-input the product to be added into the smart white cane 3, and the voice input may be transmitted from the communication unit 304 of the smart white cane 3 to the mobile terminal 2 and accepted by the first communication unit 204 of the mobile terminal 2 (S28).

[0115] Alternatively, the user may delete unnecessary products from the shopping list during route guidance using the smart white cane 3. For example, during route guidance, the user voice-inputs the name of the product to be deleted into the microphone 208 of the mobile terminal 2 or the microphone 311 of the smart white cane 3 (S28).

[0116] Then, the route generation unit 22 of the mobile terminal 2 updates the shopping list by adding the products that are in stock among the added products (S32). Alternatively, the route generation unit 22 of the mobile terminal 2 updates the shopping list based on the deleted products (S32).

[0117] The route generation unit 22 of the mobile terminal 2 regenerates a shopping route based on the updated shopping list (S33).

[0118] Then, the route generation unit 22 of the mobile terminal 2 transmits the regenerated shopping route to the smart white cane 3 via the first communication unit 204 (S34).

[0119] Then, the guidance unit 32 of the smart white cane 3 starts route guidance based on the regenerated shopping route (S35).

[0120] The processes of S13 to S35 are repeatedly executed after the route guidance starts in S12, until the user leaves the store or a payment instruction is given to start the payment process. When the payment process starts, the route guidance using the smart white cane 3 ends.

[0121] Here, we will explain the payment process when the user tries to leave the store without paying for the product. When the user approaches the store exit, the first communication unit 204 of the mobile terminal 2 carried by the user receives a store exit signal from the wireless access point 5 located near the exit (S36).

[0122] When the first communication unit 204 of the portable terminal 2 receives the store exit signal from the wireless access point 5, the payment processing unit 25 of the portable terminal 2 notifies the smart white cane 3 to start payment (S37).

[0123] In this case, the guiding unit 32 of the smart white cane 3 ends the route guidance. Then, the notification unit 35 of the smart white cane 3 controls the speaker 310 to output a voice message notifying the user that payment will begin (S38).

[0124] The payment processing unit 25 of the mobile terminal 2 cooperates with the store server 4 to start the payment process using the default payment method registered in advance by the user (S39).

[0125] The payment processing unit 43 of the store server 4 receives a notification of the start of payment processing from the mobile terminal 2 and executes the payment processing by a default payment method (S40). The payment processing unit 43 transmits a payment result to the mobile terminal 2 (S41).

[0126] Next, a case where payment is started in response to an instruction to start payment from the user will be described. When the user inputs a voice instruction to start payment into the microphone 311 of the smart white cane 3 during route guidance with the smart white cane 3 (S42), the guidance unit 32 of the smart white cane 3 ends the route guidance.

[0127] Then, the selection operation receiving unit 34 of the smart white cane 3 receives an operation by the user to select a payment method (S43). For example, the selection operation receiving unit 34 receives an action of the user poking the smart white cane 3 once as an operation to select credit card payment, an action of the user poking twice as an operation to select electronic money payment, an action of the user poking three times as an operation to select code payment, and an action of the user poking four times as an operation to select virtual currency payment. Note that the selection method is not limited to the action of the user poking the smart white cane 3, and may be, for example, a voice input to the smart white cane 3.

[0128] Then, the selection operation receiving unit 34 of the smart white cane 3 transmits the payment method selected by the user to the mobile terminal 2 via the communication unit 304 (S44).

[0129] The payment processing unit 25 of the mobile terminal 2 cooperates with the store server 4 to start the payment process using the payment method selected by the user (S45).

[0130] The payment processing unit 43 of the store server 4 receives notification of the start of the payment process from the mobile terminal 2 and executes the payment process using the payment method selected by the user (S46). The payment processing unit 43 transmits the payment result to the mobile terminal 2 (S47). The processing of this sequence diagram then ends.

[0131] In this way, according to the mobile terminal 2 of this embodiment, a shopping list is generated based on the user's voice information, and if the shopping list contains product information corresponding to a product image acquired from the smart white cane 3 carried by the user, the product corresponding to the product information is registered as the payment target, thereby assisting a visually impaired user in selecting and registering products when shopping.

[0132] In addition, the mobile terminal 2 of this embodiment notifies the user when the product information corresponding to the product image acquired from the store server is not included in the shopping list. Therefore, according to the mobile terminal 2 of this embodiment, when a visually impaired user picks up the wrong product, the user can be promptly notified of the mistake. This reduces the occurrence of a situation where the user purchases a product that he or she does not want.

[0133] Furthermore, the mobile terminal 2 of this embodiment generates a shopping route based on the shopping list and store map information. When product information is added by the user's voice information while shopping, the mobile terminal 2 of this embodiment adds the product information to the shopping list and changes the shopping route so that the shopping route passes in front of the product shelf on which the product corresponding to the added product information is placed. Therefore, the mobile terminal 2 of this embodiment can support the user even if the user wants more products while shopping.

[0134] Furthermore, when product information is deleted by the user's voice information during shopping, the mobile terminal 2 of this embodiment deletes the product information from the shopping list and changes the shopping route so as not to pass in front of the product shelf on which the product corresponding to the deleted product information is placed. Therefore, the mobile terminal 2 of this embodiment can support the user even when the user encounters a product that he or she no longer needs while shopping.

[0135] Furthermore, the portable terminal 2 of this embodiment accepts the selection of a payment method from the user's operation of the smart white cane 3. Therefore, the portable terminal 2 of this embodiment makes it easy for the user to select a desired payment method.

[0136] In addition, when a user leaves a store without making payment for a product registered as a payment target in the store, the mobile terminal 2 of this embodiment pays for the product registered as a payment target by a default method. Therefore, according to the mobile terminal 2 of this embodiment, even if a user forgets to make a payment and tries to leave the store, it is possible to reduce the occurrence of missed payments. In addition, even if a malicious user tries to leave the store without making a payment, the payment is forcibly made by a default method, so shoplifting can be prevented.

[0137] Moreover, the product sales data processing system 1 of this embodiment includes a mobile terminal 2 and a smart white cane 3. The smart white cane 3 guides the user based on a shopping route generated by the mobile terminal 2, and transmits product images captured by the camera 309 of the smart white cane 3 to the mobile terminal 2. If a product image acquired from the smart white cane 3 is included in a shopping list, the mobile terminal 2 registers that product as a payment target. Therefore, the product sales data processing system 1 of this embodiment can assist visually impaired users in selecting and registering products when shopping.

[0138] In the above embodiment, the microphone 208 of the mobile terminal 2 is an example of a voice input means, but the smart white cane 3 may receive voice input of a product list by the user, the communication unit 304 of the smart white cane 3 may transmit the voice input to the mobile terminal 2, and the first communication unit 204 of the mobile terminal 2 may receive the voice input. In this case, the first communication unit 204, which is a wireless communication interface of the mobile terminal 2, becomes a voice input means that receives the voice-input product list from the smart white cane 3. When this configuration is adopted, the first communication unit 204 becomes an example of a voice input means. Also, either the microphone 208 or the first communication unit 204 of the mobile terminal 2 may be a first voice input means, and the other may be a second voice input means.

[0139] As described above, the mobile terminal 2 executes processes such as obtaining product images and product information, notifying the user, accepting the selection of the payment method, and registering the product to be paid for, via the wireless networks Nb, Nc or ​​the wireless access point 5, using the first communication unit 204. For this reason, the first communication unit 204 may be an example of a product image obtaining means, a product information obtaining means, a transmission means, a registration means, a notification means, a map obtaining means, and a selection accepting means in this embodiment.

[0140] In the above embodiment, a method for assisting a user in shopping using the respective functions of the mobile terminal 2 and the smart white cane 3 has been described, but the division of functions between the mobile terminal 2 and the smart white cane 3 is not limited to the above example. The smart white cane 3 may have the functions described as the functions of the mobile terminal 2 in the above embodiment, or the mobile terminal 2 may have the functions described as the functions of the smart white cane 3.

[0141] In the above embodiment, a configuration was adopted in which the product picked up by the user was identified by the product image captured by the camera 309 of the smart white cane 3, but a method other than the product image may also be adopted. For example, instead of the product image, a barcode, a QR code (registered trademark), an RFID tag, etc. read by the smart white cane 3 may be used.

[0142] In addition, in the above-described embodiment, the mobile terminal 2 communicates with both the smart white cane 3 and the store server 4 to send and receive information, but the smart white cane 3 and the store server 4 may also send and receive information directly via wireless communication.

[0143] In the above embodiment, an example is shown in which the mobile terminal 2 and the store server 4 notify a store clerk when the product corresponding to the product image is not included in the shopping list, but the response when the user picks up a product that is not included in the shopping list is not limited to this. For example, the user may use a shopping cart equipped with two baskets, one of which is for placing products to be paid for and the other is for placing products to be returned. When the user is notified by the smart white cane 3 that the photographed product is not included in the shopping list, the user places the product in the basket for return. A camera or the like provided in the cart recognizes which basket the user has placed the product in.

[0144] In the above embodiment, if the product corresponding to the product image does not belong to the shopping list, i.e., if the user picks up the wrong product, the store server 4 notifies the user to call a store clerk to request a return, but the method of return is not limited to this. For example, if the product corresponding to the product image does not belong to the shopping list, the route generation unit 22 of the mobile terminal 2 may generate a return route for returning the product to the store shelf. The return route is a route that passes in front of the shelf on which the product not belonging to the shopping list was placed. In addition, if a return occurs during shopping, the return route may be incorporated as part of the shopping route.

[0145] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0146] (Additional Note) (1) An acquisition means for acquiring a shopping route within a store based on a user's desired shopping list from a mobile terminal; a guidance means for guiding the user to a shelf on which products included in the shopping list are placed based on the shopping route; An imaging means capable of imaging products in the store; A transmission means for transmitting the product image captured by the imaging means to an external device (a mobile terminal or a store server); a notification means for notifying the user that the product depicted in the product image does not correspond to the shopping list when the external device notifies the user that the product does not correspond to the shopping list; A walking aid cane equipped with (2) when notified by the external device that the product depicted in the product image corresponds to the shopping list, the notification means notifies the user that the product corresponds to the shopping list and the name of the product. A walking aid cane as described in (1) above. (3) Further comprising a sensor capable of detecting a user's action of pushing the walking aid cane against a floor surface, the transmitting means transmits to the mobile terminal a payment method according to the number of times that the motion of hitting the floor surface with the walking cane is detected by the sensor within a predetermined time after a payment is instructed. A walking aid cane as described in (1) or (2) above. [Explanation of symbols]

[0147] 1. Product sales data processing system 2. Mobile devices 3. Smart White Cane 4. Store Server 5. Wireless Access Points 20 Control section 21 Shopping list generator 22 Route Generation Unit 23 Matching processing unit 24 Registration processing section 25 Payment Processing Unit 30 Control section 31 Voice input reception unit 32 Information Department 33 Product image transmission unit 34 Selection operation reception section 35 Notification Department 40 Control section 41 Product image matching unit 42 Registration processing section 43 Payment Processing Unit 44 Staff Call Department 45 Store Information Department 46 Inventory Information Department 204 First Communication Department 205 Second Communication Department 206 Display section 207 Operation section 208 Microphone 209 Speaker 210 Storage section 211 Program 304 Communications Department 305 Motor 306 Acceleration Sensor 307 Gyroscope 308 Ultrasonic Sensor 309 Camera 310 Speaker 311 Microphone 312 Storage section 313 Programs 404 Communications Department 405 Display section 406 Operation section 407 Storage section 408 Program 409 Store Information 410 Stock Information 411 Product Image Database 412 Product Information Database Na Network Nb,Nc wireless network [Prior art documents] [Patent documents]

[0148] [Patent Document 1] JP 2015-007846 A

Claims

1. A product sales data processing device configured as a mobile terminal, A voice input means for accepting a voice input; a shopping list generating means for generating a shopping list based on the voice input; A product image acquisition means for acquiring product images of products in a store from the walking aid cane; a registration means for registering a product corresponding to the product information as a payment target when the product information corresponding to the product image is included in the shopping list; A product sales data processing device comprising:

2. and a notification means for notifying a user when the product information corresponding to the product image is not included in the shopping list. The product sales data processing device according to claim 1.

3. A map acquisition means for acquiring map information indicating the positions of product shelves on which products are placed in the store; a route generating means for generating a shopping route that passes in front of a product shelf on which a product corresponding to the product information included in the shopping list is placed, based on the shopping list and the map information; the voice input means accepts addition or deletion of product information by voice; the shopping list generating means adds to or deletes from the shopping list the product information added or deleted by the voice input means; the route generating means, when product information is added to the shopping list, changes the shopping route so as to pass in front of a product shelf on which a product corresponding to the added product information is placed, or, when product information is deleted from the shopping list, changes the shopping route so as not to pass in front of a product shelf on which a product corresponding to the deleted product information is placed. The product sales data processing device according to claim 1.

4. The walking aid further includes a selection receiving means for receiving a selection of a payment method from a user through the operation of the walking aid cane. The product sales data processing device according to claim 1.

5. The method further includes a payment means for paying for the registered items as payment targets in a predetermined manner when the user leaves the store without making payment for the registered items as payment targets in the store. The product sales data processing device according to any one of claims 1 to 4.

6. On the computer, a voice input step of accepting a voice input; a shopping list generation step of generating a shopping list based on the voice input; a product image acquisition step of acquiring product images of products in a store from the walking aid cane; a registration step of registering a product corresponding to the product information as a payment target when the product information corresponding to the product image is included in the shopping list; A program for executing the above.

7. A product sales data processing device configured as a mobile terminal; A walking stick, The product sales data processing device includes: A voice input means for accepting a voice input; a shopping list generating means for generating a shopping list based on the voice input; an image acquisition means for acquiring product images of products in a store from the walking aid cane; a registration means for registering a product corresponding to the product information as a payment target when the product information corresponding to the product image is included in the shopping list; The walking aid cane is a guidance means for guiding a user to a shelf on which a product corresponding to the product information included in the shopping list is placed; An imaging means capable of imaging products in the store; and an image transmission means for transmitting the photographed product image to the product sales data processing device. Product sales data processing system.

Images