System

Abstract

To provide a system.SOLUTION: A system includes: image analysis means for recognizing the prices, item names, payment method, and date and time of purchase of the articles purchased by a user included in a receipt image photographed by photographing means provided in an eyeglass-type wearable device worn by the user by inputting a specific prompt text into a generative AI model; recording means for recording the prices, item names, payment method, and date and time of purchase of the articles recognized from the receipt image in a household account book including the respective items of the prices, item names, payment method, and date and time of purchase of the articles; a feeling engine which recognizes the feeling of the user from the image photographed by the photographing means; and means for recording the recognized feeling in the household account book.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional household accounting requires manual effort and time, and it is difficult to accurately record information such as the amount, price, and payment method of purchased items. [Means for solving the problem]

[0005] The system of the present invention includes an image analysis means that recognizes the amount, item name, payment method, and purchase date and time of an item purchased by a user from an image of a receipt captured by a camera provided on a glasses-type wearable device worn by the user by inputting a specific prompt sentence into a generative AI model; a recording means that records the amount, item name, payment method, and purchase date and time of the item recognized from the receipt image in a household ledger that includes items for each of the amount, item name, payment method, and purchase date and time; an emotion engine that recognizes the user's emotion from the image captured by the camera means; and a means for recording the recognized emotion in the household ledger. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The eyeglass-type wearable device of the present invention has a camera mounted on the frame that takes a picture of the item purchased by the user. The captured image is analyzed by a built-in image recognition system, and information such as the item's amount, price, and payment method is automatically extracted. For example, if a receipt is photographed, the necessary information can be extracted from the letters, numbers, barcodes, etc. on the receipt.

[0029] "Example 2"

[0030] The extracted information is automatically recorded in a household ledger by the built-in household ledger creation system. The household ledger consists of items such as the date, item name, amount, and payment method, and is saved so that the user can check it later. For example, if you purchase a book for 1,000 yen with cash on January 1, 2022, the information recorded in the household ledger will be "January 1, 2022, book, 1,000 yen, cash."

[0031] "Example 3"

[0032] Furthermore, the glasses-type wearable device of the present invention also has the function of automatically saving household accounting information to the cloud, allowing users to check their household accounting information anytime, anywhere. Furthermore, even if the device breaks down or is lost, the household accounting information saved in the cloud is safely protected.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[0036] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[0037] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0038] "Example 2"

[0039] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[0040] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[0041] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0042] Step 4: The recorded household accounting information is stored internally on the device for the user to review later.

[0043] "Example 3"

[0044] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[0045] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[0046] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0047] Step 4: The recorded household accounting information is automatically saved on the cloud, allowing users to check their household accounting information anytime, anywhere.

[0048] Example 1

[0049] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0050] Conventional glasses-type wearable devices require users to manually enter information about purchased items, which is time-consuming for the user. Furthermore, there is a lack of systems for accurately managing purchase information, making it difficult to automatically create a household ledger or save it in the cloud. This makes it difficult for users to efficiently manage their purchase history, making household management cumbersome.

[0051] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0052] In this invention, the server includes a photographing means, a storage means for storing images acquired by the photographing means, an analysis means for analyzing the images stored in the storage means, an extraction means for extracting letters and numbers using optical character recognition technology by the analysis means, an information analysis means for analyzing the amount, price, and payment method of the item from the information extracted by the extraction means, and a display means for displaying the information analyzed by the information analysis means. This allows the user to automatically obtain information on purchased items and create a household account book and store it in the cloud.

[0053] The "photography means" is a device such as a camera for taking pictures of items purchased by the user and receipts.

[0054] The "storage means" refers to a memory or storage device for storing image data acquired by the imaging means.

[0055] The "analysis means" refers to software or hardware for analyzing the image data stored in the storage means.

[0056] The "extraction means" is a function that allows the analysis means to extract characters and numbers from image data using optical character recognition technology.

[0057] The "information analysis means" is a function for analyzing information such as the amount, price, and payment method of the item from the characters and numbers extracted by the extraction means.

[0058] The "display means" is a display or monitor for displaying the information analyzed by the information analysis means to the user.

[0059] The "cloud" refers to remote servers and storage services for storing and managing data over the Internet.

[0060] MODE FOR CARRYING OUT THE INVENTION

[0061] This invention is a system that uses a glasses-type wearable device to capture images of items and receipts purchased by a user, analyzes the image data, and automatically extracts information such as the item amount, price, payment method, etc. Specific embodiments of this system are described below.

[0062] Hardware and Software Configuration

[0063] The glasses-type wearable device worn by the user includes the following hardware and software:

[0064] Photography method: High-resolution camera

[0065] Storage method: Built-in memory

[0066] Analysis method: Image recognition system (TENSORFLOW (registered trademark), OpenCV)

[0067] Extraction method: OCR technology (optical character recognition)

[0068] Information analysis tools: Data analysis algorithms

[0069] Display means: Display

[0070] System Operation

[0071] The user puts on the glasses-type wearable device and takes a picture of the purchased item or receipt. The captured image data is saved in the device's internal memory. The image recognition system then launches and analyzes the image data using TensorFlow and OpenCV. OCR technology is used to extract characters and numbers from the image data and analyze information such as the item's amount, price, and payment method. The analyzed information is then displayed on the screen.

[0072] Specific examples

[0073] For example, consider the case where a user goes shopping at a supermarket and takes a photo of their receipt. The user puts on a glasses-type wearable device and takes a photo of the receipt with the camera. The device saves the captured image data in its internal memory. Next, the device launches its image recognition system and analyzes the image data using TensorFlow and OpenCV. OCR technology is used to extract letters and numbers from the receipt, reading information such as "rice ball 150 yen" and "drink 350 yen." The device analyzes this information and calculates the total amount as "500 yen" and the payment method as "credit card." Finally, the device displays the analysis results on the glasses' display so that the user can confirm them.

[0074] Prompt Sentence Examples

[0075] An example of a prompt to input to a generative AI model is as follows:

[0076] The user puts on the glasses-type wearable device and takes a photo of the receipt for the items purchased at the supermarket. The device analyzes the image, extracts information such as the purchased items, amount, and payment method, and displays it on the display.

[0077] In this way, users can easily check information about the items they have purchased, and their household accounts are automatically created and saved to the cloud.

[0078] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0079] Step 1:

[0080] The user puts on the glasses-type wearable device and takes photos of purchased items and receipts.

[0081] Input: An image taken by the user using the camera.

[0082] Output: High resolution image data.

[0083] How it works: A user uses the camera on their glasses-type wearable device to take a picture of a receipt or purchase, which captures a high-resolution image.

[0084] Step 2:

[0085] The device stores the captured image data in its internal memory.

[0086] Input: High resolution image data captured.

[0087] Output: Image data stored in the internal memory.

[0088] Specific operation: The device stores the captured image data in its internal memory. The stored data is used for later analysis.

[0089] Step 3:

[0090] The device activates the image recognition system and analyzes the image data.

[0091] Input: Image data stored in the internal memory.

[0092] Output: Image data ready for analysis.

[0093] Specific operation: The device launches an image recognition system such as TensorFlow or OpenCV and reads image data from the built-in memory.

[0094] Step 4:

[0095] The device uses OCR technology to extract letters and numbers from the image data.

[0096] Input: Image data ready for analysis.

[0097] Output: Extracted character and number data.

[0098] How it works: The device uses OCR technology to extract letters and numbers from image data, for example, reading information such as "rice ball 150 yen" or "drink 350 yen" on a receipt.

[0099] Step 5:

[0100] The terminal analyzes the extracted letters and numbers to determine the item's amount, price, payment method, etc.

[0101] Input: Extracted character and number data.

[0102] Output: Information such as the parsed item amount, price, and payment method.

[0103] Specific operation: The device analyzes the extracted characters and numbers to determine the price of the item and the payment method. For example, "rice ball 150 yen" and "drink 350 yen" are added together to determine "500 yen."

[0104] Step 6:

[0105] The terminal displays the analysis results to the user.

[0106] Input: Information such as the parsed item amount, price, and payment method.

[0107] Output: The analysis results that are displayed to the user.

[0108] Specific operation: The device displays the analysis results on the glasses' display so that the user can confirm them. For example, it displays "Purchased items: rice ball, drink," "Total amount: 500 yen," and "Payment method: credit card."

[0109] (Application example 1)

[0110] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] While conventional glasses-type wearable devices can automatically recognize the amount and payment method of a user's purchases, they lack the functionality to display detailed product information in real time. Furthermore, there are limited ways to efficiently manage purchase history, which requires users to manually enter information.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0113] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a barcode recognition means for automatically recognizing a barcode from an image acquired by the camera means, and a display means for displaying detailed information about the item based on the information recognized by the barcode recognition means. This allows a user to check detailed information about purchased items in real time and efficiently manage their purchase history. The household account book creation means may be interpreted as a recording means for recording the amount, item name, payment method, and purchase date and time of the item recognized by the image analysis means in the household account book.

[0114] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and a display.

[0115] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[0116] "Image recognition means" refers to technology or equipment for analyzing and recognizing specific information from acquired images.

[0117] "Means for creating a household account book" refers to technology or equipment for automatically creating a household account book based on recognized information.

[0118] "Barcode recognition means" refers to technology or equipment for analyzing and recognizing barcodes from acquired images.

[0119] "Display means" refers to the technology or device used to visually present recognized information to the user.

[0120] The system for implementing this invention includes a camera, an image recognition unit, a household accounting book creation unit, a barcode recognition unit, and a display unit, all of which are provided in a glasses-type wearable device. The specific operation of each unit and the hardware and software used will be described below.

[0121] Camera Means

[0122] The camera means is a camera built into the eyeglass-type wearable device that takes pictures of items purchased by the user and receipts. The camera uses the latest image sensor technology to capture high-resolution images.

[0123] Image Recognition Method

[0124] The image recognition module uses OCR (Optical Character Recognition) technology to automatically recognize the amount, price, and payment method of an item from a captured image. Specifically, it uses the Pytesseract library to extract text from the image. Furthermore, it uses the Pyzbar library to recognize barcodes in the image.

[0125] How to create a household budget

[0126] The household account book creation means automatically records the user's purchase history based on the information recognized by the image recognition means and creates a household account book. The created household account book is automatically saved on the cloud. Cloud services used include Amazon Web Services (AWS (registered trademark)) and Google (registered trademark) Cloud Platform (GCP).

[0127] Barcode Recognition Method

[0128] The barcode recognition means automatically recognizes barcodes from images captured by the camera means and obtains detailed information about the item. The Pyzbar library is used to decode the barcodes.

[0129] Display means

[0130] The display of the glasses-type wearable device is used to visually present the recognized information to the user, and by using OLED or MicroLED technology for the display, high resolution and low power consumption are possible.

[0131] Specific examples

[0132] For example, when a user buys apples at a supermarket, they scan the barcode on the apples using the glasses-type wearable device. The price and payment method for the apples are then displayed on the screen. Also, by taking a photo of the receipt, the purchase history is automatically recorded in the household accounting app.

[0133] Example prompts to input to a generative AI model:

[0134] "I want to develop an application that uses smart glasses to display the price and payment methods of items purchased in a physical store in real time. Using Python, write a program to extract text and barcodes from images taken by the camera."

[0135] In this way, users can check detailed information about purchased items in real time and efficiently manage their purchase history.

[0136] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0137] Step 1:

[0138] The user puts on the glasses-type wearable device and takes a picture of the purchased item or receipt with the camera. The input is the image captured by the camera, and the output is high-resolution image data.

[0139] Step 2:

[0140] The terminal sends the acquired image data to the image recognition means, which uses the Pytesseract library to extract text from the image. The input is the image data, and the output is the extracted text data.

[0141] Step 3:

[0142] The terminal analyzes the text data acquired from the image recognition means and identifies the amount, price, and payment method of the item. The input is text data, and the output is information on the amount, price, and payment method of the item.

[0143] Step 4:

[0144] The terminal sends the acquired image data to the barcode recognition means, which uses the Pyzbar library to extract and decode the barcode from the image. The input is the image data, and the output is the barcode data.

[0145] Step 5:

[0146] The terminal acquires detailed information about the product based on the barcode data and sends it to the display means. The input is the barcode data, and the output is detailed information about the product.

[0147] Step 6:

[0148] The display means visually presents the acquired detailed information of the item to the user. The input is the detailed information of the item, and the output is the information displayed on the display.

[0149] Step 7:

[0150] The device automatically creates a household ledger based on information acquired from the image recognition system and stores it on the cloud. The input is the amount, price, and payment method of the item, and the output is the household ledger data stored on the cloud.

[0151] In this way, the user can check detailed information about the purchased items in real time, and can efficiently manage the purchase history.

[0152] Example 2

[0153] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, they lack the ability to store purchase information on the cloud, making it difficult to back up or share data. Furthermore, they lack support for users to efficiently analyze and understand the contents of their household accounting records.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0156] In this invention, the server includes a means for a user to input purchase information, a means for a terminal to transmit the input information to the server, a means for the server to pass the information to the household accounting system, a means for the household accounting system to analyze the information and record it in the household accounting, and a means for the user to check the household accounting. This allows users to easily create accurate household accounting and store it on the cloud, enabling data backup and sharing. In addition, by using a generative AI model to analyze the contents of the household accounting and generate prompt sentences for the user, the contents of the household accounting can be efficiently understood.

[0157] "Means for users to input purchase information" refers to a device or software that provides an interface for users to input information about the items they have purchased (e.g., date, name of item, amount, payment method, etc.).

[0158] The "means for the terminal to transmit input information to the server" refers to a device or software having a communication function for transmitting the purchase information input by the user to the server.

[0159] The "means for the server to pass information to the household account book creation system" refers to a device or software having a function for transferring the purchase information received by the server to the household account book creation system.

[0160] The "means by which the household accounting system analyzes information and records it in the household accounting book" refers to a device or software that analyzes the received purchase information and automatically records it in the household accounting book in an appropriate format.

[0161] The "means for the user to check the household account book" refers to a device or software that provides an interface for the user to view the contents of the recorded household account book.

[0162] "Means for automatically saving on the cloud" refers to a device or software that has the function of automatically saving information recorded by the household accounting system to cloud storage via the Internet.

[0163] A "means for generating prompt sentences using a generative AI model" is a device or software that uses an artificial intelligence model to analyze information recorded in a household account book and generate appropriate prompt sentences for the user.

[0164] MODE FOR CARRYING OUT THE INVENTION

[0165] This invention is a system that automatically creates and saves a household account book based on purchase information entered by a user. A specific embodiment of this system will be described below.

[0166] Hardware and software used

[0167] Hardware: Servers, devices (PCs, smartphones, etc.)

[0168] Software: Household accounting system, generative AI model

[0169] System configuration

[0170] 1. How users enter their purchase information

[0171] The user uses the terminal to input information about the purchased item, such as the date, the name of the item, the amount, and the payment method.

[0172] For example, if a user purchases a book for 1,000 yen in cash on January 1, 2022, they would enter this information into the input form.

[0173] 2. How the device sends input information to the server

[0174] The terminal sends the information entered by the user to the server. Specifically, it sends the information to the server using an HTTP request.

[0175] For example, the input information "January 1, 2022, book, 1,000 yen, cash" is sent to the server in JSON format.

[0176] 3. How the server passes information to the household accounting system

[0177] The server passes the received information to the household accounting system. Specifically, it calls the API in the server and passes the information to the household accounting system.

[0178] For example, the server passes the information "January 1, 2022, books, 1,000 yen, cash" to the household accounting system.

[0179] 4. How the household accounting system analyzes and records information in the household accounting book

[0180] The household accounting system analyzes the received information and automatically records it in the household accounting book, specifically by storing the information in a database and recording it in the household accounting book in the appropriate format.

[0181] For example, information such as "January 1, 2022, books, 1,000 yen, cash" is saved in a database and recorded in a household ledger.

[0182] 5. How users can check their household accounts

[0183] The user uses the terminal to check the recorded household account book. Specifically, the user views the recorded information through the interface of the household account book creation system.

[0184] For example, a user checks a household account book that contains the following information: "January 1, 2022, books, 1,000 yen, cash."

[0185] 6. Automatically save to the cloud

[0186] The household accounting system automatically saves recorded information to cloud storage, which allows data to be backed up and shared.

[0187] For example, recorded household accounting information is stored on the cloud and can be accessed by users from any device.

[0188] 7. How to generate prompts using a generative AI model

[0189] It uses a generative AI model to analyze the information recorded in the household ledger and generate appropriate prompts for the user.

[0190] For example, by inputting the following prompt sentence into a generative AI model, it can efficiently understand the contents of a household budget.

[0191] If a user purchases a book for 1,000 yen with cash on January 1, 2022, explain how this will be recorded in the household ledger.

[0192] In this way, users can effortlessly create accurate household accounts and store them on the cloud for data backup and sharing. Furthermore, by analyzing the contents of the household account book using a generative AI model and generating prompts for the user, the contents of the household account book can be efficiently understood.

[0193] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0194] Step 1:

[0195] The user enters purchase information.

[0196] Input: Information about the item purchased by the user (e.g., date, item name, amount, payment method)

[0197] Specific operation: The user enters purchase information into the input form on the device. For example, if a user purchases a book for 1,000 yen in cash on January 1, 2022, the user enters "January 1, 2022, book, 1,000 yen, cash."

[0198] Output: The entered purchase information is saved on the device.

[0199] Step 2:

[0200] The terminal sends the input information to the server.

[0201] Input: Purchase information entered by the user

[0202] Specific operation: The device uses an HTTP request to send the entered purchase information to the server. For example, the entered information "January 1, 2022, book, 1,000 yen, cash" is sent to the server in JSON format.

[0203] Output: The server receives the purchase information.

[0204] Step 3:

[0205] The server passes the information to the household accounting system.

[0206] Input: Purchase information sent from the device

[0207] Specific operation: The server calls the internal API to pass the received purchase information to the household accounting system. For example, the server passes the information "January 1, 2022, Books, 1,000 yen, Cash" to the household accounting system.

[0208] Output: The household accounting system receives the purchase information.

[0209] Step 4:

[0210] The household accounting system analyzes the information and records it in the household accounting book.

[0211] Input: Purchase information passed from the server

[0212] Specific operation: The household accounting system analyzes the received information and stores it in a database. For example, information such as "January 1, 2022, Books, 1,000 yen, Cash" is stored in the database and recorded in the household accounting book in the appropriate format.

[0213] Output: Purchase information recorded in the household ledger

[0214] Step 5:

[0215] The user checks the household account book.

[0216] Input: Purchase information recorded in the household ledger

[0217] Specific operation: The user uses the terminal to view the recorded information through the interface of the household accounting system. For example, the user checks the household accounting book that contains the information "January 1, 2022, Books, 1,000 yen, Cash."

[0218] Output: The contents of the household account book checked by the user

[0219] Step 6:

[0220] The household accounting system automatically saves information on the cloud.

[0221] Input: Purchase information recorded in the household ledger

[0222] Specific operation: The household accounting system automatically saves recorded information to cloud storage. For example, the recorded household accounting information is stored on the cloud and users can access it from any device.

[0223] Output: Household accounting information stored on the cloud

[0224] Step 7:

[0225] Generate prompts using a generative AI model.

[0226] Input: Purchase information recorded in the household ledger

[0227] Specific operation: The household accounting system uses the generative AI model to analyze the contents of the household accounting and generate appropriate prompts for the user. For example, it generates the following prompts:

[0228] If a user purchases a book for 1,000 yen with cash on January 1, 2022, explain how this will be recorded in the household ledger.

[0229] Output: Generated prompt statement

[0230] (Application example 2)

[0231] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, purchase information using electronic payment services cannot be automatically updated in the household accounting system, making it difficult for users to centrally manage their spending across multiple platforms.

[0233] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means that automatically recognizes the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means that automatically creates a household account book based on the information recognized by the image recognition means, a means for acquiring purchase information from an electronic payment service, and a means for automatically recording the acquired purchase information in the household account book. This eliminates the need for the user to manually enter purchase information, and enables the purchase information to be accurately and quickly reflected in the household account book.

[0234] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[0235] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[0236] "Image recognition means" is a technology for automatically identifying specific information from an image acquired by a camera means, and has the function of recognizing the amount, price, payment method, etc. of an item.

[0237] The "household account book creation means" is a system for recording the user's expenditures and automatically creating a household account book based on the information recognized by the image recognition means.

[0238] "Electronic payment service" refers to a payment service conducted over the Internet, and is a payment method used by users when purchasing goods or services online or offline.

[0239] "Purchase Information" refers to detailed information when a user purchases a product or service, including the date, name of the item, amount, and payment method.

[0240] "Cloud" means a data storage and processing service provided over the Internet that allows users to store and access data remotely.

[0241] An embodiment of the present invention will be described.

[0242] First, the user puts on the eyeglass-type wearable device. This device has a built-in camera means that captures images of the product when the user purchases it. The image captured by the camera means is analyzed by image recognition means, and information such as the product amount, price, and payment method is automatically recognized.

[0243] Next, the information recognized by the image recognition means is sent to the household account book creation means. The household account book creation means automatically creates a household account book based on the recognized information and saves it on the cloud. This allows the user to check their spending information anytime, anywhere.

[0244] Furthermore, this system is also linked to an electronic payment service. The server has a means of obtaining purchase information from the electronic payment service and a means of automatically recording the obtained purchase information in the household account book. As a result, information about purchases made by users using the electronic payment service is automatically reflected in the household account book.

[0245] As a specific example, if a user purchases a book for 1,000 yen using an electronic payment service on October 1, 2023, that information will be automatically recorded in the household account book. This saves users the trouble of manually entering the information, and allows them to accurately and quickly update their household account book with purchase information.

[0246] The following hardware and software are used to implement this system:

[0247] Hardware: Glasses-type wearable device, camera means

[0248] Software: Image recognition, household accounting, electronic payment service integration

[0249] It is also possible to create prompts using generative AI models, such as the following:

[0250] Example prompt sentence:

[0251] Create an application that automatically records purchases made by users using an electronic payment service in a household ledger. The household ledger will include the date, item name, amount, and payment method. Use Python to generate a program that retrieves data from the API and records it in a CSV file.

[0252] In this way, the user can create an accurate household account book without much effort.

[0253] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0254] Step 1:

[0255] A user puts on the eyeglass-type wearable device and purchases a product. At this time, the camera means takes a picture of the product and its price tag. The input is the image captured by the camera means, and the output is image data.

[0256] Step 2:

[0257] The image recognition means of the terminal analyzes the image data acquired by the camera means and automatically recognizes the amount, price, and payment method of the item. The input is image data, and the output is the recognized amount, price, and payment method of the item. Specifically, the system uses an image recognition algorithm to extract text information and identify the item information.

[0258] Step 3:

[0259] The household account book creation means of the terminal automatically creates a household account book based on the information recognized by the image recognition means. The input is the amount, price, and payment method information of the recognized items, and the output is household account book data. Specifically, the recognized information is converted into a specified format and saved in the household account book database.

[0260] Step 4:

[0261] The server obtains purchase information from the electronic payment service. The input is the purchase information obtained from the electronic payment service's API, and the output is the purchase information data. Specifically, the server sends an API request and receives the purchase information as a response.

[0262] Step 5:

[0263] The server automatically records the purchase information it acquires in the household account book. The input is purchase information data, and the output is updated household account book data. Specifically, it analyzes the purchase information and adds it to the existing household account book data.

[0264] Step 6:

[0265] The server saves the household accounting data on the cloud. The input is the updated household accounting data, and the output is the household accounting data saved on the cloud. Specifically, the data is uploaded to a cloud storage service.

[0266] Step 7:

[0267] The user checks the household accounting data stored on the cloud. The input is the household accounting data stored on the cloud, and the output is the household accounting information displayed on the user's device. Specifically, the data is downloaded from cloud storage and displayed on the user interface.

[0268] Example 3

[0269] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] Conventional household accounting management systems require users to manually input data, which is time-consuming and prone to input errors. There is also the risk of data being lost if the device breaks down or is lost. Another issue is that it is difficult for users to check their household accounting information while on the go.

[0271] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a voice input means, a voice recognition means for converting voice data acquired by the voice input means into text data, a communication means for transmitting the text data to a cloud server, and a display means for displaying the household account book information stored in the cloud server. This allows the user to manage the household account book without hassle, ensures that data is safely protected even if the device breaks down or is lost, and enables the user to check the household account book information while on the go.

[0272] A "wearable eyeglasses device" is an eyeglasses-type electronic device that can be worn by a user and has the function of acquiring visual and audio information.

[0273] "Camera means" refers to a device for taking images or videos and storing or transmitting them as digital data.

[0274] "Image recognition means" is a technology for automatically extracting and analyzing specific information from captured images and videos.

[0275] The "household account book creation means" is a function for organizing income and expenditure information based on the acquired data and recording it as a household account book.

[0276] "Voice input means" is a device for capturing the user's voice and processing it as digital data.

[0277] "Speech recognition means" is a technology for converting voice data into text data.

[0278] "Communication means" is a function for transmitting data to other devices or servers.

[0279] A "cloud server" is a remote server that stores and processes data over the Internet.

[0280] A "display means" is a device for visually displaying digital data.

[0281] MODE FOR CARRYING OUT THE INVENTION

[0282] This invention relates to a system that uses a glasses-type wearable device to automatically store household accounting information on the cloud, allowing users to check the information anytime, anywhere. Specific embodiments of this system are described below.

[0283] Hardware and Software Configuration

[0284] Glasses-type wearable device

[0285] The user wears a wearable eyeglass device that includes the following hardware and software:

[0286] Camera means: a camera for taking images or videos.

[0287] Audio input means: A microphone for capturing the user's voice.

[0288] Speech recognition tool: Software that converts voice data into text data using the Google Cloud Speech-to-Text API.

[0289] Communication means: A communication module for sending data to a cloud server via the Internet.

[0290] Cloud Server

[0291] The cloud server has the following functions:

[0292] Data receiving means: Receives data transmitted from the terminal.

[0293] Data storage method: Data is stored using Google Cloud Firestore.

[0294] Data display means: Allows users to view household accounting information from any internet-connected device.

[0295] System Operation

[0296] After shopping at the supermarket, a user speaks to the glasses-type wearable device, saying, "Today's shopping is for 200 yen for milk and 150 yen for bread." The device captures this speech and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is then sent to a cloud server. The server receives the data and stores it in Google Cloud Firestore. When the user returns home, they can open a dedicated app on their smartphone to check their shopping information for the day.

[0297] Specific examples

[0298] For example, if a user says, "Today's shopping is 200 yen for milk and 150 yen for bread," the glasses-type wearable device captures this voice and uses voice recognition to convert it into text data: "Today's shopping is 200 yen for milk and 150 yen for bread." This text data is then sent to a cloud server, which receives the data and stores it in Google Cloud Firestore. The user can then open a dedicated app on their smartphone to check the saved household accounting information.

[0299] Example prompts to input to a generative AI model:

[0300] "How can I use a wearable eyeglass device to store my household accounting information in the cloud? Please explain, including the names of specific APIs and cloud services."

[0301] In this way, the user can manage the household account book without hassle, the data is safely protected even if the device breaks down or is lost, and the user can check the information in the household account book even when they are away from home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0302] Step 1:

[0303] The user inputs household account information. The user puts on a glasses-type wearable device and inputs household account information by voice. For example, the user might say, "Today's shopping is 200 yen for milk and 150 yen for bread." The input is voice data, and the output is the voice data itself.

[0304] Step 2:

[0305] The device converts input information into digital data. The device uses a built-in microphone to capture the user's voice. The device then converts the voice data into text data using the Google Cloud Speech-to-Text API. For example, a speech saying "Today's shopping is milk for 200 yen and bread for 150 yen" is converted into text data saying "Today's shopping is milk for 200 yen and bread for 150 yen." The input is voice data, and the output is text data.

[0306] Step 3:

[0307] The device sends digital data to the cloud server. The device then sends the converted text data to the cloud server. The HTTPS protocol is used for transmission to ensure data security. The input is text data, and the output is data sent to the cloud server.

[0308] Step 4:

[0309] The server receives the data and stores it in cloud storage. The server receives the data sent from the device. The received data is stored in Google Cloud Firestore. Firestore is a real-time database, and data is quickly stored and retrieved. The input is text data sent to the cloud server, and the output is data stored in Firestore.

[0310] Step 5:

[0311] A user checks household accounting information stored in the cloud. The user checks household accounting information stored in the cloud using an internet-connected device such as a smartphone or PC. The user can access Google Cloud Firestore through a dedicated application or a web browser and view the stored household accounting data. The input is the data stored in Firestore, and the output is the household accounting information displayed on the user's device.

[0312] (Application example 3)

[0313] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0314] Conventional household accounting management systems require manual input, which is time-consuming for users and prone to input errors. Furthermore, because expenditure information cannot be managed in real time, it is difficult for users to immediately grasp their current spending situation. Furthermore, there is no function to issue an alert if expenditures exceed a certain range, making budget management difficult. To solve these problems, there is a need for a system that can automatically acquire expenditure information and manage household accounting in real time.

[0315] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0316] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a means for encrypting payment information and saving it on the cloud, a means for displaying the household account book information saved on the cloud in real time, and a means for generating an alert when a certain expenditure amount is exceeded, thereby enabling automatic acquisition and real-time management of expenditure information.

[0317] A "wearable eyeglasses device" is a wearable electronic device in the shape of eyeglasses that can be worn by users on a daily basis.

[0318] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[0319] The "image recognition means" is a technology for automatically identifying and analyzing specific information from an image acquired by the camera means.

[0320] The "household accounting means" is a system for recording and managing a user's income and expenditure based on information recognized by the image recognition means.

[0321] "Payment information" refers to data such as the amount, payment method, date and time incurred when a user purchases a product or service.

[0322] "Encryption" is a technology that converts data into a format that cannot be deciphered by third parties and is used to ensure the security of data.

[0323] "Storage on the cloud" is a technology that stores data on a remote server via the Internet.

[0324] "Real-time display means" refers to technology that instantly displays data stored in the cloud on the user's device.

[0325] An "alert generating means" is a system for notifying a user when certain conditions are met.

[0326] As an embodiment of this invention, a system is constructed that uses a glasses-type wearable device, a cloud storage service, a database, an encryption library, and a generative AI model.

[0327] Hardware and software used

[0328] Hardware: Wearable glasses, smartphones

[0329] Software: Cloud storage services (e.g., AWS S3), databases (e.g., Firebase), encryption libraries (e.g., PyCryptodome), generative AI models (e.g., GPT-4®)

[0330] System Overview

[0331] 1. Capture spending information:

[0332] A camera means built into the glasses-type wearable device captures payment information when a user purchases goods or services, for example, by using NFC (Near Field Communication).

[0333] The payment information obtained is temporarily stored on the device.

[0334] 2. Cloud storage of data:

[0335] Payment information is encrypted and uploaded to a cloud storage service (AWS S3).

[0336] Metadata of expenditure information (date, time, amount, category, etc.) is stored in the database (Firebase).

[0337] 3. Real-time household budget display:

[0338] Household accounting information retrieved from the cloud is displayed on the display of a smartphone app or glasses-type device.

[0339] Users can check their spending status in real time.

[0340] 4. Generate spending alerts:

[0341] If a certain amount of spending is exceeded, the application will generate an alert and notify the user.

[0342] Specific examples

[0343] For example, when a user goes shopping at a supermarket and pays through the glasses, the payment information is automatically saved in the cloud. The user can then check their spending in real time on their smartphone app and receive an alert if they exceed their budget.

[0344] Prompt Sentence Examples

[0345] An example of a prompt sentence when using a generative AI model is as follows:

[0346] "How can I capture my grocery shopping information, store it in the cloud, and display it on my phone in real time?"

[0347] In this way, the expenditure management system for the electronic payment service that uses the glasses-type wearable device can make it easier and more efficient for users to manage their expenditures.

[0348] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0349] Step 1:

[0350] Capture spending information

[0351] Input: Payment information when a user purchases a product or service (e.g., amount, payment method, date and time)

[0352] Processing: A camera means built into the glasses-type wearable device captures payment information. Specifically, the payment information is acquired using NFC (Near Field Communication).

[0353] Output: The payment information is temporarily stored on the device.

[0354] Step 2:

[0355] Data encryption

[0356] Input: Payment information stored on your device

[0357] Processing: Encrypt the payment information using a cryptography library (e.g. PyCryptodome).

[0358] Output: Encrypted payment information

[0359] Step 3:

[0360] Data storage in the cloud

[0361] Input: Encrypted payment information

[0362] Processing: Uploading encrypted payment information to a cloud storage service (e.g. AWS S3).

[0363] Output: Payment information stored in the cloud

[0364] Step 4:

[0365] Metadata storage

[0366] Input: Payment information metadata (e.g., date, time, amount, category)

[0367] Process: Store expenditure metadata in a database (e.g. Firebase).

[0368] Output: Metadata stored in a database

[0369] Step 5:

[0370] Real-time household budget display

[0371] Input: Payment information stored in the cloud and metadata stored in a database

[0372] Processing: Display the household accounting information retrieved from the cloud on the smartphone app or the display of the glasses-type device.

[0373] Output: Real-time household accounting information available to users

[0374] Step 6:

[0375] Generate spending alerts

[0376] Input: Metadata of expenditure information stored in the database

[0377] Action: If a certain spending amount is exceeded, the application generates an alert and notifies the user.

[0378] Output: The alert notification that is displayed to the user

[0379] In this way, through the specific operations performed at each step, a system is realized that automatically acquires and manages user expenditure information in real time.

[0380] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0381] "Example 1"

[0382] In one embodiment of the present invention, a user puts on a glasses-type wearable device and takes a photo of the purchased items and receipts with a camera. The captured image is sent to an image recognition system inside the device and analyzed. This analysis automatically extracts information such as the amount, price, and payment method of the items. Furthermore, an emotion engine recognizes the user's emotions and extracts this information as well. The extracted information is sent to a household accounting system inside the device and is automatically recorded in the household accounting book. Specifically, when a user buys food at a supermarket, the receipts are automatically recorded.

[0383] When the user photographs the sheet, the amount, item, and payment method are automatically recognized and recorded in the household ledger. In addition, the emotion engine recognizes the user's emotions at the time and records this information as well.

[0384] "Example 2"

[0385] In another embodiment of the present invention, a user puts on a glasses-type wearable device and takes a photo of the purchased item and receipt with a camera. The captured image is sent to an image recognition system inside the device and analyzed. This analysis automatically extracts information such as the item's amount, price, payment method, and purchase date and time. Furthermore, an emotion engine recognizes the user's emotion and extracts that information as well. The extracted information is sent to a household accounting system inside the device and automatically recorded in the household accounting book. Specifically, when a user purchases a book at a bookstore, they take a photo of the receipt, and the amount, item, payment method, and purchase date and time are automatically recognized and recorded in the household accounting book. Furthermore, the emotion engine recognizes the user's emotion at that time and records that information as well.

[0386] "Example 3"

[0387] In yet another embodiment of the present invention, a user wears a glasses-type wearable device and takes a photo of the purchased items and receipt with a camera. The captured image is sent to an image recognition system within the device and analyzed. This analysis automatically extracts information such as the item's amount, price, payment method, and purchase date and time. An emotion engine then recognizes the user's emotion and extracts that information. The extracted information is sent to a household accounting system within the device, where it is automatically recorded in the household accounting system and stored in the cloud. Specifically, when a user purchases clothes from an online shop and takes a photo of the receipt, the amount, item, payment method, and purchase date and time are automatically recognized, recorded in the household accounting system, and stored in the cloud. The emotion engine also recognizes the user's emotion at the time, and this information is also recorded and stored in the cloud.

[0388] The processing flow of each embodiment will be described below.

[0389] "Example 1"

[0390] Step 1: The user puts on the glasses-type wearable device.

[0391] Step 2: The user takes a photo of the purchased item or receipt with their camera.

[0392] Step 3: The captured image is sent to the device's internal image recognition system for analysis. Step 4: Information such as the item's amount, price, and payment method is automatically extracted.

[0393] Step 5: The emotion engine recognizes the user's emotions and extracts their information.

[0394] Step 6: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0395] "Example 2"

[0396] Step 1: The user puts on the glasses-type wearable device.

[0397] Step 2: The user takes a photo of the purchased item or receipt with their camera.

[0398] Step 3: The captured image is sent to the device's internal image recognition system and analyzed. Step 4: Information such as the item's amount, price, payment method, and purchase date and time is automatically extracted. Step 5: The emotion engine recognizes the user's emotion, and that information is also extracted.

[0399] Step 6: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0400] "Example 3"

[0401] Step 1: The user puts on the glasses-type wearable device.

[0402] Step 2: The user takes a photo of the purchased item or receipt with their camera.

[0403] Step 3: The captured image is sent to the device's internal image recognition system and analyzed. Step 4: Information such as the item's amount, price, payment method, and purchase date and time is automatically extracted. Step 5: The emotion engine recognizes the user's emotion, and that information is also extracted.

[0404] Step 6: The extracted information is sent to the device's internal household accounting system, which automatically

[0405] It is recorded in the ledger.

[0406] Step 7: The recorded information is stored on the cloud.

[0407] Example 1

[0408] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Conventional eyeglass-type wearable devices have the ability to automatically recognize the amount and payment method of purchased items and create a household ledger, but they lack the ability to recognize and record the user's emotions. As a result, they are unable to grasp changes in emotions that accompany a user's purchasing behavior, making it difficult to manage household finances or analyze consumption behavior in more detail. In addition, the automatic recognition of purchase dates and times and automatic saving of data to the cloud are insufficient, making data management and access cumbersome.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0411] In this invention, the server includes a camera mounted on the glasses-type wearable device, an image analysis unit that automatically recognizes the amount, price, and payment method of an item from an image acquired by the camera, a recording unit that automatically creates a household ledger based on the information recognized by the image analysis unit, an emotion analysis unit that recognizes the user's emotions, and a unit that transmits the emotion information recognized by the emotion analysis unit to the recording unit. This allows for recording of changes in emotions accompanying the user's purchasing behavior, enabling more detailed household management and analysis of consumption behavior. Furthermore, adding functions such as automatic recognition of purchase dates and times and automatic storage on the cloud facilitates data management and access.

[0412] A "wearable eyeglasses device" is a device that can be worn by the user and has various sensors, cameras, and computer systems built into the frame.

[0413] "Photographing means" refers to a camera or other image capturing device provided in the glasses-type wearable device, and is a means for capturing images of items purchased by the user, receipts, etc.

[0414] The "image analysis means" is a means for analyzing the image acquired by the photographing means and automatically recognizing information such as the amount, price, and payment method of the item.

[0415] The "recording means" is a means for automatically creating a household ledger based on information recognized by the image analysis means and storing it on the cloud as necessary.

[0416] "Emotion analysis means" refers to a means for analyzing a user's facial expressions and other biometric information to recognize the user's emotions.

[0417] "Emotion information" refers to information about the user's emotions recognized by the emotion analysis means.

[0418] "Purchase date and time" refers to the specific date and time when the user purchased the item, and is information that is automatically recognized by the image analysis means.

[0419] "Cloud" refers to data storage and computing resources provided over the internet, a place for storing information identified by a storage medium.

[0420] This invention relates to a system that uses a glasses-type wearable device to record a user's purchasing behavior and emotions and automatically creates a household account book. Specific embodiments of this system are described below.

[0421] The user puts on a glasses-type wearable device. The device has a camera (photography means) mounted on the frame, which allows the user to take pictures of the items and receipts they have purchased. The captured images are sent to an image analysis system inside the device. This image analysis system uses machine learning libraries such as TensorFlow to analyze the images and automatically extract information such as the item amount, price, and payment method.

[0422] The device also has a built-in emotion analysis system that uses facial recognition libraries such as OpenCV and Dlib to analyze the user's facial expressions and extract emotional information. For example, a smiling face is recognized as "joy."

[0423] The extracted information is sent to the device's internal recording system, which uses a database such as SQLite to manage the information and automatically creates a household ledger. The recognized information is also automatically saved in the cloud, allowing users to access their household ledger anytime, anywhere.

[0424] As a concrete example, imagine a user buys groceries at a supermarket and takes a photo of the receipt. The device sends the image to an image analysis system, which uses TensorFlow to analyze the letters, numbers, and barcodes on the receipt. The analysis results extract the price, item, and payment method of the food. Next, the device uses OpenCV to analyze the user's facial expression, and the emotion analysis system recognizes it as "happiness." All of this information is recorded in a record system using an SQLite database.

[0425] Example prompt sentence:

[0426] "A user wears a glasses-type wearable device and takes a picture of a receipt for food purchased at a supermarket. The device should use TensorFlow to analyze the image of the receipt, automatically recognize the amount, item, and payment method, and record these in an SQLite database. It should also use OpenCV to recognize the user's emotions and record this information as well."

[0427] In this way, the glasses-type wearable device can automatically record the user's purchasing information and emotions and create a household ledger.

[0428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0429] Step 1:

[0430] The user puts on the glasses-type wearable device.

[0431] Specific operation: The user puts the device on their head and checks that it is working properly. The device starts up and the camera and various sensors become ready.

[0432] Input: User wearing behavior

[0433] Output: Device boot status

[0434] Step 2:

[0435] The user takes a photo of the purchased item or receipt with the camera.

[0436] What happens: A user uses the device's camera to take a picture of a purchase or receipt. For example, they point the camera at a receipt for groceries they bought at a supermarket.

[0437] Input: Image of item or receipt

[0438] Output: Captured image data

[0439] Step 3:

[0440] The device sends the captured image to an image analysis system.

[0441] How it works: The system inside the device acquires the captured image data and sends it to an image analysis system, which uses machine learning libraries such as TensorFlow.

[0442] Input: Photographed image data

[0443] Output: Sending image data to an image analysis system

[0444] Step 4:

[0445] The terminal analyzes the image using an image analysis system to extract information such as the item's amount, price, and payment method.

[0446] Specific operation: The image analysis system analyzes image data and recognizes the letters, numbers, barcodes, etc. on the receipt. For example, it extracts the amount, item name, and payment method from the receipt.

[0447] Input: Image data

[0448] Output: Information such as the amount, price, and payment method of the extracted item

[0449] Step 5:

[0450] The device recognizes the user's emotions using an emotion analysis system.

[0451] How it works: The device's internal emotion analysis system analyzes the user's facial expressions and extracts emotional information. The emotion analysis system uses face recognition libraries such as OpenCV and Dlib. For example, if the user is smiling, it will recognize the emotion as "joy."

[0452] Input: User's facial expression data

[0453] Output: Extracted emotion information

[0454] Step 6:

[0455] The terminal transmits the extracted information to the recording system.

[0456] How it works: The system inside the device sends the information extracted from the image analysis system and emotion analysis system to a recording system, which uses a database such as SQLite.

[0457] Input: Extracted item information and emotion information

[0458] Output: Sending information to a system of record

[0459] Step 7:

[0460] The terminal records the information in the recording system.

[0461] Specific operation: The recording system automatically creates a household account book based on the information received and stores it in the cloud as needed. For example, the system records the amount, items, payment method, and user emotions of food purchased at the supermarket in the household account book.

[0462] Input: Information into the system of record

[0463] Output: Create a household budget and save it to the cloud

[0464] (Application example 1)

[0465] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0466] Conventional eyeglass-type wearable devices have the ability to automatically recognize the amount of purchases and the payment method, and create a household account book, but they are unable to take the user's emotions into account. As a result, they are unable to record and analyze the user's emotional reactions to purchasing behavior and spending. In addition, they lack the ability to store household account information on the cloud, making centralized management and backup of data difficult. To solve these issues, there is a need for a system that recognizes the user's emotions, adds the ability to record information in a household account book, and stores data on the cloud.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0468] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for recognizing a user's emotion, and a means for recording the emotion information recognized by the emotion recognition means in the household account book. This makes it possible to record and analyze a user's emotional response to purchasing behavior. Furthermore, storing household account book information on the cloud facilitates centralized data management and backup.

[0469] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera, sensor, and display.

[0470] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[0471] "Image recognition means" is a system for automatically extracting and analyzing specific information from captured images.

[0472] The "household accounting means" is a system for recording and managing a user's income and expenditure based on information extracted by the image recognition means.

[0473] An "emotion recognition means" is a system that analyzes and recognizes emotions from a user's facial expressions, voice, etc.

[0474] The "means for recording emotion information in a household account book" is a system for adding and recording emotion information recognized by the emotion recognition means in a household account book.

[0475] "Means for automatically saving on the cloud" is a system for automatically saving recognized information on a cloud server via the Internet.

[0476] As an embodiment of the present invention, a system using a glasses-type wearable device will be described below. This system includes a glasses-type wearable device worn by a user, a camera means built into the device, an image recognition means, an emotion recognition means, a household accounting means, and a means for storing data on the cloud.

[0477] System Program

[0478] The program of this system operates as follows.

[0479] 1. Camera means: A camera built into the glasses-type wearable device takes images of the items purchased by the user and the receipt.

[0480] 2. Image recognition: The captured image is sent to the device's internal image recognition system (e.g., OpenCV or pytesseract), which automatically extracts information such as the item's amount, price, and payment method.

[0481] 3. Emotion recognition means: At the same time, an emotion recognition system (e.g., EmotionRecognizer) operates to analyze emotions from the user's facial expressions and voice and recognize the user's emotional information.

[0482] 4. Household accounting system: The extracted information and emotion information are sent to a household accounting system (e.g., FinanceManager) and automatically recorded in the household accounting system.

[0483] 5. Cloud storage: Household accounting information is automatically saved to a cloud server via the Internet.

[0484] Hardware and Software

[0485] Hardware: Camera built into a wearable eyeglasses device

[0486] Software: OpenCV (image capture), pytesseract (OCR), EmotionRecognizer (emotion recognition), FinanceManager (household finance management)

[0487] Specific examples

[0488] Consider a scenario where a user buys food at a supermarket and takes a photo of the receipt. The user puts on the smart glasses and takes a photo of the receipt. The amount, item, and payment method information are automatically extracted from the captured image and recorded in a household ledger. The user's emotions are also recognized, and this information is also recorded in the household ledger.

[0489] Example prompt sentence:

[0490] The user puts on the smart glasses and takes a picture of a supermarket receipt. The system automatically extracts information about the amount, item, and payment method from the image and records it in a household ledger. The system also recognizes the user's emotions and records that information in the household ledger.

[0491] In this way, it is possible to record and analyze users' emotional responses to purchasing behavior. Furthermore, by storing household accounting information on the cloud, it becomes easier to centrally manage and back up data.

[0492] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0493] Step 1:

[0494] The user puts on the glasses-type wearable device and takes pictures of purchased items and receipts with the camera.

[0495] Input: Physical image of purchase or receipt

[0496] Output: Digital image data

[0497] How it works: The camera built into the glasses-type wearable device takes pictures of items and receipts from the user's point of view and saves them as digital image data.

[0498] Step 2:

[0499] The terminal transmits the captured image to an image recognition means, which automatically recognizes the amount, price and payment method of the item.

[0500] Input: Digital image data

[0501] Output: Extracted text information (amount, item, payment method)

[0502] Specific operation: Image recognition tools (e.g., OpenCV or pytesseract) analyze digital image data and extract text information using OCR technology.

[0503] Step 3:

[0504] The terminal transmits the extracted text information to the household account book creation means, and the household account book is automatically created.

[0505] Input: Extracted text information (amount, item, payment method)

[0506] Output: Household accounting data

[0507] Specific operation: A household account book creation means (for example, FinanceManager) records the user's income and expenditure based on the extracted text information and generates household account book data.

[0508] Step 4:

[0509] The terminal transmits the user's facial expression and voice to the emotion recognition means, and recognizes the user's emotion.

[0510] Input: User facial and voice data

[0511] Output: Emotional information

[0512] Specific operation: The emotion recognition means (e.g., EmotionRecognizer) analyzes the user's facial expressions and voice and recognizes emotional information.

[0513] Step 5:

[0514] The terminal transmits the recognized emotion information to the household account book creating means and records it in the household account book.

[0515] Input: Emotion information

[0516] Output: Updated household accounting data

[0517] Specific operation: The household account book creation means adds the emotional information to the household account book data and generates updated household account book data.

[0518] Step 6:

[0519] The device automatically saves updated household accounting data to the cloud.

[0520] Input: Updated household accounting data

[0521] Output: Household accounting data stored on the cloud

[0522] Specific operation: The household accounting creation means connects to the cloud server via the Internet and saves the updated household accounting data on the cloud.

[0523] Example 2

[0524] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0525] Conventional household accounting systems require users to manually input product information, which is time-consuming and prone to input errors. Furthermore, there is no function to record the user's emotions, meaning that household management cannot reflect the emotions felt at the time of purchase. Furthermore, when data is limited to local storage, there is also the problem of a high risk of data loss or corruption.

[0526] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0527] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the price, name, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for analyzing the user's emotions, and a means for recording the emotion information recognized by the emotion recognition means in the household account book. This eliminates the need for the user to manually input information, enabling an accurate household account book to be automatically created. Furthermore, since the user's emotion information is also recorded, household management that reflects emotions can be realized. Furthermore, since data is automatically stored on the cloud, the risk of data loss or corruption is reduced.

[0528] A "wearable eyeglasses device" is an eyeglass-type electronic device that can be worn by the user and has a built-in camera and various sensors.

[0529] The "camera means" is a device for taking images, and refers to a camera built into the eyeglass-type wearable device.

[0530] "Image recognition means" refers to software or hardware for automatically extracting specific information from a captured image.

[0531] The "household accounting book creation means" is software or hardware for automatically creating a household accounting book based on the information extracted by the image recognition means.

[0532] The "emotion recognition means" is software or hardware for analyzing the user's emotions and extracting that information.

[0533] "Cloud" is a general term for data storage and processing services provided over the Internet.

[0534] "Amount of Goods" refers to the price of the purchased goods.

[0535] "Name" refers to the name of the purchased item.

[0536] "Payment Method" means the payment method used at the time of purchase.

[0537] "Purchase date and time" refers to the specific date and time when the item was purchased.

[0538] "Emotion information" is data that indicates the emotional state of the user.

[0539] MODE FOR CARRYING OUT THE INVENTION

[0540] The present invention relates to a system that uses a glasses-type wearable device to take photos of items purchased by a user and receipts, and automatically records the information in a household account book. Specific embodiments of this system are described below.

[0541] Wearing a glasses-type wearable device

[0542] The user wears a glasses-type wearable device that is equipped with a camera, an image recognition system, and an emotion engine.

[0543] Taking photos of items and receipts

[0544] Users take pictures of purchases or receipts with the device's camera, and the images are sent to the device's internal processing system.

[0545] Image analysis

[0546] The device uses a built-in image recognition system to analyze the captured image and extract information such as the item's price, name, payment method, and purchase date and time.

[0547] Emotion recognition

[0548] The device uses the emotion engine to analyze the user's emotions, for example, to extract emotional information such as whether the user is happy or sad.

[0549] Sending information

[0550] The extracted information is sent to the device's internal household accounting system, where item information and user emotion information are processed together.

[0551] Recording in a household account book

[0552] The server receives the information sent from the device and records it in a household ledger, which consists of items such as the date, the name of the item, the amount, the payment method, and the user's emotions.

[0553] Data storage

[0554] The server stores the recorded household accounting data on the cloud, where it is stored securely so that users can check it later.

[0555] Specific examples

[0556] For example, if a user purchases a book for 1,000 yen in cash at a bookstore on January 1, 2022, the user takes a photo of the receipt with a glasses-type wearable device. The device uses an image recognition system to extract the information "January 1, 2022, book, 1,000 yen, cash," and uses an emotion engine to recognize the user's emotion as "joy." This information is sent to the device's internal household accounting system. The server receives the information sent from the device and records the data "January 1, 2022, book, 1,000 yen, cash, joy" in the household accounting book. The recorded data is securely stored on the server, allowing the user to view it later.

[0557] Prompt Sentence Examples

[0558] "Describe a system that uses a wearable eyeglass device to take pictures of purchased items and receipts, extracts information using an image recognition system and an emotion engine, and automatically records the information in a household ledger."

[0559] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0560] Step 1:

[0561] Wearing a glasses-type wearable device

[0562] The user wears a glasses-type wearable device. This device has a built-in camera, image recognition system, and emotion engine. The input is the user's wearing behavior, and the output is the device's activation status.

[0563] Step 2:

[0564] Taking photos of items and receipts

[0565] The user takes a photo of the purchased item or receipt with the device's camera. The input is the item or receipt to be photographed, and the output is the captured image data. Specifically, the user presses the camera's shutter button.

[0566] Step 3:

[0567] Image analysis

[0568] The device uses a built-in image recognition system to analyze the captured image. The input is the captured image data, and the output is extracted information such as the item's price, name, payment method, and purchase date and time. Specifically, the image recognition algorithm analyzes the image data and extracts text information.

[0569] Step 4:

[0570] Emotion recognition

[0571] The device uses an emotion engine to analyze the user's emotions. The input is the user's facial expression and voice data, and the output is the user's emotional information. Specifically, the emotion recognition algorithm analyzes the facial expression and voice data to identify the emotion.

[0572] Step 5:

[0573] Sending information

[0574] The extracted information is sent to the device's internal household accounting system. The input is item information and user emotion information, and the output is data transmission to the household accounting system. Specifically, the device stores the data in its internal memory and transmits it to the household accounting system.

[0575] Step 6:

[0576] Recording in a household account book

[0577] The server receives the information sent from the device and records it in the household ledger. The input is the item information and emotion information sent from the device, and the output is the household ledger data. Specifically, the server analyzes the received data and records it according to the household ledger format.

[0578] Step 7:

[0579] Data storage

[0580] The server stores the recorded household accounting data on the cloud. The input is the household accounting data, and the output is the data stored on the cloud. Specifically, the server uploads the data to a cloud storage service.

[0581] (Application example 2)

[0582] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0583] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, since the system is unable to record shopping experiences that take into account the user's emotions, it is difficult to evaluate shopping satisfaction or stress. Furthermore, data management on the cloud is inadequate, leading to issues such as data loss and difficulty in accessing it.

[0584] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0585] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for recognizing a user's emotion, and a means for recording the emotion information recognized by the emotion recognition means in the household account book. This makes it possible to automatically record information about items purchased by a user in the household account book, and also to simultaneously record the user's emotion information.

[0586] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[0587] The "camera means" is a device for capturing images and videos, and is built into the eyeglass-type wearable device.

[0588] "Image recognition means" refers to technology or equipment for automatically extracting and analyzing specific information from captured images.

[0589] The "household account book creation means" refers to a technique or device for automatically creating a household account book based on the acquired purchase information.

[0590] "Emotion recognition means" refers to technology or devices that analyze and recognize emotions from a user's facial expressions, voice, etc.

[0591] "Cloud" is a data storage and computing resource service provided over the Internet, allowing users to store and manage data on remote servers.

[0592] As an embodiment of the present invention, a system using a glasses-type wearable device will be described below.

[0593] System configuration

[0594] Hardware

[0595] Glasses-type wearable device: An electronic device in the shape of glasses worn by the user, equipped with a camera and sensors.

[0596] Camera means: A device built into the glasses-type wearable device for taking images and videos.

[0597] software

[0598] Image recognition means: Technology or equipment for automatically extracting and analyzing the amount, price, and payment method of an item from a captured image. Specifically, it uses libraries such as OpenCV and Pytesseract.

[0599] Household accounting means: This refers to the technology or device that automatically creates a household accounting record based on acquired purchase information. It uses a database management system or cloud storage.

[0600] Emotion recognition means: Technology and devices for analyzing and recognizing emotions from the user's facial expressions, voice, etc. Libraries such as EmotionRecognizer are used.

[0601] Processing flow

[0602] 1. Photographing a receipt: The user puts on the glasses-type wearable device and takes a picture of the receipt using the camera.

[0603] 2. Text extraction: Extract text from the captured receipt image using Pytesseract.

[0604] 3. Information analysis: Analyze the extracted text to obtain information such as the item amount, price, payment method, and purchase date and time.

[0605] 4. Emotion Recognition: Take another image and use EmotionRecognizer to recognize the user's emotions.

[0606] 5. Household account book update: Based on the acquired purchase information and emotional information, a household account book is automatically created and saved on the cloud.

[0607] Specific examples

[0608] When a user purchases a book at a bookstore and takes a photo of the receipt with the glasses-type wearable device, the following information is recorded in the household ledger:

[0609] Date: October 1, 2023

[0610] Item: Book

[0611] Price: 1000 yen

[0612] Payment Method: Cash

[0613] Emotion: Joy

[0614] Prompt Sentence Examples

[0615] "Put on the smart glasses and take a photo of the receipt for a book you bought at a bookstore. The application will automatically analyze the purchase information and emotional information and record it in your household account book."

[0616] In this way, information about the items purchased by the user can be automatically recorded in the household ledger, and the user's emotional information can also be recorded at the same time. This makes it easier to evaluate the satisfaction and stress of the shopping experience, and allows for more efficient household management.

[0617] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0618] Step 1:

[0619] A user puts on the glasses-type wearable device and takes a picture of a receipt using the camera means. The input is an image of the receipt, and the output is the captured image of the receipt. This image is used in subsequent processing steps.

[0620] Step 2:

[0621] The server receives the captured receipt image and extracts the text using image recognition. Specifically, it uses the Pytesseract library to extract text data from the image. The input is the receipt image, and the output is the extracted text data.

[0622] Step 3:

[0623] The server analyzes the extracted text data and obtains information such as the item amount, price, payment method, and purchase date and time. Specifically, the text data is divided into lines and information is extracted based on specific keywords (e.g., "yen," "cash," etc.). The input is text data, and the output is the analyzed purchase information.

[0624] Step 4:

[0625] The user then takes a photo of their face using the glasses-type wearable device, and their emotions are recognized using the emotion recognition means. Specifically, the EmotionRecognizer library is used to analyze emotions from the facial image. The input is the user's facial image, and the output is the recognized emotion data.

[0626] Step 5:

[0627] The server automatically creates a household account book based on the purchase information and emotion data it acquires. Specifically, the purchase information and emotion data are stored in a database as a single record. The input is the purchase information and emotion data, and the output is updated household account book data.

[0628] Step 6:

[0629] The server saves the updated household accounting data on the cloud. Specifically, the data is saved on a remote server using a cloud storage service. The input is the updated household accounting data, and the output is the household accounting data saved on the cloud.

[0630] In this way, information about the items purchased by the user and emotional information can be automatically recorded in a household account book and managed on the cloud.

[0631] Example 3

[0632] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0633] Conventional household accounting management systems require users to manually input purchase information, which is time-consuming and prone to input errors. There is also the risk of losing household accounting information if the device breaks down or is lost. Furthermore, because household accounting management cannot take into account the user's emotions, it is difficult to perform detailed household accounting that reflects the psychological state at the time of purchase.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0635] In this invention, the server includes a photographing means provided in the glasses-type wearable device, an image analysis means for automatically recognizing the price, name, and payment method of the item from the image acquired by the photographing means, a recording means for automatically creating a household account book based on the information recognized by the image analysis means, a storage means for automatically saving the household account book information created by the recording means on the cloud, an emotion analysis means for recognizing the user's emotion from the image acquired by the photographing means, and an emotion recording means for recording the emotion information recognized by the emotion analysis means in the household account book. This saves the user the trouble of manually entering information and prevents input errors. Furthermore, since the information is stored on the cloud, the risk of losing household account book information even if the device breaks down or is lost is reduced. Furthermore, detailed household management that takes the user's emotions into consideration is possible.

[0636] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[0637] "Photographing means" refers to a camera or other photographing device provided in the glasses-type wearable device, which is used to capture images of items purchased by the user and receipts.

[0638] "Image analysis means" refers to software or hardware for automatically recognizing and analyzing information such as the price, name, and payment method of an item from an image acquired by a photographing means.

[0639] The "recording means" refers to a system or device that automatically creates a household ledger based on information recognized by the image analysis means and records that information.

[0640] "Storage means" refers to a system or device for automatically storing the household accounting information created by the recording means on the cloud.

[0641] The "emotion analysis means" refers to software or hardware for recognizing the user's emotions from the image captured by the image capture means and analyzing that information.

[0642] The "emotion recording means" refers to a system or device for recording the emotional information recognized by the emotion analysis means in a household account book.

[0643] "Cloud" refers to remote servers and storage services for storing, managing, and accessing data over the Internet.

[0644] The present invention relates to a system that uses a glasses-type wearable device to automatically store household accounting information on the cloud. Specific embodiments of this system will be described below.

[0645] The user puts on the eyeglass-type wearable device and takes a picture of the purchased item or receipt with the camera. The eyeglass-type wearable device has a built-in camera as a photographing means, and the user captures the image data by taking a picture of the purchased item or receipt.

[0646] The terminal sends the image captured by the image capture means to the image analysis means, which uses image recognition software such as Google Cloud Vision API, and automatically extracts information such as the price, name, payment method, and purchase date and time of the item.

[0647] Furthermore, the device recognizes the user's emotions using an emotion analysis means, such as the Emotion API from Microsoft® Azure®, to analyze the emotions the user felt when making a purchase.

[0648] The extracted information is sent to a recording means, which may use, for example, MICROSOFT EXCEL (registered trademark) API or Google Sheets API. The recording means automatically creates a household ledger based on the extracted information and records emotional information as well.

[0649] The recorded household accounting information is automatically saved to the cloud using a storage method, such as a cloud storage service like Amazon S3 or Google Drive. This allows users to check their household accounting information anytime, anywhere. Furthermore, even if the device breaks down or is lost, the household accounting information saved to the cloud is safely protected.

[0650] For example, when a user purchases clothes from an online shop and takes a photo of the receipt, the amount, item name, payment method, and purchase date and time are automatically recognized, recorded in a household account book, and saved on the cloud. Furthermore, the user's emotions at the time are recognized by emotion analysis means, and this information is also recorded and saved on the cloud.

[0651] Example prompt sentence:

[0652] "Please explain how when a user puts on a wearable eyeglass device and takes pictures of purchased items and receipts with the camera, the information is recorded in a household ledger and stored in the cloud."

[0653] In this way, the present invention provides a system that saves the user time and effort, prevents input errors, and enables detailed household management.The flow of the identification process in the third embodiment will be described with reference to FIG.

[0654] Step 1:

[0655] The user puts on the glasses-type wearable device and takes pictures of purchased items and receipts with the camera.

[0656] Input: Purchases and receipts

[0657] Output: Captured image data

[0658] Specific operation: The user uses the camera of the glasses-type wearable device to take a picture of the purchased items and receipt. For example, after shopping at a supermarket, the user points the camera at the receipt and takes a picture.

[0659] Step 2:

[0660] The terminal transmits the captured image to the image analysis means.

[0661] Input: Photographed image data

[0662] Output: Sending image data to image analysis means

[0663] Specific operation: The terminal uses an internal communication module to send captured image data to the image analysis means. For example, the terminal sends image data using Wi-Fi or Bluetooth.

[0664] Step 3:

[0665] The terminal analyzes the image using an image analysis means and extracts information such as the price of the item, the item name, the payment method, and the date and time of purchase.

[0666] Input: Image data

[0667] Output: Extracted information (item price, item name, payment method, purchase date and time)

[0668] How it works: The device uses image recognition software such as Google Cloud Vision API to analyze image data. For example, it can extract information from a receipt, such as "Milk 200 yen," "Payment method: credit card," and "Purchase date: October 1, 2023."

[0669] Step 4:

[0670] The terminal recognizes the user's emotions using emotion analysis means and extracts that information.

[0671] Input: User's face image data

[0672] Output: Extracted emotion information

[0673] How it works: The device uses emotion recognition software, such as Microsoft Azure's Emotion API, to analyze the user's facial image data. For example, the device may recognize the user's emotion as "joy" while shopping.

[0674] Step 5:

[0675] The terminal transmits the extracted information to the recording means and records it in the household account book.

[0676] Input: Extracted information (item price, item name, payment method, purchase date and time, emotional information)

[0677] Output: Information recorded in the household ledger

[0678] Specific operation: The device uses the Microsoft Excel API, Google Sheets API, etc. to record the extracted information in a household ledger. For example, information such as "Milk 200 yen," "Payment method: Credit card," "Purchase date: October 1, 2023," and "Emotion: Joy" is recorded in the household ledger.

[0679] Step 6:

[0680] The terminal stores the recorded household account book information on the cloud using a storage means.

[0681] Input: Information recorded in the household ledger

[0682] Output: Household accounting information stored on the cloud

[0683] Specific operation: The device uses cloud storage services such as Amazon S3 and Google Drive to store the household accounting information in the cloud. For example, the household accounting information is uploaded to the cloud storage.

[0684] Step 7:

[0685] The server manages the household accounting information on the cloud, allowing users to access it anytime, anywhere.

[0686] Input: Household accounting information stored on the cloud

[0687] Output: User-accessible household accounting information

[0688] How it works: The server manages the cloud storage service and allows users to access their household accounting information from their smartphones or PCs. For example, users can check their household accounting information through a dedicated app or web browser.

[0689] (Application example 3)

[0690] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0691] Conventional household accounting management systems require users to manually enter purchase and expenditure information, which is time-consuming and prone to input errors. Furthermore, they are unable to take into account the user's emotions, making it difficult to create a household accounting system that reflects their emotions and satisfaction at the time of purchase. Furthermore, data storage is limited to local storage, which poses the risk of data loss if the device breaks down or is lost.

[0692] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0693] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of items from images acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for recognizing the user's emotions, and a means for automatically saving the recognized information and emotion information on the cloud. This saves the user the trouble of manually inputting purchase and expenditure information, enables emotion-based household account management, and safely protects data even if the device breaks down or is lost.

[0694] A "wearable eyeglasses device" is an electronic device in the form of glasses that, when worn by the user, acquires visual information and processes and displays various data.

[0695] The "camera means" is a device for capturing images and videos, and is a camera built into the eyeglass-type wearable device.

[0696] "Image recognition means" is a technology for automatically extracting and analyzing specific information from captured images.

[0697] The "household account book creation means" is a system for automatically creating a household account book based on the acquired expenditure information.

[0698] "Emotion recognition means" is a technology for analyzing and recognizing emotions from a user's facial expressions, voice, etc.

[0699] "Means for automatic storage on the cloud" refers to technology for automatically storing acquired data on a remote server via the Internet.

[0700] "Amount of Goods" refers to the price of the purchased goods.

[0701] "Price" refers to the cost of a good or service.

[0702] "Payment method" refers to the payment method used at the time of purchase (e.g., cash, credit card, electronic money, etc.).

[0703] "Purchase date and time" refers to the specific date and time when a good or service was purchased.

[0704] The embodiments of the present invention will be described in detail below.

[0705] System Overview

[0706] This system uses a wearable eyeglasses-type device to take photos of items purchased by the user and receipts, automatically records the information in a household ledger, and stores it in the cloud. It also recognizes the user's emotions and records that information.

[0707] Hardware and software used

[0708] Hardware:

[0709] Glasses-type wearable device (camera, image recognition system, emotion recognition system)

[0710] software:

[0711] OpenCV (image processing library)

[0712] Tesseract OCR (Optical Character Recognition Library)

[0713] cloud_storage_api (Cloud Storage API)

[0714] emotion_recognition_api (emotion recognition API)

[0715] Data processing and calculation

[0716] Camera means:

[0717] The camera built into the glasses-type wearable device takes pictures of the items and receipts purchased by the user, and the images are sent to the device's internal image recognition system.

[0718] Image Recognition Methods:

[0719] The image recognition system uses OpenCV and Tesseract OCR to automatically extract information from captured images, such as the item's amount, price, payment method, and purchase date and time, saving users the trouble of manually entering this information.

[0720] Emotion recognition means:

[0721] The emotion recognition system uses emotion_recognition_api to analyze and recognize emotions from the user's facial expressions and voice. This information is also recorded in the household ledger.

[0722] How to create a household budget:

[0723] The extracted information and emotional information are sent to a household accounting system and automatically recorded in the household accounting book, enabling users to manage their household finances in a way that reflects their emotions and satisfaction at the time of purchase.

[0724] Cloud storage options:

[0725] The recognized information and emotion information are automatically stored in the cloud using cloud_storage_api, which ensures data security even in the event of device failure or loss.

[0726] Specific examples

[0727] When a user purchases clothing at a physical store and takes a photo of the receipt with the glasses-type wearable device, the amount, item, payment method, and purchase date and time are automatically recognized and recorded in a household ledger. The device also recognizes the user's emotions at the time (e.g., happiness) and records this information. This information is stored in the cloud, and the user can check the household ledger information anytime, anywhere via a smartphone app.

[0728] Prompt Sentence Examples

[0729] When shopping in a physical store, users put on the glasses-type wearable device and take pictures of the items they purchased and the receipt. The captured image is analyzed by the device's internal image recognition system, and information such as the amount, item, payment method, and purchase date and time is automatically extracted. An emotion engine also recognizes the user's emotions and records this information. The extracted information is automatically recorded in a household account book and saved in the cloud. Users can check their household account book information anytime, anywhere via a smartphone app.

[0730] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0731] Step 1:

[0732] The user puts on the glasses-type wearable device and takes pictures of purchased items and receipts with the camera.

[0733] Input: Items purchased by the user and receipts

[0734] Output: Captured image file

[0735] Specific operation: The camera in the glasses-type wearable device captures an image and stores it in the internal memory.

[0736] Step 2:

[0737] The device sends the captured image to an image recognition system.

[0738] Input: Captured image file

[0739] Output: Input data to the image recognition system

[0740] Specific operation: The device transfers the image file to the image recognition system.

[0741] Step 3:

[0742] The image recognition system uses OpenCV and Tesseract OCR to extract information from the image, such as the item's amount, price, payment method, and purchase date and time.

[0743] Input: Input data to the image recognition system

[0744] Output: Extracted text information (item amount, price, payment method, purchase date and time)

[0745] How it works: The image recognition system analyzes the image and extracts text information.

[0746] Step 4:

[0747] The device transmits the user's facial expressions and voice to the emotion recognition system.

[0748] Input: User's facial expressions and voice data

[0749] Output: Input data to the emotion recognition system

[0750] Specific operation: The device captures the user's facial expressions and voice and transmits them to the emotion recognition system.

[0751] Step 5:

[0752] The emotion recognition system uses emotion_recognition_api to analyze and recognize the user's emotions.

[0753] Input: Input data to the emotion recognition system

[0754] Output: Recognized emotion information

[0755] Specific operation: The emotion recognition system analyzes facial expressions and voice data to recognize emotions.

[0756] Step 6:

[0757] The terminal transmits the extracted text information and the recognized emotion information to a household accounting system.

[0758] Input: Extracted text information, recognized emotion information

[0759] Output: Input data to the household accounting system

[0760] Specific operation: The terminal transfers text information and emotional information to the household accounting system.

[0761] Step 7:

[0762] The household accounting system automatically creates a household accounting book based on the received information.

[0763] Input: Data to be input into the household accounting system

[0764] Output: Automatically generated household accounting data

[0765] Specific operation: The household accounting system integrates text information and emotional information to generate a household accounting book.

[0766] Step 8:

[0767] The server stores the household accounting data on the cloud.

[0768] Input: Automatically generated household accounting data

[0769] Output: Household accounting data stored on the cloud

[0770] Specific operation: The server uploads the household accounting data to the cloud using the cloud storage API.

[0771] Step 9:

[0772] Users can check their household accounting data on the cloud through a smartphone app.

[0773] Input: Household accounting data stored on the cloud

[0774] Output: Household accounting information displayed on a smartphone app

[0775] Specific operation: The user launches the smartphone app, retrieves household accounting data from the cloud, and displays it.

[0776] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0777] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0778] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0779] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0780] [Second embodiment]

[0781] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0782] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0783] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0784] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0785] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0786] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0787] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0788] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0789] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0790] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0791] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0792] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0793] "Example 1"

[0794] The eyeglass-type wearable device of the present invention has a camera mounted on the frame that takes a picture of the item purchased by the user. The captured image is analyzed by a built-in image recognition system, and information such as the item's amount, price, and payment method is automatically extracted. For example, if a receipt is photographed, the necessary information can be extracted from the letters, numbers, barcodes, etc. on the receipt.

[0795] "Example 2"

[0796] The extracted information is automatically recorded in a household ledger by the built-in household ledger creation system. The household ledger consists of items such as the date, item name, amount, and payment method, and is saved so that the user can check it later. For example, if you purchase a book for 1,000 yen with cash on January 1, 2022, the information recorded in the household ledger will be "January 1, 2022, book, 1,000 yen, cash."

[0797] "Example 3"

[0798] Furthermore, the glasses-type wearable device of the present invention also has the function of automatically saving household accounting information to the cloud, allowing users to check their household accounting information anytime, anywhere. Furthermore, even if the device breaks down or is lost, the household accounting information saved in the cloud is safely protected.

[0799] The processing flow of each embodiment will be described below.

[0800] "Example 1"

[0801] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[0802] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[0803] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0804] "Example 2"

[0805] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[0806] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[0807] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0808] Step 4: The recorded household accounting information is stored internally on the device for the user to review later.

[0809] "Example 3"

[0810] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[0811] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[0812] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[0813] Step 4: The recorded household accounting information is automatically saved on the cloud, allowing users to check their household accounting information anytime, anywhere.

[0814] Example 1

[0815] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0816] Conventional glasses-type wearable devices require users to manually enter information about purchased items, which is time-consuming for the user. Furthermore, there is a lack of systems for accurately managing purchase information, making it difficult to automatically create a household ledger or save it in the cloud. This makes it difficult for users to efficiently manage their purchase history, making household management cumbersome.

[0817] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0818] In this invention, the server includes a photographing means, a storage means for storing images acquired by the photographing means, an analysis means for analyzing the images stored in the storage means, an extraction means for extracting letters and numbers using optical character recognition technology by the analysis means, an information analysis means for analyzing the amount, price, and payment method of the item from the information extracted by the extraction means, and a display means for displaying the information analyzed by the information analysis means. This allows the user to automatically obtain information on purchased items and create a household account book and store it in the cloud.

[0819] The "photography means" is a device such as a camera for taking pictures of items purchased by the user and receipts.

[0820] The "storage means" refers to a memory or storage device for storing image data acquired by the imaging means.

[0821] The "analysis means" refers to software or hardware for analyzing the image data stored in the storage means.

[0822] The "extraction means" is a function that allows the analysis means to extract characters and numbers from image data using optical character recognition technology.

[0823] The "information analysis means" is a function for analyzing information such as the amount, price, and payment method of the item from the characters and numbers extracted by the extraction means.

[0824] The "display means" is a display or monitor for displaying the information analyzed by the information analysis means to the user.

[0825] The "cloud" refers to remote servers and storage services for storing and managing data over the Internet.

[0826] MODE FOR CARRYING OUT THE INVENTION

[0827] This invention is a system that uses a glasses-type wearable device to capture images of items and receipts purchased by a user, analyzes the image data, and automatically extracts information such as the item amount, price, payment method, etc. Specific embodiments of this system are described below.

[0828] Hardware and Software Configuration

[0829] The glasses-type wearable device worn by the user includes the following hardware and software:

[0830] Photography method: High-resolution camera

[0831] Storage method: Built-in memory

[0832] Analysis method: Image recognition system (TensorFlow, OpenCV)

[0833] Extraction method: OCR technology (optical character recognition)

[0834] Information analysis tools: Data analysis algorithms

[0835] Display means: Display

[0836] System Operation

[0837] The user puts on the glasses-type wearable device and takes a picture of the purchased item or receipt. The captured image data is saved in the device's internal memory. The image recognition system then launches and analyzes the image data using TensorFlow and OpenCV. OCR technology is used to extract characters and numbers from the image data and analyze information such as the item's amount, price, and payment method. The analyzed information is then displayed on the screen.

[0838] Specific examples

[0839] For example, consider the case where a user goes shopping at a supermarket and takes a photo of their receipt. The user puts on a glasses-type wearable device and takes a photo of the receipt with the camera. The device saves the captured image data in its internal memory. Next, the device launches its image recognition system and analyzes the image data using TensorFlow and OpenCV. OCR technology is used to extract letters and numbers from the receipt, reading information such as "rice ball 150 yen" and "drink 350 yen." The device analyzes this information and calculates the total amount as "500 yen" and the payment method as "credit card." Finally, the device displays the analysis results on the glasses' display so that the user can confirm them.

[0840] Prompt Sentence Examples

[0841] An example of a prompt to input to a generative AI model is as follows:

[0842] The user puts on the glasses-type wearable device and takes a photo of the receipt for the items purchased at the supermarket. The device analyzes the image, extracts information such as the purchased items, amount, and payment method, and displays it on the display.

[0843] In this way, users can easily check information about the items they have purchased, and their household accounts are automatically created and saved to the cloud.

[0844] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0845] Step 1:

[0846] The user puts on the glasses-type wearable device and takes photos of purchased items and receipts.

[0847] Input: An image taken by the user using the camera.

[0848] Output: High resolution image data.

[0849] How it works: A user uses the camera on their glasses-type wearable device to take a picture of a receipt or purchase, which captures a high-resolution image.

[0850] Step 2:

[0851] The device stores the captured image data in its internal memory.

[0852] Input: High resolution image data captured.

[0853] Output: Image data stored in the internal memory.

[0854] Specific operation: The device stores the captured image data in its internal memory. The stored data is used for later analysis.

[0855] Step 3:

[0856] The device activates the image recognition system and analyzes the image data.

[0857] Input: Image data stored in the internal memory.

[0858] Output: Image data ready for analysis.

[0859] Specific operation: The device launches an image recognition system such as TensorFlow or OpenCV and reads image data from the built-in memory.

[0860] Step 4:

[0861] The device uses OCR technology to extract letters and numbers from the image data.

[0862] Input: Image data ready for analysis.

[0863] Output: Extracted character and number data.

[0864] How it works: The device uses OCR technology to extract letters and numbers from image data, for example, reading information such as "rice ball 150 yen" or "drink 350 yen" on a receipt.

[0865] Step 5:

[0866] The terminal analyzes the extracted letters and numbers to determine the item's amount, price, payment method, etc.

[0867] Input: Extracted character and number data.

[0868] Output: Information such as the parsed item amount, price, and payment method.

[0869] Specific operation: The device analyzes the extracted characters and numbers to determine the price of the item and the payment method. For example, "rice ball 150 yen" and "drink 350 yen" are added together to determine "500 yen."

[0870] Step 6:

[0871] The terminal displays the analysis results to the user.

[0872] Input: Information such as the parsed item amount, price, and payment method.

[0873] Output: The analysis results that are displayed to the user.

[0874] Specific operation: The device displays the analysis results on the glasses' display so that the user can confirm them. For example, it displays "Purchased items: rice ball, drink," "Total amount: 500 yen," and "Payment method: credit card."

[0875] (Application example 1)

[0876] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0877] While conventional glasses-type wearable devices can automatically recognize the amount and payment method of a user's purchases, they lack the functionality to display detailed product information in real time. Furthermore, there are limited ways to efficiently manage purchase history, which requires users to manually enter information.

[0878] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0879] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a barcode recognition means for automatically recognizing a barcode from an image acquired by the camera means, and a display means for displaying detailed information about the item based on the information recognized by the barcode recognition means. This allows a user to check detailed information about purchased items in real time and efficiently manage their purchase history.

[0880] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and a display.

[0881] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[0882] "Image recognition means" refers to technology or equipment for analyzing and recognizing specific information from acquired images.

[0883] "Means for creating a household account book" refers to technology or equipment for automatically creating a household account book based on recognized information.

[0884] "Barcode recognition means" refers to technology or equipment for analyzing and recognizing barcodes from acquired images.

[0885] "Display means" refers to the technology or device used to visually present recognized information to the user.

[0886] The system for implementing this invention includes a camera, an image recognition unit, a household accounting book creation unit, a barcode recognition unit, and a display unit, all of which are provided in a glasses-type wearable device. The specific operation of each unit and the hardware and software used will be described below.

[0887] Camera Means

[0888] The camera means is a camera built into the eyeglass-type wearable device that takes pictures of items purchased by the user and receipts. The camera uses the latest image sensor technology to capture high-resolution images.

[0889] Image Recognition Method

[0890] The image recognition module uses OCR (Optical Character Recognition) technology to automatically recognize the amount, price, and payment method of an item from a captured image. Specifically, it uses the Pytesseract library to extract text from the image. Furthermore, it uses the Pyzbar library to recognize barcodes in the image.

[0891] How to create a household budget

[0892] The household accounting creation means automatically records the user's purchase history based on information recognized by the image recognition means and creates a household accounting record. The created household accounting record is automatically saved in the cloud. Cloud services such as Amazon Web Services (AWS) and Google Cloud Platform (GCP) are used.

[0893] Barcode Recognition Method

[0894] The barcode recognition means automatically recognizes barcodes from images captured by the camera means and obtains detailed information about the item. The Pyzbar library is used to decode the barcodes.

[0895] Display means

[0896] The display of the glasses-type wearable device is used to visually present the recognized information to the user, and by using OLED or MicroLED technology for the display, high resolution and low power consumption are possible.

[0897] Specific examples

[0898] For example, when a user buys apples at a supermarket, they scan the barcode on the apples using the glasses-type wearable device. The price and payment method for the apples are then displayed on the screen. Also, by taking a photo of the receipt, the purchase history is automatically recorded in the household accounting app.

[0899] Example prompts to input to a generative AI model:

[0900] "I want to develop an application that uses smart glasses to display the price and payment methods of items purchased in a physical store in real time. Using Python, write a program to extract text and barcodes from images taken by the camera."

[0901] In this way, users can check detailed information about purchased items in real time and efficiently manage their purchase history.

[0902] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0903] Step 1:

[0904] The user puts on the glasses-type wearable device and takes a picture of the purchased item or receipt with the camera. The input is the image captured by the camera, and the output is high-resolution image data.

[0905] Step 2:

[0906] The terminal sends the acquired image data to the image recognition means, which uses the Pytesseract library to extract text from the image. The input is the image data, and the output is the extracted text data.

[0907] Step 3:

[0908] The terminal analyzes the text data acquired from the image recognition means and identifies the amount, price, and payment method of the item. The input is text data, and the output is information on the amount, price, and payment method of the item.

[0909] Step 4:

[0910] The terminal sends the acquired image data to the barcode recognition means, which uses the Pyzbar library to extract and decode the barcode from the image. The input is the image data, and the output is the barcode data.

[0911] Step 5:

[0912] The terminal acquires detailed information about the product based on the barcode data and sends it to the display means. The input is the barcode data, and the output is detailed information about the product.

[0913] Step 6:

[0914] The display means visually presents the acquired detailed information of the item to the user. The input is the detailed information of the item, and the output is the information displayed on the display.

[0915] Step 7:

[0916] The device automatically creates a household ledger based on information acquired from the image recognition system and stores it on the cloud. The input is the amount, price, and payment method of the item, and the output is the household ledger data stored on the cloud.

[0917] In this way, the user can check detailed information about the purchased items in real time, and can efficiently manage the purchase history.

[0918] Example 2

[0919] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0920] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, they lack the ability to store purchase information on the cloud, making it difficult to back up or share data. Furthermore, they lack support for users to efficiently analyze and understand the contents of their household accounting records.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0922] In this invention, the server includes a means for a user to input purchase information, a means for a terminal to transmit the input information to the server, a means for the server to pass the information to the household accounting system, a means for the household accounting system to analyze the information and record it in the household accounting, and a means for the user to check the household accounting. This allows users to easily create accurate household accounting and store it on the cloud, enabling data backup and sharing. In addition, by using a generative AI model to analyze the contents of the household accounting and generate prompt sentences for the user, the contents of the household accounting can be efficiently understood.

[0923] "Means for users to input purchase information" refers to a device or software that provides an interface for users to input information about the items they have purchased (e.g., date, name of item, amount, payment method, etc.).

[0924] The "means for the terminal to transmit input information to the server" refers to a device or software having a communication function for transmitting the purchase information input by the user to the server.

[0925] The "means for the server to pass information to the household account book creation system" refers to a device or software having a function for transferring the purchase information received by the server to the household account book creation system.

[0926] The "means by which the household accounting system analyzes information and records it in the household accounting book" refers to a device or software that analyzes the received purchase information and automatically records it in the household accounting book in an appropriate format.

[0927] The "means for the user to check the household account book" refers to a device or software that provides an interface for the user to view the contents of the recorded household account book.

[0928] "Means for automatically saving on the cloud" refers to a device or software that has the function of automatically saving information recorded by the household accounting system to cloud storage via the Internet.

[0929] A "means for generating prompt sentences using a generative AI model" is a device or software that uses an artificial intelligence model to analyze information recorded in a household account book and generate appropriate prompt sentences for the user.

[0930] MODE FOR CARRYING OUT THE INVENTION

[0931] This invention is a system that automatically creates and saves a household account book based on purchase information entered by a user. A specific embodiment of this system will be described below.

[0932] Hardware and software used

[0933] Hardware: Servers, devices (PCs, smartphones, etc.)

[0934] Software: Household accounting system, generative AI model

[0935] System configuration

[0936] 1. How users enter their purchase information

[0937] The user uses the terminal to input information about the purchased item, such as the date, the name of the item, the amount, and the payment method.

[0938] For example, if a user purchases a book for 1,000 yen in cash on January 1, 2022, they would enter this information into the input form.

[0939] 2. How the device sends input information to the server

[0940] The terminal sends the information entered by the user to the server. Specifically, it sends the information to the server using an HTTP request.

[0941] For example, the input information "January 1, 2022, book, 1,000 yen, cash" is sent to the server in JSON format.

[0942] 3. How the server passes information to the household accounting system

[0943] The server passes the received information to the household accounting system. Specifically, it calls the API in the server and passes the information to the household accounting system.

[0944] For example, the server passes the information "January 1, 2022, books, 1,000 yen, cash" to the household accounting system.

[0945] 4. How the household accounting system analyzes and records information in the household accounting book

[0946] The household accounting system analyzes the received information and automatically records it in the household accounting book, specifically by storing the information in a database and recording it in the household accounting book in the appropriate format.

[0947] For example, information such as "January 1, 2022, books, 1,000 yen, cash" is saved in a database and recorded in a household ledger.

[0948] 5. How users can check their household accounts

[0949] The user uses the terminal to check the recorded household account book. Specifically, the user views the recorded information through the interface of the household account book creation system.

[0950] For example, a user checks a household account book that contains the following information: "January 1, 2022, books, 1,000 yen, cash."

[0951] 6. Automatically save to the cloud

[0952] The household accounting system automatically saves recorded information to cloud storage, which allows data to be backed up and shared.

[0953] For example, recorded household accounting information is stored on the cloud and can be accessed by users from any device.

[0954] 7. How to generate prompts using a generative AI model

[0955] It uses a generative AI model to analyze the information recorded in the household ledger and generate appropriate prompts for the user.

[0956] For example, by inputting the following prompt sentence into a generative AI model, it can efficiently understand the contents of a household budget.

[0957] If a user purchases a book for 1,000 yen with cash on January 1, 2022, explain how this will be recorded in the household ledger.

[0958] In this way, users can effortlessly create accurate household accounts and store them on the cloud for data backup and sharing. Furthermore, by analyzing the contents of the household account book using a generative AI model and generating prompts for the user, the contents of the household account book can be efficiently understood.

[0959] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0960] Step 1:

[0961] The user enters purchase information.

[0962] Input: Information about the item purchased by the user (e.g., date, item name, amount, payment method)

[0963] Specific operation: The user enters purchase information into the input form on the device. For example, if a user purchases a book for 1,000 yen in cash on January 1, 2022, the user enters "January 1, 2022, book, 1,000 yen, cash."

[0964] Output: The entered purchase information is saved on the device.

[0965] Step 2:

[0966] The terminal sends the input information to the server.

[0967] Input: Purchase information entered by the user

[0968] Specific operation: The device uses an HTTP request to send the entered purchase information to the server. For example, the entered information "January 1, 2022, book, 1,000 yen, cash" is sent to the server in JSON format.

[0969] Output: The server receives the purchase information.

[0970] Step 3:

[0971] The server passes the information to the household accounting system.

[0972] Input: Purchase information sent from the device

[0973] Specific operation: The server calls the internal API to pass the received purchase information to the household accounting system. For example, the server passes the information "January 1, 2022, Books, 1,000 yen, Cash" to the household accounting system.

[0974] Output: The household accounting system receives the purchase information.

[0975] Step 4:

[0976] The household accounting system analyzes the information and records it in the household accounting book.

[0977] Input: Purchase information passed from the server

[0978] Specific operation: The household accounting system analyzes the received information and stores it in a database. For example, information such as "January 1, 2022, Books, 1,000 yen, Cash" is stored in the database and recorded in the household accounting book in the appropriate format.

[0979] Output: Purchase information recorded in the household ledger

[0980] Step 5:

[0981] The user checks the household account book.

[0982] Input: Purchase information recorded in the household ledger

[0983] Specific operation: The user uses the terminal to view the recorded information through the interface of the household accounting system. For example, the user checks the household accounting book that contains the information "January 1, 2022, Books, 1,000 yen, Cash."

[0984] Output: The contents of the household account book checked by the user

[0985] Step 6:

[0986] The household accounting system automatically saves information on the cloud.

[0987] Input: Purchase information recorded in the household ledger

[0988] Specific operation: The household accounting system automatically saves recorded information to cloud storage. For example, the recorded household accounting information is stored on the cloud and users can access it from any device.

[0989] Output: Household accounting information stored on the cloud

[0990] Step 7:

[0991] Generate prompts using a generative AI model.

[0992] Input: Purchase information recorded in the household ledger

[0993] Specific operation: The household accounting system uses the generative AI model to analyze the contents of the household accounting and generate appropriate prompts for the user. For example, it generates the following prompts:

[0994] If a user purchases a book for 1,000 yen with cash on January 1, 2022, explain how this will be recorded in the household ledger.

[0995] Output: Generated prompt statement

[0996] (Application example 2)

[0997] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0998] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, purchase information using electronic payment services cannot be automatically updated in the household accounting system, making it difficult for users to centrally manage their spending across multiple platforms.

[0999] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means that automatically recognizes the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means that automatically creates a household account book based on the information recognized by the image recognition means, a means for acquiring purchase information from an electronic payment service, and a means for automatically recording the acquired purchase information in the household account book. This eliminates the need for the user to manually enter purchase information, and enables the purchase information to be accurately and quickly reflected in the household account book.

[1000] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[1001] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[1002] "Image recognition means" is a technology for automatically identifying specific information from an image acquired by a camera means, and has the function of recognizing the amount, price, payment method, etc. of an item.

[1003] The "household account book creation means" is a system for recording the user's expenditures and automatically creating a household account book based on the information recognized by the image recognition means.

[1004] "Electronic payment service" refers to a payment service conducted over the Internet, and is a payment method used by users when purchasing goods or services online or offline.

[1005] "Purchase Information" refers to detailed information when a user purchases a product or service, including the date, name of the item, amount, and payment method.

[1006] "Cloud" means a data storage and processing service provided over the Internet that allows users to store and access data remotely.

[1007] An embodiment of the present invention will be described.

[1008] First, the user puts on the eyeglass-type wearable device. This device has a built-in camera means that captures images of the product when the user purchases it. The image captured by the camera means is analyzed by image recognition means, and information such as the product amount, price, and payment method is automatically recognized.

[1009] Next, the information recognized by the image recognition means is sent to the household account book creation means. The household account book creation means automatically creates a household account book based on the recognized information and saves it on the cloud. This allows the user to check their spending information anytime, anywhere.

[1010] Furthermore, this system is also linked to an electronic payment service. The server has a means of obtaining purchase information from the electronic payment service and a means of automatically recording the obtained purchase information in the household account book. As a result, information about purchases made by users using the electronic payment service is automatically reflected in the household account book.

[1011] As a specific example, if a user purchases a book for 1,000 yen using an electronic payment service on October 1, 2023, that information will be automatically recorded in the household account book. This saves users the trouble of manually entering the information, and allows them to accurately and quickly update their household account book with purchase information.

[1012] The following hardware and software are used to implement this system:

[1013] Hardware: Glasses-type wearable device, camera means

[1014] Software: Image recognition, household accounting, electronic payment service integration

[1015] It is also possible to create prompts using generative AI models, such as the following:

[1016] Example prompt sentence:

[1017] Create an application that automatically records purchases made by users using an electronic payment service in a household ledger. The household ledger will include the date, item name, amount, and payment method. Use Python to generate a program that retrieves data from the API and records it in a CSV file.

[1018] In this way, the user can create an accurate household account book without much effort.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] A user puts on the eyeglass-type wearable device and purchases a product. At this time, the camera means takes a picture of the product and its price tag. The input is the image captured by the camera means, and the output is image data.

[1022] Step 2:

[1023] The image recognition means of the terminal analyzes the image data acquired by the camera means and automatically recognizes the amount, price, and payment method of the item. The input is image data, and the output is the recognized amount, price, and payment method of the item. Specifically, the system uses an image recognition algorithm to extract text information and identify the item information.

[1024] Step 3:

[1025] The household account book creation means of the terminal automatically creates a household account book based on the information recognized by the image recognition means. The input is the amount, price, and payment method information of the recognized items, and the output is household account book data. Specifically, the recognized information is converted into a specified format and saved in the household account book database.

[1026] Step 4:

[1027] The server obtains purchase information from the electronic payment service. The input is the purchase information obtained from the electronic payment service's API, and the output is the purchase information data. Specifically, the server sends an API request and receives the purchase information as a response.

[1028] Step 5:

[1029] The server automatically records the purchase information it acquires in the household account book. The input is purchase information data, and the output is updated household account book data. Specifically, it analyzes the purchase information and adds it to the existing household account book data.

[1030] Step 6:

[1031] The server saves the household accounting data on the cloud. The input is the updated household accounting data, and the output is the household accounting data saved on the cloud. Specifically, the data is uploaded to a cloud storage service.

[1032] Step 7:

[1033] The user checks the household accounting data stored on the cloud. The input is the household accounting data stored on the cloud, and the output is the household accounting information displayed on the user's device. Specifically, the data is downloaded from cloud storage and displayed on the user interface.

[1034] Example 3

[1035] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1036] Conventional household accounting management systems require users to manually input data, which is time-consuming and prone to input errors. There is also the risk of data being lost if the device breaks down or is lost. Another issue is that it is difficult for users to check their household accounting information while on the go.

[1037] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a voice input means, a voice recognition means for converting voice data acquired by the voice input means into text data, a communication means for transmitting the text data to a cloud server, and a display means for displaying the household account book information stored in the cloud server. This allows the user to manage the household account book without hassle, ensures that data is safely protected even if the device breaks down or is lost, and enables the user to check the household account book information while on the go.

[1038] A "wearable eyeglasses device" is an eyeglasses-type electronic device that can be worn by a user and has the function of acquiring visual and audio information.

[1039] "Camera means" refers to a device for taking images or videos and storing or transmitting them as digital data.

[1040] "Image recognition means" is a technology for automatically extracting and analyzing specific information from captured images and videos.

[1041] The "household account book creation means" is a function for organizing income and expenditure information based on the acquired data and recording it as a household account book.

[1042] "Voice input means" is a device for capturing the user's voice and processing it as digital data.

[1043] "Speech recognition means" is a technology for converting voice data into text data.

[1044] "Communication means" is a function for transmitting data to other devices or servers.

[1045] A "cloud server" is a remote server that stores and processes data over the Internet.

[1046] A "display means" is a device for visually displaying digital data.

[1047] MODE FOR CARRYING OUT THE INVENTION

[1048] This invention relates to a system that uses a glasses-type wearable device to automatically store household accounting information on the cloud, allowing users to check the information anytime, anywhere. Specific embodiments of this system are described below.

[1049] Hardware and Software Configuration

[1050] Glasses-type wearable device

[1051] The user wears a wearable eyeglass device that includes the following hardware and software:

[1052] Camera means: a camera for taking images or videos.

[1053] Audio input means: A microphone for capturing the user's voice.

[1054] Speech recognition tool: Software that converts voice data into text data using the Google Cloud Speech-to-Text API.

[1055] Communication means: A communication module for sending data to a cloud server via the Internet.

[1056] Cloud Server

[1057] The cloud server has the following functions:

[1058] Data receiving means: Receives data transmitted from the terminal.

[1059] Data storage method: Data is stored using Google Cloud Firestore.

[1060] Data display means: Allows users to view household accounting information from any internet-connected device.

[1061] System Operation

[1062] After shopping at the supermarket, a user speaks to the glasses-type wearable device, saying, "Today's shopping is for 200 yen for milk and 150 yen for bread." The device captures this speech and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is then sent to a cloud server. The server receives the data and stores it in Google Cloud Firestore. When the user returns home, they can open a dedicated app on their smartphone to check their shopping information for the day.

[1063] Specific examples

[1064] For example, if a user says, "Today's shopping is 200 yen for milk and 150 yen for bread," the glasses-type wearable device captures this voice and uses voice recognition to convert it into text data: "Today's shopping is 200 yen for milk and 150 yen for bread." This text data is then sent to a cloud server, which receives the data and stores it in Google Cloud Firestore. The user can then open a dedicated app on their smartphone to check the saved household accounting information.

[1065] Example prompts to input to a generative AI model:

[1066] "How can I use a wearable eyeglass device to store my household accounting information in the cloud? Please explain, including the names of specific APIs and cloud services."

[1067] In this way, the user can manage the household account book without hassle, the data is safely protected even if the device breaks down or is lost, and the user can check the information in the household account book even when they are away from home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1068] Step 1:

[1069] The user inputs household account information. The user puts on a glasses-type wearable device and inputs household account information by voice. For example, the user might say, "Today's shopping is 200 yen for milk and 150 yen for bread." The input is voice data, and the output is the voice data itself.

[1070] Step 2:

[1071] The device converts input information into digital data. The device uses a built-in microphone to capture the user's voice. The device then converts the voice data into text data using the Google Cloud Speech-to-Text API. For example, a speech saying "Today's shopping is milk for 200 yen and bread for 150 yen" is converted into text data saying "Today's shopping is milk for 200 yen and bread for 150 yen." The input is voice data, and the output is text data.

[1072] Step 3:

[1073] The device sends digital data to the cloud server. The device then sends the converted text data to the cloud server. The HTTPS protocol is used for transmission to ensure data security. The input is text data, and the output is data sent to the cloud server.

[1074] Step 4:

[1075] The server receives the data and stores it in cloud storage. The server receives the data sent from the device. The received data is stored in Google Cloud Firestore. Firestore is a real-time database, and data is quickly stored and retrieved. The input is text data sent to the cloud server, and the output is data stored in Firestore.

[1076] Step 5:

[1077] A user checks household accounting information stored in the cloud. The user checks household accounting information stored in the cloud using an internet-connected device such as a smartphone or PC. The user can access Google Cloud Firestore through a dedicated application or a web browser and view the stored household accounting data. The input is the data stored in Firestore, and the output is the household accounting information displayed on the user's device.

[1078] (Application example 3)

[1079] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1080] Conventional household accounting management systems require manual input, which is time-consuming for users and prone to input errors. Furthermore, because expenditure information cannot be managed in real time, it is difficult for users to immediately grasp their current spending situation. Furthermore, there is no function to issue an alert if expenditures exceed a certain range, making budget management difficult. To solve these problems, there is a need for a system that can automatically acquire expenditure information and manage household accounting in real time.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1082] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a means for encrypting payment information and saving it on the cloud, a means for displaying the household account book information saved on the cloud in real time, and a means for generating an alert when a certain expenditure amount is exceeded, thereby enabling automatic acquisition and real-time management of expenditure information.

[1083] A "wearable eyeglasses device" is a wearable electronic device in the shape of eyeglasses that can be worn by users on a daily basis.

[1084] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[1085] The "image recognition means" is a technology for automatically identifying and analyzing specific information from an image acquired by the camera means.

[1086] The "household accounting means" is a system for recording and managing a user's income and expenditure based on information recognized by the image recognition means.

[1087] "Payment information" refers to data such as the amount, payment method, date and time incurred when a user purchases a product or service.

[1088] "Encryption" is a technology that converts data into a format that cannot be deciphered by third parties and is used to ensure the security of data.

[1089] "Storage on the cloud" is a technology that stores data on a remote server via the Internet.

[1090] "Real-time display means" refers to technology that instantly displays data stored in the cloud on the user's device.

[1091] An "alert generating means" is a system for notifying a user when certain conditions are met.

[1092] As an embodiment of this invention, a system is constructed that uses a glasses-type wearable device, a cloud storage service, a database, an encryption library, and a generative AI model.

[1093] Hardware and software used

[1094] Hardware: Wearable glasses, smartphones

[1095] Software: Cloud storage services (e.g., AWS S3), databases (e.g., Firebase), encryption libraries (e.g., PyCryptodome), generative AI models (e.g., GPT-4)

[1096] System Overview

[1097] 1. Capture spending information:

[1098] A camera means built into the glasses-type wearable device captures payment information when a user purchases goods or services, for example, by using NFC (Near Field Communication).

[1099] The payment information obtained is temporarily stored on the device.

[1100] 2. Cloud storage of data:

[1101] Payment information is encrypted and uploaded to a cloud storage service (AWS S3).

[1102] Metadata of expenditure information (date, time, amount, category, etc.) is stored in the database (Firebase).

[1103] 3. Real-time household budget display:

[1104] Household accounting information retrieved from the cloud is displayed on the display of a smartphone app or glasses-type device.

[1105] Users can check their spending status in real time.

[1106] 4. Generate spending alerts:

[1107] If a certain amount of spending is exceeded, the application will generate an alert and notify the user.

[1108] Specific examples

[1109] For example, when a user goes shopping at a supermarket and pays through the glasses, the payment information is automatically saved in the cloud. The user can then check their spending in real time on their smartphone app and receive an alert if they exceed their budget.

[1110] Prompt Sentence Examples

[1111] An example of a prompt sentence when using a generative AI model is as follows:

[1112] "How can I capture my grocery shopping information, store it in the cloud, and display it on my phone in real time?"

[1113] In this way, the expenditure management system for the electronic payment service that uses the glasses-type wearable device can make it easier and more efficient for users to manage their expenditures.

[1114] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1115] Step 1:

[1116] Capture spending information

[1117] Input: Payment information when a user purchases a product or service (e.g., amount, payment method, date and time)

[1118] Processing: A camera means built into the glasses-type wearable device captures payment information. Specifically, the payment information is acquired using NFC (Near Field Communication).

[1119] Output: The payment information is temporarily stored on the device.

[1120] Step 2:

[1121] Data encryption

[1122] Input: Payment information stored on your device

[1123] Processing: Encrypt the payment information using a cryptography library (e.g. PyCryptodome).

[1124] Output: Encrypted payment information

[1125] Step 3:

[1126] Data storage in the cloud

[1127] Input: Encrypted payment information

[1128] Processing: Uploading encrypted payment information to a cloud storage service (e.g. AWS S3).

[1129] Output: Payment information stored in the cloud

[1130] Step 4:

[1131] Metadata storage

[1132] Input: Payment information metadata (e.g., date, time, amount, category)

[1133] Process: Store expenditure metadata in a database (e.g. Firebase).

[1134] Output: Metadata stored in a database

[1135] Step 5:

[1136] Real-time household budget display

[1137] Input: Payment information stored in the cloud and metadata stored in a database

[1138] Processing: Display the household accounting information retrieved from the cloud on the smartphone app or the display of the glasses-type device.

[1139] Output: Real-time household accounting information available to users

[1140] Step 6:

[1141] Generate spending alerts

[1142] Input: Metadata of expenditure information stored in the database

[1143] Action: If a certain spending amount is exceeded, the application generates an alert and notifies the user.

[1144] Output: The alert notification that is displayed to the user

[1145] In this way, through the specific operations performed at each step, a system is realized that automatically acquires and manages user expenditure information in real time.

[1146] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1147] "Example 1"

[1148] In one embodiment of the present invention, a user puts on a glasses-type wearable device and takes a photo of the purchased items and receipt with a camera. The captured image is sent to an image recognition system inside the device and analyzed. This analysis automatically extracts information such as the item's amount, price, and payment method. An emotion engine then recognizes the user's emotion and extracts that information. The extracted information is sent to a household accounting system inside the device and automatically recorded in the household accounting book. Specifically, when a user purchases food at a supermarket, they take a photo of the receipt, and the amount, item, and payment method are automatically recognized and recorded in the household accounting book. The emotion engine also recognizes the user's emotion at the time and records that information.

[1149] "Example 2"

[1150] In another embodiment of the present invention, a user puts on a glasses-type wearable device and takes a photo of the purchased item and receipt with a camera. The captured image is sent to an image recognition system inside the device and analyzed. This analysis automatically extracts information such as the item's amount, price, payment method, and purchase date and time. Furthermore, an emotion engine recognizes the user's emotion and extracts that information as well. The extracted information is sent to a household accounting system inside the device and automatically recorded in the household accounting book. Specifically, when a user purchases a book at a bookstore, they take a photo of the receipt, and the amount, item, payment method, and purchase date and time are automatically recognized and recorded in the household accounting book. Furthermore, the emotion engine recognizes the user's emotion at that time and records that information as well.

[1151] "Example 3"

[1152] In yet another embodiment of the present invention, a user wears a glasses-type wearable device and takes a photo of the purchased items and receipt with a camera. The captured image is sent to an image recognition system within the device and analyzed. This analysis automatically extracts information such as the item's amount, price, payment method, and purchase date and time. An emotion engine then recognizes the user's emotion and extracts that information. The extracted information is sent to a household accounting system within the device, where it is automatically recorded in the household accounting system and stored in the cloud. Specifically, when a user purchases clothes from an online shop and takes a photo of the receipt, the amount, item, payment method, and purchase date and time are automatically recognized, recorded in the household accounting system, and stored in the cloud. The emotion engine also recognizes the user's emotion at the time, and this information is also recorded and stored in the cloud.

[1153] The processing flow of each embodiment will be described below.

[1154] "Example 1"

[1155] Step 1: The user puts on the glasses-type wearable device.

[1156] Step 2: The user takes a photo of the purchased item or receipt with their camera.

[1157] Step 3: The captured image is sent to the device's internal image recognition system for analysis. Step 4: Information such as the item's amount, price, and payment method is automatically extracted.

[1158] Step 5: The emotion engine recognizes the user's emotions and extracts their information.

[1159] Step 6: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[1160] "Example 2"

[1161] Step 1: The user puts on the glasses-type wearable device.

[1162] Step 2: The user takes a photo of the purchased item or receipt with their camera.

[1163] Step 3: The captured image is sent to the device's internal image recognition system and analyzed. Step 4: Information such as the item's amount, price, payment method, and purchase date and time is automatically extracted. Step 5: The emotion engine recognizes the user's emotion, and that information is also extracted.

[1164] Step 6: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[1165] "Example 3"

[1166] Step 1: The user puts on the glasses-type wearable device.

[1167] Step 2: The user takes a photo of the purchased item or receipt with their camera.

[1168] Step 3: The captured image is sent to the device's internal image recognition system and analyzed. Step 4: Information such as the item's amount, price, payment method, and purchase date and time is automatically extracted. Step 5: The emotion engine recognizes the user's emotion, and that information is also extracted.

[1169] Step 6: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[1170] Step 7: The recorded information is stored on the cloud.

[1171] Example 1

[1172] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1173] Conventional eyeglass-type wearable devices have the ability to automatically recognize the amount and payment method of purchased items and create a household ledger, but they lack the ability to recognize and record the user's emotions. As a result, they are unable to grasp changes in emotions that accompany a user's purchasing behavior, making it difficult to manage household finances or analyze consumption behavior in more detail. In addition, the automatic recognition of purchase dates and times and automatic saving of data to the cloud are insufficient, making data management and access cumbersome.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1175] In this invention, the server includes a camera mounted on the glasses-type wearable device, an image analysis unit that automatically recognizes the amount, price, and payment method of an item from an image acquired by the camera, a recording unit that automatically creates a household ledger based on the information recognized by the image analysis unit, an emotion analysis unit that recognizes the user's emotions, and a unit that transmits the emotion information recognized by the emotion analysis unit to the recording unit. This allows for recording of changes in emotions accompanying the user's purchasing behavior, enabling more detailed household management and analysis of consumption behavior. Furthermore, adding functions such as automatic recognition of purchase dates and times and automatic storage on the cloud facilitates data management and access.

[1176] A "wearable eyeglasses device" is a device that can be worn by the user and has various sensors, cameras, and computer systems built into the frame.

[1177] "Photographing means" refers to a camera or other image capturing device provided in the glasses-type wearable device, and is a means for capturing images of items purchased by the user, receipts, etc.

[1178] The "image analysis means" is a means for analyzing the image acquired by the photographing means and automatically recognizing information such as the amount, price, and payment method of the item.

[1179] The "recording means" is a means for automatically creating a household ledger based on information recognized by the image analysis means and storing it on the cloud as necessary.

[1180] "Emotion analysis means" refers to a means for analyzing a user's facial expressions and other biometric information to recognize the user's emotions.

[1181] "Emotion information" refers to information about the user's emotions recognized by the emotion analysis means.

[1182] "Purchase date and time" refers to the specific date and time when the user purchased the item, and is information that is automatically recognized by the image analysis means.

[1183] "Cloud" refers to data storage and computing resources provided over the internet, a place for storing information identified by a storage medium.

[1184] This invention relates to a system that uses a glasses-type wearable device to record a user's purchasing behavior and emotions and automatically creates a household account book. Specific embodiments of this system are described below.

[1185] The user puts on a glasses-type wearable device. The device has a camera (photography means) mounted on the frame, which allows the user to take pictures of the items and receipts they have purchased. The captured images are sent to an image analysis system inside the device. This image analysis system uses machine learning libraries such as TensorFlow to analyze the images and automatically extract information such as the item amount, price, and payment method.

[1186] The device also has a built-in emotion analysis system that uses facial recognition libraries such as OpenCV and Dlib to analyze the user's facial expressions and extract emotional information. For example, a smiling face is recognized as "joy."

[1187] The extracted information is sent to the device's internal recording system, which uses a database such as SQLite to manage the information and automatically creates a household ledger. The recognized information is also automatically saved in the cloud, allowing users to access their household ledger anytime, anywhere.

[1188] As a concrete example, imagine a user buys groceries at a supermarket and takes a photo of the receipt. The device sends the image to an image analysis system, which uses TensorFlow to analyze the letters, numbers, and barcodes on the receipt. The analysis results extract the price, item, and payment method of the food. Next, the device uses OpenCV to analyze the user's facial expression, and the emotion analysis system recognizes it as "happiness." All of this information is recorded in a record system using an SQLite database.

[1189] Example prompt sentence:

[1190] "A user wears a glasses-type wearable device and takes a picture of a receipt for food purchased at a supermarket. The device should use TensorFlow to analyze the image of the receipt, automatically recognize the amount, item, and payment method, and record these in an SQLite database. It should also use OpenCV to recognize the user's emotions and record this information as well."

[1191] In this way, the glasses-type wearable device can automatically record the user's purchasing information and emotions and create a household ledger.

[1192] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1193] Step 1:

[1194] The user puts on the glasses-type wearable device.

[1195] Specific operation: The user puts the device on their head and checks that it is working properly. The device starts up and the camera and various sensors become ready.

[1196] Input: User wearing behavior

[1197] Output: Device boot status

[1198] Step 2:

[1199] The user takes a photo of the purchased item or receipt with the camera.

[1200] What happens: A user uses the device's camera to take a picture of a purchase or receipt. For example, they point the camera at a receipt for groceries they bought at a supermarket.

[1201] Input: Image of item or receipt

[1202] Output: Captured image data

[1203] Step 3:

[1204] The device sends the captured image to an image analysis system.

[1205] How it works: The system inside the device acquires the captured image data and sends it to an image analysis system, which uses machine learning libraries such as TensorFlow.

[1206] Input: Photographed image data

[1207] Output: Sending image data to an image analysis system

[1208] Step 4:

[1209] The terminal analyzes the image using an image analysis system to extract information such as the item's amount, price, and payment method.

[1210] Specific operation: The image analysis system analyzes image data and recognizes the letters, numbers, barcodes, etc. on the receipt. For example, it extracts the amount, item name, and payment method from the receipt.

[1211] Input: Image data

[1212] Output: Information such as the amount, price, and payment method of the extracted item

[1213] Step 5:

[1214] The device recognizes the user's emotions using an emotion analysis system.

[1215] How it works: The device's internal emotion analysis system analyzes the user's facial expressions and extracts emotional information. The emotion analysis system uses face recognition libraries such as OpenCV and Dlib. For example, if the user is smiling, it will recognize the emotion as "joy."

[1216] Input: User's facial expression data

[1217] Output: Extracted emotion information

[1218] Step 6:

[1219] The terminal transmits the extracted information to the recording system.

[1220] How it works: The system inside the device sends the information extracted from the image analysis system and emotion analysis system to a recording system, which uses a database such as SQLite.

[1221] Input: Extracted item information and emotion information

[1222] Output: Sending information to a system of record

[1223] Step 7:

[1224] The terminal records the information in the recording system.

[1225] Specific operation: The recording system automatically creates a household account book based on the information received and stores it in the cloud as needed. For example, the system records the amount, items, payment method, and user emotions of food purchased at the supermarket in the household account book.

[1226] Input: Information into the system of record

[1227] Output: Create a household budget and save it to the cloud

[1228] (Application example 1)

[1229] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1230] Conventional eyeglass-type wearable devices have the ability to automatically recognize the amount of purchases and the payment method, and create a household account book, but they are unable to take the user's emotions into account. As a result, they are unable to record and analyze the user's emotional reactions to purchasing behavior and spending. In addition, they lack the ability to store household account information on the cloud, making centralized management and backup of data difficult. To solve these issues, there is a need for a system that recognizes the user's emotions, adds the ability to record information in a household account book, and stores data on the cloud.

[1231] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1232] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for recognizing a user's emotion, and a means for recording the emotion information recognized by the emotion recognition means in the household account book. This makes it possible to record and analyze a user's emotional response to purchasing behavior. Furthermore, storing household account book information on the cloud facilitates centralized data management and backup.

[1233] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera, sensor, and display.

[1234] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[1235] "Image recognition means" is a system for automatically extracting and analyzing specific information from captured images.

[1236] The "household accounting means" is a system for recording and managing a user's income and expenditure based on information extracted by the image recognition means.

[1237] An "emotion recognition means" is a system that analyzes and recognizes emotions from a user's facial expressions, voice, etc.

[1238] The "means for recording emotion information in a household account book" is a system for adding and recording emotion information recognized by the emotion recognition means in a household account book.

[1239] "Means for automatically saving on the cloud" is a system for automatically saving recognized information on a cloud server via the Internet.

[1240] As an embodiment of the present invention, a system using a glasses-type wearable device will be described below. This system includes a glasses-type wearable device worn by a user, a camera means built into the device, an image recognition means, an emotion recognition means, a household accounting means, and a means for storing data on the cloud.

[1241] System Program

[1242] The program of this system operates as follows.

[1243] 1. Camera means: A camera built into the glasses-type wearable device takes images of the items purchased by the user and the receipt.

[1244] 2. Image recognition: The captured image is sent to the device's internal image recognition system (e.g., OpenCV or pytesseract), which automatically extracts information such as the item's amount, price, and payment method.

[1245] 3. Emotion recognition means: At the same time, an emotion recognition system (e.g., EmotionRecognizer) operates to analyze emotions from the user's facial expressions and voice and recognize the user's emotional information.

[1246] 4. Household accounting system: The extracted information and emotion information are sent to a household accounting system (e.g., FinanceManager) and automatically recorded in the household accounting system.

[1247] 5. Cloud storage: Household accounting information is automatically saved to a cloud server via the Internet.

[1248] Hardware and Software

[1249] Hardware: Camera built into a wearable eyeglasses device

[1250] Software: OpenCV (image capture), pytesseract (OCR), EmotionRecognizer (emotion recognition), FinanceManager (household finance management)

[1251] Specific examples

[1252] Consider a scenario where a user buys food at a supermarket and takes a photo of the receipt. The user puts on the smart glasses and takes a photo of the receipt. The amount, item, and payment method information are automatically extracted from the captured image and recorded in a household ledger. The user's emotions are also recognized, and this information is also recorded in the household ledger.

[1253] Example prompt sentence:

[1254] The user puts on the smart glasses and takes a picture of a supermarket receipt. The system automatically extracts information about the amount, item, and payment method from the image and records it in a household ledger. The system also recognizes the user's emotions and records that information in the household ledger.

[1255] In this way, it is possible to record and analyze users' emotional responses to purchasing behavior. Furthermore, by storing household accounting information on the cloud, it becomes easier to centrally manage and back up data.

[1256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1257] Step 1:

[1258] The user puts on the glasses-type wearable device and takes pictures of purchased items and receipts with the camera.

[1259] Input: Physical image of purchase or receipt

[1260] Output: Digital image data

[1261] How it works: The camera built into the glasses-type wearable device takes pictures of items and receipts from the user's point of view and saves them as digital image data.

[1262] Step 2:

[1263] The terminal transmits the captured image to an image recognition means, which automatically recognizes the amount, price and payment method of the item.

[1264] Input: Digital image data

[1265] Output: Extracted text information (amount, item, payment method)

[1266] Specific operation: Image recognition tools (e.g., OpenCV or pytesseract) analyze digital image data and extract text information using OCR technology.

[1267] Step 3:

[1268] The terminal transmits the extracted text information to the household account book creation means, and the household account book is automatically created.

[1269] Input: Extracted text information (amount, item, payment method)

[1270] Output: Household accounting data

[1271] Specific operation: A household account book creation means (for example, FinanceManager) records the user's income and expenditure based on the extracted text information and generates household account book data.

[1272] Step 4:

[1273] The terminal transmits the user's facial expression and voice to the emotion recognition means, and recognizes the user's emotion.

[1274] Input: User facial and voice data

[1275] Output: Emotional information

[1276] Specific operation: The emotion recognition means (e.g., EmotionRecognizer) analyzes the user's facial expressions and voice and recognizes emotional information.

[1277] Step 5:

[1278] The terminal transmits the recognized emotion information to the household account book creating means and records it in the household account book.

[1279] Input: Emotion information

[1280] Output: Updated household accounting data

[1281] Specific operation: The household account book creation means adds the emotional information to the household account book data and generates updated household account book data.

[1282] Step 6:

[1283] The device automatically saves updated household accounting data to the cloud.

[1284] Input: Updated household accounting data

[1285] Output: Household accounting data stored on the cloud

[1286] Specific operation: The household accounting creation means connects to the cloud server via the Internet and saves the updated household accounting data on the cloud.

[1287] Example 2

[1288] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1289] Conventional household accounting systems require users to manually input product information, which is time-consuming and prone to input errors. Furthermore, there is no function to record the user's emotions, meaning that household management cannot reflect the emotions felt at the time of purchase. Furthermore, when data is limited to local storage, there is also the problem of a high risk of data loss or corruption.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1291] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the price, name, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for analyzing the user's emotions, and a means for recording the emotion information recognized by the emotion recognition means in the household account book. This eliminates the need for the user to manually input information, enabling an accurate household account book to be automatically created. Furthermore, since the user's emotion information is also recorded, household management that reflects emotions can be realized. Furthermore, since data is automatically stored on the cloud, the risk of data loss or corruption is reduced.

[1292] A "wearable eyeglasses device" is an eyeglass-type electronic device that can be worn by the user and has a built-in camera and various sensors.

[1293] The "camera means" is a device for taking images, and refers to a camera built into the eyeglass-type wearable device.

[1294] "Image recognition means" refers to software or hardware for automatically extracting specific information from a captured image.

[1295] The "household accounting book creation means" is software or hardware for automatically creating a household accounting book based on the information extracted by the image recognition means.

[1296] The "emotion recognition means" is software or hardware for analyzing the user's emotions and extracting that information.

[1297] "Cloud" is a general term for data storage and processing services provided over the Internet.

[1298] "Amount of Goods" refers to the price of the purchased goods.

[1299] "Name" refers to the name of the purchased item.

[1300] "Payment Method" means the payment method used at the time of purchase.

[1301] "Purchase date and time" refers to the specific date and time when the item was purchased.

[1302] "Emotion information" is data that indicates the emotional state of the user.

[1303] MODE FOR CARRYING OUT THE INVENTION

[1304] The present invention relates to a system that uses a glasses-type wearable device to take photos of items purchased by a user and receipts, and automatically records the information in a household account book. Specific embodiments of this system are described below.

[1305] Wearing a glasses-type wearable device

[1306] The user wears a glasses-type wearable device that is equipped with a camera, an image recognition system, and an emotion engine.

[1307] Taking photos of items and receipts

[1308] Users take pictures of purchases or receipts with the device's camera, and the images are sent to the device's internal processing system.

[1309] Image analysis

[1310] The device uses a built-in image recognition system to analyze the captured image and extract information such as the item's price, name, payment method, and purchase date and time.

[1311] Emotion recognition

[1312] The device uses the emotion engine to analyze the user's emotions, for example, to extract emotional information such as whether the user is happy or sad.

[1313] Sending information

[1314] The extracted information is sent to the device's internal household accounting system, where item information and user emotion information are processed together.

[1315] Recording in a household account book

[1316] The server receives the information sent from the device and records it in a household ledger, which consists of items such as the date, the name of the item, the amount, the payment method, and the user's emotions.

[1317] Data storage

[1318] The server stores the recorded household accounting data on the cloud, where it is stored securely so that users can check it later.

[1319] Specific examples

[1320] For example, if a user purchases a book for 1,000 yen in cash at a bookstore on January 1, 2022, the user takes a photo of the receipt with a glasses-type wearable device. The device uses an image recognition system to extract the information "January 1, 2022, book, 1,000 yen, cash," and uses an emotion engine to recognize the user's emotion as "joy." This information is sent to the device's internal household accounting system. The server receives the information sent from the device and records the data "January 1, 2022, book, 1,000 yen, cash, joy" in the household accounting book. The recorded data is securely stored on the server, allowing the user to view it later.

[1321] Prompt Sentence Examples

[1322] "Describe a system that uses a wearable eyeglass device to take pictures of purchased items and receipts, extracts information using an image recognition system and an emotion engine, and automatically records the information in a household ledger."

[1323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1324] Step 1:

[1325] Wearing a glasses-type wearable device

[1326] The user wears a glasses-type wearable device. This device has a built-in camera, image recognition system, and emotion engine. The input is the user's wearing behavior, and the output is the device's activation status.

[1327] Step 2:

[1328] Taking photos of items and receipts

[1329] The user takes a photo of the purchased item or receipt with the device's camera. The input is the item or receipt to be photographed, and the output is the captured image data. Specifically, the user presses the camera's shutter button.

[1330] Step 3:

[1331] Image analysis

[1332] The device uses a built-in image recognition system to analyze the captured image. The input is the captured image data, and the output is extracted information such as the item's price, name, payment method, and purchase date and time. Specifically, the image recognition algorithm analyzes the image data and extracts text information.

[1333] Step 4:

[1334] Emotion recognition

[1335] The device uses an emotion engine to analyze the user's emotions. The input is the user's facial expression and voice data, and the output is the user's emotional information. Specifically, the emotion recognition algorithm analyzes the facial expression and voice data to identify the emotion.

[1336] Step 5:

[1337] Sending information

[1338] The extracted information is sent to the device's internal household accounting system. The input is item information and user emotion information, and the output is data transmission to the household accounting system. Specifically, the device stores the data in its internal memory and transmits it to the household accounting system.

[1339] Step 6:

[1340] Recording in a household account book

[1341] The server receives the information sent from the device and records it in the household ledger. The input is the item information and emotion information sent from the device, and the output is the household ledger data. Specifically, the server analyzes the received data and records it according to the household ledger format.

[1342] Step 7:

[1343] Data storage

[1344] The server stores the recorded household accounting data on the cloud. The input is the household accounting data, and the output is the data stored on the cloud. Specifically, the server uploads the data to a cloud storage service.

[1345] (Application example 2)

[1346] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1347] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, since the system is unable to record shopping experiences that take into account the user's emotions, it is difficult to evaluate shopping satisfaction or stress. Furthermore, data management on the cloud is inadequate, leading to issues such as data loss and difficulty in accessing it.

[1348] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1349] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for recognizing a user's emotion, and a means for recording the emotion information recognized by the emotion recognition means in the household account book. This makes it possible to automatically record information about items purchased by a user in the household account book, and also to simultaneously record the user's emotion information.

[1350] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[1351] The "camera means" is a device for capturing images and videos, and is built into the eyeglass-type wearable device.

[1352] "Image recognition means" refers to technology or equipment for automatically extracting and analyzing specific information from captured images.

[1353] The "household account book creation means" refers to a technique or device for automatically creating a household account book based on the acquired purchase information.

[1354] "Emotion recognition means" refers to technology or devices that analyze and recognize emotions from a user's facial expressions, voice, etc.

[1355] "Cloud" is a data storage and computing resource service provided over the Internet, allowing users to store and manage data on remote servers.

[1356] As an embodiment of the present invention, a system using a glasses-type wearable device will be described below.

[1357] System configuration

[1358] Hardware

[1359] Glasses-type wearable device: An electronic device in the shape of glasses worn by the user, equipped with a camera and sensors.

[1360] Camera means: A device built into the glasses-type wearable device for taking images and videos.

[1361] software

[1362] Image recognition means: Technology or equipment for automatically extracting and analyzing the amount, price, and payment method of an item from a captured image. Specifically, it uses libraries such as OpenCV and Pytesseract.

[1363] Household accounting means: This refers to the technology or device that automatically creates a household accounting record based on acquired purchase information. It uses a database management system or cloud storage.

[1364] Emotion recognition means: Technology and devices for analyzing and recognizing emotions from the user's facial expressions, voice, etc. Libraries such as EmotionRecognizer are used.

[1365] Processing flow

[1366] 1. Photographing a receipt: The user puts on the glasses-type wearable device and takes a picture of the receipt using the camera.

[1367] 2. Text extraction: Extract text from the captured receipt image using Pytesseract.

[1368] 3. Information analysis: Analyze the extracted text to obtain information such as the item amount, price, payment method, and purchase date and time.

[1369] 4. Emotion Recognition: Take another image and use EmotionRecognizer to recognize the user's emotions.

[1370] 5. Household account book update: Based on the acquired purchase information and emotional information, a household account book is automatically created and saved on the cloud.

[1371] Specific examples

[1372] When a user purchases a book at a bookstore and takes a photo of the receipt with the glasses-type wearable device, the following information is recorded in the household ledger:

[1373] Date: October 1, 2023

[1374] Item: Book

[1375] Price: 1000 yen

[1376] Payment Method: Cash

[1377] Emotion: Joy

[1378] Prompt Sentence Examples

[1379] "Put on the smart glasses and take a photo of the receipt for a book you bought at a bookstore. The application will automatically analyze the purchase information and emotional information and record it in your household account book."

[1380] In this way, information about the items purchased by the user can be automatically recorded in the household ledger, and the user's emotional information can also be recorded at the same time. This makes it easier to evaluate the satisfaction and stress of the shopping experience, and allows for more efficient household management.

[1381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1382] Step 1:

[1383] A user puts on the glasses-type wearable device and takes a picture of a receipt using the camera means. The input is an image of the receipt, and the output is the captured image of the receipt. This image is used in subsequent processing steps.

[1384] Step 2:

[1385] The server receives the captured receipt image and extracts the text using image recognition. Specifically, it uses the Pytesseract library to extract text data from the image. The input is the receipt image, and the output is the extracted text data.

[1386] Step 3:

[1387] The server analyzes the extracted text data and obtains information such as the item amount, price, payment method, and purchase date and time. Specifically, the text data is divided into lines and information is extracted based on specific keywords (e.g., "yen," "cash," etc.). The input is text data, and the output is the analyzed purchase information.

[1388] Step 4:

[1389] The user then takes a photo of their face using the glasses-type wearable device, and their emotions are recognized using the emotion recognition means. Specifically, the EmotionRecognizer library is used to analyze emotions from the facial image. The input is the user's facial image, and the output is the recognized emotion data.

[1390] Step 5:

[1391] The server automatically creates a household account book based on the purchase information and emotion data it acquires. Specifically, the purchase information and emotion data are stored in a database as a single record. The input is the purchase information and emotion data, and the output is updated household account book data.

[1392] Step 6:

[1393] The server saves the updated household accounting data on the cloud. Specifically, the data is saved on a remote server using a cloud storage service. The input is the updated household accounting data, and the output is the household accounting data saved on the cloud.

[1394] In this way, information about the items purchased by the user and emotional information can be automatically recorded in a household account book and managed on the cloud.

[1395] Example 3

[1396] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1397] Conventional household accounting management systems require users to manually input purchase information, which is time-consuming and prone to input errors. There is also the risk of losing household accounting information if the device breaks down or is lost. Furthermore, because household accounting management cannot take into account the user's emotions, it is difficult to perform detailed household accounting that reflects the psychological state at the time of purchase.

[1398] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1399] In this invention, the server includes a photographing means provided in the glasses-type wearable device, an image analysis means for automatically recognizing the price, name, and payment method of the item from the image acquired by the photographing means, a recording means for automatically creating a household account book based on the information recognized by the image analysis means, a storage means for automatically saving the household account book information created by the recording means on the cloud, an emotion analysis means for recognizing the user's emotion from the image acquired by the photographing means, and an emotion recording means for recording the emotion information recognized by the emotion analysis means in the household account book. This saves the user the trouble of manually entering information and prevents input errors. Furthermore, since the information is stored on the cloud, the risk of losing household account book information even if the device breaks down or is lost is reduced. Furthermore, detailed household management that takes the user's emotions into consideration is possible.

[1400] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[1401] "Photographing means" refers to a camera or other photographing device provided in the glasses-type wearable device, which is used to capture images of items purchased by the user and receipts.

[1402] "Image analysis means" refers to software or hardware for automatically recognizing and analyzing information such as the price, name, and payment method of an item from an image acquired by a photographing means.

[1403] The "recording means" refers to a system or device that automatically creates a household ledger based on information recognized by the image analysis means and records that information.

[1404] "Storage means" refers to a system or device for automatically storing the household accounting information created by the recording means on the cloud.

[1405] The "emotion analysis means" refers to software or hardware for recognizing the user's emotions from the image captured by the image capture means and analyzing that information.

[1406] The "emotion recording means" refers to a system or device for recording the emotional information recognized by the emotion analysis means in a household account book.

[1407] "Cloud" refers to remote servers and storage services for storing, managing, and accessing data over the Internet.

[1408] The present invention relates to a system that uses a glasses-type wearable device to automatically store household accounting information on the cloud. Specific embodiments of this system will be described below.

[1409] The user puts on the eyeglass-type wearable device and takes a picture of the purchased item or receipt with the camera. The eyeglass-type wearable device has a built-in camera as a photographing means, and the user captures the image data by taking a picture of the purchased item or receipt.

[1410] The terminal sends the image captured by the image capture means to the image analysis means, which uses image recognition software such as Google Cloud Vision API, and automatically extracts information such as the price, name, payment method, and purchase date and time of the item.

[1411] Furthermore, the device recognizes the user's emotions using emotion analysis means, such as Microsoft Azure's Emotion API, to analyze the emotions the user felt when making a purchase.

[1412] The extracted information is sent to a recording means, such as Microsoft Excel API or Google Sheets API, which automatically creates a household ledger based on the extracted information and records emotional information.

[1413] The recorded household accounting information is automatically saved to the cloud using a storage method, such as a cloud storage service like Amazon S3 or Google Drive. This allows users to check their household accounting information anytime, anywhere. Furthermore, even if the device breaks down or is lost, the household accounting information saved to the cloud is safely protected.

[1414] For example, when a user purchases clothes from an online shop and takes a photo of the receipt, the amount, item name, payment method, and purchase date and time are automatically recognized, recorded in a household account book, and saved on the cloud. Furthermore, the user's emotions at the time are recognized by emotion analysis means, and this information is also recorded and saved on the cloud.

[1415] Example prompt sentence:

[1416] "Please explain how when a user puts on a wearable eyeglass device and takes pictures of purchased items and receipts with the camera, the information is recorded in a household ledger and stored in the cloud."

[1417] In this way, the present invention provides a system that saves the user time and effort, prevents input errors, and enables detailed household management.The flow of the identification process in the third embodiment will be described with reference to FIG.

[1418] Step 1:

[1419] The user puts on the glasses-type wearable device and takes pictures of purchased items and receipts with the camera.

[1420] Input: Purchases and receipts

[1421] Output: Captured image data

[1422] Specific operation: The user uses the camera of the glasses-type wearable device to take a picture of the purchased items and receipt. For example, after shopping at a supermarket, the user points the camera at the receipt and takes a picture.

[1423] Step 2:

[1424] The terminal transmits the captured image to the image analysis means.

[1425] Input: Photographed image data

[1426] Output: Sending image data to image analysis means

[1427] Specific operation: The terminal uses an internal communication module to send captured image data to the image analysis means. For example, the terminal sends image data using Wi-Fi or Bluetooth.

[1428] Step 3:

[1429] The terminal analyzes the image using an image analysis means and extracts information such as the price of the item, the item name, the payment method, and the date and time of purchase.

[1430] Input: Image data

[1431] Output: Extracted information (item price, item name, payment method, purchase date and time)

[1432] How it works: The device uses image recognition software such as Google Cloud Vision API to analyze image data. For example, it can extract information from a receipt, such as "Milk 200 yen," "Payment method: credit card," and "Purchase date: October 1, 2023."

[1433] Step 4:

[1434] The terminal recognizes the user's emotions using emotion analysis means and extracts that information.

[1435] Input: User's face image data

[1436] Output: Extracted emotion information

[1437] How it works: The device uses emotion recognition software, such as Microsoft Azure's Emotion API, to analyze the user's facial image data. For example, the device may recognize the user's emotion as "joy" while shopping.

[1438] Step 5:

[1439] The terminal transmits the extracted information to the recording means and records it in the household account book.

[1440] Input: Extracted information (item price, item name, payment method, purchase date and time, emotional information)

[1441] Output: Information recorded in the household ledger

[1442] Specific operation: The device uses the Microsoft Excel API, Google Sheets API, etc. to record the extracted information in a household ledger. For example, information such as "Milk 200 yen," "Payment method: Credit card," "Purchase date: October 1, 2023," and "Emotion: Joy" is recorded in the household ledger.

[1443] Step 6:

[1444] The terminal stores the recorded household account book information on the cloud using a storage means.

[1445] Input: Information recorded in the household ledger

[1446] Output: Household accounting information stored on the cloud

[1447] Specific operation: The device uses cloud storage services such as Amazon S3 and Google Drive to store the household accounting information in the cloud. For example, the household accounting information is uploaded to the cloud storage.

[1448] Step 7:

[1449] The server manages the household accounting information on the cloud, allowing users to access it anytime, anywhere.

[1450] Input: Household accounting information stored on the cloud

[1451] Output: User-accessible household accounting information

[1452] How it works: The server manages the cloud storage service and allows users to access their household accounting information from their smartphones or PCs. For example, users can check their household accounting information through a dedicated app or web browser.

[1453] (Application example 3)

[1454] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1455] Conventional household accounting management systems require users to manually enter purchase and expenditure information, which is time-consuming and prone to input errors. Furthermore, they are unable to take into account the user's emotions, making it difficult to create a household accounting system that reflects their emotions and satisfaction at the time of purchase. Furthermore, data storage is limited to local storage, which poses the risk of data loss if the device breaks down or is lost.

[1456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1457] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of items from images acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, an emotion recognition means for recognizing the user's emotions, and a means for automatically saving the recognized information and emotion information on the cloud. This saves the user the trouble of manually inputting purchase and expenditure information, enables emotion-based household account management, and safely protects data even if the device breaks down or is lost.

[1458] A "wearable eyeglasses device" is an electronic device in the form of glasses that, when worn by the user, acquires visual information and processes and displays various data.

[1459] The "camera means" is a device for capturing images and videos, and is a camera built into the eyeglass-type wearable device.

[1460] "Image recognition means" is a technology for automatically extracting and analyzing specific information from captured images.

[1461] The "household account book creation means" is a system for automatically creating a household account book based on the acquired expenditure information.

[1462] "Emotion recognition means" is a technology for analyzing and recognizing emotions from a user's facial expressions, voice, etc.

[1463] "Means for automatic storage on the cloud" refers to technology for automatically storing acquired data on a remote server via the Internet.

[1464] "Amount of Goods" refers to the price of the purchased goods.

[1465] "Price" refers to the cost of a good or service.

[1466] "Payment method" refers to the payment method used at the time of purchase (e.g., cash, credit card, electronic money, etc.).

[1467] "Purchase date and time" refers to the specific date and time when a good or service was purchased.

[1468] The embodiments of the present invention will be described in detail below.

[1469] System Overview

[1470] This system uses a wearable eyeglasses-type device to take photos of items purchased by the user and receipts, automatically records the information in a household ledger, and stores it in the cloud. It also recognizes the user's emotions and records that information.

[1471] Hardware and software used

[1472] Hardware:

[1473] Glasses-type wearable device (camera, image recognition system, emotion recognition system)

[1474] software:

[1475] OpenCV (image processing library)

[1476] Tesseract OCR (Optical Character Recognition Library)

[1477] cloud_storage_api (Cloud Storage API)

[1478] emotion_recognition_api (emotion recognition API)

[1479] Data processing and calculation

[1480] Camera means:

[1481] The camera built into the glasses-type wearable device takes pictures of the items and receipts purchased by the user, and the images are sent to the device's internal image recognition system.

[1482] Image Recognition Methods:

[1483] The image recognition system uses OpenCV and Tesseract OCR to automatically extract information from captured images, such as the item's amount, price, payment method, and purchase date and time, saving users the trouble of manually entering this information.

[1484] Emotion recognition means:

[1485] The emotion recognition system uses emotion_recognition_api to analyze and recognize emotions from the user's facial expressions and voice. This information is also recorded in the household ledger.

[1486] How to create a household budget:

[1487] The extracted information and emotional information are sent to a household accounting system and automatically recorded in the household accounting book, enabling users to manage their household finances in a way that reflects their emotions and satisfaction at the time of purchase.

[1488] Cloud storage options:

[1489] The recognized information and emotion information are automatically stored in the cloud using cloud_storage_api, which ensures data security even in the event of device failure or loss.

[1490] Specific examples

[1491] When a user purchases clothing at a physical store and takes a photo of the receipt with the glasses-type wearable device, the amount, item, payment method, and purchase date and time are automatically recognized and recorded in a household ledger. The device also recognizes the user's emotions at the time (e.g., happiness) and records this information. This information is stored in the cloud, and the user can check the household ledger information anytime, anywhere via a smartphone app.

[1492] Prompt Sentence Examples

[1493] When shopping in a physical store, users put on the glasses-type wearable device and take pictures of the items they purchased and the receipt. The captured image is analyzed by the device's internal image recognition system, and information such as the amount, item, payment method, and purchase date and time is automatically extracted. An emotion engine also recognizes the user's emotions and records this information. The extracted information is automatically recorded in a household account book and saved in the cloud. Users can check their household account book information anytime, anywhere via a smartphone app.

[1494] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1495] Step 1:

[1496] The user puts on the glasses-type wearable device and takes pictures of purchased items and receipts with the camera.

[1497] Input: Items purchased by the user and receipts

[1498] Output: Captured image file

[1499] Specific operation: The camera in the glasses-type wearable device captures an image and stores it in the internal memory.

[1500] Step 2:

[1501] The device sends the captured image to an image recognition system.

[1502] Input: Captured image file

[1503] Output: Input data to the image recognition system

[1504] Specific operation: The device transfers the image file to the image recognition system.

[1505] Step 3:

[1506] The image recognition system uses OpenCV and Tesseract OCR to extract information from the image, such as the item's amount, price, payment method, and purchase date and time.

[1507] Input: Input data to the image recognition system

[1508] Output: Extracted text information (item amount, price, payment method, purchase date and time)

[1509] How it works: The image recognition system analyzes the image and extracts text information.

[1510] Step 4:

[1511] The device transmits the user's facial expressions and voice to the emotion recognition system.

[1512] Input: User's facial expressions and voice data

[1513] Output: Input data to the emotion recognition system

[1514] Specific operation: The device captures the user's facial expressions and voice and transmits them to the emotion recognition system.

[1515] Step 5:

[1516] The emotion recognition system uses emotion_recognition_api to analyze and recognize the user's emotions.

[1517] Input: Input data to the emotion recognition system

[1518] Output: Recognized emotion information

[1519] Specific operation: The emotion recognition system analyzes facial expressions and voice data to recognize emotions.

[1520] Step 6:

[1521] The terminal transmits the extracted text information and the recognized emotion information to a household accounting system.

[1522] Input: Extracted text information, recognized emotion information

[1523] Output: Input data to the household accounting system

[1524] Specific operation: The terminal transfers text information and emotional information to the household accounting system.

[1525] Step 7:

[1526] The household accounting system automatically creates a household accounting book based on the received information.

[1527] Input: Data to be input into the household accounting system

[1528] Output: Automatically generated household accounting data

[1529] Specific operation: The household accounting system integrates text information and emotional information to generate a household accounting book.

[1530] Step 8:

[1531] The server stores the household accounting data on the cloud.

[1532] Input: Automatically generated household accounting data

[1533] Output: Household accounting data stored on the cloud

[1534] Specific operation: The server uploads the household accounting data to the cloud using the cloud storage API.

[1535] Step 9:

[1536] Users can check their household accounting data on the cloud through a smartphone app.

[1537] Input: Household accounting data stored on the cloud

[1538] Output: Household accounting information displayed on a smartphone app

[1539] Specific operation: The user launches the smartphone app, retrieves household accounting data from the cloud, and displays it.

[1540] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1542] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1543] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1544] [Third embodiment]

[1545] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1546] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1547] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1548] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1549] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1551] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1552] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1553] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1554] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1555] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1556] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1557] "Example 1"

[1558] The eyeglass-type wearable device of the present invention has a camera mounted on the frame that takes a picture of the item purchased by the user. The captured image is analyzed by a built-in image recognition system, and information such as the item's amount, price, and payment method is automatically extracted. For example, if a receipt is photographed, the necessary information can be extracted from the letters, numbers, barcodes, etc. on the receipt.

[1559] "Example 2"

[1560] The extracted information is automatically recorded in a household ledger by the built-in household ledger creation system. The household ledger consists of items such as the date, item name, amount, and payment method, and is saved so that the user can check it later. For example, if you purchase a book for 1,000 yen with cash on January 1, 2022, the information recorded in the household ledger will be "January 1, 2022, book, 1,000 yen, cash."

[1561] "Example 3"

[1562] Furthermore, the glasses-type wearable device of the present invention also has the function of automatically saving household accounting information to the cloud, allowing users to check their household accounting information anytime, anywhere. Furthermore, even if the device breaks down or is lost, the household accounting information saved in the cloud is safely protected.

[1563] The processing flow of each embodiment will be described below.

[1564] "Example 1"

[1565] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[1566] Step 2: The captured image is sent to the device's internal image recognition system and analyzed.

[1567] This analysis automatically extracts information such as the item's amount, price, and payment method.

[1568] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[1569] "Example 2"

[1570] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[1571] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[1572] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[1573] Step 4: The recorded household accounting information is stored internally on the device for the user to review later.

[1574] "Example 3"

[1575] Step 1: The user puts on the glasses-type wearable device and takes a photo of the purchased item or receipt with the camera.

[1576] Step 2: The captured image is sent to the device's internal image recognition system for analysis, which automatically extracts information such as the item's value, price, and payment method.

[1577] Step 3: The extracted information is sent to the device's internal accounting system and automatically recorded in the accounting book.

[1578] Step 4: The recorded household accounting information is automatically saved on the cloud, allowing users to check their household accounting information anytime, anywhere.

[1579] Example 1

[1580] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1581] Conventional glasses-type wearable devices require users to manually enter information about purchased items, which is time-consuming for the user. Furthermore, there is a lack of systems for accurately managing purchase information, making it difficult to automatically create a household ledger or save it in the cloud. This makes it difficult for users to efficiently manage their purchase history, making household management cumbersome.

[1582] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1583] In this invention, the server includes a photographing means, a storage means for storing images acquired by the photographing means, an analysis means for analyzing the images stored in the storage means, an extraction means for extracting letters and numbers using optical character recognition technology by the analysis means, an information analysis means for analyzing the amount, price, and payment method of the item from the information extracted by the extraction means, and a display means for displaying the information analyzed by the information analysis means. This allows the user to automatically obtain information on purchased items and create a household account book and store it in the cloud.

[1584] The "photography means" is a device such as a camera for taking pictures of items purchased by the user and receipts.

[1585] The "storage means" refers to a memory or storage device for storing image data acquired by the imaging means.

[1586] The "analysis means" refers to software or hardware for analyzing the image data stored in the storage means.

[1587] The "extraction means" is a function that allows the analysis means to extract characters and numbers from image data using optical character recognition technology.

[1588] The "information analysis means" is a function for analyzing information such as the amount, price, and payment method of the item from the characters and numbers extracted by the extraction means.

[1589] The "display means" is a display or monitor for displaying the information analyzed by the information analysis means to the user.

[1590] The "cloud" refers to remote servers and storage services for storing and managing data over the Internet.

[1591] MODE FOR CARRYING OUT THE INVENTION

[1592] This invention is a system that uses a glasses-type wearable device to capture images of items and receipts purchased by a user, analyzes the image data, and automatically extracts information such as the item amount, price, payment method, etc. Specific embodiments of this system are described below.

[1593] Hardware and Software Configuration

[1594] The glasses-type wearable device worn by the user includes the following hardware and software:

[1595] Photography method: High-resolution camera

[1596] Storage method: Built-in memory

[1597] Analysis method: Image recognition system (TensorFlow, OpenCV)

[1598] Extraction method: OCR technology (optical character recognition)

[1599] Information analysis tools: Data analysis algorithms

[1600] Display means: Display

[1601] System Operation

[1602] The user puts on the glasses-type wearable device and takes a picture of the purchased item or receipt. The captured image data is saved in the device's internal memory. The image recognition system then launches and analyzes the image data using TensorFlow and OpenCV. OCR technology is used to extract characters and numbers from the image data and analyze information such as the item's amount, price, and payment method. The analyzed information is then displayed on the screen.

[1603] Specific examples

[1604] For example, consider the case where a user goes shopping at a supermarket and takes a photo of their receipt. The user puts on a glasses-type wearable device and takes a photo of the receipt with the camera. The device saves the captured image data in its internal memory. Next, the device launches its image recognition system and analyzes the image data using TensorFlow and OpenCV. OCR technology is used to extract letters and numbers from the receipt, reading information such as "rice ball 150 yen" and "drink 350 yen." The device analyzes this information and calculates the total amount as "500 yen" and the payment method as "credit card." Finally, the device displays the analysis results on the glasses' display so that the user can confirm them.

[1605] Prompt Sentence Examples

[1606] An example of a prompt to input to a generative AI model is as follows:

[1607] The user puts on the glasses-type wearable device and takes a photo of the receipt for the items purchased at the supermarket. The device analyzes the image, extracts information such as the purchased items, amount, and payment method, and displays it on the display.

[1608] In this way, users can easily check information about the items they have purchased, and their household accounts are automatically created and saved to the cloud.

[1609] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1610] Step 1:

[1611] The user puts on the glasses-type wearable device and takes photos of purchased items and receipts.

[1612] Input: An image taken by the user using the camera.

[1613] Output: High resolution image data.

[1614] How it works: A user uses the camera on their glasses-type wearable device to take a picture of a receipt or purchase, which captures a high-resolution image.

[1615] Step 2:

[1616] The device stores the captured image data in its internal memory.

[1617] Input: High resolution image data captured.

[1618] Output: Image data stored in the internal memory.

[1619] Specific operation: The device stores the captured image data in its internal memory. The stored data is used for later analysis.

[1620] Step 3:

[1621] The device activates the image recognition system and analyzes the image data.

[1622] Input: Image data stored in the internal memory.

[1623] Output: Image data ready for analysis.

[1624] Specific operation: The device launches an image recognition system such as TensorFlow or OpenCV and reads image data from the built-in memory.

[1625] Step 4:

[1626] The device uses OCR technology to extract letters and numbers from the image data.

[1627] Input: Image data ready for analysis.

[1628] Output: Extracted character and number data.

[1629] How it works: The device uses OCR technology to extract letters and numbers from image data, for example, reading information such as "rice ball 150 yen" or "drink 350 yen" on a receipt.

[1630] Step 5:

[1631] The terminal analyzes the extracted letters and numbers to determine the item's amount, price, payment method, etc.

[1632] Input: Extracted character and number data.

[1633] Output: Information such as the parsed item amount, price, and payment method.

[1634] Specific operation: The device analyzes the extracted characters and numbers to determine the price of the item and the payment method. For example, "rice ball 150 yen" and "drink 350 yen" are added together to determine "500 yen."

[1635] Step 6:

[1636] The terminal displays the analysis results to the user.

[1637] Input: Information such as the parsed item amount, price, and payment method.

[1638] Output: The analysis results that are displayed to the user.

[1639] Specific operation: The device displays the analysis results on the glasses' display so that the user can confirm them. For example, it displays "Purchased items: rice ball, drink," "Total amount: 500 yen," and "Payment method: credit card."

[1640] (Application example 1)

[1641] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1642] While conventional glasses-type wearable devices can automatically recognize the amount and payment method of a user's purchases, they lack the functionality to display detailed product information in real time. Furthermore, there are limited ways to efficiently manage purchase history, which requires users to manually enter information.

[1643] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1644] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a barcode recognition means for automatically recognizing a barcode from an image acquired by the camera means, and a display means for displaying detailed information about the item based on the information recognized by the barcode recognition means. This allows a user to check detailed information about purchased items in real time and efficiently manage their purchase history.

[1645] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and a display.

[1646] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[1647] "Image recognition means" refers to technology or equipment for analyzing and recognizing specific information from acquired images.

[1648] "Means for creating a household account book" refers to technology or equipment for automatically creating a household account book based on recognized information.

[1649] "Barcode recognition means" refers to technology or equipment for analyzing and recognizing barcodes from acquired images.

[1650] "Display means" refers to the technology or device used to visually present recognized information to the user.

[1651] The system for implementing this invention includes a camera, an image recognition unit, a household accounting book creation unit, a barcode recognition unit, and a display unit, all of which are provided in a glasses-type wearable device. The specific operation of each unit and the hardware and software used will be described below.

[1652] Camera Means

[1653] The camera means is a camera built into the eyeglass-type wearable device that takes pictures of items purchased by the user and receipts. The camera uses the latest image sensor technology to capture high-resolution images.

[1654] Image Recognition Method

[1655] The image recognition module uses OCR (Optical Character Recognition) technology to automatically recognize the amount, price, and payment method of an item from a captured image. Specifically, it uses the Pytesseract library to extract text from the image. Furthermore, it uses the Pyzbar library to recognize barcodes in the image.

[1656] How to create a household budget

[1657] The household accounting creation means automatically records the user's purchase history based on information recognized by the image recognition means and creates a household accounting record. The created household accounting record is automatically saved in the cloud. Cloud services such as Amazon Web Services (AWS) and Google Cloud Platform (GCP) are used.

[1658] Barcode Recognition Method

[1659] The barcode recognition means automatically recognizes barcodes from images captured by the camera means and obtains detailed information about the item. The Pyzbar library is used to decode the barcodes.

[1660] Display means

[1661] The display of the glasses-type wearable device is used to visually present the recognized information to the user, and by using OLED or MicroLED technology for the display, high resolution and low power consumption are possible.

[1662] Specific examples

[1663] For example, when a user buys apples at a supermarket, they scan the barcode on the apples using the glasses-type wearable device. The price and payment method for the apples are then displayed on the screen. Also, by taking a photo of the receipt, the purchase history is automatically recorded in the household accounting app.

[1664] Example prompts to input to a generative AI model:

[1665] "I want to develop an application that uses smart glasses to display the price and payment methods of items purchased in a physical store in real time. Using Python, write a program to extract text and barcodes from images taken by the camera."

[1666] In this way, users can check detailed information about purchased items in real time and efficiently manage their purchase history.

[1667] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1668] Step 1:

[1669] The user puts on the glasses-type wearable device and takes a picture of the purchased item or receipt with the camera. The input is the image captured by the camera, and the output is high-resolution image data.

[1670] Step 2:

[1671] The terminal sends the acquired image data to the image recognition means, which uses the Pytesseract library to extract text from the image. The input is the image data, and the output is the extracted text data.

[1672] Step 3:

[1673] The terminal analyzes the text data acquired from the image recognition means and identifies the amount, price, and payment method of the item. The input is text data, and the output is information on the amount, price, and payment method of the item.

[1674] Step 4:

[1675] The terminal sends the acquired image data to the barcode recognition means, which uses the Pyzbar library to extract and decode the barcode from the image. The input is the image data, and the output is the barcode data.

[1676] Step 5:

[1677] The terminal acquires detailed information about the product based on the barcode data and sends it to the display means. The input is the barcode data, and the output is detailed information about the product.

[1678] Step 6:

[1679] The display means visually presents the acquired detailed information of the item to the user. The input is the detailed information of the item, and the output is the information displayed on the display.

[1680] Step 7:

[1681] The device automatically creates a household ledger based on information acquired from the image recognition system and stores it on the cloud. The input is the amount, price, and payment method of the item, and the output is the household ledger data stored on the cloud.

[1682] In this way, the user can check detailed information about the purchased items in real time, and can efficiently manage the purchase history.

[1683] Example 2

[1684] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1685] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, they lack the ability to store purchase information on the cloud, making it difficult to back up or share data. Furthermore, they lack support for users to efficiently analyze and understand the contents of their household accounting records.

[1686] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1687] In this invention, the server includes a means for a user to input purchase information, a means for a terminal to transmit the input information to the server, a means for the server to pass the information to the household accounting system, a means for the household accounting system to analyze the information and record it in the household accounting, and a means for the user to check the household accounting. This allows users to easily create accurate household accounting and store it on the cloud, enabling data backup and sharing. In addition, by using a generative AI model to analyze the contents of the household accounting and generate prompt sentences for the user, the contents of the household accounting can be efficiently understood.

[1688] "Means for users to input purchase information" refers to a device or software that provides an interface for users to input information about the items they have purchased (e.g., date, name of item, amount, payment method, etc.).

[1689] The "means for the terminal to transmit input information to the server" refers to a device or software having a communication function for transmitting the purchase information input by the user to the server.

[1690] The "means for the server to pass information to the household account book creation system" refers to a device or software having a function for transferring the purchase information received by the server to the household account book creation system.

[1691] The "means by which the household accounting system analyzes information and records it in the household accounting book" refers to a device or software that analyzes the received purchase information and automatically records it in the household accounting book in an appropriate format.

[1692] The "means for the user to check the household account book" refers to a device or software that provides an interface for the user to view the contents of the recorded household account book.

[1693] "Means for automatically saving on the cloud" refers to a device or software that has the function of automatically saving information recorded by the household accounting system to cloud storage via the Internet.

[1694] A "means for generating prompt sentences using a generative AI model" is a device or software that uses an artificial intelligence model to analyze information recorded in a household account book and generate appropriate prompt sentences for the user.

[1695] MODE FOR CARRYING OUT THE INVENTION

[1696] This invention is a system that automatically creates and saves a household account book based on purchase information entered by a user. A specific embodiment of this system will be described below.

[1697] Hardware and software used

[1698] Hardware: Servers, devices (PCs, smartphones, etc.)

[1699] Software: Household accounting system, generative AI model

[1700] System configuration

[1701] 1. How users enter their purchase information

[1702] The user uses the terminal to input information about the purchased item, such as the date, the name of the item, the amount, and the payment method.

[1703] For example, if a user purchases a book for 1,000 yen in cash on January 1, 2022, they would enter this information into the input form.

[1704] 2. How the device sends input information to the server

[1705] The terminal sends the information entered by the user to the server. Specifically, it sends the information to the server using an HTTP request.

[1706] For example, the input information "January 1, 2022, book, 1,000 yen, cash" is sent to the server in JSON format.

[1707] 3. How the server passes information to the household accounting system

[1708] The server passes the received information to the household accounting system. Specifically, it calls the API in the server and passes the information to the household accounting system.

[1709] For example, the server passes the information "January 1, 2022, books, 1,000 yen, cash" to the household accounting system.

[1710] 4. How the household accounting system analyzes and records information in the household accounting book

[1711] The household accounting system analyzes the received information and automatically records it in the household accounting book, specifically by storing the information in a database and recording it in the household accounting book in the appropriate format.

[1712] For example, information such as "January 1, 2022, books, 1,000 yen, cash" is saved in a database and recorded in a household ledger.

[1713] 5. How users can check their household accounts

[1714] The user uses the terminal to check the recorded household account book. Specifically, the user views the recorded information through the interface of the household account book creation system.

[1715] For example, a user checks a household account book that contains the following information: "January 1, 2022, books, 1,000 yen, cash."

[1716] 6. Automatically save to the cloud

[1717] The household accounting system automatically saves recorded information to cloud storage, which allows data to be backed up and shared.

[1718] For example, recorded household accounting information is stored on the cloud and can be accessed by users from any device.

[1719] 7. How to generate prompts using a generative AI model

[1720] It uses a generative AI model to analyze the information recorded in the household ledger and generate appropriate prompts for the user.

[1721] For example, by inputting the following prompt sentence into a generative AI model, it can efficiently understand the contents of a household budget.

[1722] If a user purchases a book for 1,000 yen with cash on January 1, 2022, explain how this will be recorded in the household ledger.

[1723] In this way, users can effortlessly create accurate household accounts and store them on the cloud for data backup and sharing. Furthermore, by analyzing the contents of the household account book using a generative AI model and generating prompts for the user, the contents of the household account book can be efficiently understood.

[1724] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1725] Step 1:

[1726] The user enters purchase information.

[1727] Input: Information about the item purchased by the user (e.g., date, item name, amount, payment method)

[1728] Specific operation: The user enters purchase information into the input form on the device. For example, if a user purchases a book for 1,000 yen in cash on January 1, 2022, the user enters "January 1, 2022, book, 1,000 yen, cash."

[1729] Output: The entered purchase information is saved on the device.

[1730] Step 2:

[1731] The terminal sends the input information to the server.

[1732] Input: Purchase information entered by the user

[1733] Specific operation: The device uses an HTTP request to send the entered purchase information to the server. For example, the entered information "January 1, 2022, book, 1,000 yen, cash" is sent to the server in JSON format.

[1734] Output: The server receives the purchase information.

[1735] Step 3:

[1736] The server passes the information to the household accounting system.

[1737] Input: Purchase information sent from the device

[1738] Specific operation: The server calls the internal API to pass the received purchase information to the household accounting system. For example, the server passes the information "January 1, 2022, Books, 1,000 yen, Cash" to the household accounting system.

[1739] Output: The household accounting system receives the purchase information.

[1740] Step 4:

[1741] The household accounting system analyzes the information and records it in the household accounting book.

[1742] Input: Purchase information passed from the server

[1743] Specific operation: The household accounting system analyzes the received information and stores it in a database. For example, information such as "January 1, 2022, Books, 1,000 yen, Cash" is stored in the database and recorded in the household accounting book in the appropriate format.

[1744] Output: Purchase information recorded in the household ledger

[1745] Step 5:

[1746] The user checks the household account book.

[1747] Input: Purchase information recorded in the household ledger

[1748] Specific operation: The user uses the terminal to view the recorded information through the interface of the household accounting system. For example, the user checks the household accounting book that contains the information "January 1, 2022, Books, 1,000 yen, Cash."

[1749] Output: The contents of the household account book checked by the user

[1750] Step 6:

[1751] The household accounting system automatically saves information on the cloud.

[1752] Input: Purchase information recorded in the household ledger

[1753] Specific operation: The household accounting system automatically saves recorded information to cloud storage. For example, the recorded household accounting information is stored on the cloud and users can access it from any device.

[1754] Output: Household accounting information stored on the cloud

[1755] Step 7:

[1756] Generate prompts using a generative AI model.

[1757] Input: Purchase information recorded in the household ledger

[1758] Specific operation: The household accounting system uses the generative AI model to analyze the contents of the household accounting and generate appropriate prompts for the user. For example, it generates the following prompts:

[1759] If a user purchases a book for 1,000 yen with cash on January 1, 2022, explain how this will be recorded in the household ledger.

[1760] Output: Generated prompt statement

[1761] (Application example 2)

[1762] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1763] Conventional household accounting systems require users to manually enter purchase information, which is time-consuming and prone to input errors. Furthermore, purchase information using electronic payment services cannot be automatically updated in the household accounting system, making it difficult for users to centrally manage their spending across multiple platforms.

[1764] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means that automatically recognizes the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means that automatically creates a household account book based on the information recognized by the image recognition means, a means for acquiring purchase information from an electronic payment service, and a means for automatically recording the acquired purchase information in the household account book. This eliminates the need for the user to manually enter purchase information, and enables the purchase information to be accurately and quickly reflected in the household account book.

[1765] A "wearable eyeglasses device" is an electronic device in the shape of glasses that can be worn by the user and has built-in functions such as a camera and sensors.

[1766] "Camera means" refers to a device for capturing images and videos, and refers to a camera built into the eyeglass-type wearable device.

[1767] "Image recognition means" is a technology for automatically identifying specific information from an image acquired by a camera means, and has the function of recognizing the amount, price, payment method, etc. of an item.

[1768] The "household account book creation means" is a system for recording the user's expenditures and automatically creating a household account book based on the information recognized by the image recognition means.

[1769] "Electronic payment service" refers to a payment service conducted over the Internet, and is a payment method used by users when purchasing goods or services online or offline.

[1770] "Purchase Information" refers to detailed information when a user purchases a product or service, including the date, name of the item, amount, and payment method.

[1771] "Cloud" means a data storage and processing service provided over the Internet that allows users to store and access data remotely.

[1772] An embodiment of the present invention will be described.

[1773] First, the user puts on the eyeglass-type wearable device. This device has a built-in camera means that captures images of the product when the user purchases it. The image captured by the camera means is analyzed by image recognition means, and information such as the product amount, price, and payment method is automatically recognized.

[1774] Next, the information recognized by the image recognition means is sent to the household account book creation means. The household account book creation means automatically creates a household account book based on the recognized information and saves it on the cloud. This allows the user to check their spending information anytime, anywhere.

[1775] Furthermore, this system is also linked to an electronic payment service. The server has a means of obtaining purchase information from the electronic payment service and a means of automatically recording the obtained purchase information in the household account book. As a result, information about purchases made by users using the electronic payment service is automatically reflected in the household account book.

[1776] As a specific example, if a user purchases a book for 1,000 yen using an electronic payment service on October 1, 2023, that information will be automatically recorded in the household account book. This saves users the trouble of manually entering the information, and allows them to accurately and quickly update their household account book with purchase information.

[1777] The following hardware and software are used to implement this system:

[1778] Hardware: Glasses-type wearable device, camera means

[1779] Software: Image recognition, household accounting, electronic payment service integration

[1780] It is also possible to create prompts using generative AI models, such as the following:

[1781] Example prompt sentence:

[1782] Create an application that automatically records purchases made by users using an electronic payment service in a household ledger. The household ledger will include the date, item name, amount, and payment method. Use Python to generate a program that retrieves data from the API and records it in a CSV file.

[1783] In this way, the user can create an accurate household account book without much effort.

[1784] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1785] Step 1:

[1786] A user puts on the eyeglass-type wearable device and purchases a product. At this time, the camera means takes a picture of the product and its price tag. The input is the image captured by the camera means, and the output is image data.

[1787] Step 2:

[1788] The image recognition means of the terminal analyzes the image data acquired by the camera means and automatically recognizes the amount, price, and payment method of the item. The input is image data, and the output is the recognized amount, price, and payment method of the item. Specifically, the system uses an image recognition algorithm to extract text information and identify the item information.

[1789] Step 3:

[1790] The household account book creation means of the terminal automatically creates a household account book based on the information recognized by the image recognition means. The input is the amount, price, and payment method information of the recognized items, and the output is household account book data. Specifically, the recognized information is converted into a specified format and saved in the household account book database.

[1791] Step 4:

[1792] The server obtains purchase information from the electronic payment service. The input is the purchase information obtained from the electronic payment service's API, and the output is the purchase information data. Specifically, the server sends an API request and receives the purchase information as a response.

[1793] Step 5:

[1794] The server automatically records the purchase information it acquires in the household account book. The input is purchase information data, and the output is updated household account book data. Specifically, it analyzes the purchase information and adds it to the existing household account book data.

[1795] Step 6:

[1796] The server saves the household accounting data on the cloud. The input is the updated household accounting data, and the output is the household accounting data saved on the cloud. Specifically, the data is uploaded to a cloud storage service.

[1797] Step 7:

[1798] The user checks the household accounting data stored on the cloud. The input is the household accounting data stored on the cloud, and the output is the household accounting information displayed on the user's device. Specifically, the data is downloaded from cloud storage and displayed on the user interface.

[1799] Example 3

[1800] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1801] Conventional household accounting management systems require users to manually input data, which is time-consuming and prone to input errors. There is also the risk of data being lost if the device breaks down or is lost. Another issue is that it is difficult for users to check their household accounting information while on the go.

[1802] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a voice input means, a voice recognition means for converting voice data acquired by the voice input means into text data, a communication means for transmitting the text data to a cloud server, and a display means for displaying the household account book information stored in the cloud server. This allows the user to manage the household account book without hassle, ensures that data is safely protected even if the device breaks down or is lost, and enables the user to check the household account book information while on the go.

[1803] A "wearable eyeglasses device" is an eyeglasses-type electronic device that can be worn by a user and has the function of acquiring visual and audio information.

[1804] "Camera means" refers to a device for taking images or videos and storing or transmitting them as digital data.

[1805] "Image recognition means" is a technology for automatically extracting and analyzing specific information from captured images and videos.

[1806] The "household account book creation means" is a function for organizing income and expenditure information based on the acquired data and recording it as a household account book.

[1807] "Voice input means" is a device for capturing the user's voice and processing it as digital data.

[1808] "Speech recognition means" is a technology for converting voice data into text data.

[1809] "Communication means" is a function for transmitting data to other devices or servers.

[1810] A "cloud server" is a remote server that stores and processes data over the Internet.

[1811] A "display means" is a device for visually displaying digital data.

[1812] MODE FOR CARRYING OUT THE INVENTION

[1813] This invention relates to a system that uses a glasses-type wearable device to automatically store household accounting information on the cloud, allowing users to check the information anytime, anywhere. Specific embodiments of this system are described below.

[1814] Hardware and Software Configuration

[1815] Glasses-type wearable device

[1816] The user wears a wearable eyeglass device that includes the following hardware and software:

[1817] Camera means: a camera for taking images or videos.

[1818] Audio input means: A microphone for capturing the user's voice.

[1819] Speech recognition tool: Software that converts voice data into text data using the Google Cloud Speech-to-Text API.

[1820] Communication means: A communication module for sending data to a cloud server via the Internet.

[1821] Cloud Server

[1822] The cloud server has the following functions:

[1823] Data receiving means: Receives data transmitted from the terminal.

[1824] Data storage method: Data is stored using Google Cloud Firestore.

[1825] Data display means: Allows users to view household accounting information from any internet-connected device.

[1826] System Operation

[1827] After shopping at the supermarket, a user speaks to the glasses-type wearable device, saying, "Today's shopping is for 200 yen for milk and 150 yen for bread." The device captures this speech and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is then sent to a cloud server. The server receives the data and stores it in Google Cloud Firestore. When the user returns home, they can open a dedicated app on their smartphone to check their shopping information for the day.

[1828] Specific examples

[1829] For example, if a user says, "Today's shopping is 200 yen for milk and 150 yen for bread," the glasses-type wearable device captures this voice and uses voice recognition to convert it into text data: "Today's shopping is 200 yen for milk and 150 yen for bread." This text data is then sent to a cloud server, which receives the data and stores it in Google Cloud Firestore. The user can then open a dedicated app on their smartphone to check the saved household accounting information.

[1830] Example prompts to input to a generative AI model:

[1831] "How can I use a wearable eyeglass device to store my household accounting information in the cloud? Please explain, including the names of specific APIs and cloud services."

[1832] In this way, the user can manage the household account book without hassle, the data is safely protected even if the device breaks down or is lost, and the user can check the information in the household account book even when they are away from home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1833] Step 1:

[1834] The user inputs household account information. The user puts on a glasses-type wearable device and inputs household account information by voice. For example, the user might say, "Today's shopping is 200 yen for milk and 150 yen for bread." The input is voice data, and the output is the voice data itself.

[1835] Step 2:

[1836] The device converts input information into digital data. The device uses a built-in microphone to capture the user's voice. The device then converts the voice data into text data using the Google Cloud Speech-to-Text API. For example, a speech saying "Today's shopping is milk for 200 yen and bread for 150 yen" is converted into text data saying "Today's shopping is milk for 200 yen and bread for 150 yen." The input is voice data, and the output is text data.

[1837] Step 3:

[1838] The device sends digital data to the cloud server. The device then sends the converted text data to the cloud server. The HTTPS protocol is used for transmission to ensure data security. The input is text data, and the output is data sent to the cloud server.

[1839] Step 4:

[1840] The server receives the data and stores it in cloud storage. The server receives the data sent from the device. The received data is stored in Google Cloud Firestore. Firestore is a real-time database, and data is quickly stored and retrieved. The input is text data sent to the cloud server, and the output is data stored in Firestore.

[1841] Step 5:

[1842] A user checks household accounting information stored in the cloud. The user checks household accounting information stored in the cloud using an internet-connected device such as a smartphone or PC. The user can access Google Cloud Firestore through a dedicated application or a web browser and view the stored household accounting data. The input is the data stored in Firestore, and the output is the household accounting information displayed on the user's device.

[1843] (Application example 3)

[1844] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1845] Conventional household accounting management systems require manual input, which is time-consuming for users and prone to input errors. Furthermore, because expenditure information cannot be managed in real time, it is difficult for users to immediately grasp their current spending situation. Furthermore, there is no function to issue an alert if expenditures exceed a certain range, making budget management difficult. To solve these problems, there is a need for a system that can automatically acquire expenditure information and manage household accounting in real time.

[1846] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1847] In this invention, the server includes a camera means provided in the glasses-type wearable device, an image recognition means for automatically recognizing the amount, price, and payment method of an item from an image acquired by the camera means, a household account book creation means for automatically creating a household account book based on the information recognized by the image recognition means, a means for encrypting payment information and saving it on the cloud, a means for displaying the household account book information saved on the cloud in re...

Claims

1. an image analysis means for inputting a specific prompt sentence into a generation AI model, and recognizing, from an image of a receipt captured by a capturing means provided in a glasses-type wearable device worn by a user, the amount, the name of the item, the payment method, and the purchase date and time of the item purchased by the user, which are included in the image; a recording means for recording the price, name, payment method, and purchase date and time of the item recognized from an image of a receipt in a household ledger including items of the price, name, payment method, and purchase date and time of the item; an emotion engine that recognizes the emotion of the user from the image captured by the image capture means; A means for recording the recognized emotion in the household account book; A system including:

2. The system according to claim 1 , wherein the recording means stores the household account book on a cloud.

3. The system of claim 1 further comprising means for generating an alert if a certain spending amount is exceeded.

Images