System

Abstract

To provide a system for reducing the oversight of useful information, and for achieving efficient reminder notification and the reduction of an environmental load.SOLUTION: The system includes means for capturing visual information and usage history from a user's mobile device, means for converting the captured visual information and usage history into text data, means for storing the converted text data in a database, means for analyzing the stored text data to identify user behavior patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, and means for receiving and analyzing user assessment data and distributing rewards to information providers and app developers.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's information society, users often miss useful information due to information overload, and setting reminders is often a hassle. It has also been pointed out that traditional notification methods have a high environmental impact. To solve these problems, a system that provides users with personalized information and reminders at the appropriate time is needed. [Means for solving the problem]

[0005] The present invention provides a means for capturing visual information and usage history from a user's mobile device, converting it into text data, and saving it. Furthermore, the saved text data is analyzed to identify the user's behavioral patterns and preferences, and personalized advertisements and reminders are generated based on the analysis results. The generated advertisements and reminders are sent to the user's mobile device, and a system is provided for receiving and analyzing evaluation data from users and distributing rewards to information providers and app developers. This system reduces the oversight of useful information, while also achieving efficient reminder notifications and reducing environmental impact.

[0006] "User" refers to an individual or organization that uses the system of the present invention.

[0007] "Mobile device" refers to electronic devices that are carried and used by users, such as smart glasses and smartphones.

[0008] "Visual information" refers to image and video information that comes into the user's field of vision.

[0009] "Usage History" refers to a record of activities performed by a user through a mobile device.

[0010] "Text data" refers to data extracted from visual information or usage history and saved as a string of characters.

[0011] "Database" refers to an information collection system for managing stored text data.

[0012] "Analysis" refers to the processing of data using stored text data to identify user behavioral patterns and preferences.

[0013] "Behavioral patterns" refers to information that shows the trends and periodicity of the user's behavior.

[0014] "Preferences" refer to the things a user is interested in or likes.

[0015] "Personalized advertising" refers to advertising that is generated based on analysis results and tailored to the interests and concerns of each individual user.

[0016] A "reminder" is a message or alarm that reminds you of something and is sent at a specific time based on the user's schedule or actions.

[0017] "Rating Data" refers to information about the ratings users give to advertisements and reminders provided to them.

[0018] "Information Provider" refers to a third party that provides information to the system.

[0019] "App Developer" refers to a person who develops and provides applications that run within this system.

[0020] "Means for distributing rewards" refers to a system for allocating rewards to information providers and app developers based on evaluation data obtained from users. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores the text data. Specific embodiments will be described below.

[0043] 1. Data Collection

[0044] Visual information capture and translation

[0045] The device (smart glasses) captures the video and images the user is viewing in real time using built-in cameras and sensors to obtain visual information.

[0046] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as the name and price of the product the user is looking at.

[0047] Collection and conversion of usage history

[0048] The device (smartphone) collects usage history, such as which apps the user has used and which websites they have visited, and also periodically records location information.

[0049] The device (smartphone) converts the collected usage history into text data, for example, storing the places the user visited and the services they used as text data.

[0050] 2. Data transmission and storage

[0051] The device (smart glasses, smartphone) sends the converted text data to the server, where it is encrypted to ensure security.

[0052] The server stores the received text data in a database, including timestamps and location information.

[0053] 3. Data Analysis

[0054] The server performs preprocessing to analyze the stored text data, including removing noise and extracting necessary data.

[0055] The server uses machine learning algorithms to analyze the text data and identify user behavioral patterns and preferences, such as the stores users frequently visit and the categories of products they purchase.

[0056] 4. Generate and send ads and reminders

[0057] The server then generates personalized ads and reminders based on the analysis, such as ads for new products related to the user's past purchases or appointment reminders.

[0058] The server then sends the generated advertisements and reminders to the user's mobile device, either in real time or at a time specified by the user.

[0059] 5. User rating and reward system

[0060] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[0061] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[0062] The server analyzes the user evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are given rewards.

[0063] For example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as an advertisement. The smartphone also analyzes the user's calendar and sends a reminder of the next appointment.

[0064] This system improves user convenience, reduces the risk of missing useful information, provides efficient reminder notifications, and reduces environmental impact.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[0068] Step 2:

[0069] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, image recognition technology can be used to extract product names and prices and convert them into text.

[0070] Step 3:

[0071] The device (smartphone) collects the user's location information using a GPS sensor, and also records the history of applications used and web browsing.

[0072] Step 4:

[0073] The device (smartphone) converts the collected location information and app usage history into text data, such as the names of stores visited or the titles of web pages viewed.

[0074] Step 5:

[0075] The device (smartglasses, smartphone) then sends the converted text data to the server. The data is encrypted before being transferred, ensuring security.

[0076] Step 6:

[0077] The server stores the received text data in a database, along with a timestamp and location information.

[0078] Step 7:

[0079] The server performs preprocessing on the stored text data, specifically removing noise and extracting necessary data.

[0080] Step 8:

[0081] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns and preferences, such as the stores they frequently visit or the categories of products they prefer to purchase.

[0082] Step 9:

[0083] The server then generates personalized ads and reminders based on the analysis, for example, ads for new products or appointment reminders tailored to the user's preferences.

[0084] Step 10:

[0085] The server then sends the generated advertisements and reminders to the user's mobile device, allowing the user to receive notifications in real time.

[0086] Step 11:

[0087] Users can rate the ads and reminders they receive through the application's rating system.

[0088] Step 12:

[0089] The device (smartglasses, smartphone) sends the user's evaluation data to the server. The data includes the evaluation results and the identification information of the evaluated content.

[0090] Step 13:

[0091] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the evaluation results. Information providers who receive high evaluations are given rewards.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Accurate data collection and analysis based on user activity is necessary to efficiently deliver information and relevant advertisements that interest users in their daily lives. However, current systems struggle to effectively capture, convert, and analyze visual information and usage history, and they also struggle to deliver personalized notifications and advertisements in a timely manner. Furthermore, systems must efficiently reflect user feedback and distribute rewards to information providers and app developers.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for capturing visual information and usage history from a user's mobile device, means for converting the captured visual information and usage history into text data, means for encrypting the converted text data and transmitting and storing it in a database, means for preprocessing the stored text data and removing noise data, means for analyzing the preprocessed data to identify user behavior patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device in real time, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables a system that efficiently performs everything from capture to analysis and reward distribution.

[0097] "User's mobile device" refers to a mobile device that a user uses on a daily basis, including smart glasses and smartphones.

[0098] "Visual information" refers to the video and image information that users see, and is obtained through devices such as smart glasses.

[0099] "Usage history" refers to a record of the applications a user uses, the websites they visit, and their location.

[0100] "Text data" refers to data that has been analyzed from visual information and usage history and converted into a format that can be saved or transmitted in text form.

[0101] "Encryption" refers to the technology of transforming data using specific algorithms to protect it from unauthorized access.

[0102] "Database" refers to a system for systematically storing and managing collected and analyzed data.

[0103] "Noise data" refers to unnecessary data or information that is not the subject of analysis.

[0104] "Preprocessing" refers to preparatory work such as organizing, cleaning, and filtering data before data analysis.

[0105] "Behavioral patterns" refer to specific behavioral tendencies derived from a user's behavioral history.

[0106] "Preferences" refers to the preferences or tendencies a user has toward certain activities or products.

[0107] "Personalized advertising" refers to advertising that is individually customized based on a user's behavioral patterns and preferences.

[0108] "Reminder" refers to an alert or message that notifies the user of an event or task.

[0109] "Rating Data" refers to data that records the ratings users give of advertisements and reminders provided to them.

[0110] "Rewards" refers to compensation or incentives given to information providers and app developers based on user ratings.

[0111] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores and analyzes the data. Specific embodiments for carrying out the invention are described below.

[0112] 1. Data Collection

[0113] Visual information capture and translation

[0114] The device (smart glasses) captures the video and images the user is viewing in real time, using built-in cameras and sensors to obtain high-resolution visual information.

[0115] The device (smart glasses) analyzes the acquired visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the necessary text data, such as the name and price of a product being viewed on a shelf.

[0116] Collection and conversion of usage history

[0117] The device (smartphone) collects the history of applications used and websites visited by the user, and periodically records location information using the Google Maps API.

[0118] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, it stores the locations visited and the services used as text data.

[0119] 2. Data transmission and storage

[0120] Data transmission

[0121] The device (smart glasses, smartphone) then sends the converted text data to the server, where AES encryption technology is used to ensure security during data transmission.

[0122] Data storage

[0123] The server stores the received text data in a database (e.g., MongoDB), adding timestamps and location information to the data for easier later analysis.

[0124] 3. Data Analysis

[0125] Pretreatment

[0126] The server performs preprocessing on the stored text data before analyzing it. In preprocessing, noise data is removed and necessary data is extracted. For example, unnecessary advertising information and duplicate data are deleted.

[0127] analysis

[0128] The server then analyzes the stored text data using machine learning algorithms such as TensorFlow, and the results of the analysis identify user behavioral patterns and preferences, such as the stores the user frequently visits and the categories of products they purchase.

[0129] 4. Generate and send ads and reminders

[0130] Advertisement and reminder generation

[0131] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or reminders for important appointments.

[0132] Sending notifications

[0133] The server then sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, which the user can easily check.

[0134] 5. User rating and reward system

[0135] Collecting ratings

[0136] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[0137] Evaluation data transmission and analysis

[0138] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[0139] The server analyzes the evaluation data and distributes rewards to information providers and app developers who receive high ratings. Rewards are distributed based on user feedback.

[0140] Specific examples

[0141] For example, imagine a user browsing a supermarket shelf. Smart glasses capture this visual information and send the product name and price as text data to a server. The server then analyzes the user's past purchase history and sends push notifications to the user's smartphone about sales on related products that the user might be interested in. If the user makes a movie reservation using an app installed on their smartphone on their way home, that history is also sent to the server as text data. At that time, the server can send a reminder of the next movie release date.

[0142] Example prompt sentence:

[0143] "Explain how smart glasses capture visual information when a user visits a nearby supermarket, analyze it as text data, and then display advertisements for related products."

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] The device (smart glasses) captures the video and images the user is looking at in real time. The smart glasses use built-in cameras and sensors to obtain high-resolution visual information. Specifically, as the user looks at a supermarket shelf, they continuously capture video of the shelf. The input is visual information (video), and the output is the captured visual data.

[0147] Step 2:

[0148] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the required text data. For example, it extracts text from the captured video and extracts product names and prices. Specifically, it sends image data to the API and receives the analysis results in return. The input is the captured visual data, and the output is the analyzed text data.

[0149] Step 3:

[0150] The device (smartphone) collects the history of applications used by the user and websites visited. It also periodically records location information using the Google Maps API. It collects which apps the user used, when, which websites they visited, and their location at that time. The input is the app usage history and location information, and the output is the collected usage history data.

[0151] Step 4:

[0152] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, the places the user visited and the services they used are saved as text data. Specifically, the collected usage history data is input into the algorithm and the analysis results are received. The input is the usage history data, and the output is the analyzed text data.

[0153] Step 5:

[0154] The device (smart glasses, smartphone) sends the converted text data to the server. The data is secured using AES encryption technology. Specifically, the acquired text data is encrypted and sent to the server using a secure communication protocol. The input is the analyzed text data, and the output is the encrypted data sent to the server.

[0155] Step 6:

[0156] The server receives the sent data and stores it in a database (e.g., MongoDB). When storing it, it adds a timestamp and location information to the text data. Specifically, it inserts the received data into the database and adds the necessary metadata. The input is encrypted text data, and the output is the data stored in the database.

[0157] Step 7:

[0158] The server performs preprocessing on the stored text data. This involves removing noise and extracting necessary data. For example, it removes duplicate data and unnecessary advertising information. The input is the stored text data, and the output is the preprocessed data.

[0159] Step 8:

[0160] The server analyzes the preprocessed data using machine learning algorithms (e.g., TensorFlow) to identify the user's behavioral patterns and preferences. For example, it identifies the stores the user frequently visits and the categories of products they purchase. The analysis results are used to generate advertisements and reminders based on the user's preferences. The input is the preprocessed data, and the output is the analysis results.

[0161] Step 9:

[0162] The server generates personalized advertisements and reminders based on the analysis results. For example, it generates advertisements for new products related to the user's past purchase history or reminders for important appointments. The input is the analysis results, and the output is the generated advertisements and reminders.

[0163] Step 10:

[0164] The server sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, so the user can easily check them. The input is the generated advertisements and reminders, and the output is the notification to the user.

[0165] Step 11:

[0166] Users rate the ads and reminders they receive. Ratings are recorded as positive (high rating) or negative (low rating). The inputs are ads and reminders, and the output is rating data.

[0167] Step 12:

[0168] The device (smart glasses, smartphone) sends the user's rating data to the server. The rating data is again sent using AES encryption technology. The input is the rating data, and the output is the encrypted rating data sent to the server.

[0169] Step 13:

[0170] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the results. The input is evaluation data, and the output is reward distribution data. Reward distribution is fair and efficient because it is based on user feedback.

[0171] (Application example 1)

[0172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] Conventional systems have had difficulty efficiently analyzing users' visual information and usage history to provide personalized advertisements and reminders tailored to individual users. Furthermore, they have been unable to provide real-time product information or display recommended products when users are actually browsing products in a store. The challenge is to provide a system that overcomes these drawbacks and improves user convenience.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0175] In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for storing the converted text data in a database, means for analyzing the stored text data to identify the user's behavioral patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers, means for a visual device attached to the user's mobile device to capture visual information of product shelves and convert it into text data in real time, and means for displaying personalized advertisements and reminders on the visual display device of the user's mobile device, thereby making it possible to provide product information on the spot when the user is shopping in a physical store and display personalized reminders and recommended products in real time.

[0176] "User's mobile device" refers to electronic devices such as smartphones and smart glasses carried by the user.

[0177] "Visual information" refers to visual data such as images and videos that the user is viewing.

[0178] "Usage history" refers to behavioral data such as which applications a user used, which websites they visited, and their location information.

[0179] "Text data" refers to textual information obtained by analyzing and converting visual information and usage history.

[0180] "Database" refers to a system that stores and manages data in a structured format.

[0181] "Behavioral patterns" refer to the tendency of a user to repeatedly perform certain actions.

[0182] "Preferences" refers to the tendency of users to prefer certain products and services.

[0183] "Personalized advertising" refers to advertising that is specifically tailored to a user's behavioral patterns and preferences.

[0184] "Reminders" refer to notifications or messages sent based on a user's schedule or actions.

[0185] "Visual device" refers to a device that can capture visual information when worn by a user, such as smart glasses.

[0186] "Product shelf" refers to a shelf on which products are displayed in a store.

[0187] "Rating Data" refers to the ratings you give to advertisements and reminders you receive.

[0188] "Information Provider" refers to a party that provides advertising or reminder content.

[0189] "App Developer" refers to a person who develops an application that is installed on a user's mobile device.

[0190] The system of this invention uses a user's mobile device (such as smart glasses or a smartphone) to collect visual information and usage history, converts it into text data, analyzes it, and provides personalized advertisements and reminders. Specific embodiments are described below.

[0191] 1. Hardware Configuration

[0192] Smart glasses use built-in cameras and sensors to capture video and images of what the user is seeing in real time.

[0193] Smartphones collect usage history, such as which apps users use and which websites they visit, and also regularly record location information.

[0194] The servers have high-performance data processing capabilities and store and analyze data, using cloud services such as Amazon AWS and Microsoft Azure.

[0195] 2. Software Configuration

[0196] OpenCV and TensorFlow are used as image recognition algorithms.

[0197] Use Scikit-learn and Keras as machine learning libraries.

[0198] MySQL or PostgreSQL is used as the database management system.

[0199] React Native and Flutter are used as mobile app development environments.

[0200] 3. Data collection and conversion

[0201] The smart glasses capture visual information about the product shelves the user is looking at. The captured images are temporarily stored in the smart glasses, and image analysis is performed using OpenCV and TensorFlow. The analyzed information is converted into text data such as product name, price, and ingredient information.

[0202] Smartphones regularly collect users' location information and app usage history and convert it into text data.

[0203] 4. Data transmission and storage

[0204] The smart glasses and smartphone send the converted text data to the server, where it is encrypted using TLS / SSL.

[0205] The server stores the received text data in a MySQL or PostgreSQL database, including timestamps and location information.

[0206] 5. Data Analysis

[0207] The server performs preprocessing to analyze the stored text data, removing noise and extracting necessary data.

[0208] Machine learning algorithms (Scikit-learn and Keras) are used to analyze text data and identify user behavior patterns and preferences, which can then be used to identify the stores users frequently visit and the categories of products they purchase.

[0209] 6. Generate and send ads and reminders

[0210] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or appointment reminders.

[0211] The generated advertisements and reminders are displayed on the visual display of the user's mobile device (smart glasses or smartphone).

[0212] 7. User rating and reward system

[0213] Users rate the ads and reminders they receive, which is recorded as positive (thumbs up) or negative (thumbs down).

[0214] Smart glasses and smartphones send user evaluation data to a server, which analyzes the data and distributes rewards to information providers and app developers based on the results.

[0215] Specific examples

[0216] When a user visits a nearby supermarket, the smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as advertisements. The smartphone also analyzes the user's calendar and sends reminders of upcoming appointments.

[0217] Prompt Sentence Examples

[0218] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] Visual information capture

[0222] The device (smart glasses) captures images of the shelves the user is looking at.

[0223] Input: Video captured by the camera built into the smart glasses.

[0224] Output: Captured shelf image data.

[0225] Specific operation: The smart glasses' camera periodically captures still images and stores the video data in its internal memory.

[0226] Step 2:

[0227] Converting visual information into text

[0228] The device (smart glasses) analyzes the captured image using OpenCV and TensorFlow and extracts text from the image (such as product name and price).

[0229] Input: Captured shelf image data.

[0230] Output: Extracted text data (product name, price, ingredient information, etc.).

[0231] Specific operation: Image preprocessing (grayscale conversion, thresholding) is performed using OpenCV, and the text portion is identified using a TensorFlow model, followed by character recognition.

[0232] Step 3:

[0233] Collection and storage of usage history

[0234] The device (smartphone) collects the user's location information and app usage history and stores it as text data.

[0235] Input: Location information obtained by the smartphone, app usage history.

[0236] Output: Location information and usage history stored as text data.

[0237] Specific operation: Using the smartphone's GPS function and app history function, data is collected at regular intervals and stored in the internal memory.

[0238] Step 4:

[0239] Sending data

[0240] The device (smart glasses, smartphone) sends the converted text data to the server.

[0241] Input: Extracted text data.

[0242] Output: The encrypted data sent to the server.

[0243] What it does: Sends text data over the internet to a server using TLS / SSL encryption.

[0244] Step 5:

[0245] Data storage

[0246] The server stores the received text data in a database.

[0247] Input: The encrypted data sent to the server.

[0248] Output: Text data stored in a database.

[0249] Specific operation: The server confirms receipt of the data and stores it in a MySQL or PostgreSQL database.

[0250] Step 6:

[0251] Data analysis

[0252] The server analyzes the stored text data to identify the user's behavioral patterns and preferences.

[0253] Input: Text data stored in a database.

[0254] Output: Analysis of user behavior patterns and preferences.

[0255] What it does: It uses machine learning algorithms to analyze text data and extract features related to behavioral patterns and preferences.

[0256] Step 7:

[0257] Advertisement and reminder generation

[0258] The server generates personalized advertisements and reminders based on the analysis results.

[0259] Input: Analysis of user behavior patterns and preferences.

[0260] Output: Personalized ads and reminders.

[0261] Specific operation: Extracts relevant products and services from the analysis results and generates ads and reminders that are best suited to the user.

[0262] Step 8:

[0263] Sending Advertisements and Reminders

[0264] The server transmits the generated advertisements and reminders to the user's mobile device.

[0265] Enter: personalized ads and reminders.

[0266] Output: Advertisements and reminders displayed on the user's mobile device.

[0267] Specific operation: The server sends a notification to the user's smart glasses or smartphone, which is then displayed on the visual device.

[0268] Prompt Sentence Examples

[0269] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[0270] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0271] This invention relates to a system that captures visual information and usage history from a user's mobile device, converts it into text data, and stores it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized advertisements and reminders.

[0272] 1. Data Collection

[0273] Visual information capture and translation

[0274] The device (smart glasses) captures the video and images the user is viewing in real time, using the built-in camera and sensors to acquire visual information and store it in temporary memory.

[0275] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as extracting product names and prices from a shop shelf.

[0276] Collecting and converting emotional data

[0277] The device (smartglasses, smartphone) captures the user's facial expressions and voice and extracts emotional data using an emotion engine. For example, facial expression recognition technology and voice analysis are used to identify the user's emotions, such as joy or surprise.

[0278] The device (smart glasses, smartphone) sends the extracted emotion data together with the text data to the server.

[0279] Collection and conversion of usage history

[0280] The device (smartphone) periodically collects the user's location information and app usage history, as well as records the user's web browsing history.

[0281] The device (smartphone) converts the collected usage history into text data, such as the names of places visited and apps used.

[0282] 2. Data transmission and storage

[0283] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[0284] The server stores the received data in a database, including timestamps and location information.

[0285] 3. Data Analysis

[0286] The server performs preprocessing on the stored text data and emotion data, removing noise and extracting only the necessary information.

[0287] The server uses machine learning algorithms to analyze the text and emotional data to identify user behavioral patterns, preferences, and emotional tendencies, such as frequently visited places, favorite product categories, and the emotions users have toward specific products.

[0288] 4. Generate and send ads and reminders

[0289] The server generates personalized advertisements and reminders based on the analysis results, matching the user's preferences and emotions. For example, it notifies the user of information related to a product they are interested in, along with a feeling of joy.

[0290] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[0291] 5. User rating and reward system

[0292] Users can rate the ads and reminders they receive, either as a thumbs up or a thumbs down, through the application.

[0293] The device (smart glasses, smartphone) sends user rating data to the server, which includes identification information for advertisements and reminders.

[0294] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are rewarded, which increases the usefulness of the system.

[0295] As a specific example, when a user visits a nearby supermarket and their smart glasses capture visual information about the shelves, the emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[0296] This system will enable users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[0297] The processing flow will be explained below.

[0298] Step 1:

[0299] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[0300] Step 2:

[0301] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, it recognizes product names and prices and extracts them as text data.

[0302] Step 3:

[0303] The device (smartphone) periodically acquires the user's location information using a GPS sensor, and also records the history of applications used and web browsing history.

[0304] Step 4:

[0305] The device (smartphone) converts the collected location information and usage history into text data. For example, it stores the names of places visited and applications used as text data.

[0306] Step 5:

[0307] The device (smartglasses, smartphone) captures the user's facial expressions and voice and uses an emotion engine to extract emotional data. For example, facial expression recognition technology can analyze the user's smiling or surprised expressions to recognize their emotions.

[0308] Step 6:

[0309] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[0310] Step 7:

[0311] The server stores the received text data and emotion data in a database, along with timestamps and location information.

[0312] Step 8:

[0313] The server performs preprocessing on the stored data, specifically removing noise data and extracting necessary information.

[0314] Step 9:

[0315] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns, preferences, and emotional trends, such as the stores a user frequently visits and the product categories they prefer.

[0316] Step 10:

[0317] The server generates personalized advertisements and reminders based on the analysis results, and also takes into account emotional data to provide information that matches the user's current emotions, such as providing information related to products that the user has shown interest in.

[0318] Step 11:

[0319] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[0320] Step 12:

[0321] Users can rate the ads and reminders they receive through the application's rating system.

[0322] Step 13:

[0323] The device (smartglasses, smartphone) sends user rating data to the server, including identification information for advertisements and reminders.

[0324] Step 14:

[0325] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are given rewards.

[0326] Example 2

[0327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] Conventional systems required large amounts of manual data entry to identify user preferences and behavioral patterns, making it difficult to provide personalized advertisements and reminders in real time. Furthermore, they were unable to understand user emotions and respond based on those emotions, limiting the user experience. Furthermore, they lacked a mechanism for properly evaluating user feedback and providing it back to information providers and app developers. There is a need to resolve these issues and provide a more accurate personalization system.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0330] In this invention, the server includes means for capturing visual information, audio data, and usage history from the user's mobile device, means for converting the captured visual information, audio data, and usage history into text data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables real-time personalization based on the user's preferences and emotions, improving the user experience and realizing an efficient feedback system.

[0331] "User's mobile device" refers to an electronic device that the user can carry and use at all times, including smartphones and smart glasses.

[0332] "Visual information" refers to video and image data captured from the user's perspective.

[0333] "Audio data" refers to data recorded from the user's voice and surrounding sounds.

[0334] "Usage history" refers to recorded data about the applications a user has used and the places they have visited.

[0335] "Text data" refers to textual information extracted by analyzing visual information, audio data, and usage history.

[0336] "Emotional data" refers to information about the user's emotional state analyzed from their facial expressions and voice.

[0337] "Database" refers to a system for storing and managing text data and emotional data.

[0338] "Behavioral patterns" refers to information that shows a user's habits and tendencies toward frequent actions.

[0339] "Preferences" refers to information that indicates a user's preferences and interests.

[0340] "Emotional trends" refer to trends that indicate changes in a user's mood or emotions.

[0341] "Personalized advertising" refers to advertising that is customized based on a user's behavioral patterns, preferences, and emotional tendencies.

[0342] "Reminder" refers to a message that notifies the user of a specific event or matter.

[0343] "Rating Data" refers to information about the ratings you give to advertisements and reminders you receive.

[0344] "Information Provider" refers to an individual or organization that provides information or data to the System.

[0345] "App developer" refers to an individual or organization that develops application software that runs on the system.

[0346] The present invention is a system that captures visual information, audio data, and usage history from a user's mobile device, converts it into text data, stores it, and combines it with an emotion engine to provide more personalized advertisements and reminders.

[0347] First, the device (such as smart glasses or a smartphone) captures the user's visual information and audio data. Specifically, the smart glasses' built-in camera and sensors are used to capture the video and images the user is viewing in real time. The smart glasses or smartphone also record the user's voice and analyze it to extract emotional data. This data is temporarily stored in the device's memory.

[0348] Next, the device analyzes the captured visual information using an image recognition algorithm (e.g., OpenCV or TensorFlow) and converts it into text data. Similarly, the audio data is converted into text data using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text). For example, product names and prices on store shelves, and user comments can be extracted as text data.

[0349] The converted text and emotion data is then sent from the device to a server, where it is encrypted to ensure security. The server then stores the received data in a database, including timestamps and location information.

[0350] The server preprocesses the stored text and emotion data, filtering out noise and extracting only the necessary information. It then uses machine learning algorithms (e.g., sparse vectors, natural language processing models) to analyze the text and emotion data and identify user behavioral patterns, preferences, and emotional trends. This allows it to understand frequently visited locations, favorite product categories, and emotions toward specific products.

[0351] Based on the analysis results, the server generates personalized advertisements and reminders that match the user's preferences and emotions. The generated advertisements and reminders are sent from the server to the user's mobile device. Notifications are delivered in real time, with timing adjusted based on the user's current situation and emotions.

[0352] Finally, users rate the ads and reminders they receive. Ratings are made through the application as either likes or dislikes, and the rating data is sent from the device to the server. The server analyzes the rating data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are rewarded, which increases the usefulness of the system.

[0353] As a concrete example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves, and an emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[0354] Example prompt for a generative AI model:

[0355] "Please provide details about the system that captures visual information and usage history and converts it into text data. Also, please explain specifically how the emotion engine is used to provide personalized advertising."

[0356] This system allows users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[0357] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0358] Detailed flow of processing steps

[0359] Step 1: Capture the data

[0360] Input: Device worn by the user (smart glasses, smartphone)

[0361] How it works: The device captures the visual and audio data of what the user sees in real time. Specifically, the smart glasses' built-in camera captures the video, and the smartphone's microphone captures the audio.

[0362] Output: Visual information (video data) and audio data

[0363] Step 2: Temporarily save data

[0364] Input: Visual and audio data

[0365] Specific operation: The captured data is stored in the device's temporary memory and used for subsequent processing.

[0366] Output: Temporarily saved visual information and audio data

[0367] Step 3: Image recognition of visual information

[0368] Input: Temporarily saved visual information

[0369] Specific operation: The device uses an image recognition algorithm (e.g., OpenCV or TensorFlow) to analyze the captured video and convert it into text data. For example, it can extract product names and prices from video of a product shelf and convert them into text data.

[0370] Output: Text data (e.g. product name and price)

[0371] Step 4: Analyzing the audio data

[0372] Input: Temporarily saved audio data

[0373] Specific operation: The device uses a speech analysis algorithm (e.g., Google Cloud Speech-to-Text) to analyze the voice data and convert it into text data, for example, transcribing what the user is saying.

[0374] Output: Text data (e.g., speech content)

[0375] Step 5: Extracting emotion data

[0376] Input: Captured visual information (user facial expressions) and audio data

[0377] Specific operation: The device combines facial expression recognition technology and voice analysis to input data into the emotion engine, analyze the user's emotional state (e.g., joy, surprise), and extract emotion data.

[0378] Output: Emotion data (e.g., joy, surprise)

[0379] Step 6: Encrypt and send data

[0380] Input: Text data and emotion data

[0381] Specific operation: The terminal encrypts this data using an encryption algorithm such as AES and transmits the data to the server using the HTTPS protocol.

[0382] Output: Encrypted text data and sentiment data

[0383] Step 7: Receiving and storing data

[0384] Input: Encrypted text data and emotion data

[0385] How it works: The server receives the data, decrypts it, and stores it in a database, along with a timestamp and location information.

[0386] Output: Text data and emotion data stored in a database

[0387] Step 8: Data Preprocessing

[0388] Input: Stored text data and emotion data

[0389] Specific operation: The server performs preprocessing such as removing noise data and normalizing it, and prepares the data in a format suitable for analysis.

[0390] Output: Preprocessed text and sentiment data

[0391] Step 9: Data analysis

[0392] Input: Preprocessed text data and sentiment data

[0393] How it works: The server uses machine learning algorithms to analyze the data and identify user behavioral patterns, preferences, and emotional tendencies.

[0394] Output: Analysis results (e.g., behavioral patterns, preferences, emotional tendencies)

[0395] Step 10: Generate Ads and Reminders

[0396] Input: Analysis results

[0397] Specific operation: Based on the analysis results, the server automatically generates personalized advertisements and reminders appropriate for the user.

[0398] Output: Generated ads and reminders

[0399] Step 11: Sending Ads and Reminders

[0400] Input: Generated ads and reminders

[0401] How it works: The server sends the generated advertisements and reminders to the user's mobile device, again in real time, with the timing of notifications adjusted accordingly.

[0402] Output: Advertisements and reminders sent to the user's mobile device

[0403] Step 12: Gather user ratings and feedback

[0404] Input: User's ratings of the ads and reminders they received

[0405] Specific operation: Users use a smartphone application to make ratings, and the rating data is sent from the device to the server.

[0406] Output: Rating data sent to the server

[0407] Step 13: Analyze evaluation data and distribute rewards

[0408] Input: Evaluation data

[0409] Specific operation: The server analyzes the evaluation data and automatically distributes rewards to information providers and app developers based on the results.

[0410] Output: Reward distribution to informants and app developers

[0411] The above is the flow of specific processing steps of this system.

[0412] (Application example 2)

[0413] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] Conventional content delivery systems have difficulty in providing personalized content based on individual user preferences and emotions, making it difficult to improve user experience. In addition, user evaluation data is not effectively reflected, and reward distribution to information providers and application developers is insufficient.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for capturing and extracting user emotional data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized content and reminders based on the analysis results, means for transmitting the generated content and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and application developers. This makes it possible to provide more personalized content based on the user's preferences and emotions and to realize a reward system based on user evaluations.

[0416] "User's mobile device" refers to an electronic device that a user carries and uses on a daily basis, including a smartphone, smart glasses, or tablet device.

[0417] "Visual information" refers to the images and videos that a user sees with their eyes, captured through cameras and sensors.

[0418] "Usage history" refers to a record of the operations and actions a user performs using a mobile device, including application usage data and internet browsing history.

[0419] "Text data" refers to captured visual information, usage history, and emotional data expressed as text information.

[0420] "Emotion data" refers to data that indicates the emotional state of a user extracted from their facial expressions and voice, and includes emotional information such as joy, anger, sadness, and happiness.

[0421] A "database" is a computer system for storing and managing collected text data and emotion data.

[0422] "Analyzing" means processing stored data using machine learning algorithms and data analysis techniques to identify user behavioral patterns, preferences, and emotional tendencies.

[0423] "Personalized content" refers to information and entertainment tailored to a user's individual preferences and emotions based on analytical results, including advertisements and reminders.

[0424] A "reminder" is a notification that reminds the user of a specific action or event.

[0425] "Rating Data" means the ratings and feedback you provide about the content and reminders you receive.

[0426] "Information Provider" means an individual or organization that provides content or information to Users.

[0427] "Application Developer" means the individual or organization that develops the applicable application software.

[0428] "Reward" means monetary or material consideration given to information providers or application developers based on user ratings.

[0429] This system captures visual information and usage history from a user's mobile device, converts it into text data, saves it, and analyzes it to provide personalized content. Furthermore, by incorporating user emotional data, more precise personalization can be achieved.

[0430] 1. Data Collection

[0431] Visual information capture and translation

[0432] The device (smart glasses) captures the video and images the user is looking at in real time. It uses the built-in camera and sensors to acquire visual information and stores it in temporary memory. The captured visual information is analyzed using image recognition algorithms such as TensorFlow and converted into text data. For example, if a user views a movie poster, the movie title and related information are extracted as text data.

[0433] Collecting and converting emotional data

[0434] The device (smart glasses, smartphone) captures the user's facial expressions and voice and uses facial recognition and speech analysis technologies to extract emotional data, such as whether the user is enjoying a movie trailer.

[0435] Collection and conversion of usage history

[0436] The device (smartphone) periodically collects the user's location information and app usage history, and converts this information into text data using the Google Maps API and Firebase. For example, it records the names of recently visited places and used applications.

[0437] 2. Data transmission and storage

[0438] The device (smart glasses, smartphone) sends the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The server then stores this data in a database.

[0439] 3. Data Analysis

[0440] The server preprocesses the stored text and sentiment data and removes noise, leaving only the necessary information. It then uses machine learning algorithms to analyze the text and sentiment data and identify user behavioral patterns, preferences, and emotional trends. Specifically, it uses Amazon SageMaker and Azure Machine Learning.

[0441] 4. Content and reminder generation and sending

[0442] The server uses the analytics to generate personalized content and reminders, such as trailers for upcoming movies the user enjoys, using Firebase Cloud Messaging and Apple Push Notification Service.

[0443] 5. User rating and reward system

[0444] Users rate the content and reminders they receive and send the rating data to the server. The server analyzes the rating data and distributes rewards to information providers and application developers. Reward distribution uses Firebase Realtime Database and Google Analytics.

[0445] Specific examples

[0446] A concrete example would be a scenario where a user looks at a movie poster with smart glasses, and the visual and emotional information is captured, and a notification is sent to the smartphone saying, "Here's the next trending movie. Would you like to watch it now?"

[0447] Prompt Sentence Examples

[0448] "If a user is enjoying looking at a movie poster on their smart glasses, capture that information and send a movie trailer notification to their smartphone saying: 'Here's the next trending movie. Want to watch it now?'"

[0449] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0450] Step 1:

[0451] The device (smart glasses) captures the video the user is looking at in real time using a built-in camera and stores the video data in temporary memory. The input is real-time video data, and the output is captured video data stored in temporary memory.

[0452] Step 2:

[0453] The device (smart glasses) analyzes the captured video data using an image recognition algorithm (e.g., TensorFlow) and converts specific information in the video into text data. For example, in the case of a movie poster, the title and director's name are extracted. The input is video data, and the output is text data.

[0454] Step 3:

[0455] The device (smart glasses and smartphone) captures the user's facial expressions and voice and converts the data into emotion data using an emotion analysis engine (e.g., OpenCV or Google Cloud Speech-to-Text API). The input is the user's facial expression data and voice data, and the output is the analyzed emotion data.

[0456] Step 4:

[0457] The device (smartphone) periodically collects the user's location information and application usage history. This is done using the Google Maps API and Firebase, and this usage history is converted into text data. The input is location data and application usage data, and the output is text data.

[0458] Step 5:

[0459] The devices (smart glasses and smartphones) transmit the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The input is text data and emotion data, and the output is data transmission to the server.

[0460] Step 6:

[0461] The server stores the received data in a database, including timestamps and location information. The input is text data and emotion data, and the output is an entry stored in the database.

[0462] Step 7:

[0463] The server preprocesses the stored text and emotion data to remove noise. Filtering techniques are used for preprocessing to extract only the necessary information. The input is the stored data, and the output is the preprocessed data.

[0464] Step 8:

[0465] The server analyzes the preprocessed data using machine learning algorithms (e.g., Amazon SageMaker or Azure Machine Learning) to identify user behavioral patterns, preferences, and emotional trends. The input is the preprocessed data, and the output is the analysis results.

[0466] Step 9:

[0467] The server generates personalized content and reminders based on the analysis results. It uses a generative AI model to create content that matches the user's preferences and emotions. The input is the analysis results, and the output is the generated content and reminders.

[0468] Step 10:

[0469] The server sends the generated content and reminders to the user's device using Firebase Cloud Messaging or Apple Push Notification Service. The input is the generated content and reminders, and the output is notifications sent to the user's device.

[0470] Step 11:

[0471] Users rate the content and reminders they receive and send the rating data to the server via their devices (smart glasses or smartphones). The input is the user rating data, and the output is the transmission of the rating data to the server.

[0472] Step 12:

[0473] The server analyzes the evaluation data and distributes rewards to information providers and application developers based on the results. This uses Firebase Realtime Database and Google Analytics. The input is the evaluation data, and the output is the distribution of rewards.

[0474] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0476] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0477] [Second embodiment]

[0478] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0479] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0480] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0481] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0482] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0483] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0484] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0485] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0488] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0489] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0490] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores the text data. Specific embodiments will be described below.

[0491] 1. Data Collection

[0492] Visual information capture and translation

[0493] The device (smart glasses) captures the video and images the user is viewing in real time using built-in cameras and sensors to obtain visual information.

[0494] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as the name and price of the product the user is looking at.

[0495] Collection and conversion of usage history

[0496] The device (smartphone) collects usage history, such as which apps the user has used and which websites they have visited, and also periodically records location information.

[0497] The device (smartphone) converts the collected usage history into text data, for example, storing the places the user visited and the services they used as text data.

[0498] 2. Data transmission and storage

[0499] The device (smart glasses, smartphone) sends the converted text data to the server, where it is encrypted to ensure security.

[0500] The server stores the received text data in a database, including timestamps and location information.

[0501] 3. Data Analysis

[0502] The server performs preprocessing to analyze the stored text data, including removing noise and extracting necessary data.

[0503] The server uses machine learning algorithms to analyze the text data and identify user behavioral patterns and preferences, such as the stores users frequently visit and the categories of products they purchase.

[0504] 4. Generate and send ads and reminders

[0505] The server then generates personalized ads and reminders based on the analysis, such as ads for new products related to the user's past purchases or appointment reminders.

[0506] The server then sends the generated advertisements and reminders to the user's mobile device, either in real time or at a time specified by the user.

[0507] 5. User rating and reward system

[0508] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[0509] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[0510] The server analyzes the user evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are given rewards.

[0511] For example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as an advertisement. The smartphone also analyzes the user's calendar and sends a reminder of the next appointment.

[0512] This system improves user convenience, reduces the risk of missing useful information, provides efficient reminder notifications, and reduces environmental impact.

[0513] The processing flow will be explained below.

[0514] Step 1:

[0515] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[0516] Step 2:

[0517] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, image recognition technology can be used to extract product names and prices and convert them into text.

[0518] Step 3:

[0519] The device (smartphone) collects the user's location information using a GPS sensor, and also records the history of applications used and web browsing.

[0520] Step 4:

[0521] The device (smartphone) converts the collected location information and app usage history into text data, such as the names of stores visited or the titles of web pages viewed.

[0522] Step 5:

[0523] The device (smartglasses, smartphone) then sends the converted text data to the server. The data is encrypted before being transferred, ensuring security.

[0524] Step 6:

[0525] The server stores the received text data in a database, along with a timestamp and location information.

[0526] Step 7:

[0527] The server performs preprocessing on the stored text data, specifically removing noise and extracting necessary data.

[0528] Step 8:

[0529] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns and preferences, such as the stores they frequently visit or the categories of products they prefer to purchase.

[0530] Step 9:

[0531] The server then generates personalized ads and reminders based on the analysis, for example, ads for new products or appointment reminders tailored to the user's preferences.

[0532] Step 10:

[0533] The server then sends the generated advertisements and reminders to the user's mobile device, allowing the user to receive notifications in real time.

[0534] Step 11:

[0535] Users can rate the ads and reminders they receive through the application's rating system.

[0536] Step 12:

[0537] The device (smartglasses, smartphone) sends the user's evaluation data to the server. The data includes the evaluation results and the identification information of the evaluated content.

[0538] Step 13:

[0539] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the evaluation results. Information providers who receive high evaluations are given rewards.

[0540] Example 1

[0541] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0542] Accurate data collection and analysis based on user activity is necessary to efficiently deliver information and relevant advertisements that interest users in their daily lives. However, current systems struggle to effectively capture, convert, and analyze visual information and usage history, and they also struggle to deliver personalized notifications and advertisements in a timely manner. Furthermore, systems must efficiently reflect user feedback and distribute rewards to information providers and app developers.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0544] In this invention, the server includes means for capturing visual information and usage history from a user's mobile device, means for converting the captured visual information and usage history into text data, means for encrypting the converted text data and transmitting and storing it in a database, means for preprocessing the stored text data and removing noise data, means for analyzing the preprocessed data to identify user behavior patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device in real time, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables a system that efficiently performs everything from capture to analysis and reward distribution.

[0545] "User's mobile device" refers to a mobile device that a user uses on a daily basis, including smart glasses and smartphones.

[0546] "Visual information" refers to the video and image information that users see, and is obtained through devices such as smart glasses.

[0547] "Usage history" refers to a record of the applications a user uses, the websites they visit, and their location.

[0548] "Text data" refers to data that has been analyzed from visual information and usage history and converted into a format that can be saved or transmitted in text form.

[0549] "Encryption" refers to the technology of transforming data using specific algorithms to protect it from unauthorized access.

[0550] "Database" refers to a system for systematically storing and managing collected and analyzed data.

[0551] "Noise data" refers to unnecessary data or information that is not the subject of analysis.

[0552] "Preprocessing" refers to preparatory work such as organizing, cleaning, and filtering data before data analysis.

[0553] "Behavioral patterns" refer to specific behavioral tendencies derived from a user's behavioral history.

[0554] "Preferences" refers to the preferences or tendencies a user has toward certain activities or products.

[0555] "Personalized advertising" refers to advertising that is individually customized based on a user's behavioral patterns and preferences.

[0556] "Reminder" refers to an alert or message that notifies the user of an event or task.

[0557] "Rating Data" refers to data that records the ratings users give of advertisements and reminders provided to them.

[0558] "Rewards" refers to compensation or incentives given to information providers and app developers based on user ratings.

[0559] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores and analyzes the data. Specific embodiments for carrying out the invention are described below.

[0560] 1. Data Collection

[0561] Visual information capture and translation

[0562] The device (smart glasses) captures the video and images the user is viewing in real time, using built-in cameras and sensors to obtain high-resolution visual information.

[0563] The device (smart glasses) analyzes the acquired visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the necessary text data, such as the name and price of a product being viewed on a shelf.

[0564] Collection and conversion of usage history

[0565] The device (smartphone) collects the history of applications used and websites visited by the user, and periodically records location information using the Google Maps API.

[0566] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, it stores the locations visited and the services used as text data.

[0567] 2. Data transmission and storage

[0568] Data transmission

[0569] The device (smart glasses, smartphone) then sends the converted text data to the server, where AES encryption technology is used to ensure security during data transmission.

[0570] Data storage

[0571] The server stores the received text data in a database (e.g., MongoDB), adding timestamps and location information to the data for easier later analysis.

[0572] 3. Data Analysis

[0573] Pretreatment

[0574] The server performs preprocessing on the stored text data before analyzing it. In preprocessing, noise data is removed and necessary data is extracted. For example, unnecessary advertising information and duplicate data are deleted.

[0575] analysis

[0576] The server then analyzes the stored text data using machine learning algorithms such as TensorFlow, and the results of the analysis identify user behavioral patterns and preferences, such as the stores the user frequently visits and the categories of products they purchase.

[0577] 4. Generate and send ads and reminders

[0578] Advertisement and reminder generation

[0579] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or reminders for important appointments.

[0580] Sending notifications

[0581] The server then sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, which the user can easily check.

[0582] 5. User rating and reward system

[0583] Collecting ratings

[0584] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[0585] Evaluation data transmission and analysis

[0586] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[0587] The server analyzes the evaluation data and distributes rewards to information providers and app developers who receive high ratings. Rewards are distributed based on user feedback.

[0588] Specific examples

[0589] For example, imagine a user browsing a supermarket shelf. Smart glasses capture this visual information and send the product name and price as text data to a server. The server then analyzes the user's past purchase history and sends push notifications to the user's smartphone about sales on related products that the user might be interested in. If the user makes a movie reservation using an app installed on their smartphone on their way home, that history is also sent to the server as text data. At that time, the server can send a reminder of the next movie release date.

[0590] Example prompt sentence:

[0591] "Explain how smart glasses capture visual information when a user visits a nearby supermarket, analyze it as text data, and then display advertisements for related products."

[0592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0593] Step 1:

[0594] The device (smart glasses) captures the video and images the user is looking at in real time. The smart glasses use built-in cameras and sensors to obtain high-resolution visual information. Specifically, as the user looks at a supermarket shelf, they continuously capture video of the shelf. The input is visual information (video), and the output is the captured visual data.

[0595] Step 2:

[0596] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the required text data. For example, it extracts text from the captured video and extracts product names and prices. Specifically, it sends image data to the API and receives the analysis results in return. The input is the captured visual data, and the output is the analyzed text data.

[0597] Step 3:

[0598] The device (smartphone) collects the history of applications used by the user and websites visited. It also periodically records location information using the Google Maps API. It collects which apps the user used, when, which websites they visited, and their location at that time. The input is the app usage history and location information, and the output is the collected usage history data.

[0599] Step 4:

[0600] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, the places the user visited and the services they used are saved as text data. Specifically, the collected usage history data is input into the algorithm and the analysis results are received. The input is the usage history data, and the output is the analyzed text data.

[0601] Step 5:

[0602] The device (smart glasses, smartphone) sends the converted text data to the server. The data is secured using AES encryption technology. Specifically, the acquired text data is encrypted and sent to the server using a secure communication protocol. The input is the analyzed text data, and the output is the encrypted data sent to the server.

[0603] Step 6:

[0604] The server receives the sent data and stores it in a database (e.g., MongoDB). When storing it, it adds a timestamp and location information to the text data. Specifically, it inserts the received data into the database and adds the necessary metadata. The input is encrypted text data, and the output is the data stored in the database.

[0605] Step 7:

[0606] The server performs preprocessing on the stored text data. This involves removing noise and extracting necessary data. For example, it removes duplicate data and unnecessary advertising information. The input is the stored text data, and the output is the preprocessed data.

[0607] Step 8:

[0608] The server analyzes the preprocessed data using machine learning algorithms (e.g., TensorFlow) to identify the user's behavioral patterns and preferences. For example, it identifies the stores the user frequently visits and the categories of products they purchase. The analysis results are used to generate advertisements and reminders based on the user's preferences. The input is the preprocessed data, and the output is the analysis results.

[0609] Step 9:

[0610] The server generates personalized advertisements and reminders based on the analysis results. For example, it generates advertisements for new products related to the user's past purchase history or reminders for important appointments. The input is the analysis results, and the output is the generated advertisements and reminders.

[0611] Step 10:

[0612] The server sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, so the user can easily check them. The input is the generated advertisements and reminders, and the output is the notification to the user.

[0613] Step 11:

[0614] Users rate the ads and reminders they receive. Ratings are recorded as positive (high rating) or negative (low rating). The inputs are ads and reminders, and the output is rating data.

[0615] Step 12:

[0616] The device (smart glasses, smartphone) sends the user's rating data to the server. The rating data is again sent using AES encryption technology. The input is the rating data, and the output is the encrypted rating data sent to the server.

[0617] Step 13:

[0618] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the results. The input is evaluation data, and the output is reward distribution data. Reward distribution is fair and efficient because it is based on user feedback.

[0619] (Application example 1)

[0620] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0621] Conventional systems have had difficulty efficiently analyzing users' visual information and usage history to provide personalized advertisements and reminders tailored to individual users. Furthermore, they have been unable to provide real-time product information or display recommended products when users are actually browsing products in a store. The challenge is to provide a system that overcomes these drawbacks and improves user convenience.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0623] In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for storing the converted text data in a database, means for analyzing the stored text data to identify the user's behavioral patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers, means for a visual device attached to the user's mobile device to capture visual information of product shelves and convert it into text data in real time, and means for displaying personalized advertisements and reminders on the visual display device of the user's mobile device, thereby making it possible to provide product information on the spot when the user is shopping in a physical store and display personalized reminders and recommended products in real time.

[0624] "User's mobile device" refers to electronic devices such as smartphones and smart glasses carried by the user.

[0625] "Visual information" refers to visual data such as images and videos that the user is viewing.

[0626] "Usage history" refers to behavioral data such as which applications a user used, which websites they visited, and their location information.

[0627] "Text data" refers to textual information obtained by analyzing and converting visual information and usage history.

[0628] "Database" refers to a system that stores and manages data in a structured format.

[0629] "Behavioral patterns" refer to the tendency of a user to repeatedly perform certain actions.

[0630] "Preferences" refers to the tendency of users to prefer certain products and services.

[0631] "Personalized advertising" refers to advertising that is specifically tailored to a user's behavioral patterns and preferences.

[0632] "Reminders" refer to notifications or messages sent based on a user's schedule or actions.

[0633] "Visual device" refers to a device that can capture visual information when worn by a user, such as smart glasses.

[0634] "Product shelf" refers to a shelf on which products are displayed in a store.

[0635] "Rating Data" refers to the ratings you give to advertisements and reminders you receive.

[0636] "Information Provider" refers to a party that provides advertising or reminder content.

[0637] "App Developer" refers to a person who develops an application that is installed on a user's mobile device.

[0638] The system of this invention uses a user's mobile device (such as smart glasses or a smartphone) to collect visual information and usage history, converts it into text data, analyzes it, and provides personalized advertisements and reminders. Specific embodiments are described below.

[0639] 1. Hardware Configuration

[0640] Smart glasses use built-in cameras and sensors to capture video and images of what the user is seeing in real time.

[0641] Smartphones collect usage history, such as which apps users use and which websites they visit, and also regularly record location information.

[0642] The servers have high-performance data processing capabilities and store and analyze data, using cloud services such as Amazon AWS and Microsoft Azure.

[0643] 2. Software Configuration

[0644] OpenCV and TensorFlow are used as image recognition algorithms.

[0645] Use Scikit-learn and Keras as machine learning libraries.

[0646] MySQL or PostgreSQL is used as the database management system.

[0647] React Native and Flutter are used as mobile app development environments.

[0648] 3. Data collection and conversion

[0649] The smart glasses capture visual information about the product shelves the user is looking at. The captured images are temporarily stored in the smart glasses, and image analysis is performed using OpenCV and TensorFlow. The analyzed information is converted into text data such as product name, price, and ingredient information.

[0650] Smartphones regularly collect users' location information and app usage history and convert it into text data.

[0651] 4. Data transmission and storage

[0652] The smart glasses and smartphone send the converted text data to the server, where it is encrypted using TLS / SSL.

[0653] The server stores the received text data in a MySQL or PostgreSQL database, including timestamps and location information.

[0654] 5. Data Analysis

[0655] The server performs preprocessing to analyze the stored text data, removing noise and extracting necessary data.

[0656] Machine learning algorithms (Scikit-learn and Keras) are used to analyze text data and identify user behavior patterns and preferences, which can then be used to identify the stores users frequently visit and the categories of products they purchase.

[0657] 6. Generate and send ads and reminders

[0658] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or appointment reminders.

[0659] The generated advertisements and reminders are displayed on the visual display of the user's mobile device (smart glasses or smartphone).

[0660] 7. User rating and reward system

[0661] Users rate the ads and reminders they receive, which is recorded as positive (thumbs up) or negative (thumbs down).

[0662] Smart glasses and smartphones send user evaluation data to a server, which analyzes the data and distributes rewards to information providers and app developers based on the results.

[0663] Specific examples

[0664] When a user visits a nearby supermarket, the smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as advertisements. The smartphone also analyzes the user's calendar and sends reminders of upcoming appointments.

[0665] Prompt Sentence Examples

[0666] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[0667] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0668] Step 1:

[0669] Visual information capture

[0670] The device (smart glasses) captures images of the shelves the user is looking at.

[0671] Input: Video captured by the camera built into the smart glasses.

[0672] Output: Captured shelf image data.

[0673] Specific operation: The smart glasses' camera periodically captures still images and stores the video data in its internal memory.

[0674] Step 2:

[0675] Converting visual information into text

[0676] The device (smart glasses) analyzes the captured image using OpenCV and TensorFlow and extracts text from the image (such as product name and price).

[0677] Input: Captured shelf image data.

[0678] Output: Extracted text data (product name, price, ingredient information, etc.).

[0679] Specific operation: Image preprocessing (grayscale conversion, thresholding) is performed using OpenCV, and the text portion is identified using a TensorFlow model, followed by character recognition.

[0680] Step 3:

[0681] Collection and storage of usage history

[0682] The device (smartphone) collects the user's location information and app usage history and stores it as text data.

[0683] Input: Location information obtained by the smartphone, app usage history.

[0684] Output: Location information and usage history stored as text data.

[0685] Specific operation: Using the smartphone's GPS function and app history function, data is collected at regular intervals and stored in the internal memory.

[0686] Step 4:

[0687] Sending data

[0688] The device (smart glasses, smartphone) sends the converted text data to the server.

[0689] Input: Extracted text data.

[0690] Output: The encrypted data sent to the server.

[0691] What it does: Sends text data over the internet to a server using TLS / SSL encryption.

[0692] Step 5:

[0693] Data storage

[0694] The server stores the received text data in a database.

[0695] Input: The encrypted data sent to the server.

[0696] Output: Text data stored in a database.

[0697] Specific operation: The server confirms receipt of the data and stores it in a MySQL or PostgreSQL database.

[0698] Step 6:

[0699] Data analysis

[0700] The server analyzes the stored text data to identify the user's behavioral patterns and preferences.

[0701] Input: Text data stored in a database.

[0702] Output: Analysis of user behavior patterns and preferences.

[0703] What it does: It uses machine learning algorithms to analyze text data and extract features related to behavioral patterns and preferences.

[0704] Step 7:

[0705] Advertisement and reminder generation

[0706] The server generates personalized advertisements and reminders based on the analysis results.

[0707] Input: Analysis of user behavior patterns and preferences.

[0708] Output: Personalized ads and reminders.

[0709] Specific operation: Extracts relevant products and services from the analysis results and generates ads and reminders that are best suited to the user.

[0710] Step 8:

[0711] Sending Advertisements and Reminders

[0712] The server transmits the generated advertisements and reminders to the user's mobile device.

[0713] Enter: personalized ads and reminders.

[0714] Output: Advertisements and reminders displayed on the user's mobile device.

[0715] Specific operation: The server sends a notification to the user's smart glasses or smartphone, which is then displayed on the visual device.

[0716] Prompt Sentence Examples

[0717] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[0718] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0719] This invention relates to a system that captures visual information and usage history from a user's mobile device, converts it into text data, and stores it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized advertisements and reminders.

[0720] 1. Data Collection

[0721] Visual information capture and translation

[0722] The device (smart glasses) captures the video and images the user is viewing in real time, using the built-in camera and sensors to acquire visual information and store it in temporary memory.

[0723] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as extracting product names and prices from a shop shelf.

[0724] Collecting and converting emotional data

[0725] The device (smartglasses, smartphone) captures the user's facial expressions and voice and extracts emotional data using an emotion engine. For example, facial expression recognition technology and voice analysis are used to identify the user's emotions, such as joy or surprise.

[0726] The device (smart glasses, smartphone) sends the extracted emotion data together with the text data to the server.

[0727] Collection and conversion of usage history

[0728] The device (smartphone) periodically collects the user's location information and app usage history, as well as records the user's web browsing history.

[0729] The device (smartphone) converts the collected usage history into text data, such as the names of places visited and apps used.

[0730] 2. Data transmission and storage

[0731] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[0732] The server stores the received data in a database, including timestamps and location information.

[0733] 3. Data Analysis

[0734] The server performs preprocessing on the stored text data and emotion data, removing noise and extracting only the necessary information.

[0735] The server uses machine learning algorithms to analyze the text and emotional data to identify user behavioral patterns, preferences, and emotional tendencies, such as frequently visited places, favorite product categories, and the emotions users have toward specific products.

[0736] 4. Generate and send ads and reminders

[0737] The server generates personalized advertisements and reminders based on the analysis results, matching the user's preferences and emotions. For example, it notifies the user of information related to a product they are interested in, along with a feeling of joy.

[0738] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[0739] 5. User rating and reward system

[0740] Users can rate the ads and reminders they receive, either as a thumbs up or a thumbs down, through the application.

[0741] The device (smart glasses, smartphone) sends user rating data to the server, which includes identification information for advertisements and reminders.

[0742] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are rewarded, which increases the usefulness of the system.

[0743] As a specific example, when a user visits a nearby supermarket and their smart glasses capture visual information about the shelves, the emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[0744] This system will enable users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[0745] The processing flow will be explained below.

[0746] Step 1:

[0747] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[0748] Step 2:

[0749] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, it recognizes product names and prices and extracts them as text data.

[0750] Step 3:

[0751] The device (smartphone) periodically acquires the user's location information using a GPS sensor, and also records the history of applications used and web browsing history.

[0752] Step 4:

[0753] The device (smartphone) converts the collected location information and usage history into text data. For example, it stores the names of places visited and applications used as text data.

[0754] Step 5:

[0755] The device (smartglasses, smartphone) captures the user's facial expressions and voice and uses an emotion engine to extract emotional data. For example, facial expression recognition technology can analyze the user's smiling or surprised expressions to recognize their emotions.

[0756] Step 6:

[0757] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[0758] Step 7:

[0759] The server stores the received text data and emotion data in a database, along with timestamps and location information.

[0760] Step 8:

[0761] The server performs preprocessing on the stored data, specifically removing noise data and extracting necessary information.

[0762] Step 9:

[0763] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns, preferences, and emotional trends, such as the stores a user frequently visits and the product categories they prefer.

[0764] Step 10:

[0765] The server generates personalized advertisements and reminders based on the analysis results, and also takes into account emotional data to provide information that matches the user's current emotions, such as providing information related to products that the user has shown interest in.

[0766] Step 11:

[0767] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[0768] Step 12:

[0769] Users can rate the ads and reminders they receive through the application's rating system.

[0770] Step 13:

[0771] The device (smartglasses, smartphone) sends user rating data to the server, including identification information for advertisements and reminders.

[0772] Step 14:

[0773] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are given rewards.

[0774] Example 2

[0775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] Conventional systems required large amounts of manual data entry to identify user preferences and behavioral patterns, making it difficult to provide personalized advertisements and reminders in real time. Furthermore, they were unable to understand user emotions and respond based on those emotions, limiting the user experience. Furthermore, they lacked a mechanism for properly evaluating user feedback and providing it back to information providers and app developers. There is a need to resolve these issues and provide a more accurate personalization system.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0778] In this invention, the server includes means for capturing visual information, audio data, and usage history from the user's mobile device, means for converting the captured visual information, audio data, and usage history into text data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables real-time personalization based on the user's preferences and emotions, improving the user experience and realizing an efficient feedback system.

[0779] "User's mobile device" refers to an electronic device that the user can carry and use at all times, including smartphones and smart glasses.

[0780] "Visual information" refers to video and image data captured from the user's perspective.

[0781] "Audio data" refers to data recorded from the user's voice and surrounding sounds.

[0782] "Usage history" refers to recorded data about the applications a user has used and the places they have visited.

[0783] "Text data" refers to textual information extracted by analyzing visual information, audio data, and usage history.

[0784] "Emotional data" refers to information about the user's emotional state analyzed from their facial expressions and voice.

[0785] "Database" refers to a system for storing and managing text data and emotional data.

[0786] "Behavioral patterns" refers to information that shows a user's habits and tendencies toward frequent actions.

[0787] "Preferences" refers to information that indicates a user's preferences and interests.

[0788] "Emotional trends" refer to trends that indicate changes in a user's mood or emotions.

[0789] "Personalized advertising" refers to advertising that is customized based on a user's behavioral patterns, preferences, and emotional tendencies.

[0790] "Reminder" refers to a message that notifies the user of a specific event or matter.

[0791] "Rating Data" refers to information about the ratings you give to advertisements and reminders you receive.

[0792] "Information Provider" refers to an individual or organization that provides information or data to the System.

[0793] "App developer" refers to an individual or organization that develops application software that runs on the system.

[0794] The present invention is a system that captures visual information, audio data, and usage history from a user's mobile device, converts it into text data, stores it, and combines it with an emotion engine to provide more personalized advertisements and reminders.

[0795] First, the device (such as smart glasses or a smartphone) captures the user's visual information and audio data. Specifically, the smart glasses' built-in camera and sensors are used to capture the video and images the user is viewing in real time. The smart glasses or smartphone also record the user's voice and analyze it to extract emotional data. This data is temporarily stored in the device's memory.

[0796] Next, the device analyzes the captured visual information using an image recognition algorithm (e.g., OpenCV or TensorFlow) and converts it into text data. Similarly, the audio data is converted into text data using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text). For example, product names and prices on store shelves, and user comments can be extracted as text data.

[0797] The converted text and emotion data is then sent from the device to a server, where it is encrypted to ensure security. The server then stores the received data in a database, including timestamps and location information.

[0798] The server preprocesses the stored text and emotion data, filtering out noise and extracting only the necessary information. It then uses machine learning algorithms (e.g., sparse vectors, natural language processing models) to analyze the text and emotion data and identify user behavioral patterns, preferences, and emotional trends. This allows it to understand frequently visited locations, favorite product categories, and emotions toward specific products.

[0799] Based on the analysis results, the server generates personalized advertisements and reminders that match the user's preferences and emotions. The generated advertisements and reminders are sent from the server to the user's mobile device. Notifications are delivered in real time, with timing adjusted based on the user's current situation and emotions.

[0800] Finally, users rate the ads and reminders they receive. Ratings are made through the application as either likes or dislikes, and the rating data is sent from the device to the server. The server analyzes the rating data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are rewarded, which increases the usefulness of the system.

[0801] As a concrete example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves, and an emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[0802] Example prompt for a generative AI model:

[0803] "Please provide details about the system that captures visual information and usage history and converts it into text data. Also, please explain specifically how the emotion engine is used to provide personalized advertising."

[0804] This system allows users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[0805] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0806] Detailed flow of processing steps

[0807] Step 1: Capture the data

[0808] Input: Device worn by the user (smart glasses, smartphone)

[0809] How it works: The device captures the visual and audio data of what the user sees in real time. Specifically, the smart glasses' built-in camera captures the video, and the smartphone's microphone captures the audio.

[0810] Output: Visual information (video data) and audio data

[0811] Step 2: Temporarily save data

[0812] Input: Visual and audio data

[0813] Specific operation: The captured data is stored in the device's temporary memory and used for subsequent processing.

[0814] Output: Temporarily saved visual information and audio data

[0815] Step 3: Image recognition of visual information

[0816] Input: Temporarily saved visual information

[0817] Specific operation: The device uses an image recognition algorithm (e.g., OpenCV or TensorFlow) to analyze the captured video and convert it into text data. For example, it can extract product names and prices from video of a product shelf and convert them into text data.

[0818] Output: Text data (e.g. product name and price)

[0819] Step 4: Analyzing the audio data

[0820] Input: Temporarily saved audio data

[0821] Specific operation: The device uses a speech analysis algorithm (e.g., Google Cloud Speech-to-Text) to analyze the voice data and convert it into text data, for example, transcribing what the user is saying.

[0822] Output: Text data (e.g., speech content)

[0823] Step 5: Extracting emotion data

[0824] Input: Captured visual information (user facial expressions) and audio data

[0825] Specific operation: The device combines facial expression recognition technology and voice analysis to input data into the emotion engine, analyze the user's emotional state (e.g., joy, surprise), and extract emotion data.

[0826] Output: Emotion data (e.g., joy, surprise)

[0827] Step 6: Encrypt and send data

[0828] Input: Text data and emotion data

[0829] Specific operation: The terminal encrypts this data using an encryption algorithm such as AES and transmits the data to the server using the HTTPS protocol.

[0830] Output: Encrypted text data and sentiment data

[0831] Step 7: Receiving and storing data

[0832] Input: Encrypted text data and emotion data

[0833] How it works: The server receives the data, decrypts it, and stores it in a database, along with a timestamp and location information.

[0834] Output: Text data and emotion data stored in a database

[0835] Step 8: Data Preprocessing

[0836] Input: Stored text data and emotion data

[0837] Specific operation: The server performs preprocessing such as removing noise data and normalizing it, and prepares the data in a format suitable for analysis.

[0838] Output: Preprocessed text and sentiment data

[0839] Step 9: Data analysis

[0840] Input: Preprocessed text data and sentiment data

[0841] How it works: The server uses machine learning algorithms to analyze the data and identify user behavioral patterns, preferences, and emotional tendencies.

[0842] Output: Analysis results (e.g., behavioral patterns, preferences, emotional tendencies)

[0843] Step 10: Generate Ads and Reminders

[0844] Input: Analysis results

[0845] Specific operation: Based on the analysis results, the server automatically generates personalized advertisements and reminders appropriate for the user.

[0846] Output: Generated ads and reminders

[0847] Step 11: Sending Ads and Reminders

[0848] Input: Generated ads and reminders

[0849] How it works: The server sends the generated advertisements and reminders to the user's mobile device, again in real time, with the timing of notifications adjusted accordingly.

[0850] Output: Advertisements and reminders sent to the user's mobile device

[0851] Step 12: Gather user ratings and feedback

[0852] Input: User's ratings of the ads and reminders they received

[0853] Specific operation: Users use a smartphone application to make ratings, and the rating data is sent from the device to the server.

[0854] Output: Rating data sent to the server

[0855] Step 13: Analyze evaluation data and distribute rewards

[0856] Input: Evaluation data

[0857] Specific operation: The server analyzes the evaluation data and automatically distributes rewards to information providers and app developers based on the results.

[0858] Output: Reward distribution to informants and app developers

[0859] The above is the flow of specific processing steps of this system.

[0860] (Application example 2)

[0861] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0862] Conventional content delivery systems have difficulty in providing personalized content based on individual user preferences and emotions, making it difficult to improve user experience. In addition, user evaluation data is not effectively reflected, and reward distribution to information providers and application developers is insufficient.

[0863] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for capturing and extracting user emotional data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized content and reminders based on the analysis results, means for transmitting the generated content and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and application developers. This makes it possible to provide more personalized content based on the user's preferences and emotions and to realize a reward system based on user evaluations.

[0864] "User's mobile device" refers to an electronic device that a user carries and uses on a daily basis, including a smartphone, smart glasses, or tablet device.

[0865] "Visual information" refers to the images and videos that a user sees with their eyes, captured through cameras and sensors.

[0866] "Usage history" refers to a record of the operations and actions a user performs using a mobile device, including application usage data and internet browsing history.

[0867] "Text data" refers to captured visual information, usage history, and emotional data expressed as text information.

[0868] "Emotion data" refers to data that indicates the emotional state of a user extracted from their facial expressions and voice, and includes emotional information such as joy, anger, sadness, and happiness.

[0869] A "database" is a computer system for storing and managing collected text data and emotion data.

[0870] "Analyzing" means processing stored data using machine learning algorithms and data analysis techniques to identify user behavioral patterns, preferences, and emotional tendencies.

[0871] "Personalized content" refers to information and entertainment tailored to a user's individual preferences and emotions based on analytical results, including advertisements and reminders.

[0872] A "reminder" is a notification that reminds the user of a specific action or event.

[0873] "Rating Data" means the ratings and feedback you provide about the content and reminders you receive.

[0874] "Information Provider" means an individual or organization that provides content or information to Users.

[0875] "Application Developer" means the individual or organization that develops the applicable application software.

[0876] "Reward" means monetary or material consideration given to information providers or application developers based on user ratings.

[0877] This system captures visual information and usage history from a user's mobile device, converts it into text data, saves it, and analyzes it to provide personalized content. Furthermore, by incorporating user emotional data, more precise personalization can be achieved.

[0878] 1. Data Collection

[0879] Visual information capture and translation

[0880] The device (smart glasses) captures the video and images the user is looking at in real time. It uses the built-in camera and sensors to acquire visual information and stores it in temporary memory. The captured visual information is analyzed using image recognition algorithms such as TensorFlow and converted into text data. For example, if a user views a movie poster, the movie title and related information are extracted as text data.

[0881] Collecting and converting emotional data

[0882] The device (smart glasses, smartphone) captures the user's facial expressions and voice and uses facial recognition and speech analysis technologies to extract emotional data, such as whether the user is enjoying a movie trailer.

[0883] Collection and conversion of usage history

[0884] The device (smartphone) periodically collects the user's location information and app usage history, and converts this information into text data using the Google Maps API and Firebase. For example, it records the names of recently visited places and used applications.

[0885] 2. Data transmission and storage

[0886] The device (smart glasses, smartphone) sends the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The server then stores this data in a database.

[0887] 3. Data Analysis

[0888] The server preprocesses the stored text and sentiment data and removes noise, leaving only the necessary information. It then uses machine learning algorithms to analyze the text and sentiment data and identify user behavioral patterns, preferences, and emotional trends. Specifically, it uses Amazon SageMaker and Azure Machine Learning.

[0889] 4. Content and reminder generation and sending

[0890] The server uses the analytics to generate personalized content and reminders, such as trailers for upcoming movies the user enjoys, using Firebase Cloud Messaging and Apple Push Notification Service.

[0891] 5. User rating and reward system

[0892] Users rate the content and reminders they receive and send the rating data to the server. The server analyzes the rating data and distributes rewards to information providers and application developers. Reward distribution uses Firebase Realtime Database and Google Analytics.

[0893] Specific examples

[0894] A concrete example would be a scenario where a user looks at a movie poster with smart glasses, and the visual and emotional information is captured, and a notification is sent to the smartphone saying, "Here's the next trending movie. Would you like to watch it now?"

[0895] Prompt Sentence Examples

[0896] "If a user is enjoying looking at a movie poster on their smart glasses, capture that information and send a movie trailer notification to their smartphone saying: 'Here's the next trending movie. Want to watch it now?'"

[0897] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0898] Step 1:

[0899] The device (smart glasses) captures the video the user is looking at in real time using a built-in camera and stores the video data in temporary memory. The input is real-time video data, and the output is captured video data stored in temporary memory.

[0900] Step 2:

[0901] The device (smart glasses) analyzes the captured video data using an image recognition algorithm (e.g., TensorFlow) and converts specific information in the video into text data. For example, in the case of a movie poster, the title and director's name are extracted. The input is video data, and the output is text data.

[0902] Step 3:

[0903] The device (smart glasses and smartphone) captures the user's facial expressions and voice and converts the data into emotion data using an emotion analysis engine (e.g., OpenCV or Google Cloud Speech-to-Text API). The input is the user's facial expression data and voice data, and the output is the analyzed emotion data.

[0904] Step 4:

[0905] The device (smartphone) periodically collects the user's location information and application usage history. This is done using the Google Maps API and Firebase, and this usage history is converted into text data. The input is location data and application usage data, and the output is text data.

[0906] Step 5:

[0907] The devices (smart glasses and smartphones) transmit the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The input is text data and emotion data, and the output is data transmission to the server.

[0908] Step 6:

[0909] The server stores the received data in a database, including timestamps and location information. The input is text data and emotion data, and the output is an entry stored in the database.

[0910] Step 7:

[0911] The server preprocesses the stored text and emotion data to remove noise. Filtering techniques are used for preprocessing to extract only the necessary information. The input is the stored data, and the output is the preprocessed data.

[0912] Step 8:

[0913] The server analyzes the preprocessed data using machine learning algorithms (e.g., Amazon SageMaker or Azure Machine Learning) to identify user behavioral patterns, preferences, and emotional trends. The input is the preprocessed data, and the output is the analysis results.

[0914] Step 9:

[0915] The server generates personalized content and reminders based on the analysis results. It uses a generative AI model to create content that matches the user's preferences and emotions. The input is the analysis results, and the output is the generated content and reminders.

[0916] Step 10:

[0917] The server sends the generated content and reminders to the user's device using Firebase Cloud Messaging or Apple Push Notification Service. The input is the generated content and reminders, and the output is notifications sent to the user's device.

[0918] Step 11:

[0919] Users rate the content and reminders they receive and send the rating data to the server via their devices (smart glasses or smartphones). The input is the user rating data, and the output is the transmission of the rating data to the server.

[0920] Step 12:

[0921] The server analyzes the evaluation data and distributes rewards to information providers and application developers based on the results. This uses Firebase Realtime Database and Google Analytics. The input is the evaluation data, and the output is the distribution of rewards.

[0922] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0923] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0924] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0925] [Third embodiment]

[0926] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0927] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0928] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0929] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0930] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0931] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0932] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0933] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0934] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0935] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0936] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0937] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0938] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores the text data. Specific embodiments will be described below.

[0939] 1. Data Collection

[0940] Visual information capture and translation

[0941] The device (smart glasses) captures the video and images the user is viewing in real time using built-in cameras and sensors to obtain visual information.

[0942] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as the name and price of the product the user is looking at.

[0943] Collection and conversion of usage history

[0944] The device (smartphone) collects usage history, such as which apps the user has used and which websites they have visited, and also periodically records location information.

[0945] The device (smartphone) converts the collected usage history into text data, for example, storing the places the user visited and the services they used as text data.

[0946] 2. Data transmission and storage

[0947] The device (smart glasses, smartphone) sends the converted text data to the server, where it is encrypted to ensure security.

[0948] The server stores the received text data in a database, including timestamps and location information.

[0949] 3. Data Analysis

[0950] The server performs preprocessing to analyze the stored text data, including removing noise and extracting necessary data.

[0951] The server uses machine learning algorithms to analyze the text data and identify user behavioral patterns and preferences, such as the stores users frequently visit and the categories of products they purchase.

[0952] 4. Generate and send ads and reminders

[0953] The server then generates personalized ads and reminders based on the analysis, such as ads for new products related to the user's past purchases or appointment reminders.

[0954] The server then sends the generated advertisements and reminders to the user's mobile device, either in real time or at a time specified by the user.

[0955] 5. User rating and reward system

[0956] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[0957] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[0958] The server analyzes the user evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are given rewards.

[0959] For example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as an advertisement. The smartphone also analyzes the user's calendar and sends a reminder of the next appointment.

[0960] This system improves user convenience, reduces the risk of missing useful information, provides efficient reminder notifications, and reduces environmental impact.

[0961] The processing flow will be explained below.

[0962] Step 1:

[0963] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[0964] Step 2:

[0965] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, image recognition technology can be used to extract product names and prices and convert them into text.

[0966] Step 3:

[0967] The device (smartphone) collects the user's location information using a GPS sensor, and also records the history of applications used and web browsing.

[0968] Step 4:

[0969] The device (smartphone) converts the collected location information and app usage history into text data, such as the names of stores visited or the titles of web pages viewed.

[0970] Step 5:

[0971] The device (smartglasses, smartphone) then sends the converted text data to the server. The data is encrypted before being transferred, ensuring security.

[0972] Step 6:

[0973] The server stores the received text data in a database, along with a timestamp and location information.

[0974] Step 7:

[0975] The server performs preprocessing on the stored text data, specifically removing noise and extracting necessary data.

[0976] Step 8:

[0977] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns and preferences, such as the stores they frequently visit or the categories of products they prefer to purchase.

[0978] Step 9:

[0979] The server then generates personalized ads and reminders based on the analysis, for example, ads for new products or appointment reminders tailored to the user's preferences.

[0980] Step 10:

[0981] The server then sends the generated advertisements and reminders to the user's mobile device, allowing the user to receive notifications in real time.

[0982] Step 11:

[0983] Users can rate the ads and reminders they receive through the application's rating system.

[0984] Step 12:

[0985] The device (smartglasses, smartphone) sends the user's evaluation data to the server. The data includes the evaluation results and the identification information of the evaluated content.

[0986] Step 13:

[0987] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the evaluation results. Information providers who receive high evaluations are given rewards.

[0988] Example 1

[0989] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0990] Accurate data collection and analysis based on user activity is necessary to efficiently deliver information and relevant advertisements that interest users in their daily lives. However, current systems struggle to effectively capture, convert, and analyze visual information and usage history, and they also struggle to deliver personalized notifications and advertisements in a timely manner. Furthermore, systems must efficiently reflect user feedback and distribute rewards to information providers and app developers.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0992] In this invention, the server includes means for capturing visual information and usage history from a user's mobile device, means for converting the captured visual information and usage history into text data, means for encrypting the converted text data and transmitting and storing it in a database, means for preprocessing the stored text data and removing noise data, means for analyzing the preprocessed data to identify user behavior patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device in real time, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables a system that efficiently performs everything from capture to analysis and reward distribution.

[0993] "User's mobile device" refers to a mobile device that a user uses on a daily basis, including smart glasses and smartphones.

[0994] "Visual information" refers to the video and image information that users see, and is obtained through devices such as smart glasses.

[0995] "Usage history" refers to a record of the applications a user uses, the websites they visit, and their location.

[0996] "Text data" refers to data that has been analyzed from visual information and usage history and converted into a format that can be saved or transmitted in text form.

[0997] "Encryption" refers to the technology of transforming data using specific algorithms to protect it from unauthorized access.

[0998] "Database" refers to a system for systematically storing and managing collected and analyzed data.

[0999] "Noise data" refers to unnecessary data or information that is not the subject of analysis.

[1000] "Preprocessing" refers to preparatory work such as organizing, cleaning, and filtering data before data analysis.

[1001] "Behavioral patterns" refer to specific behavioral tendencies derived from a user's behavioral history.

[1002] "Preferences" refers to the preferences or tendencies a user has toward certain activities or products.

[1003] "Personalized advertising" refers to advertising that is individually customized based on a user's behavioral patterns and preferences.

[1004] "Reminder" refers to an alert or message that notifies the user of an event or task.

[1005] "Rating Data" refers to data that records the ratings users give of advertisements and reminders provided to them.

[1006] "Rewards" refers to compensation or incentives given to information providers and app developers based on user ratings.

[1007] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores and analyzes the data. Specific embodiments for carrying out the invention are described below.

[1008] 1. Data Collection

[1009] Visual information capture and translation

[1010] The device (smart glasses) captures the video and images the user is viewing in real time, using built-in cameras and sensors to obtain high-resolution visual information.

[1011] The device (smart glasses) analyzes the acquired visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the necessary text data, such as the name and price of a product being viewed on a shelf.

[1012] Collection and conversion of usage history

[1013] The device (smartphone) collects the history of applications used and websites visited by the user, and periodically records location information using the Google Maps API.

[1014] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, it stores the locations visited and the services used as text data.

[1015] 2. Data transmission and storage

[1016] Data transmission

[1017] The device (smart glasses, smartphone) then sends the converted text data to the server, where AES encryption technology is used to ensure security during data transmission.

[1018] Data storage

[1019] The server stores the received text data in a database (e.g., MongoDB), adding timestamps and location information to the data for easier later analysis.

[1020] 3. Data Analysis

[1021] Pretreatment

[1022] The server performs preprocessing on the stored text data before analyzing it. In preprocessing, noise data is removed and necessary data is extracted. For example, unnecessary advertising information and duplicate data are deleted.

[1023] analysis

[1024] The server then analyzes the stored text data using machine learning algorithms such as TensorFlow, and the results of the analysis identify user behavioral patterns and preferences, such as the stores the user frequently visits and the categories of products they purchase.

[1025] 4. Generate and send ads and reminders

[1026] Advertisement and reminder generation

[1027] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or reminders for important appointments.

[1028] Sending notifications

[1029] The server then sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, which the user can easily check.

[1030] 5. User rating and reward system

[1031] Collecting ratings

[1032] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[1033] Evaluation data transmission and analysis

[1034] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[1035] The server analyzes the evaluation data and distributes rewards to information providers and app developers who receive high ratings. Rewards are distributed based on user feedback.

[1036] Specific examples

[1037] For example, imagine a user browsing a supermarket shelf. Smart glasses capture this visual information and send the product name and price as text data to a server. The server then analyzes the user's past purchase history and sends push notifications to the user's smartphone about sales on related products that the user might be interested in. If the user makes a movie reservation using an app installed on their smartphone on their way home, that history is also sent to the server as text data. At that time, the server can send a reminder of the next movie release date.

[1038] Example prompt sentence:

[1039] "Explain how smart glasses capture visual information when a user visits a nearby supermarket, analyze it as text data, and then display advertisements for related products."

[1040] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1041] Step 1:

[1042] The device (smart glasses) captures the video and images the user is looking at in real time. The smart glasses use built-in cameras and sensors to obtain high-resolution visual information. Specifically, as the user looks at a supermarket shelf, they continuously capture video of the shelf. The input is visual information (video), and the output is the captured visual data.

[1043] Step 2:

[1044] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the required text data. For example, it extracts text from the captured video and extracts product names and prices. Specifically, it sends image data to the API and receives the analysis results in return. The input is the captured visual data, and the output is the analyzed text data.

[1045] Step 3:

[1046] The device (smartphone) collects the history of applications used by the user and websites visited. It also periodically records location information using the Google Maps API. It collects which apps the user used, when, which websites they visited, and their location at that time. The input is the app usage history and location information, and the output is the collected usage history data.

[1047] Step 4:

[1048] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, the places the user visited and the services they used are saved as text data. Specifically, the collected usage history data is input into the algorithm and the analysis results are received. The input is the usage history data, and the output is the analyzed text data.

[1049] Step 5:

[1050] The device (smart glasses, smartphone) sends the converted text data to the server. The data is secured using AES encryption technology. Specifically, the acquired text data is encrypted and sent to the server using a secure communication protocol. The input is the analyzed text data, and the output is the encrypted data sent to the server.

[1051] Step 6:

[1052] The server receives the sent data and stores it in a database (e.g., MongoDB). When storing it, it adds a timestamp and location information to the text data. Specifically, it inserts the received data into the database and adds the necessary metadata. The input is encrypted text data, and the output is the data stored in the database.

[1053] Step 7:

[1054] The server performs preprocessing on the stored text data. This involves removing noise and extracting necessary data. For example, it removes duplicate data and unnecessary advertising information. The input is the stored text data, and the output is the preprocessed data.

[1055] Step 8:

[1056] The server analyzes the preprocessed data using machine learning algorithms (e.g., TensorFlow) to identify the user's behavioral patterns and preferences. For example, it identifies the stores the user frequently visits and the categories of products they purchase. The analysis results are used to generate advertisements and reminders based on the user's preferences. The input is the preprocessed data, and the output is the analysis results.

[1057] Step 9:

[1058] The server generates personalized advertisements and reminders based on the analysis results. For example, it generates advertisements for new products related to the user's past purchase history or reminders for important appointments. The input is the analysis results, and the output is the generated advertisements and reminders.

[1059] Step 10:

[1060] The server sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, so the user can easily check them. The input is the generated advertisements and reminders, and the output is the notification to the user.

[1061] Step 11:

[1062] Users rate the ads and reminders they receive. Ratings are recorded as positive (high rating) or negative (low rating). The inputs are ads and reminders, and the output is rating data.

[1063] Step 12:

[1064] The device (smart glasses, smartphone) sends the user's rating data to the server. The rating data is again sent using AES encryption technology. The input is the rating data, and the output is the encrypted rating data sent to the server.

[1065] Step 13:

[1066] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the results. The input is evaluation data, and the output is reward distribution data. Reward distribution is fair and efficient because it is based on user feedback.

[1067] (Application example 1)

[1068] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1069] Conventional systems have had difficulty efficiently analyzing users' visual information and usage history to provide personalized advertisements and reminders tailored to individual users. Furthermore, they have been unable to provide real-time product information or display recommended products when users are actually browsing products in a store. The challenge is to provide a system that overcomes these drawbacks and improves user convenience.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1071] In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for storing the converted text data in a database, means for analyzing the stored text data to identify the user's behavioral patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers, means for a visual device attached to the user's mobile device to capture visual information of product shelves and convert it into text data in real time, and means for displaying personalized advertisements and reminders on the visual display device of the user's mobile device, thereby making it possible to provide product information on the spot when the user is shopping in a physical store and display personalized reminders and recommended products in real time.

[1072] "User's mobile device" refers to electronic devices such as smartphones and smart glasses carried by the user.

[1073] "Visual information" refers to visual data such as images and videos that the user is viewing.

[1074] "Usage history" refers to behavioral data such as which applications a user used, which websites they visited, and their location information.

[1075] "Text data" refers to textual information obtained by analyzing and converting visual information and usage history.

[1076] "Database" refers to a system that stores and manages data in a structured format.

[1077] "Behavioral patterns" refer to the tendency of a user to repeatedly perform certain actions.

[1078] "Preferences" refers to the tendency of users to prefer certain products and services.

[1079] "Personalized advertising" refers to advertising that is specifically tailored to a user's behavioral patterns and preferences.

[1080] "Reminders" refer to notifications or messages sent based on a user's schedule or actions.

[1081] "Visual device" refers to a device that can capture visual information when worn by a user, such as smart glasses.

[1082] "Product shelf" refers to a shelf on which products are displayed in a store.

[1083] "Rating Data" refers to the ratings you give to advertisements and reminders you receive.

[1084] "Information Provider" refers to a party that provides advertising or reminder content.

[1085] "App Developer" refers to a person who develops an application that is installed on a user's mobile device.

[1086] The system of this invention uses a user's mobile device (such as smart glasses or a smartphone) to collect visual information and usage history, converts it into text data, analyzes it, and provides personalized advertisements and reminders. Specific embodiments are described below.

[1087] 1. Hardware Configuration

[1088] Smart glasses use built-in cameras and sensors to capture video and images of what the user is seeing in real time.

[1089] Smartphones collect usage history, such as which apps users use and which websites they visit, and also regularly record location information.

[1090] The servers have high-performance data processing capabilities and store and analyze data, using cloud services such as Amazon AWS and Microsoft Azure.

[1091] 2. Software Configuration

[1092] OpenCV and TensorFlow are used as image recognition algorithms.

[1093] Use Scikit-learn and Keras as machine learning libraries.

[1094] MySQL or PostgreSQL is used as the database management system.

[1095] React Native and Flutter are used as mobile app development environments.

[1096] 3. Data collection and conversion

[1097] The smart glasses capture visual information about the product shelves the user is looking at. The captured images are temporarily stored in the smart glasses, and image analysis is performed using OpenCV and TensorFlow. The analyzed information is converted into text data such as product name, price, and ingredient information.

[1098] Smartphones regularly collect users' location information and app usage history and convert it into text data.

[1099] 4. Data transmission and storage

[1100] The smart glasses and smartphone send the converted text data to the server, where it is encrypted using TLS / SSL.

[1101] The server stores the received text data in a MySQL or PostgreSQL database, including timestamps and location information.

[1102] 5. Data Analysis

[1103] The server performs preprocessing to analyze the stored text data, removing noise and extracting necessary data.

[1104] Machine learning algorithms (Scikit-learn and Keras) are used to analyze text data and identify user behavior patterns and preferences, which can then be used to identify the stores users frequently visit and the categories of products they purchase.

[1105] 6. Generate and send ads and reminders

[1106] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or appointment reminders.

[1107] The generated advertisements and reminders are displayed on the visual display of the user's mobile device (smart glasses or smartphone).

[1108] 7. User rating and reward system

[1109] Users rate the ads and reminders they receive, which is recorded as positive (thumbs up) or negative (thumbs down).

[1110] Smart glasses and smartphones send user evaluation data to a server, which analyzes the data and distributes rewards to information providers and app developers based on the results.

[1111] Specific examples

[1112] When a user visits a nearby supermarket, the smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as advertisements. The smartphone also analyzes the user's calendar and sends reminders of upcoming appointments.

[1113] Prompt Sentence Examples

[1114] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[1115] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1116] Step 1:

[1117] Visual information capture

[1118] The device (smart glasses) captures images of the shelves the user is looking at.

[1119] Input: Video captured by the camera built into the smart glasses.

[1120] Output: Captured shelf image data.

[1121] Specific operation: The smart glasses' camera periodically captures still images and stores the video data in its internal memory.

[1122] Step 2:

[1123] Converting visual information into text

[1124] The device (smart glasses) analyzes the captured image using OpenCV and TensorFlow and extracts text from the image (such as product name and price).

[1125] Input: Captured shelf image data.

[1126] Output: Extracted text data (product name, price, ingredient information, etc.).

[1127] Specific operation: Image preprocessing (grayscale conversion, thresholding) is performed using OpenCV, and the text portion is identified using a TensorFlow model, followed by character recognition.

[1128] Step 3:

[1129] Collection and storage of usage history

[1130] The device (smartphone) collects the user's location information and app usage history and stores it as text data.

[1131] Input: Location information obtained by the smartphone, app usage history.

[1132] Output: Location information and usage history stored as text data.

[1133] Specific operation: Using the smartphone's GPS function and app history function, data is collected at regular intervals and stored in the internal memory.

[1134] Step 4:

[1135] Sending data

[1136] The device (smart glasses, smartphone) sends the converted text data to the server.

[1137] Input: Extracted text data.

[1138] Output: The encrypted data sent to the server.

[1139] What it does: Sends text data over the internet to a server using TLS / SSL encryption.

[1140] Step 5:

[1141] Data storage

[1142] The server stores the received text data in a database.

[1143] Input: The encrypted data sent to the server.

[1144] Output: Text data stored in a database.

[1145] Specific operation: The server confirms receipt of the data and stores it in a MySQL or PostgreSQL database.

[1146] Step 6:

[1147] Data analysis

[1148] The server analyzes the stored text data to identify the user's behavioral patterns and preferences.

[1149] Input: Text data stored in a database.

[1150] Output: Analysis of user behavior patterns and preferences.

[1151] What it does: It uses machine learning algorithms to analyze text data and extract features related to behavioral patterns and preferences.

[1152] Step 7:

[1153] Advertisement and reminder generation

[1154] The server generates personalized advertisements and reminders based on the analysis results.

[1155] Input: Analysis of user behavior patterns and preferences.

[1156] Output: Personalized ads and reminders.

[1157] Specific operation: Extracts relevant products and services from the analysis results and generates ads and reminders that are best suited to the user.

[1158] Step 8:

[1159] Sending Advertisements and Reminders

[1160] The server transmits the generated advertisements and reminders to the user's mobile device.

[1161] Enter: personalized ads and reminders.

[1162] Output: Advertisements and reminders displayed on the user's mobile device.

[1163] Specific operation: The server sends a notification to the user's smart glasses or smartphone, which is then displayed on the visual device.

[1164] Prompt Sentence Examples

[1165] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[1166] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1167] This invention relates to a system that captures visual information and usage history from a user's mobile device, converts it into text data, and stores it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized advertisements and reminders.

[1168] 1. Data Collection

[1169] Visual information capture and translation

[1170] The device (smart glasses) captures the video and images the user is viewing in real time, using the built-in camera and sensors to acquire visual information and store it in temporary memory.

[1171] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as extracting product names and prices from a shop shelf.

[1172] Collecting and converting emotional data

[1173] The device (smartglasses, smartphone) captures the user's facial expressions and voice and extracts emotional data using an emotion engine. For example, facial expression recognition technology and voice analysis are used to identify the user's emotions, such as joy or surprise.

[1174] The device (smart glasses, smartphone) sends the extracted emotion data together with the text data to the server.

[1175] Collection and conversion of usage history

[1176] The device (smartphone) periodically collects the user's location information and app usage history, as well as records the user's web browsing history.

[1177] The device (smartphone) converts the collected usage history into text data, such as the names of places visited and apps used.

[1178] 2. Data transmission and storage

[1179] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[1180] The server stores the received data in a database, including timestamps and location information.

[1181] 3. Data Analysis

[1182] The server performs preprocessing on the stored text data and emotion data, removing noise and extracting only the necessary information.

[1183] The server uses machine learning algorithms to analyze the text and emotional data to identify user behavioral patterns, preferences, and emotional tendencies, such as frequently visited places, favorite product categories, and the emotions users have toward specific products.

[1184] 4. Generate and send ads and reminders

[1185] The server generates personalized advertisements and reminders based on the analysis results, matching the user's preferences and emotions. For example, it notifies the user of information related to a product they are interested in, along with a feeling of joy.

[1186] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[1187] 5. User rating and reward system

[1188] Users can rate the ads and reminders they receive, either as a thumbs up or a thumbs down, through the application.

[1189] The device (smart glasses, smartphone) sends user rating data to the server, which includes identification information for advertisements and reminders.

[1190] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are rewarded, which increases the usefulness of the system.

[1191] As a specific example, when a user visits a nearby supermarket and their smart glasses capture visual information about the shelves, the emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[1192] This system will enable users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[1193] The processing flow will be explained below.

[1194] Step 1:

[1195] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[1196] Step 2:

[1197] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, it recognizes product names and prices and extracts them as text data.

[1198] Step 3:

[1199] The device (smartphone) periodically acquires the user's location information using a GPS sensor, and also records the history of applications used and web browsing history.

[1200] Step 4:

[1201] The device (smartphone) converts the collected location information and usage history into text data. For example, it stores the names of places visited and applications used as text data.

[1202] Step 5:

[1203] The device (smartglasses, smartphone) captures the user's facial expressions and voice and uses an emotion engine to extract emotional data. For example, facial expression recognition technology can analyze the user's smiling or surprised expressions to recognize their emotions.

[1204] Step 6:

[1205] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[1206] Step 7:

[1207] The server stores the received text data and emotion data in a database, along with timestamps and location information.

[1208] Step 8:

[1209] The server performs preprocessing on the stored data, specifically removing noise data and extracting necessary information.

[1210] Step 9:

[1211] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns, preferences, and emotional trends, such as the stores a user frequently visits and the product categories they prefer.

[1212] Step 10:

[1213] The server generates personalized advertisements and reminders based on the analysis results, and also takes into account emotional data to provide information that matches the user's current emotions, such as providing information related to products that the user has shown interest in.

[1214] Step 11:

[1215] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[1216] Step 12:

[1217] Users can rate the ads and reminders they receive through the application's rating system.

[1218] Step 13:

[1219] The device (smartglasses, smartphone) sends user rating data to the server, including identification information for advertisements and reminders.

[1220] Step 14:

[1221] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are given rewards.

[1222] Example 2

[1223] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1224] Conventional systems required large amounts of manual data entry to identify user preferences and behavioral patterns, making it difficult to provide personalized advertisements and reminders in real time. Furthermore, they were unable to understand user emotions and respond based on those emotions, limiting the user experience. Furthermore, they lacked a mechanism for properly evaluating user feedback and providing it back to information providers and app developers. There is a need to resolve these issues and provide a more accurate personalization system.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1226] In this invention, the server includes means for capturing visual information, audio data, and usage history from the user's mobile device, means for converting the captured visual information, audio data, and usage history into text data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables real-time personalization based on the user's preferences and emotions, improving the user experience and realizing an efficient feedback system.

[1227] "User's mobile device" refers to an electronic device that the user can carry and use at all times, including smartphones and smart glasses.

[1228] "Visual information" refers to video and image data captured from the user's perspective.

[1229] "Audio data" refers to data recorded from the user's voice and surrounding sounds.

[1230] "Usage history" refers to recorded data about the applications a user has used and the places they have visited.

[1231] "Text data" refers to textual information extracted by analyzing visual information, audio data, and usage history.

[1232] "Emotional data" refers to information about the user's emotional state analyzed from their facial expressions and voice.

[1233] "Database" refers to a system for storing and managing text data and emotional data.

[1234] "Behavioral patterns" refers to information that shows a user's habits and tendencies toward frequent actions.

[1235] "Preferences" refers to information that indicates a user's preferences and interests.

[1236] "Emotional trends" refer to trends that indicate changes in a user's mood or emotions.

[1237] "Personalized advertising" refers to advertising that is customized based on a user's behavioral patterns, preferences, and emotional tendencies.

[1238] "Reminder" refers to a message that notifies the user of a specific event or matter.

[1239] "Rating Data" refers to information about the ratings you give to advertisements and reminders you receive.

[1240] "Information Provider" refers to an individual or organization that provides information or data to the System.

[1241] "App developer" refers to an individual or organization that develops application software that runs on the system.

[1242] The present invention is a system that captures visual information, audio data, and usage history from a user's mobile device, converts it into text data, stores it, and combines it with an emotion engine to provide more personalized advertisements and reminders.

[1243] First, the device (such as smart glasses or a smartphone) captures the user's visual information and audio data. Specifically, the smart glasses' built-in camera and sensors are used to capture the video and images the user is viewing in real time. The smart glasses or smartphone also record the user's voice and analyze it to extract emotional data. This data is temporarily stored in the device's memory.

[1244] Next, the device analyzes the captured visual information using an image recognition algorithm (e.g., OpenCV or TensorFlow) and converts it into text data. Similarly, the audio data is converted into text data using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text). For example, product names and prices on store shelves, and user comments can be extracted as text data.

[1245] The converted text and emotion data is then sent from the device to a server, where it is encrypted to ensure security. The server then stores the received data in a database, including timestamps and location information.

[1246] The server preprocesses the stored text and emotion data, filtering out noise and extracting only the necessary information. It then uses machine learning algorithms (e.g., sparse vectors, natural language processing models) to analyze the text and emotion data and identify user behavioral patterns, preferences, and emotional trends. This allows it to understand frequently visited locations, favorite product categories, and emotions toward specific products.

[1247] Based on the analysis results, the server generates personalized advertisements and reminders that match the user's preferences and emotions. The generated advertisements and reminders are sent from the server to the user's mobile device. Notifications are delivered in real time, with timing adjusted based on the user's current situation and emotions.

[1248] Finally, users rate the ads and reminders they receive. Ratings are made through the application as either likes or dislikes, and the rating data is sent from the device to the server. The server analyzes the rating data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are rewarded, which increases the usefulness of the system.

[1249] As a concrete example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves, and an emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[1250] Example prompt for a generative AI model:

[1251] "Please provide details about the system that captures visual information and usage history and converts it into text data. Also, please explain specifically how the emotion engine is used to provide personalized advertising."

[1252] This system allows users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[1253] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1254] Detailed flow of processing steps

[1255] Step 1: Capture the data

[1256] Input: Device worn by the user (smart glasses, smartphone)

[1257] How it works: The device captures the visual and audio data of what the user sees in real time. Specifically, the smart glasses' built-in camera captures the video, and the smartphone's microphone captures the audio.

[1258] Output: Visual information (video data) and audio data

[1259] Step 2: Temporarily save data

[1260] Input: Visual and audio data

[1261] Specific operation: The captured data is stored in the device's temporary memory and used for subsequent processing.

[1262] Output: Temporarily saved visual information and audio data

[1263] Step 3: Image recognition of visual information

[1264] Input: Temporarily saved visual information

[1265] Specific operation: The device uses an image recognition algorithm (e.g., OpenCV or TensorFlow) to analyze the captured video and convert it into text data. For example, it can extract product names and prices from video of a product shelf and convert them into text data.

[1266] Output: Text data (e.g. product name and price)

[1267] Step 4: Analyzing the audio data

[1268] Input: Temporarily saved audio data

[1269] Specific operation: The device uses a speech analysis algorithm (e.g., Google Cloud Speech-to-Text) to analyze the voice data and convert it into text data, for example, transcribing what the user is saying.

[1270] Output: Text data (e.g., speech content)

[1271] Step 5: Extracting emotion data

[1272] Input: Captured visual information (user facial expressions) and audio data

[1273] Specific operation: The device combines facial expression recognition technology and voice analysis to input data into the emotion engine, analyze the user's emotional state (e.g., joy, surprise), and extract emotion data.

[1274] Output: Emotion data (e.g., joy, surprise)

[1275] Step 6: Encrypt and send data

[1276] Input: Text data and emotion data

[1277] Specific operation: The terminal encrypts this data using an encryption algorithm such as AES and transmits the data to the server using the HTTPS protocol.

[1278] Output: Encrypted text data and sentiment data

[1279] Step 7: Receiving and storing data

[1280] Input: Encrypted text data and emotion data

[1281] How it works: The server receives the data, decrypts it, and stores it in a database, along with a timestamp and location information.

[1282] Output: Text data and emotion data stored in a database

[1283] Step 8: Data Preprocessing

[1284] Input: Stored text data and emotion data

[1285] Specific operation: The server performs preprocessing such as removing noise data and normalizing it, and prepares the data in a format suitable for analysis.

[1286] Output: Preprocessed text and sentiment data

[1287] Step 9: Data analysis

[1288] Input: Preprocessed text data and sentiment data

[1289] How it works: The server uses machine learning algorithms to analyze the data and identify user behavioral patterns, preferences, and emotional tendencies.

[1290] Output: Analysis results (e.g., behavioral patterns, preferences, emotional tendencies)

[1291] Step 10: Generate Ads and Reminders

[1292] Input: Analysis results

[1293] Specific operation: Based on the analysis results, the server automatically generates personalized advertisements and reminders appropriate for the user.

[1294] Output: Generated ads and reminders

[1295] Step 11: Sending Ads and Reminders

[1296] Input: Generated ads and reminders

[1297] How it works: The server sends the generated advertisements and reminders to the user's mobile device, again in real time, with the timing of notifications adjusted accordingly.

[1298] Output: Advertisements and reminders sent to the user's mobile device

[1299] Step 12: Gather user ratings and feedback

[1300] Input: User's ratings of the ads and reminders they received

[1301] Specific operation: Users use a smartphone application to make ratings, and the rating data is sent from the device to the server.

[1302] Output: Rating data sent to the server

[1303] Step 13: Analyze evaluation data and distribute rewards

[1304] Input: Evaluation data

[1305] Specific operation: The server analyzes the evaluation data and automatically distributes rewards to information providers and app developers based on the results.

[1306] Output: Reward distribution to informants and app developers

[1307] The above is the flow of specific processing steps of this system.

[1308] (Application example 2)

[1309] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1310] Conventional content delivery systems have difficulty in providing personalized content based on individual user preferences and emotions, making it difficult to improve user experience. In addition, user evaluation data is not effectively reflected, and reward distribution to information providers and application developers is insufficient.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for capturing and extracting user emotional data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized content and reminders based on the analysis results, means for transmitting the generated content and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and application developers. This makes it possible to provide more personalized content based on the user's preferences and emotions and to realize a reward system based on user evaluations.

[1312] "User's mobile device" refers to an electronic device that a user carries and uses on a daily basis, including a smartphone, smart glasses, or tablet device.

[1313] "Visual information" refers to the images and videos that a user sees with their eyes, captured through cameras and sensors.

[1314] "Usage history" refers to a record of the operations and actions a user performs using a mobile device, including application usage data and internet browsing history.

[1315] "Text data" refers to captured visual information, usage history, and emotional data expressed as text information.

[1316] "Emotion data" refers to data that indicates the emotional state of a user extracted from their facial expressions and voice, and includes emotional information such as joy, anger, sadness, and happiness.

[1317] A "database" is a computer system for storing and managing collected text data and emotion data.

[1318] "Analyzing" means processing stored data using machine learning algorithms and data analysis techniques to identify user behavioral patterns, preferences, and emotional tendencies.

[1319] "Personalized content" refers to information and entertainment tailored to a user's individual preferences and emotions based on analytical results, including advertisements and reminders.

[1320] A "reminder" is a notification that reminds the user of a specific action or event.

[1321] "Rating Data" means the ratings and feedback you provide about the content and reminders you receive.

[1322] "Information Provider" means an individual or organization that provides content or information to Users.

[1323] "Application Developer" means the individual or organization that develops the applicable application software.

[1324] "Reward" means monetary or material consideration given to information providers or application developers based on user ratings.

[1325] This system captures visual information and usage history from a user's mobile device, converts it into text data, saves it, and analyzes it to provide personalized content. Furthermore, by incorporating user emotional data, more precise personalization can be achieved.

[1326] 1. Data Collection

[1327] Visual information capture and translation

[1328] The device (smart glasses) captures the video and images the user is looking at in real time. It uses the built-in camera and sensors to acquire visual information and stores it in temporary memory. The captured visual information is analyzed using image recognition algorithms such as TensorFlow and converted into text data. For example, if a user views a movie poster, the movie title and related information are extracted as text data.

[1329] Collecting and converting emotional data

[1330] The device (smart glasses, smartphone) captures the user's facial expressions and voice and uses facial recognition and speech analysis technologies to extract emotional data, such as whether the user is enjoying a movie trailer.

[1331] Collection and conversion of usage history

[1332] The device (smartphone) periodically collects the user's location information and app usage history, and converts this information into text data using the Google Maps API and Firebase. For example, it records the names of recently visited places and used applications.

[1333] 2. Data transmission and storage

[1334] The device (smart glasses, smartphone) sends the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The server then stores this data in a database.

[1335] 3. Data Analysis

[1336] The server preprocesses the stored text and sentiment data and removes noise, leaving only the necessary information. It then uses machine learning algorithms to analyze the text and sentiment data and identify user behavioral patterns, preferences, and emotional trends. Specifically, it uses Amazon SageMaker and Azure Machine Learning.

[1337] 4. Content and reminder generation and sending

[1338] The server uses the analytics to generate personalized content and reminders, such as trailers for upcoming movies the user enjoys, using Firebase Cloud Messaging and Apple Push Notification Service.

[1339] 5. User rating and reward system

[1340] Users rate the content and reminders they receive and send the rating data to the server. The server analyzes the rating data and distributes rewards to information providers and application developers. Reward distribution uses Firebase Realtime Database and Google Analytics.

[1341] Specific examples

[1342] A concrete example would be a scenario where a user looks at a movie poster with smart glasses, and the visual and emotional information is captured, and a notification is sent to the smartphone saying, "Here's the next trending movie. Would you like to watch it now?"

[1343] Prompt Sentence Examples

[1344] "If a user is enjoying looking at a movie poster on their smart glasses, capture that information and send a movie trailer notification to their smartphone saying: 'Here's the next trending movie. Want to watch it now?'"

[1345] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1346] Step 1:

[1347] The device (smart glasses) captures the video the user is looking at in real time using a built-in camera and stores the video data in temporary memory. The input is real-time video data, and the output is captured video data stored in temporary memory.

[1348] Step 2:

[1349] The device (smart glasses) analyzes the captured video data using an image recognition algorithm (e.g., TensorFlow) and converts specific information in the video into text data. For example, in the case of a movie poster, the title and director's name are extracted. The input is video data, and the output is text data.

[1350] Step 3:

[1351] The device (smart glasses and smartphone) captures the user's facial expressions and voice and converts the data into emotion data using an emotion analysis engine (e.g., OpenCV or Google Cloud Speech-to-Text API). The input is the user's facial expression data and voice data, and the output is the analyzed emotion data.

[1352] Step 4:

[1353] The device (smartphone) periodically collects the user's location information and application usage history. This is done using the Google Maps API and Firebase, and this usage history is converted into text data. The input is location data and application usage data, and the output is text data.

[1354] Step 5:

[1355] The devices (smart glasses and smartphones) transmit the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The input is text data and emotion data, and the output is data transmission to the server.

[1356] Step 6:

[1357] The server stores the received data in a database, including timestamps and location information. The input is text data and emotion data, and the output is an entry stored in the database.

[1358] Step 7:

[1359] The server preprocesses the stored text and emotion data to remove noise. Filtering techniques are used for preprocessing to extract only the necessary information. The input is the stored data, and the output is the preprocessed data.

[1360] Step 8:

[1361] The server analyzes the preprocessed data using machine learning algorithms (e.g., Amazon SageMaker or Azure Machine Learning) to identify user behavioral patterns, preferences, and emotional trends. The input is the preprocessed data, and the output is the analysis results.

[1362] Step 9:

[1363] The server generates personalized content and reminders based on the analysis results. It uses a generative AI model to create content that matches the user's preferences and emotions. The input is the analysis results, and the output is the generated content and reminders.

[1364] Step 10:

[1365] The server sends the generated content and reminders to the user's device using Firebase Cloud Messaging or Apple Push Notification Service. The input is the generated content and reminders, and the output is notifications sent to the user's device.

[1366] Step 11:

[1367] Users rate the content and reminders they receive and send the rating data to the server via their devices (smart glasses or smartphones). The input is the user rating data, and the output is the transmission of the rating data to the server.

[1368] Step 12:

[1369] The server analyzes the evaluation data and distributes rewards to information providers and application developers based on the results. This uses Firebase Realtime Database and Google Analytics. The input is the evaluation data, and the output is the distribution of rewards.

[1370] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1372] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1373] [Fourth embodiment]

[1374] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1375] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1376] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1377] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1378] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1379] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1380] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1381] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1382] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1385] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1386] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1387] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores the text data. Specific embodiments will be described below.

[1388] 1. Data Collection

[1389] Visual information capture and translation

[1390] The device (smart glasses) captures the video and images the user is viewing in real time using built-in cameras and sensors to obtain visual information.

[1391] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as the name and price of the product the user is looking at.

[1392] Collection and conversion of usage history

[1393] The device (smartphone) collects usage history, such as which apps the user has used and which websites they have visited, and also periodically records location information.

[1394] The device (smartphone) converts the collected usage history into text data, for example, storing the places the user visited and the services they used as text data.

[1395] 2. Data transmission and storage

[1396] The device (smart glasses, smartphone) sends the converted text data to the server, where it is encrypted to ensure security.

[1397] The server stores the received text data in a database, including timestamps and location information.

[1398] 3. Data Analysis

[1399] The server performs preprocessing to analyze the stored text data, including removing noise and extracting necessary data.

[1400] The server uses machine learning algorithms to analyze the text data and identify user behavioral patterns and preferences, such as the stores users frequently visit and the categories of products they purchase.

[1401] 4. Generate and send ads and reminders

[1402] The server then generates personalized ads and reminders based on the analysis, such as ads for new products related to the user's past purchases or appointment reminders.

[1403] The server then sends the generated advertisements and reminders to the user's mobile device, either in real time or at a time specified by the user.

[1404] 5. User rating and reward system

[1405] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[1406] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[1407] The server analyzes the user evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are given rewards.

[1408] For example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as an advertisement. The smartphone also analyzes the user's calendar and sends a reminder of the next appointment.

[1409] This system improves user convenience, reduces the risk of missing useful information, provides efficient reminder notifications, and reduces environmental impact.

[1410] The processing flow will be explained below.

[1411] Step 1:

[1412] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[1413] Step 2:

[1414] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, image recognition technology can be used to extract product names and prices and convert them into text.

[1415] Step 3:

[1416] The device (smartphone) collects the user's location information using a GPS sensor, and also records the history of applications used and web browsing.

[1417] Step 4:

[1418] The device (smartphone) converts the collected location information and app usage history into text data, such as the names of stores visited or the titles of web pages viewed.

[1419] Step 5:

[1420] The device (smartglasses, smartphone) then sends the converted text data to the server. The data is encrypted before being transferred, ensuring security.

[1421] Step 6:

[1422] The server stores the received text data in a database, along with a timestamp and location information.

[1423] Step 7:

[1424] The server performs preprocessing on the stored text data, specifically removing noise and extracting necessary data.

[1425] Step 8:

[1426] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns and preferences, such as the stores they frequently visit or the categories of products they prefer to purchase.

[1427] Step 9:

[1428] The server then generates personalized ads and reminders based on the analysis, for example, ads for new products or appointment reminders tailored to the user's preferences.

[1429] Step 10:

[1430] The server then sends the generated advertisements and reminders to the user's mobile device, allowing the user to receive notifications in real time.

[1431] Step 11:

[1432] Users can rate the ads and reminders they receive through the application's rating system.

[1433] Step 12:

[1434] The device (smartglasses, smartphone) sends the user's evaluation data to the server. The data includes the evaluation results and the identification information of the evaluated content.

[1435] Step 13:

[1436] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the evaluation results. Information providers who receive high evaluations are given rewards.

[1437] Example 1

[1438] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1439] Accurate data collection and analysis based on user activity is necessary to efficiently deliver information and relevant advertisements that interest users in their daily lives. However, current systems struggle to effectively capture, convert, and analyze visual information and usage history, and they also struggle to deliver personalized notifications and advertisements in a timely manner. Furthermore, systems must efficiently reflect user feedback and distribute rewards to information providers and app developers.

[1440] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1441] In this invention, the server includes means for capturing visual information and usage history from a user's mobile device, means for converting the captured visual information and usage history into text data, means for encrypting the converted text data and transmitting and storing it in a database, means for preprocessing the stored text data and removing noise data, means for analyzing the preprocessed data to identify user behavior patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device in real time, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables a system that efficiently performs everything from capture to analysis and reward distribution.

[1442] "User's mobile device" refers to a mobile device that a user uses on a daily basis, including smart glasses and smartphones.

[1443] "Visual information" refers to the video and image information that users see, and is obtained through devices such as smart glasses.

[1444] "Usage history" refers to a record of the applications a user uses, the websites they visit, and their location.

[1445] "Text data" refers to data that has been analyzed from visual information and usage history and converted into a format that can be saved or transmitted in text form.

[1446] "Encryption" refers to the technology of transforming data using specific algorithms to protect it from unauthorized access.

[1447] "Database" refers to a system for systematically storing and managing collected and analyzed data.

[1448] "Noise data" refers to unnecessary data or information that is not the subject of analysis.

[1449] "Preprocessing" refers to preparatory work such as organizing, cleaning, and filtering data before data analysis.

[1450] "Behavioral patterns" refer to specific behavioral tendencies derived from a user's behavioral history.

[1451] "Preferences" refers to the preferences or tendencies a user has toward certain activities or products.

[1452] "Personalized advertising" refers to advertising that is individually customized based on a user's behavioral patterns and preferences.

[1453] "Reminder" refers to an alert or message that notifies the user of an event or task.

[1454] "Rating Data" refers to data that records the ratings users give of advertisements and reminders provided to them.

[1455] "Rewards" refers to compensation or incentives given to information providers and app developers based on user ratings.

[1456] The present invention relates to a system that captures visual information and usage history from a user's mobile device, converts the captured information into text data, and stores and analyzes the data. Specific embodiments for carrying out the invention are described below.

[1457] 1. Data Collection

[1458] Visual information capture and translation

[1459] The device (smart glasses) captures the video and images the user is viewing in real time, using built-in cameras and sensors to obtain high-resolution visual information.

[1460] The device (smart glasses) analyzes the acquired visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the necessary text data, such as the name and price of a product being viewed on a shelf.

[1461] Collection and conversion of usage history

[1462] The device (smartphone) collects the history of applications used and websites visited by the user, and periodically records location information using the Google Maps API.

[1463] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, it stores the locations visited and the services used as text data.

[1464] 2. Data transmission and storage

[1465] Data transmission

[1466] The device (smart glasses, smartphone) then sends the converted text data to the server, where AES encryption technology is used to ensure security during data transmission.

[1467] Data storage

[1468] The server stores the received text data in a database (e.g., MongoDB), adding timestamps and location information to the data for easier later analysis.

[1469] 3. Data Analysis

[1470] Pretreatment

[1471] The server performs preprocessing on the stored text data before analyzing it. In preprocessing, noise data is removed and necessary data is extracted. For example, unnecessary advertising information and duplicate data are deleted.

[1472] analysis

[1473] The server then analyzes the stored text data using machine learning algorithms such as TensorFlow, and the results of the analysis identify user behavioral patterns and preferences, such as the stores the user frequently visits and the categories of products they purchase.

[1474] 4. Generate and send ads and reminders

[1475] Advertisement and reminder generation

[1476] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or reminders for important appointments.

[1477] Sending notifications

[1478] The server then sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, which the user can easily check.

[1479] 5. User rating and reward system

[1480] Collecting ratings

[1481] Users rate the ads and reminders they receive, which are recorded as positive (thumbs up) or negative (thumbs down).

[1482] Evaluation data transmission and analysis

[1483] The device (smart glasses, smartphone) sends the user's evaluation data to the server.

[1484] The server analyzes the evaluation data and distributes rewards to information providers and app developers who receive high ratings. Rewards are distributed based on user feedback.

[1485] Specific examples

[1486] For example, imagine a user browsing a supermarket shelf. Smart glasses capture this visual information and send the product name and price as text data to a server. The server then analyzes the user's past purchase history and sends push notifications to the user's smartphone about sales on related products that the user might be interested in. If the user makes a movie reservation using an app installed on their smartphone on their way home, that history is also sent to the server as text data. At that time, the server can send a reminder of the next movie release date.

[1487] Example prompt sentence:

[1488] "Explain how smart glasses capture visual information when a user visits a nearby supermarket, analyze it as text data, and then display advertisements for related products."

[1489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1490] Step 1:

[1491] The device (smart glasses) captures the video and images the user is looking at in real time. The smart glasses use built-in cameras and sensors to obtain high-resolution visual information. Specifically, as the user looks at a supermarket shelf, they continuously capture video of the shelf. The input is visual information (video), and the output is the captured visual data.

[1492] Step 2:

[1493] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm (e.g., Google Cloud Vision API) and converts it into the required text data. For example, it extracts text from the captured video and extracts product names and prices. Specifically, it sends image data to the API and receives the analysis results in return. The input is the captured visual data, and the output is the analyzed text data.

[1494] Step 3:

[1495] The device (smartphone) collects the history of applications used by the user and websites visited. It also periodically records location information using the Google Maps API. It collects which apps the user used, when, which websites they visited, and their location at that time. The input is the app usage history and location information, and the output is the collected usage history data.

[1496] Step 4:

[1497] The device (smartphone) analyzes the collected usage history using a natural language processing algorithm and converts it into text data. For example, the places the user visited and the services they used are saved as text data. Specifically, the collected usage history data is input into the algorithm and the analysis results are received. The input is the usage history data, and the output is the analyzed text data.

[1498] Step 5:

[1499] The device (smart glasses, smartphone) sends the converted text data to the server. The data is secured using AES encryption technology. Specifically, the acquired text data is encrypted and sent to the server using a secure communication protocol. The input is the analyzed text data, and the output is the encrypted data sent to the server.

[1500] Step 6:

[1501] The server receives the sent data and stores it in a database (e.g., MongoDB). When storing it, it adds a timestamp and location information to the text data. Specifically, it inserts the received data into the database and adds the necessary metadata. The input is encrypted text data, and the output is the data stored in the database.

[1502] Step 7:

[1503] The server performs preprocessing on the stored text data. This involves removing noise and extracting necessary data. For example, it removes duplicate data and unnecessary advertising information. The input is the stored text data, and the output is the preprocessed data.

[1504] Step 8:

[1505] The server analyzes the preprocessed data using machine learning algorithms (e.g., TensorFlow) to identify the user's behavioral patterns and preferences. For example, it identifies the stores the user frequently visits and the categories of products they purchase. The analysis results are used to generate advertisements and reminders based on the user's preferences. The input is the preprocessed data, and the output is the analysis results.

[1506] Step 9:

[1507] The server generates personalized advertisements and reminders based on the analysis results. For example, it generates advertisements for new products related to the user's past purchase history or reminders for important appointments. The input is the analysis results, and the output is the generated advertisements and reminders.

[1508] Step 10:

[1509] The server sends the generated advertisements and reminders to the user's mobile device in real time or at a specified time. The notifications are in the form of push notifications, so the user can easily check them. The input is the generated advertisements and reminders, and the output is the notification to the user.

[1510] Step 11:

[1511] Users rate the ads and reminders they receive. Ratings are recorded as positive (high rating) or negative (low rating). The inputs are ads and reminders, and the output is rating data.

[1512] Step 12:

[1513] The device (smart glasses, smartphone) sends the user's rating data to the server. The rating data is again sent using AES encryption technology. The input is the rating data, and the output is the encrypted rating data sent to the server.

[1514] Step 13:

[1515] The server analyzes the received evaluation data and distributes rewards to information providers and app developers based on the results. The input is evaluation data, and the output is reward distribution data. Reward distribution is fair and efficient because it is based on user feedback.

[1516] (Application example 1)

[1517] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1518] Conventional systems have had difficulty efficiently analyzing users' visual information and usage history to provide personalized advertisements and reminders tailored to individual users. Furthermore, they have been unable to provide real-time product information or display recommended products when users are actually browsing products in a store. The challenge is to provide a system that overcomes these drawbacks and improves user convenience.

[1519] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1520] In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for storing the converted text data in a database, means for analyzing the stored text data to identify the user's behavioral patterns and preferences, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers, means for a visual device attached to the user's mobile device to capture visual information of product shelves and convert it into text data in real time, and means for displaying personalized advertisements and reminders on the visual display device of the user's mobile device, thereby making it possible to provide product information on the spot when the user is shopping in a physical store and display personalized reminders and recommended products in real time.

[1521] "User's mobile device" refers to electronic devices such as smartphones and smart glasses carried by the user.

[1522] "Visual information" refers to visual data such as images and videos that the user is viewing.

[1523] "Usage history" refers to behavioral data such as which applications a user used, which websites they visited, and their location information.

[1524] "Text data" refers to textual information obtained by analyzing and converting visual information and usage history.

[1525] "Database" refers to a system that stores and manages data in a structured format.

[1526] "Behavioral patterns" refer to the tendency of a user to repeatedly perform certain actions.

[1527] "Preferences" refers to the tendency of users to prefer certain products and services.

[1528] "Personalized advertising" refers to advertising that is specifically tailored to a user's behavioral patterns and preferences.

[1529] "Reminders" refer to notifications or messages sent based on a user's schedule or actions.

[1530] "Visual device" refers to a device that can capture visual information when worn by a user, such as smart glasses.

[1531] "Product shelf" refers to a shelf on which products are displayed in a store.

[1532] "Rating Data" refers to the ratings you give to advertisements and reminders you receive.

[1533] "Information Provider" refers to a party that provides advertising or reminder content.

[1534] "App Developer" refers to a person who develops an application that is installed on a user's mobile device.

[1535] The system of this invention uses a user's mobile device (such as smart glasses or a smartphone) to collect visual information and usage history, converts it into text data, analyzes it, and provides personalized advertisements and reminders. Specific embodiments are described below.

[1536] 1. Hardware Configuration

[1537] Smart glasses use built-in cameras and sensors to capture video and images of what the user is seeing in real time.

[1538] Smartphones collect usage history, such as which apps users use and which websites they visit, and also regularly record location information.

[1539] The servers have high-performance data processing capabilities and store and analyze data, using cloud services such as Amazon AWS and Microsoft Azure.

[1540] 2. Software Configuration

[1541] OpenCV and TensorFlow are used as image recognition algorithms.

[1542] Use Scikit-learn and Keras as machine learning libraries.

[1543] MySQL or PostgreSQL is used as the database management system.

[1544] React Native and Flutter are used as mobile app development environments.

[1545] 3. Data collection and conversion

[1546] The smart glasses capture visual information about the product shelves the user is looking at. The captured images are temporarily stored in the smart glasses, and image analysis is performed using OpenCV and TensorFlow. The analyzed information is converted into text data such as product name, price, and ingredient information.

[1547] Smartphones regularly collect users' location information and app usage history and convert it into text data.

[1548] 4. Data transmission and storage

[1549] The smart glasses and smartphone send the converted text data to the server, where it is encrypted using TLS / SSL.

[1550] The server stores the received text data in a MySQL or PostgreSQL database, including timestamps and location information.

[1551] 5. Data Analysis

[1552] The server performs preprocessing to analyze the stored text data, removing noise and extracting necessary data.

[1553] Machine learning algorithms (Scikit-learn and Keras) are used to analyze text data and identify user behavior patterns and preferences, which can then be used to identify the stores users frequently visit and the categories of products they purchase.

[1554] 6. Generate and send ads and reminders

[1555] The server generates personalized advertisements and reminders based on the analysis results, such as advertisements for new products related to the user's past purchases or appointment reminders.

[1556] The generated advertisements and reminders are displayed on the visual display of the user's mobile device (smart glasses or smartphone).

[1557] 7. User rating and reward system

[1558] Users rate the ads and reminders they receive, which is recorded as positive (thumbs up) or negative (thumbs down).

[1559] Smart glasses and smartphones send user evaluation data to a server, which analyzes the data and distributes rewards to information providers and app developers based on the results.

[1560] Specific examples

[1561] When a user visits a nearby supermarket, the smart glasses capture visual information about the shelves and send it to a server. The server analyzes the user's past purchase history and sends related sales information to the smartphone as advertisements. The smartphone also analyzes the user's calendar and sends reminders of upcoming appointments.

[1562] Prompt Sentence Examples

[1563] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[1564] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1565] Step 1:

[1566] Visual information capture

[1567] The device (smart glasses) captures images of the shelves the user is looking at.

[1568] Input: Video captured by the camera built into the smart glasses.

[1569] Output: Captured shelf image data.

[1570] Specific operation: The smart glasses' camera periodically captures still images and stores the video data in its internal memory.

[1571] Step 2:

[1572] Converting visual information into text

[1573] The device (smart glasses) analyzes the captured image using OpenCV and TensorFlow and extracts text from the image (such as product name and price).

[1574] Input: Captured shelf image data.

[1575] Output: Extracted text data (product name, price, ingredient information, etc.).

[1576] Specific operation: Image preprocessing (grayscale conversion, thresholding) is performed using OpenCV, and the text portion is identified using a TensorFlow model, followed by character recognition.

[1577] Step 3:

[1578] Collection and storage of usage history

[1579] The device (smartphone) collects the user's location information and app usage history and stores it as text data.

[1580] Input: Location information obtained by the smartphone, app usage history.

[1581] Output: Location information and usage history stored as text data.

[1582] Specific operation: Using the smartphone's GPS function and app history function, data is collected at regular intervals and stored in the internal memory.

[1583] Step 4:

[1584] Sending data

[1585] The device (smart glasses, smartphone) sends the converted text data to the server.

[1586] Input: Extracted text data.

[1587] Output: The encrypted data sent to the server.

[1588] What it does: Sends text data over the internet to a server using TLS / SSL encryption.

[1589] Step 5:

[1590] Data storage

[1591] The server stores the received text data in a database.

[1592] Input: The encrypted data sent to the server.

[1593] Output: Text data stored in a database.

[1594] Specific operation: The server confirms receipt of the data and stores it in a MySQL or PostgreSQL database.

[1595] Step 6:

[1596] Data analysis

[1597] The server analyzes the stored text data to identify the user's behavioral patterns and preferences.

[1598] Input: Text data stored in a database.

[1599] Output: Analysis of user behavior patterns and preferences.

[1600] What it does: It uses machine learning algorithms to analyze text data and extract features related to behavioral patterns and preferences.

[1601] Step 7:

[1602] Advertisement and reminder generation

[1603] The server generates personalized advertisements and reminders based on the analysis results.

[1604] Input: Analysis of user behavior patterns and preferences.

[1605] Output: Personalized ads and reminders.

[1606] Specific operation: Extracts relevant products and services from the analysis results and generates ads and reminders that are best suited to the user.

[1607] Step 8:

[1608] Sending Advertisements and Reminders

[1609] The server transmits the generated advertisements and reminders to the user's mobile device.

[1610] Enter: personalized ads and reminders.

[1611] Output: Advertisements and reminders displayed on the user's mobile device.

[1612] Specific operation: The server sends a notification to the user's smart glasses or smartphone, which is then displayed on the visual device.

[1613] Prompt Sentence Examples

[1614] "Write Python code to capture images of product shelves and extract text from the images (product name, price, ingredient information, etc.). Use OpenCV and TensorFlow for image recognition and text extraction."

[1615] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1616] This invention relates to a system that captures visual information and usage history from a user's mobile device, converts it into text data, and stores it. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized advertisements and reminders.

[1617] 1. Data Collection

[1618] Visual information capture and translation

[1619] The device (smart glasses) captures the video and images the user is viewing in real time, using the built-in camera and sensors to acquire visual information and store it in temporary memory.

[1620] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data, such as extracting product names and prices from a shop shelf.

[1621] Collecting and converting emotional data

[1622] The device (smartglasses, smartphone) captures the user's facial expressions and voice and extracts emotional data using an emotion engine. For example, facial expression recognition technology and voice analysis are used to identify the user's emotions, such as joy or surprise.

[1623] The device (smart glasses, smartphone) sends the extracted emotion data together with the text data to the server.

[1624] Collection and conversion of usage history

[1625] The device (smartphone) periodically collects the user's location information and app usage history, as well as records the user's web browsing history.

[1626] The device (smartphone) converts the collected usage history into text data, such as the names of places visited and apps used.

[1627] 2. Data transmission and storage

[1628] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[1629] The server stores the received data in a database, including timestamps and location information.

[1630] 3. Data Analysis

[1631] The server performs preprocessing on the stored text data and emotion data, removing noise and extracting only the necessary information.

[1632] The server uses machine learning algorithms to analyze the text and emotional data to identify user behavioral patterns, preferences, and emotional tendencies, such as frequently visited places, favorite product categories, and the emotions users have toward specific products.

[1633] 4. Generate and send ads and reminders

[1634] The server generates personalized advertisements and reminders based on the analysis results, matching the user's preferences and emotions. For example, it notifies the user of information related to a product they are interested in, along with a feeling of joy.

[1635] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[1636] 5. User rating and reward system

[1637] Users can rate the ads and reminders they receive, either as a thumbs up or a thumbs down, through the application.

[1638] The device (smart glasses, smartphone) sends user rating data to the server, which includes identification information for advertisements and reminders.

[1639] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are rewarded, which increases the usefulness of the system.

[1640] As a specific example, when a user visits a nearby supermarket and their smart glasses capture visual information about the shelves, the emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[1641] This system will enable users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[1642] The processing flow will be explained below.

[1643] Step 1:

[1644] The device (smart glasses) captures the visual information of the user in real time by using a built-in camera to acquire images and store them in temporary memory.

[1645] Step 2:

[1646] The device (smart glasses) analyzes the captured visual information using an image recognition algorithm and converts it into text data. For example, it recognizes product names and prices and extracts them as text data.

[1647] Step 3:

[1648] The device (smartphone) periodically acquires the user's location information using a GPS sensor, and also records the history of applications used and web browsing history.

[1649] Step 4:

[1650] The device (smartphone) converts the collected location information and usage history into text data. For example, it stores the names of places visited and applications used as text data.

[1651] Step 5:

[1652] The device (smartglasses, smartphone) captures the user's facial expressions and voice and uses an emotion engine to extract emotional data. For example, facial expression recognition technology can analyze the user's smiling or surprised expressions to recognize their emotions.

[1653] Step 6:

[1654] The device (smart glasses, smartphone) then sends the converted text data and emotion data to the server, where the data is encrypted to ensure security.

[1655] Step 7:

[1656] The server stores the received text data and emotion data in a database, along with timestamps and location information.

[1657] Step 8:

[1658] The server performs preprocessing on the stored data, specifically removing noise data and extracting necessary information.

[1659] Step 9:

[1660] The server then uses machine learning algorithms to analyze the pre-processed data and identify user behavioral patterns, preferences, and emotional trends, such as the stores a user frequently visits and the product categories they prefer.

[1661] Step 10:

[1662] The server generates personalized advertisements and reminders based on the analysis results, and also takes into account emotional data to provide information that matches the user's current emotions, such as providing information related to products that the user has shown interest in.

[1663] Step 11:

[1664] The server then sends the generated advertisements and reminders to the user's mobile device, with notifications occurring in real time and timed based on the user's current situation and mood.

[1665] Step 12:

[1666] Users can rate the ads and reminders they receive through the application's rating system.

[1667] Step 13:

[1668] The device (smartglasses, smartphone) sends user rating data to the server, including identification information for advertisements and reminders.

[1669] Step 14:

[1670] The server analyzes the evaluation data and distributes rewards to information providers and app developers based on the results. Information providers who receive high evaluations are given rewards.

[1671] Example 2

[1672] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1673] Conventional systems required large amounts of manual data entry to identify user preferences and behavioral patterns, making it difficult to provide personalized advertisements and reminders in real time. Furthermore, they were unable to understand user emotions and respond based on those emotions, limiting the user experience. Furthermore, they lacked a mechanism for properly evaluating user feedback and providing it back to information providers and app developers. There is a need to resolve these issues and provide a more accurate personalization system.

[1674] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1675] In this invention, the server includes means for capturing visual information, audio data, and usage history from the user's mobile device, means for converting the captured visual information, audio data, and usage history into text data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized advertisements and reminders based on the analysis results, means for transmitting the generated advertisements and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers. This enables real-time personalization based on the user's preferences and emotions, improving the user experience and realizing an efficient feedback system.

[1676] "User's mobile device" refers to an electronic device that the user can carry and use at all times, including smartphones and smart glasses.

[1677] "Visual information" refers to video and image data captured from the user's perspective.

[1678] "Audio data" refers to data recorded from the user's voice and surrounding sounds.

[1679] "Usage history" refers to recorded data about the applications a user has used and the places they have visited.

[1680] "Text data" refers to textual information extracted by analyzing visual information, audio data, and usage history.

[1681] "Emotional data" refers to information about the user's emotional state analyzed from their facial expressions and voice.

[1682] "Database" refers to a system for storing and managing text data and emotional data.

[1683] "Behavioral patterns" refers to information that shows a user's habits and tendencies toward frequent actions.

[1684] "Preferences" refers to information that indicates a user's preferences and interests.

[1685] "Emotional trends" refer to trends that indicate changes in a user's mood or emotions.

[1686] "Personalized advertising" refers to advertising that is customized based on a user's behavioral patterns, preferences, and emotional tendencies.

[1687] "Reminder" refers to a message that notifies the user of a specific event or matter.

[1688] "Rating Data" refers to information about the ratings you give to advertisements and reminders you receive.

[1689] "Information Provider" refers to an individual or organization that provides information or data to the System.

[1690] "App developer" refers to an individual or organization that develops application software that runs on the system.

[1691] The present invention is a system that captures visual information, audio data, and usage history from a user's mobile device, converts it into text data, stores it, and combines it with an emotion engine to provide more personalized advertisements and reminders.

[1692] First, the device (such as smart glasses or a smartphone) captures the user's visual information and audio data. Specifically, the smart glasses' built-in camera and sensors are used to capture the video and images the user is viewing in real time. The smart glasses or smartphone also record the user's voice and analyze it to extract emotional data. This data is temporarily stored in the device's memory.

[1693] Next, the device analyzes the captured visual information using an image recognition algorithm (e.g., OpenCV or TensorFlow) and converts it into text data. Similarly, the audio data is converted into text data using a speech analysis algorithm (e.g., Google Cloud Speech-to-Text). For example, product names and prices on store shelves, and user comments can be extracted as text data.

[1694] The converted text and emotion data is then sent from the device to a server, where it is encrypted to ensure security. The server then stores the received data in a database, including timestamps and location information.

[1695] The server preprocesses the stored text and emotion data, filtering out noise and extracting only the necessary information. It then uses machine learning algorithms (e.g., sparse vectors, natural language processing models) to analyze the text and emotion data and identify user behavioral patterns, preferences, and emotional trends. This allows it to understand frequently visited locations, favorite product categories, and emotions toward specific products.

[1696] Based on the analysis results, the server generates personalized advertisements and reminders that match the user's preferences and emotions. The generated advertisements and reminders are sent from the server to the user's mobile device. Notifications are delivered in real time, with timing adjusted based on the user's current situation and emotions.

[1697] Finally, users rate the ads and reminders they receive. Ratings are made through the application as either likes or dislikes, and the rating data is sent from the device to the server. The server analyzes the rating data and distributes rewards to information providers and app developers based on the results. Information providers who receive high ratings are rewarded, which increases the usefulness of the system.

[1698] As a concrete example, when a user visits a nearby supermarket, smart glasses capture visual information about the shelves, and an emotion engine uses that information to analyze the user's interests and emotions. The server then creates personalized advertisements based on the user's past purchase history and current emotions, and notifies the smartphone of sale information. At the same time, the smartphone can analyze the next appointment reminder and notify the user at the appropriate time.

[1699] Example prompt for a generative AI model:

[1700] "Please provide details about the system that captures visual information and usage history and converts it into text data. Also, please explain specifically how the emotion engine is used to provide personalized advertising."

[1701] This system allows users to receive information in real time that is optimized to their individual preferences and emotions, helping them live more comfortable lives.

[1702] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1703] Detailed flow of processing steps

[1704] Step 1: Capture the data

[1705] Input: Device worn by the user (smart glasses, smartphone)

[1706] How it works: The device captures the visual and audio data of what the user sees in real time. Specifically, the smart glasses' built-in camera captures the video, and the smartphone's microphone captures the audio.

[1707] Output: Visual information (video data) and audio data

[1708] Step 2: Temporarily save data

[1709] Input: Visual and audio data

[1710] Specific operation: The captured data is stored in the device's temporary memory and used for subsequent processing.

[1711] Output: Temporarily saved visual information and audio data

[1712] Step 3: Image recognition of visual information

[1713] Input: Temporarily saved visual information

[1714] Specific operation: The device uses an image recognition algorithm (e.g., OpenCV or TensorFlow) to analyze the captured video and convert it into text data. For example, it can extract product names and prices from video of a product shelf and convert them into text data.

[1715] Output: Text data (e.g. product name and price)

[1716] Step 4: Analyzing the audio data

[1717] Input: Temporarily saved audio data

[1718] Specific operation: The device uses a speech analysis algorithm (e.g., Google Cloud Speech-to-Text) to analyze the voice data and convert it into text data, for example, transcribing what the user is saying.

[1719] Output: Text data (e.g., speech content)

[1720] Step 5: Extracting emotion data

[1721] Input: Captured visual information (user facial expressions) and audio data

[1722] Specific operation: The device combines facial expression recognition technology and voice analysis to input data into the emotion engine, analyze the user's emotional state (e.g., joy, surprise), and extract emotion data.

[1723] Output: Emotion data (e.g., joy, surprise)

[1724] Step 6: Encrypt and send data

[1725] Input: Text data and emotion data

[1726] Specific operation: The terminal encrypts this data using an encryption algorithm such as AES and transmits the data to the server using the HTTPS protocol.

[1727] Output: Encrypted text data and sentiment data

[1728] Step 7: Receiving and storing data

[1729] Input: Encrypted text data and emotion data

[1730] How it works: The server receives the data, decrypts it, and stores it in a database, along with a timestamp and location information.

[1731] Output: Text data and emotion data stored in a database

[1732] Step 8: Data Preprocessing

[1733] Input: Stored text data and emotion data

[1734] Specific operation: The server performs preprocessing such as removing noise data and normalizing it, and prepares the data in a format suitable for analysis.

[1735] Output: Preprocessed text and sentiment data

[1736] Step 9: Data analysis

[1737] Input: Preprocessed text data and sentiment data

[1738] How it works: The server uses machine learning algorithms to analyze the data and identify user behavioral patterns, preferences, and emotional tendencies.

[1739] Output: Analysis results (e.g., behavioral patterns, preferences, emotional tendencies)

[1740] Step 10: Generate Ads and Reminders

[1741] Input: Analysis results

[1742] Specific operation: Based on the analysis results, the server automatically generates personalized advertisements and reminders appropriate for the user.

[1743] Output: Generated ads and reminders

[1744] Step 11: Sending Ads and Reminders

[1745] Input: Generated ads and reminders

[1746] How it works: The server sends the generated advertisements and reminders to the user's mobile device, again in real time, with the timing of notifications adjusted accordingly.

[1747] Output: Advertisements and reminders sent to the user's mobile device

[1748] Step 12: Gather user ratings and feedback

[1749] Input: User's ratings of the ads and reminders they received

[1750] Specific operation: Users use a smartphone application to make ratings, and the rating data is sent from the device to the server.

[1751] Output: Rating data sent to the server

[1752] Step 13: Analyze evaluation data and distribute rewards

[1753] Input: Evaluation data

[1754] Specific operation: The server analyzes the evaluation data and automatically distributes rewards to information providers and app developers based on the results.

[1755] Output: Reward distribution to informants and app developers

[1756] The above is the flow of specific processing steps of this system.

[1757] (Application example 2)

[1758] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1759] Conventional content delivery systems have difficulty in providing personalized content based on individual user preferences and emotions, making it difficult to improve user experience. In addition, user evaluation data is not effectively reflected, and reward distribution to information providers and application developers is insufficient.

[1760] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing visual information and usage history from the user's mobile device, means for converting the captured visual information and usage history into text data, means for capturing and extracting user emotional data, means for storing the converted text data and emotional data in a database, means for analyzing the stored text data and emotional data to identify the user's behavioral patterns, preferences, and emotional tendencies, means for generating personalized content and reminders based on the analysis results, means for transmitting the generated content and reminders to the user's mobile device, and means for receiving and analyzing user evaluation data and distributing rewards to information providers and application developers. This makes it possible to provide more personalized content based on the user's preferences and emotions and to realize a reward system based on user evaluations.

[1761] "User's mobile device" refers to an electronic device that a user carries and uses on a daily basis, including a smartphone, smart glasses, or tablet device.

[1762] "Visual information" refers to the images and videos that a user sees with their eyes, captured through cameras and sensors.

[1763] "Usage history" refers to a record of the operations and actions a user performs using a mobile device, including application usage data and internet browsing history.

[1764] "Text data" refers to captured visual information, usage history, and emotional data expressed as text information.

[1765] "Emotion data" refers to data that indicates the emotional state of a user extracted from their facial expressions and voice, and includes emotional information such as joy, anger, sadness, and happiness.

[1766] A "database" is a computer system for storing and managing collected text data and emotion data.

[1767] "Analyzing" means processing stored data using machine learning algorithms and data analysis techniques to identify user behavioral patterns, preferences, and emotional tendencies.

[1768] "Personalized content" refers to information and entertainment tailored to a user's individual preferences and emotions based on analytical results, including advertisements and reminders.

[1769] A "reminder" is a notification that reminds the user of a specific action or event.

[1770] "Rating Data" means the ratings and feedback you provide about the content and reminders you receive.

[1771] "Information Provider" means an individual or organization that provides content or information to Users.

[1772] "Application Developer" means the individual or organization that develops the applicable application software.

[1773] "Reward" means monetary or material consideration given to information providers or application developers based on user ratings.

[1774] This system captures visual information and usage history from a user's mobile device, converts it into text data, saves it, and analyzes it to provide personalized content. Furthermore, by incorporating user emotional data, more precise personalization can be achieved.

[1775] 1. Data Collection

[1776] Visual information capture and translation

[1777] The device (smart glasses) captures the video and images the user is looking at in real time. It uses the built-in camera and sensors to acquire visual information and stores it in temporary memory. The captured visual information is analyzed using image recognition algorithms such as TensorFlow and converted into text data. For example, if a user views a movie poster, the movie title and related information are extracted as text data.

[1778] Collecting and converting emotional data

[1779] The device (smart glasses, smartphone) captures the user's facial expressions and voice and uses facial recognition and speech analysis technologies to extract emotional data, such as whether the user is enjoying a movie trailer.

[1780] Collection and conversion of usage history

[1781] The device (smartphone) periodically collects the user's location information and app usage history, and converts this information into text data using the Google Maps API and Firebase. For example, it records the names of recently visited places and used applications.

[1782] 2. Data transmission and storage

[1783] The device (smart glasses, smartphone) sends the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The server then stores this data in a database.

[1784] 3. Data Analysis

[1785] The server preprocesses the stored text and sentiment data and removes noise, leaving only the necessary information. It then uses machine learning algorithms to analyze the text and sentiment data and identify user behavioral patterns, preferences, and emotional trends. Specifically, it uses Amazon SageMaker and Azure Machine Learning.

[1786] 4. Content and reminder generation and sending

[1787] The server uses the analytics to generate personalized content and reminders, such as trailers for upcoming movies the user enjoys, using Firebase Cloud Messaging and Apple Push Notification Service.

[1788] 5. User rating and reward system

[1789] Users rate the content and reminders they receive and send the rating data to the server. The server analyzes the rating data and distributes rewards to information providers and application developers. Reward distribution uses Firebase Realtime Database and Google Analytics.

[1790] Specific examples

[1791] A concrete example would be a scenario where a user looks at a movie poster with smart glasses, and the visual and emotional information is captured, and a notification is sent to the smartphone saying, "Here's the next trending movie. Would you like to watch it now?"

[1792] Prompt Sentence Examples

[1793] "If a user is enjoying looking at a movie poster on their smart glasses, capture that information and send a movie trailer notification to their smartphone saying: 'Here's the next trending movie. Want to watch it now?'"

[1794] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1795] Step 1:

[1796] The device (smart glasses) captures the video the user is looking at in real time using a built-in camera and stores the video data in temporary memory. The input is real-time video data, and the output is captured video data stored in temporary memory.

[1797] Step 2:

[1798] The device (smart glasses) analyzes the captured video data using an image recognition algorithm (e.g., TensorFlow) and converts specific information in the video into text data. For example, in the case of a movie poster, the title and director's name are extracted. The input is video data, and the output is text data.

[1799] Step 3:

[1800] The device (smart glasses and smartphone) captures the user's facial expressions and voice and converts the data into emotion data using an emotion analysis engine (e.g., OpenCV or Google Cloud Speech-to-Text API). The input is the user's facial expression data and voice data, and the output is the analyzed emotion data.

[1801] Step 4:

[1802] The device (smartphone) periodically collects the user's location information and application usage history. This is done using the Google Maps API and Firebase, and this usage history is converted into text data. The input is location data and application usage data, and the output is text data.

[1803] Step 5:

[1804] The devices (smart glasses and smartphones) transmit the converted text data and emotion data to the server. The data is encrypted during transmission to ensure security. The input is text data and emotion data, and the output is data transmission to the server.

[1805] Step 6:

[1806] The server stores the received data in a database, including timestamps and location information. The input is text data and emotion data, and the output is an entry stored in the database.

[1807] Step 7:

[1808] The server preprocesses the stored text and emotion data to remove noise. Filtering techniques are used for preprocessing to extract only the necessary information. The input is the stored data, and the output is the preprocessed data.

[1809] Step 8:

[1810] The server analyzes the preprocessed data using machine learning algorithms (e.g., Amazon SageMaker or Azure Machine Learning) to identify user behavioral patterns, preferences, and emotional trends. The input is the preprocessed data, and the output is the analysis results.

[1811] Step 9:

[1812] The server generates personalized content and reminders based on the analysis results. It uses a generative AI model to create content that matches the user's preferences and emotions. The input is the analysis results, and the output is the generated content and reminders.

[1813] Step 10:

[1814] The server sends the generated content and reminders to the user's device using Firebase Cloud Messaging or Apple Push Notification Service. The input is the generated content and reminders, and the output is notifications sent to the user's device.

[1815] Step 11:

[1816] Users rate the content and reminders they receive and send the rating data to the server via their devices (smart glasses or smartphones). The input is the user rating data, and the output is the transmission of the rating data to the server.

[1817] Step 12:

[1818] The server analyzes the evaluation data and distributes rewards to information providers and application developers based on the results. This uses Firebase Realtime Database and Google Analytics. The input is the evaluation data, and the output is the distribution of rewards.

[1819] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1820] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1821] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1822] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1823] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1824] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1825] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1826] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1827] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1828] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1829] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1830] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1831] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1832] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1833] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1834] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1835] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1836] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1837] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1838] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1839] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1840] The following is further disclosed regarding the above embodiment.

[1841] (Claim 1)

[1842] means for capturing visual information and usage history from a user's mobile device;

[1843] A means for converting the captured visual information and usage history into text data;

[1844] A means for storing the converted text data in a database;

[1845] A means of analyzing the stored text data to identify user behavior patterns and preferences;

[1846] A means for generating personalized advertisements and reminders based on the analysis results;

[1847] means for transmitting the generated advertisements and reminders to the user's mobile device;

[1848] A means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers;

[1849] A system including:

[1850] (Claim 2)

[1851] The system of claim 1, wherein an image recognition algorithm is used to convert the user's visual information and usage history into text data.

[1852] (Claim 3)

[1853] 10. The system of claim 1, wherein the user's mobile device captures visual information in real time and periodically collects location information and app usage data.

[1854] "Example 1"

[1855] (Claim 1)

[1856] means for capturing visual information and usage history from a user's mobile device;

[1857] A means for converting the captured visual information and usage history into text data;

[1858] A means for encrypting the converted text data and transmitting and storing it in a database;

[1859] means for preprocessing the stored text data and removing noise data;

[1860] A means of analyzing the pre-processed data to identify user behavior patterns and preferences; and

[1861] A means for generating personalized advertisements and reminders based on the analysis results;

[1862] means for transmitting the generated advertisements and reminders in real time to the user's mobile device;

[1863] A means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers;

[1864] A system including:

[1865] (Claim 2)

[1866] The system of claim 1, wherein natural language processing algorithms and image recognition algorithms are used to convert the captured visual information and usage history into text data.

[1867] (Claim 3)

[1868] 10. The system of claim 1, wherein the user's mobile device captures visual information in real time, periodically collects location information and app usage data, and encrypts and transmits this data.

[1869] "Application Example 1"

[1870] (Claim 1)

[1871] means for capturing visual information and usage history from a user's mobile device;

[1872] A means for converting the captured visual information and usage history into text data;

[1873] A means for storing the converted text data in a database;

[1874] A means of analyzing the stored text data to identify user behavior patterns and preferences;

[1875] A means for generating personalized advertisements and reminders based on the analysis results;

[1876] means for transmitting the generated advertisements and reminders to the user's mobile device;

[1877] A means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers;

[1878] a means for capturing visual information of the shelves by a visual device attached to a user's mobile device and converting the visual information into text data in real time;

[1879] means for displaying personalized advertisements and reminders on a visual display of the user's mobile device;

[1880] A system including:

[1881] (Claim 2)

[1882] The system of claim 1, wherein an image recognition algorithm is used to convert the user's visual information and usage history into text data.

[1883] (Claim 3)

[1884] 10. The system of claim 1, wherein the user's mobile device captures visual information in real time and periodically collects location information and app usage data.

[1885] "Example 2: Combining Emotion Engines"

[1886] (Claim 1)

[1887] means for capturing visual information, audio data and usage history from a user's mobile device;

[1888] A means for converting the captured visual information, audio data, and usage history into text data;

[1889] A means for storing the converted text data and emotion data in a database;

[1890] means for analyzing the stored text data and emotion data to identify user behavioral patterns, preferences, and emotional tendencies;

[1891] A means for generating personalized advertisements and reminders based on the analysis results;

[1892] means for transmitting the generated advertisements and reminders to the user's mobile device;

[1893] A means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers;

[1894] A system including:

[1895] (Claim 2)

[1896] The system of claim 1, wherein image recognition algorithms and voice analysis algorithms are used to convert the user's visual information, voice data, and usage history into text data.

[1897] (Claim 3)

[1898] 10. The system of claim 1, wherein the user's mobile device captures visual and audio data in real time and periodically collects location and app usage data.

[1899] "Application example 2 when combining emotion engines"

[1900] (Claim 1)

[1901] means for capturing visual information and usage history from a user's mobile device;

[1902] A means for converting the captured visual information and usage history into text data;

[1903] a means for capturing and extracting user emotional data;

[1904] A means for storing the converted text data and emotion data in a database;

[1905] means for analyzing the stored text data and emotional data to identify user behavioral patterns, preferences, and emotional tendencies;

[1906] A means for generating personalized content and reminders based on the analysis results;

[1907] means for transmitting the generated content and reminders to the user's mobile device;

[1908] A means for receiving and analyzing user evaluation data and distributing rewards to information providers and application developers;

[1909] A system including:

[1910] (Claim 2)

[1911] The system of claim 1, wherein an image recognition algorithm is used to convert the user's visual information and usage history into text data.

[1912] (Claim 3)

[1913] 10. The system of claim 1, wherein the user's mobile device captures visual information in real time, analyzes emotional data, and periodically collects location information and application usage data. [Explanation of symbols]

[1914] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing visual information and usage history from a user's mobile device; A means for converting the captured visual information and usage history into text data; A means for storing the converted text data in a database; A means of analyzing the stored text data to identify user behavior patterns and preferences; A means for generating personalized advertisements and reminders based on the analysis results; means for transmitting the generated advertisements and reminders to the user's mobile device; A means for receiving and analyzing user evaluation data and distributing rewards to information providers and app developers; A system including:

2. 2. The system according to claim 1, wherein an image recognition algorithm is used to convert the user's visual information and usage history into text data.

3. The system of claim 1 , wherein the user's mobile device captures visual information in real time and periodically collects location information and app usage data.

Images