System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting data such as a photo, a moving image, a message, and a search history captured by a user terminal; means for transmitting the collected data to a server; means for analyzing the data received by the server and specifying an anniversary; means for transmitting a remind notification to the user terminal based on the specified anniversary; and means for proposing a present or making a reservation for a restaurant suitable for the anniversary.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, forgetting important anniversaries with family or loved ones is common. Forgetting such anniversaries can lead to misunderstandings and disappointment in relationships. Even if people do remember anniversaries, they often find it difficult to find the time to make tedious arrangements, such as selecting a gift or making restaurant reservations. Therefore, there is an urgent need for a system that not only reliably manages anniversaries and provides timely reminders, but also supports gift suggestions and restaurant reservations. [Means for solving the problem]

[0005] The present invention includes a means for collecting data such as photos, videos, messages, and search history taken on a user device and transmitting this data to a server. The server then provides a means for analyzing the received data and identifying anniversaries. The system includes a means for sending reminders to the user device based on the identified anniversaries, suggesting gifts suitable for the anniversaries, and making restaurant reservations. Furthermore, the data is encrypted when transmitted, and a natural language processing engine and image recognition algorithms are used for analysis, improving the accuracy of identifying anniversaries. This allows users to efficiently prepare for important anniversaries without forgetting them.

[0006] A "user terminal" is a portable information processing device such as a smartphone or tablet used by a user.

[0007] A "photo" is still image data taken by a user terminal.

[0008] "Video" refers to video data captured by a user terminal.

[0009] A "message" is text data that a user sends and receives through a messaging app.

[0010] "Search history" refers to historical data about a user's search behavior in a web browser or application.

[0011] "Data collection means" refers to the function of obtaining data such as photos, videos, messages, and search history taken on the user's device.

[0012] A "server" is a remote information processing device that receives, stores, and analyzes data.

[0013] "Data transmission means" refers to the function of transmitting data collected by the user terminal to the server.

[0014] "Data analysis means" refers to a function that analyzes data received by the server and identifies anniversaries.

[0015] "Reminder notification" refers to a notification of an anniversary sent to a user's device.

[0016] "Gift suggestion means" refers to a function that generates gift candidates suitable for anniversaries and presents them to the user.

[0017] "Restaurant Reservation Tool" refers to the functionality that suggests and reserves suitable restaurants.

[0018] "Encryption" is a security process that protects data from unauthorized access during transmission.

[0019] A "natural language processing engine" is an algorithm used to analyze text data.

[0020] An "image recognition algorithm" is an algorithm used to analyze image and video data.

[0021] "Anniversary" refers to a specific date that is significant to a user (e.g., birthday, wedding anniversary). [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention can be implemented in the form of a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations. The specific processing steps are described below.

[0044] Collection and transmission of user information

[0045] Terminal

[0046] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and sent to a server at regular intervals.

[0047] server

[0048] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[0049] Data analysis and anniversary identification

[0050] server

[0051] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[0052] Execute the reminder function

[0053] server

[0054] Generate and schedule reminders one month, one week, or the day before a specific anniversary. Reminders include messages such as "Your birthday is in one month."

[0055] Terminal

[0056] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[0057] Gift suggestions and reservation support

[0058] server

[0059] Based on past gift history and trend information, the system generates gift suggestions that are perfect for anniversaries. It also takes into consideration the user's purchasing history and preferences. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[0060] Terminal

[0061] The suggested gifts and restaurant information are displayed to the user, who can then review the suggestions and select the gift or restaurant they like.

[0062] User

[0063] Confirm selected gifts and restaurant reservations and, in some cases, provide purchase or reservation instructions.

[0064] Restaurant reservation completed

[0065] server

[0066] Based on the user's selection, a restaurant reservation is made via the affiliated reservation system. Reservation completion information is generated and sent to the user's device.

[0067] Terminal

[0068] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[0069] Specific examples

[0070] Example 1: Wedding anniversary reminders and support

[0071] User

[0072] You can use LINE to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[0073] Terminal

[0074] The app collects this interaction and sends it to a server.

[0075] server

[0076] An NLP engine analyzes the conversation, identifies an upcoming wedding anniversary, and generates a reminder notification one month in advance, sent to the device with restaurant suggestions.

[0077] Terminal

[0078] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[0079] Example 2: Gift suggestions for a friend's birthday

[0080] User

[0081] Send a message to a friend on LINE such as "Happy Birthday!"

[0082] Terminal

[0083] Collects messages and sends them to the server.

[0084] server

[0085] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generating a reminder one week in advance and suggesting gift ideas.

[0086] Terminal

[0087] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[0088] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[0089] The processing flow will be explained below.

[0090] Step 1:

[0091] Users take photos and videos with their smartphones, send messages on LINE, and their search history on web browsers and applications is also accumulated on a daily basis.

[0092] Step 2:

[0093] The device periodically collects data such as photos, videos, LINE messages, and search history, which is then used for analysis.

[0094] Step 3:

[0095] The data collected by the device is encrypted before being sent to the server. The encryption process protects the user's privacy.

[0096] Step 4:

[0097] The server stores the received data in temporary storage, making it available for analysis.

[0098] Step 5:

[0099] The server runs a natural language processing engine to extract anniversary-related keywords from LINE messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary."

[0100] Step 6:

[0101] The server uses image recognition algorithms to analyze photos and videos for events and specific dates and times, for example recognizing features such as a calendar or cake in a photo.

[0102] Step 7:

[0103] The server uses the analysis results to identify important dates such as birthdays and wedding anniversaries and reflects them on the user's calendar.

[0104] Step 8:

[0105] The server generates reminder notifications based on the identified anniversaries, with multiple reminder notifications occurring one month in advance, one week in advance, and one day in advance.

[0106] Step 9:

[0107] The device displays reminder notifications received from the server to the user, such as "Your wedding anniversary is in one month."

[0108] Step 10:

[0109] The server generates the best gift candidates for the anniversary based on past gift history and current trend information, and creates a list of suitable gifts.

[0110] Step 11:

[0111] The server searches for nearby restaurants based on the user's location information and creates a list of restaurants that match the user's preferences.

[0112] Step 12:

[0113] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0114] Step 13:

[0115] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[0116] Step 14:

[0117] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[0118] Step 15:

[0119] The server generates reservation completion information and sends it to the user's terminal.

[0120] Step 16:

[0121] The device will display a reservation completion notification to the user, such as "Reservation completed."

[0122] In this way, the system collects information from users' daily activities and provides reminders and support at the appropriate times to help them remember important anniversaries.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In today's society, it is extremely important for users to remember important anniversaries and events and to efficiently prepare for them. However, due to busy daily lives and an abundance of information, people tend to forget anniversaries, which can lead to issues such as not having time to prepare the right gift or make restaurant reservations. Collecting information while protecting user privacy and providing timely reminders are also important challenges.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for encrypting and transmitting the collected data, means for temporarily storing the data received by the server and transferring it to a secure analysis environment, means for analyzing the received data and using a natural language processing engine and an image recognition algorithm to identify anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, and means for suggesting gifts suitable for anniversaries and searching for information on available restaurants and making reservations. This allows users to efficiently prepare for important anniversaries without forgetting them.

[0128] "User device" refers to a mobile information device such as a smartphone, tablet, or PC used by a user.

[0129] "Data" includes information such as photos, videos, messages, and search history.

[0130] "Encryption" refers to the technology of transforming collected data using specific algorithms to protect it from unauthorized access.

[0131] "Transmission" refers to the act of transferring collected data from a user terminal to a server via a network.

[0132] "Server" refers to a computer system that receives, stores, analyzes, notifies, and processes data.

[0133] "Temporary storage" refers to the short-term storage of data received by the server in preparation for analysis.

[0134] "Analysis" refers to the act of analyzing received data using specific algorithms or engines to extract meaning and information.

[0135] A "natural language processing engine" refers to a software engine that analyzes text data and extracts meaning and related information.

[0136] "Image recognition algorithm" refers to an algorithm that identifies specific objects, events, dates, etc. from photos and videos.

[0137] A "reminder notification" refers to a message that notifies the user when a specific anniversary is approaching.

[0138] "Gift suggestion" refers to the act of presenting the user with gift suggestions suitable for an anniversary.

[0139] "Restaurant reservation" refers to the act of making a reservation at a restaurant selected by the user and notifying the user of the results.

[0140] "Linked reservation system" refers to an external reservation system that works in conjunction with the server to automatically make restaurant reservations.

[0141] The present invention can be implemented in a variety of ways, including a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations.

[0142] Collection and encryption of user information

[0143] Terminal

[0144] Data is collected every time a user takes a photo or video with their smartphone or sends a message using a messaging app. Search history from web browsers and applications is also collected periodically. This data is encrypted using AES-256 at regular intervals and then sent to a server using the HTTPS protocol. Specific examples of systems that use smartphone camera apps and messaging apps are:

[0145] Receiving data on the server and preparing for analysis

[0146] server

[0147] The server receives and temporarily stores the encrypted data using cloud storage, specifically Amazon Web Services' (AWS) S3 storage service. To ensure data security, access is restricted immediately after reception. The data is then transferred via Apache Kafka to a secure analytics environment for analysis.

[0148] Data analysis on the server and identification of anniversaries

[0149] server

[0150] Using a server-based natural language processing (NLP) engine and image recognition algorithms, messages and search history are analyzed for keywords and context related to "birthdays" and "wedding anniversaries." Specific examples include Python's NLTK and SpaCy. The analyzed information is stored in a database and reflected in the user's calendar using the Google Calendar API.

[0151] Generate and send reminder notifications

[0152] server

[0153] When a specific anniversary approaches, the server's scheduler (e.g., Apache Airflow) generates reminder notifications one month, one week, and one day before. The notification messages include messages such as "Your birthday is in one month."

[0154] Terminal

[0155] Reminder notifications are sent to the user's device using Firebase Cloud Messaging (FCM) and displayed to the user. The notifications may include gift suggestions or restaurant links.

[0156] Handling gift suggestions and restaurant reservations

[0157] server

[0158] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. Specific examples include recommendation engines such as Amazon Personalize and Google Recommendations AI. Furthermore, the system uses the Google Places API to search for nearby restaurant information based on the user's location, generates a list of suggestions, and sends them to the user's device.

[0159] Terminal

[0160] The suggested gifts and restaurant information are displayed to the user, who then checks the suggestions and selects the gift or restaurant they like.

[0161] User

[0162] The user confirms the selected gift or restaurant reservation and, if necessary, issues instructions for purchase or reservation.

[0163] Restaurant reservation completed

[0164] server

[0165] Based on the user's selection, the restaurant reservation is made via a partner reservation system. For example, OpenTable API or Gurunavi API is used. Reservation completion information is generated and sent to the user's device.

[0166] Terminal

[0167] A reservation completion notification is displayed to the user, allowing the user to enjoy their important anniversary with peace of mind.

[0168] Specific examples

[0169] Example 1: Wedding anniversary reminders and support

[0170] User

[0171] You can use a messaging app to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[0172] Terminal

[0173] The app collects this communication, encrypts it with AES-256, and sends it to a server.

[0174] server

[0175] An NLP engine (NLTK or SpaCy) analyzes the conversation and identifies that the wedding anniversary is approaching. A reminder notification is generated one month in advance and sent to the device with restaurant suggestions using the Google Places API.

[0176] Terminal

[0177] The notification is received and displayed to the user, who can then view the suggested restaurants and make reservations at the ones they like.

[0178] Example 2: Gift suggestions for a friend's birthday

[0179] User

[0180] Send a message to a friend using a messaging app, such as "Happy Birthday!"

[0181] Terminal

[0182] The message is collected, encrypted with AES-256, and sent to the server.

[0183] server

[0184] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generates a reminder one week in advance, and uses a recommendation engine to suggest gift ideas.

[0185] Terminal

[0186] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[0187] Prompt Sentence Examples

[0188] "I sent a message to celebrate my friend's birthday and I'd like some gift suggestions."

[0189] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[0190] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0191] Step 1:

[0192] Terminal

[0193] This data is collected on your device every time you take a photo, record a video, or send a message in a messaging app.

[0194] Specifically, the smartphone's camera app automatically collects photographic information (e.g., photo metadata), and the messaging app records sent and received text messages. The collected data is encrypted with AES-256 and then sent to a server using HTTPS at regular intervals.

[0195] input

[0196] User-generated photo, video, message, and search history data.

[0197] output

[0198] AES-256 encrypted data.

[0199] Step 2:

[0200] server

[0201] The encrypted data is received by the server and temporarily stored in Amazon Web Services (AWS) S3 storage, where it is then transferred via Apache Kafka to a secure analysis environment (e.g., AWS Glue or an Apache Spark cluster) for analysis.

[0202] input

[0203] Encrypted data sent from the device.

[0204] output

[0205] Data transmitted to a secure analysis environment.

[0206] Step 3:

[0207] server

[0208] A natural language processing (NLP) engine (e.g., NLTK or SpaCy) and image recognition algorithms analyze incoming data to identify anniversaries. Keywords and context in messages and search history are used to detect "birthdays" and "wedding anniversaries," and relevant date information is extracted. Algorithms are used to identify events and dates in images and videos. The analyzed information is stored in a database (e.g., MySQL or PostgreSQL) and updated in the user's calendar using the Google Calendar API.

[0209] input

[0210] Temporarily stored data.

[0211] output

[0212] Parsed anniversary information and events reflected in the calendar.

[0213] Step 4:

[0214] server

[0215] Generate reminder notifications one month, one week, or the day before a specified anniversary. A scheduler like Apache Airflow can manage this and generate the reminder message (e.g., "Your birthday is in one month").

[0216] input

[0217] Parsed anniversary information.

[0218] output

[0219] The generated reminder notification.

[0220] Step 5:

[0221] server

[0222] The generated reminder notification is sent to the user's device using Firebase Cloud Messaging (FCM), and once sent, the notification is recorded in the notification log.

[0223] input

[0224] The generated reminder notification.

[0225] output

[0226] Reminder message sent to user device.

[0227] Step 6:

[0228] Terminal

[0229] When a reminder notification is received, it pops up on the user's device. For example, the notification may contain gift suggestions or restaurant links. The user checks the notification and decides what to do from the options displayed.

[0230] input

[0231] Reminder notifications sent by FCM.

[0232] output

[0233] A reminder message that will be displayed on the user's device.

[0234] Step 7:

[0235] server

[0236] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. It uses recommendation engines such as Amazon Personalize and Google Recommendations AI, and also uses the Google Places API to search for restaurant information based on the user's location to generate a list of suggestions. The generated information is sent to the user's device.

[0237] input

[0238] Parsed anniversary information and user history data.

[0239] output

[0240] Generated gift suggestions and restaurant information.

[0241] Step 8:

[0242] Terminal

[0243] The suggested gifts and restaurant information are displayed on the user's device. The user can review the suggestions and select the gift or restaurant they like. They can also purchase the selected gift at an online shop or make a reservation at the restaurant.

[0244] input

[0245] Gift suggestions and restaurant information sent from the server.

[0246] output

[0247] Purchase a gift or make a restaurant reservation at the user's option.

[0248] Step 9:

[0249] server

[0250] Based on the user's selection, a restaurant reservation is made via a partner reservation system, such as the OpenTable API or Gurunavi API. Reservation completion information is generated and sent to the user's device.

[0251] input

[0252] Restaurant reservation information selected by the user.

[0253] output

[0254] The generated reservation completion information is sent to the user terminal.

[0255] Step 10:

[0256] Terminal

[0257] A reservation completion notification is received on the user's device and displayed as a pop-up. The user can check the reservation details and enjoy the anniversary with peace of mind.

[0258] input

[0259] Reservation completion information sent from the server.

[0260] output

[0261] A reservation completion notification will be displayed on the user's device.

[0262] In this way, the present invention allows the user to make efficient preparations for important anniversaries without forgetting them.

[0263] (Application example 1)

[0264] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] In modern factories, robot maintenance and part replacement are important, but there is a lack of systems that efficiently manage this and provide timely reminders. In particular, in environments where many robots are in operation, it is difficult for personnel to keep track of each individual maintenance schedule, resulting in maintenance delays and unnecessary costs. Furthermore, in order to carry out scheduled maintenance at the appropriate time, a system that not only provides reminders but also allows for the smooth preparation and ordering of necessary parts is needed.

[0266] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0267] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries and regular maintenance dates, means for sending reminder notifications to the user terminal based on the identified anniversaries and regular maintenance dates, and means for suggesting gifts suitable for anniversaries, making restaurant reservations, and suggesting and ordering maintenance parts. This allows for appropriate management of robot maintenance times and efficient reminders and preparation and ordering of maintenance parts.

[0268] "User terminal" means an electronic device utilized by a user to collect and display data.

[0269] A "server" is a computer system that receives and analyzes data sent from a user terminal and provides the necessary information.

[0270] "Photos, videos, messages, and search history" are the types of information that users generate and store on a daily basis, and from these, the data to be analyzed is identified.

[0271] "Data" refers to various types of information collected from User Devices.

[0272] "Means of collection" refers to the methods and technologies used to obtain data from user devices.

[0273] "Means of transmission" refers to the method or technology used to send the collected data to the server.

[0274] "Means for analyzing" refers to techniques for analyzing received data and extracting useful information.

[0275] An "anniversary" refers to an important date or event for a user, such as a birthday or wedding anniversary.

[0276] "Scheduled maintenance days" refer to days scheduled for maintenance or part replacement of factory robots.

[0277] "Reminder Notification" means a notification that contains information informing the User that a particular event or maintenance is approaching.

[0278] A "present" refers to a gift given on a particular anniversary.

[0279] "Restaurant" refers to a food and beverage establishment whose primary business is serving food.

[0280] "Reservation" means a reservation made in advance to use the Service at a specific date and time.

[0281] "Means of suggestion" refers to methods and technologies for providing users with appropriate options and information.

[0282] "Parts proposal and ordering" refers to the selection and purchase procedures for parts necessary for maintaining factory robots.

[0283] The present invention includes a user terminal, a server, and a system for exchanging data between them. The purpose of the system is to efficiently manage users' anniversaries and scheduled maintenance of factory robots, and to send reminder notifications at appropriate times. An embodiment of the present invention will be described in detail below.

[0284] 1. System Configuration

[0285] The system includes a user terminal, a server, a natural language processing engine, an image recognition algorithm, and a reminder notification function.

[0286] 1.1 User Device

[0287] Users use devices such as smartphones and tablets that have the ability to take photos and videos, send messages, and collect search history from web browsers and applications.

[0288] 1.2 Server

[0289] The server receives and analyzes the data sent from the user terminal. The following hardware and software are used:

[0290] Database: Stores and manages received data

[0291] Natural language processing engine: Used to analyze messages and search history

[0292] Image recognition algorithms: used to analyze photos and videos

[0293] The data is encrypted and sent to a server, where it is analyzed to identify anniversaries and scheduled maintenance dates.

[0294] 2. Data Analysis

[0295] The server uses a natural language processing engine to analyze keywords and context related to anniversaries from users' messages and search history. It also uses image recognition algorithms to identify important events and dates from photos and videos. For example, it extracts date information from messages such as "birthday" or "wedding anniversary" and stores it in a database.

[0296] 3. Reminder function

[0297] Reminder notifications are generated and sent to the user's device one month, one week, or the day before a specified anniversary or maintenance date. These notifications include information about the upcoming anniversary, suggested gifts, restaurant information, maintenance parts, etc. This allows users and staff to remember important events and maintenance and make appropriate preparations.

[0298] 4. Proposal and reservation of gifts and maintenance parts

[0299] The server uses past history and trend information to suggest ideal gifts, restaurants, and maintenance parts for anniversaries. Users and staff can review the suggestions and select and reserve their favorite gifts, restaurants, and parts. This allows for smooth preparation and execution.

[0300] Specific examples

[0301] For example, suppose factory robot 001 last underwent maintenance on October 1, 2023, and the next maintenance is scheduled for every 90 days. The next maintenance is scheduled for January 1, 2024, and a reminder notification is sent on November 1. The notification includes a list of needed parts and a link to purchase them.

[0302] Example prompts to input to the generative AI model:

[0303] We are developing a system to manage the maintenance schedule of factory robots and send reminder notifications. We would like you to create a specific program in Python that will show how the robots collect data, calculate the next scheduled maintenance date, and send reminder emails when the date approaches.

[0304] In this way, the present invention efficiently manages anniversaries and maintenance, helping users and staff to ensure they do not miss important dates and times and to make appropriate preparations.

[0305] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0306] Step 1: Collect data

[0307] It collects data such as photos, videos, messages, and search history taken on user devices. There are various data generated by users as inputs, and these data are collected as outputs. Specifically, users use their smartphones to record their daily activities and store them in a database through a specific application.

[0308] Step 2: Sending data

[0309] The collected data is sent to the server. The input is the data stored on the user's device, and the output is the state in which that data has been sent to the server. Specifically, the collected data is encrypted and then sent to the server at regular time intervals or based on certain conditions.

[0310] Step 3: Analyze the data

[0311] The data received by the server is analyzed to identify anniversaries and scheduled maintenance dates. The input is data stored on the server, and the output is the analysis results. Specifically, a natural language processing engine is used to analyze keywords and context from messages and search history, and an image recognition algorithm is used to identify events and dates from photos and videos.

[0312] Step 4: Generate a reminder notification

[0313] Reminder notifications are generated based on the identified anniversaries and maintenance dates. The input is the analysis results, and the output is the reminder notification. Specifically, reminder notifications are prepared and generated in message format one month, one week, and the day before the identified date.

[0314] Step 5: Send reminders

[0315] The generated reminder notification is sent to the user's device. The input is the reminder notification, and the output is the notification that arrives on the user's device. Specifically, the notification is sent from the server to the user's device and displayed on the user's device.

[0316] Step 6: Proposal for gifts and maintenance parts

[0317] It suggests the best gifts and parts for anniversaries and maintenance. The input is past history, trend information, and data on the user's preferences, and the output is a list of suggested items and parts. Specifically, the server generates the optimal options based on trend information and history and sends them to the user's device.

[0318] Step 7: Facilitate bookings

[0319] Users or staff can check the suggested gifts, restaurants, and maintenance parts, and make reservations or orders. The input is a list of suggested items and parts, and the output is the user's selection. Specifically, the user can check the suggestions on their device, select their favorite items or services, and make a reservation.

[0320] Step 8: Complete your reservation or order

[0321] This notifies the user that a reservation or order has been completed. The input is the user's selection, and the output is a reservation completion notification. Specifically, the reservation or order is confirmed through cooperation with a reservation system or online shop, and the confirmation information is sent to the user's device.

[0322] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0323] The present invention can be implemented in a variety of ways, including a user terminal, a server, an emotion engine, and a method for exchanging data among these. This system recognizes the user's emotions to further personalize anniversary reminders, gift suggestions, and restaurant reservations. The specific processing steps are described below.

[0324] Collection and transmission of user information

[0325] Terminal

[0326] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and then sent to a server.

[0327] server

[0328] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[0329] Data analysis and anniversary identification

[0330] server

[0331] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[0332] Emotion Analysis

[0333] server

[0334] The emotion engine analyzes the user's emotions from their message content, voice, and images. For example, it uses tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to tailor reminders and suggestions.

[0335] Execute the reminder function

[0336] server

[0337] The system generates and schedules reminders one month, one week, or the day before the specified anniversary. The tone and content of the reminder message are adjusted based on the analyzed emotional information. For example, if the user is feeling stressed, the reminder will contain more encouraging words.

[0338] Terminal

[0339] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[0340] Gift suggestions and reservation support

[0341] server

[0342] In addition to past gift history and trend information, the system generates ideal gift suggestions for anniversaries based on emotional information. The system makes suggestions that reflect the user's emotions and recent interests. Furthermore, the system searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[0343] Terminal

[0344] The app displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0345] User

[0346] The user can select a gift or restaurant from the suggestions and make a purchase or reservation. The suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that suit their own and the other person's feelings.

[0347] Restaurant reservation completed

[0348] server

[0349] Based on the user's selection, restaurant reservations are made via affiliated reservation systems.

[0350] server

[0351] Reservation completion information is generated and sent to the user's terminal.

[0352] Terminal

[0353] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[0354] Specific examples

[0355] Example 1: Wedding anniversary reminders and support with sentiment analysis

[0356] User

[0357] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[0358] Terminal

[0359] The app collects this interaction and sends it to a server.

[0360] server

[0361] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[0362] Terminal

[0363] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[0364] Example 2: Gift suggestions for a friend's birthday

[0365] User

[0366] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[0367] Terminal

[0368] Collects messages and sends them to the server.

[0369] server

[0370] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[0371] Terminal

[0372] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[0373] In this way, the present invention is a system that combines an emotion engine to provide a personalized experience according to the user's emotions, making important anniversaries more fulfilling.

[0374] The processing flow will be explained below.

[0375] Step 1:

[0376] Users take photos and videos with their smartphones, send messages on LINE, and conduct daily searches using web browsers and applications.

[0377] Step 2:

[0378] The device periodically collects data such as photos, videos, LINE messages, and search history. This data is stored in local storage each time it is collected.

[0379] Step 3:

[0380] The data collected by the device is encrypted before being sent to the server, in order to protect the user's privacy.

[0381] Step 4:

[0382] The server stores the received data in temporary storage, where it is prepared for analysis.

[0383] Step 5:

[0384] The server runs a natural language processing engine to analyze anniversary-related keywords and context from LINE messages and search history. For example, keywords such as "birthday" and "wedding anniversary" are detected.

[0385] Step 6:

[0386] The server uses image recognition algorithms to extract events or specific dates from photos or videos, for example, recognizing a calendar date or an image of a cake in a photo.

[0387] Step 7:

[0388] The server combines the results of analysis from a natural language processing engine and image recognition algorithms to identify important dates such as birthdays and wedding anniversaries, which are then stored in a database and reflected in the user's calendar.

[0389] Step 8:

[0390] The server generates reminder notifications based on the identified anniversary, with multiple reminders for one month, one week, and one day in advance.

[0391] Step 9:

[0392] The server runs an emotion engine to tailor the message content included in the reminder notification, for example, adding encouraging words to the reminder notification if the user is feeling stressed.

[0393] Step 10:

[0394] The device receives a reminder notification from the server and displays it to the user, with a message such as "Your wedding anniversary is in one month."

[0395] Step 11:

[0396] The server generates the best gift candidates for the anniversary based on the past gift history and trend information.

[0397] Step 12:

[0398] The server uses the emotion engine to further personalize gift suggestions based on the user's recent emotional data: for example, if the user has recently been feeling happy, tickets to a fun event might be suggested.

[0399] Step 13:

[0400] The server searches for nearby restaurant information based on the user's location information and generates a list of suggestions, including options that reflect the user's emotional information.

[0401] Step 14:

[0402] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0403] Step 15:

[0404] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[0405] Step 16:

[0406] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[0407] Step 17:

[0408] The server generates reservation completion information and sends it to the user's terminal.

[0409] Step 18:

[0410] The device will display a reservation completion notification to the user, for example, "Reservation completed."

[0411] In this way, the system collects information from daily activities and provides timely reminders and support to help users remember important anniversaries. It also uses an emotion engine to provide a personalized experience based on the user's emotional state.

[0412] Example 2

[0413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] Conventional reminder notification systems only notify users based on anniversaries and events individually set by the user, and do not take into account emotions or personalized suggestions, which has the drawback of not sufficiently improving the user experience. Furthermore, privacy protection is also insufficient when collected data is transmitted unencrypted.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0416] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, means for suggesting gifts suitable for anniversaries and reserving sophisticated services, and means for analyzing emotion information using an emotion engine and adjusting the reminder notifications and suggestion content based on the results. This provides a personalized experience according to the user's emotions and enables efficient reminder notifications and suggestions while protecting privacy.

[0417] A "user terminal" is a device operated by a user, such as a smartphone or tablet.

[0418] "Photos, videos, messages, search history, and other data" refers to various digital information that users generate and save on a daily basis, and this information is used for reminders and suggestions.

[0419] "Means of sending to the server" refers to the process of delivering data collected from the user's terminal to the server via communication means such as the Internet.

[0420] "Means of analysis" refers to the process of extracting and understanding information from received data using natural language processing engines and image recognition algorithms.

[0421] "Means for identifying anniversaries" refers to the process of identifying specific significant date information from the analyzed data.

[0422] "Means for sending reminder notifications" refers to a process for delivering a notification message to a user device based on the identified anniversary.

[0423] "Means for suggesting gifts and reserving sophisticated services" refers to the process of proposing products and services related to anniversaries and supporting the reservation and purchase procedures.

[0424] An "emotion engine" refers to software or technology that analyzes a user's emotions, extracting emotional information from the content of the user's messages, behavior, facial expressions, etc.

[0425] "Emotional information" refers to emotional data that can be read from a user's statements and behavior, and includes information such as joy, sadness, and stress.

[0426] "Adjustment" refers to the process of optimizing reminder notifications and suggestions based on analyzed emotional information.

[0427] "Encryption" refers to the process of converting data collected from users so that it cannot be deciphered by third parties.

[0428] The present invention is embodied in a form that uses a user terminal, a server, an emotion engine, and software for operating these in cooperation with each other.

[0429] Collection and transmission of user information

[0430] Terminal

[0431] Data is automatically collected whenever a user takes a photo or video on a smartphone, tablet, or other device, or sends a message on LINE or similar applications. Search history on web browsers and applications is also collected. The collected data is encrypted using an encryption algorithm such as AES256 and transmitted to a server via a secure communication protocol such as HTTPS.

[0432] Data analysis and anniversary identification

[0433] server

[0434] The server temporarily stores the received encrypted data. Next, the server launches an NLP engine (e.g., BERT) to analyze messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary" and extracts date information. It then uses an image recognition algorithm (e.g., an image analysis model using TensorFlow) to identify events and dates from photos and videos. The analyzed information is then stored in a database and reflected in the user's calendar application (e.g., Google Calendar).

[0435] Emotion Analysis

[0436] server

[0437] The server launches an emotion engine (such as IBM Watson's Tone Analyzer) to analyze the user's emotions from the content, voice, and image of the message. Specifically, it uses the tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[0438] Execute the reminder function

[0439] server

[0440] The server generates reminder notifications one month, one week, and one day before the specified anniversary. The tone and content of the generated notification message are adjusted based on the emotional information analyzed by the emotion engine. For example, if the user is feeling stressed, the reminder notification will contain more encouraging words.

[0441] Terminal

[0442] The device will then display the received reminder notification to the user, which may include gift suggestions or restaurant links based on the analyzed emotional information.

[0443] Gift suggestions and reservation support

[0444] server

[0445] The server generates gift suggestions for an anniversary based on past gift history, trend information, and emotional information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions, helping users make reservations for sophisticated services.

[0446] Terminal

[0447] The device displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0448] User

[0449] Users can review the suggestions and select a gift or restaurant they like, then make a purchase or reservation via their device. These suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that match their own and the other person's emotions.

[0450] Restaurant reservation completed

[0451] server

[0452] When a user requests a restaurant reservation, the server makes the reservation via a partner reservation system (e.g., OpenTable). Once the reservation is complete, the server generates reservation completion information and sends it to the user's terminal.

[0453] Terminal

[0454] The device will then display a notification to the user that the reservation is complete, allowing them to enjoy their special day with peace of mind.

[0455] Specific examples

[0456] Example 1: Wedding anniversary reminders and support with sentiment analysis

[0457] User

[0458] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[0459] Terminal

[0460] The app collects this interaction and sends it to a server.

[0461] server

[0462] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[0463] Terminal

[0464] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[0465] Example 2: Gift suggestions for a friend's birthday

[0466] User

[0467] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[0468] Terminal

[0469] Collects messages and sends them to the server.

[0470] server

[0471] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[0472] Terminal

[0473] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[0474] Example prompt sentence:

[0475] "A user sends a message to a friend on LINE saying 'Happy Birthday!' Analyze this message to determine that the friend's birthday is coming up and generate a reminder notification one week in advance. Also, use an emotion engine to suggest gift ideas that match the positive emotion."

[0476] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0477] Step 1:

[0478] Collection of User Information

[0479] Terminal

[0480] When a user takes a photo or video on their smartphone or sends a message on LINE, that data is collected on the device. Additionally, the user's search history on their web browser or app is also periodically collected. This data is encrypted using the AES256 encryption algorithm. The input data is photos, videos, messages, and search history, and the output data is encrypted data.

[0481] Step 2:

[0482] Data transmission

[0483] Terminal

[0484] The encrypted data is sent securely to the server using HTTPS. The server transmission protocol is used here as HTTPS. The input data is the encrypted user data, and the output data is the encrypted data sent to the server.

[0485] Step 3:

[0486] Data storage and analysis

[0487] server

[0488] The server temporarily stores the received encrypted data. Next, it launches a natural language processing engine (NLP engine, e.g., BERT) to analyze messages and search history. It identifies messages containing specific keywords or contexts (e.g., birthdays, wedding anniversaries) and extracts date information. It then uses an image recognition algorithm (e.g., TensorFlow) to identify events and dates from photos and videos. The input data is the encrypted data sent to the server, and the output data is the analyzed events and date information.

[0489] Step 4:

[0490] Save analysis results and display them on the calendar

[0491] server

[0492] The server saves the analysis results in a database and reflects the information in the user's calendar application (e.g., Google Calendar). The input data is the analyzed event and date information, and the output data is the information reflected in the user's calendar.

[0493] Step 5:

[0494] Emotion analysis

[0495] server

[0496] The server launches an emotion engine (e.g., IBM Watson Tone Analyzer) and analyzes emotions from the message content, voice, and images. Specifically, it determines the user's emotions (e.g., joy, sadness, stress) using the tone of the message and facial recognition technology. The input data is the message content, voice, and images, and the output data is the analyzed emotional information.

[0497] Step 6:

[0498] Generate reminder notifications

[0499] server

[0500] The server generates reminder notifications one month, one week, and the day before the specified anniversary, adjusting the tone and content of the notification message based on the emotional information analyzed by the emotion engine. The input data are the analyzed emotional information and anniversary information, and the output data is the adjusted reminder notification.

[0501] Step 7:

[0502] Sending reminders

[0503] Terminal

[0504] The server sends the generated reminder notification to the user's device. The device displays the received reminder notification to the user. The notification includes gift suggestions and restaurant links based on the analyzed emotional information. The input data is the reminder notification, and the output data is the notification displayed to the user.

[0505] Step 8:

[0506] Gift suggestions and reservation support

[0507] server

[0508] The server generates gift suggestions for the perfect anniversary based on past gift history, trend information, and emotion information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. The input data are gift history, trend information, emotion information, and location information, and the output data is the suggestion list.

[0509] Terminal

[0510] The terminal displays suggested gifts and restaurant information to the user. The list includes links for selecting gifts and making restaurant reservations. The input data is the suggested list, and the output data is the list displayed to the user.

[0511] user

[0512] The user checks the suggestions, selects a gift or restaurant they like, and issues a purchase or reservation instruction. The input data is the suggested gift or restaurant information, and the output data is the user's purchase or reservation instruction.

[0513] Step 9:

[0514] Restaurant reservation completed

[0515] server

[0516] When a user requests a restaurant reservation, the server makes the reservation via an affiliated reservation system. After the reservation is completed, the server generates reservation completion information and sends it to the terminal. The input data is the user's reservation request, and the output data is the reservation completion information.

[0517] Terminal

[0518] The terminal displays a reservation completion notification to the user, allowing the user to celebrate their important anniversary with peace of mind. The input data is the reservation completion information, and the output data is the reservation completion notification displayed to the user.

[0519] (Application example 2)

[0520] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0521] Conventional reminder notification systems and gift suggestion systems provide uniform notifications and suggestions without considering the user's emotions, which means they are unable to provide appropriate and effective suggestions.In addition, there are no systems that understand the user's current emotional state and suggest personalized content based on that, making it difficult to provide the experience the user desires.

[0522] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user information collection means, a data transmission means, a data analysis means, a reminder notification means, a gift suggestion means, an emotion analysis means, and a content personalization means. This makes it possible to provide personalized reminder notifications and gift suggestions based on the user's emotions, and further to suggest content according to the emotions, thereby improving user satisfaction.

[0523] "User terminal" refers to a portable information terminal or computer device used by a user.

[0524] A "photo" is a still or moving image taken by a user terminal.

[0525] A "message" is text data that a user sends and receives through a communication application.

[0526] "Search history" refers to historical data of searches a user has conducted on the Internet.

[0527] A "server" is a computer system connected to a network that receives and analyzes data from user terminals.

[0528] "Data analysis" is the process of converting data collected on a server into required information.

[0529] An "anniversary" is a special event that occurs on a specific date for a user.

[0530] A "reminder" is a notification message that notifies the user of a specific date, time, or event.

[0531] "Gift Suggestion" is a function that recommends suitable gifts to users on special occasions such as anniversaries.

[0532] "Restaurant Reservation" means the process of making a reservation at a restaurant designated by the User.

[0533] "Sentiment analysis" is the process of determining a user's emotions from their message content, images, and audio data.

[0534] "Content personalization" is a feature that individually suggests content such as movies, music, and articles based on the user's emotions and preferences.

[0535] The following system is an embodiment of the present invention. The system is composed of a user terminal, a server, an emotion analysis engine, and a content personalizer. The user terminal is a mobile information terminal or a computer device, and collects data such as photos and videos taken by the user, message transmissions and receptions, and search history. The collected data is sent to the server via the Internet.

[0536] Hardware and software used

[0537] Hardware: Smartphones (iPhone, Android), network-connected servers

[0538] Software: Emotion analysis library (EmotionEngine), content recommendation system (ContentRecommender)

[0539] Data collection and transmission

[0540] The user's device automatically collects data such as photos taken by the user, videos, messages sent, and search history, which is then encrypted and sent to a server using a secure protocol.

[0541] Data analysis and anniversary identification

[0542] The server runs a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. This information is then stored in a database and updated in the user's calendar.

[0543] Emotion analysis

[0544] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the content of the user's messages, images, and voice data. For example, the analysis identifies the user's emotional state, such as tired, happy, or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[0545] Reminders and gift suggestions

[0546] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the analyzed emotional information. If the user is feeling stressed, the message will contain more encouraging words. The notification is sent to the user's device and displayed to the user.

[0547] Similarly, the server also has the ability to suggest ideal gift ideas to users for anniversaries based on the results of sentiment analysis, past gift history, and current trend information.

[0548] Content Personalization

[0549] The server then recommends the most suitable content to the user based on the results of the sentiment analysis. This function uses a content recommendation system (ContentRecommender). For example, if the server determines that the user is tired, it will suggest relaxing music, movies, articles, etc. This allows the server to provide content that is appropriate for the user at that time.

[0550] Examples of specific examples and prompts

[0551] For example, if a user sends a message on LINE saying, "I'm tired today, I want to watch a relaxing movie," the emotion analysis engine will identify the emotion "tired, wanting to be relaxed" and suggest a relaxing movie based on that emotion.

[0552] Example prompt sentence:

[0553] A user sends a message saying, "I'm tired today and want to watch a relaxing movie." Use a sentiment analysis engine to identify the emotion "tired and wanting to relax" and suggest movies based on that emotion.

[0554] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0555] Step 1: Data collection

[0556] User devices collect data such as photos and videos taken by users, messages sent, and search history. These data are collected in real time or periodically based on user behavior. The input data are photos, videos, messages, and search history, and the output data is the collected user data.

[0557] Step 2: Send data

[0558] The user device encrypts the collected data and sends it to the server using a secure protocol. The input data is the collected user data, and the output data is the encrypted transmission data. Specifically, a secure communication protocol such as TLS / SSL is used.

[0559] Step 3: Data analysis

[0560] The server invokes a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. The input data is the received user data, and the output data is the identified anniversary information. The identified information is stored in a database.

[0561] Step 4: Sentiment Analysis

[0562] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the user's message content, image data, and voice data. The input data is the message content, image data, and voice data, and the output data is the user's emotional information. Specifically, it uses algorithms that determine emotions through text analysis and facial recognition technology.

[0563] Step 5: Generate reminder notifications

[0564] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the results of emotion analysis. The input data is the specified anniversary information and the user's emotion information, and the output data is the generated reminder notification. For example, if the user is feeling stressed, a notification message containing encouraging words is generated.

[0565] Step 6: Send reminder notifications

[0566] The server sends the generated reminder notification to the user terminal. The input data is the generated reminder notification, and the output data is the notification sent to the user terminal. The user terminal displays the notification and notifies the user.

[0567] Step 7: Gift Suggestion

[0568] The server proposes ideal gift candidates for the user's anniversary based on the emotion analysis results, past gift history, and current trend information. The input data is the user's emotion information, gift history, and trend information, and the output data is a list of gift candidates.

[0569] Step 8: Content proposal

[0570] The server uses a content recommendation system (ContentRecommender) to suggest optimal content to users based on the results of emotion analysis. The input data is the user's emotion information, and the output data is suggested content (movies, music, articles, etc.). For example, if the analysis finds that the user is tired, it will suggest relaxing music, movies, articles, etc.

[0571] Step 9: User Notification

[0572] The user terminal displays the gift suggestions and content suggestions sent from the server to the user. The input data is the suggestion information sent from the server, and the output data is the displayed suggestion information. The user can make purchases or reservations based on this suggestion information.

[0573] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0574] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0575] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0576] [Second embodiment]

[0577] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0578] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0579] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0580] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0581] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0582] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0583] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0584] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0585] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0586] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0587] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0588] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0589] The present invention can be implemented in the form of a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations. The specific processing steps are described below.

[0590] Collection and transmission of user information

[0591] Terminal

[0592] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and sent to a server at regular intervals.

[0593] server

[0594] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[0595] Data analysis and anniversary identification

[0596] server

[0597] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[0598] Execute the reminder function

[0599] server

[0600] Generate and schedule reminders one month, one week, or the day before a specific anniversary. Reminders include messages such as "Your birthday is in one month."

[0601] Terminal

[0602] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[0603] Gift suggestions and reservation support

[0604] server

[0605] Based on past gift history and trend information, the system generates gift suggestions that are perfect for anniversaries. It also takes into consideration the user's purchasing history and preferences. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[0606] Terminal

[0607] The suggested gifts and restaurant information are displayed to the user, who can then review the suggestions and select the gift or restaurant they like.

[0608] User

[0609] Confirm selected gifts and restaurant reservations and, in some cases, provide purchase or reservation instructions.

[0610] Restaurant reservation completed

[0611] server

[0612] Based on the user's selection, a restaurant reservation is made via the affiliated reservation system. Reservation completion information is generated and sent to the user's device.

[0613] Terminal

[0614] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[0615] Specific examples

[0616] Example 1: Wedding anniversary reminders and support

[0617] User

[0618] You can use LINE to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[0619] Terminal

[0620] The app collects this interaction and sends it to a server.

[0621] server

[0622] An NLP engine analyzes the conversation, identifies an upcoming wedding anniversary, and generates a reminder notification one month in advance, sent to the device with restaurant suggestions.

[0623] Terminal

[0624] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[0625] Example 2: Gift suggestions for a friend's birthday

[0626] User

[0627] Send a message to a friend on LINE such as "Happy Birthday!"

[0628] Terminal

[0629] Collects messages and sends them to the server.

[0630] server

[0631] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generating a reminder one week in advance and suggesting gift ideas.

[0632] Terminal

[0633] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[0634] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[0635] The processing flow will be explained below.

[0636] Step 1:

[0637] Users take photos and videos with their smartphones, send messages on LINE, and their search history on web browsers and applications is also accumulated on a daily basis.

[0638] Step 2:

[0639] The device periodically collects data such as photos, videos, LINE messages, and search history, which is then used for analysis.

[0640] Step 3:

[0641] The data collected by the device is encrypted before being sent to the server. The encryption process protects the user's privacy.

[0642] Step 4:

[0643] The server stores the received data in temporary storage, making it available for analysis.

[0644] Step 5:

[0645] The server runs a natural language processing engine to extract anniversary-related keywords from LINE messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary."

[0646] Step 6:

[0647] The server uses image recognition algorithms to analyze photos and videos for events and specific dates and times, for example recognizing features such as a calendar or cake in a photo.

[0648] Step 7:

[0649] The server uses the analysis results to identify important dates such as birthdays and wedding anniversaries and reflects them on the user's calendar.

[0650] Step 8:

[0651] The server generates reminder notifications based on the identified anniversaries, with multiple reminder notifications occurring one month in advance, one week in advance, and one day in advance.

[0652] Step 9:

[0653] The device displays reminder notifications received from the server to the user, such as "Your wedding anniversary is in one month."

[0654] Step 10:

[0655] The server generates the best gift candidates for the anniversary based on past gift history and current trend information, and creates a list of suitable gifts.

[0656] Step 11:

[0657] The server searches for nearby restaurants based on the user's location information and creates a list of restaurants that match the user's preferences.

[0658] Step 12:

[0659] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0660] Step 13:

[0661] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[0662] Step 14:

[0663] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[0664] Step 15:

[0665] The server generates reservation completion information and sends it to the user's terminal.

[0666] Step 16:

[0667] The device will display a reservation completion notification to the user, such as "Reservation completed."

[0668] In this way, the system collects information from users' daily activities and provides reminders and support at the appropriate times to help them remember important anniversaries.

[0669] Example 1

[0670] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] In today's society, it is extremely important for users to remember important anniversaries and events and to efficiently prepare for them. However, due to busy daily lives and an abundance of information, people tend to forget anniversaries, which can lead to issues such as not having time to prepare the right gift or make restaurant reservations. Collecting information while protecting user privacy and providing timely reminders are also important challenges.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0673] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for encrypting and transmitting the collected data, means for temporarily storing the data received by the server and transferring it to a secure analysis environment, means for analyzing the received data and using a natural language processing engine and an image recognition algorithm to identify anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, and means for suggesting gifts suitable for anniversaries and searching for information on available restaurants and making reservations. This allows users to efficiently prepare for important anniversaries without forgetting them.

[0674] "User device" refers to a mobile information device such as a smartphone, tablet, or PC used by a user.

[0675] "Data" includes information such as photos, videos, messages, and search history.

[0676] "Encryption" refers to the technology of transforming collected data using specific algorithms to protect it from unauthorized access.

[0677] "Transmission" refers to the act of transferring collected data from a user terminal to a server via a network.

[0678] "Server" refers to a computer system that receives, stores, analyzes, notifies, and processes data.

[0679] "Temporary storage" refers to the short-term storage of data received by the server in preparation for analysis.

[0680] "Analysis" refers to the act of analyzing received data using specific algorithms or engines to extract meaning and information.

[0681] A "natural language processing engine" refers to a software engine that analyzes text data and extracts meaning and related information.

[0682] "Image recognition algorithm" refers to an algorithm that identifies specific objects, events, dates, etc. from photos and videos.

[0683] A "reminder notification" refers to a message that notifies the user when a specific anniversary is approaching.

[0684] "Gift suggestion" refers to the act of presenting the user with gift suggestions suitable for an anniversary.

[0685] "Restaurant reservation" refers to the act of making a reservation at a restaurant selected by the user and notifying the user of the results.

[0686] "Linked reservation system" refers to an external reservation system that works in conjunction with the server to automatically make restaurant reservations.

[0687] The present invention can be implemented in a variety of ways, including a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations.

[0688] Collection and encryption of user information

[0689] Terminal

[0690] Data is collected every time a user takes a photo or video with their smartphone or sends a message using a messaging app. Search history from web browsers and applications is also collected periodically. This data is encrypted using AES-256 at regular intervals and then sent to a server using the HTTPS protocol. Specific examples of systems that use smartphone camera apps and messaging apps are:

[0691] Receiving data on the server and preparing for analysis

[0692] server

[0693] The server receives and temporarily stores the encrypted data using cloud storage, specifically Amazon Web Services' (AWS) S3 storage service. To ensure data security, access is restricted immediately after reception. The data is then transferred via Apache Kafka to a secure analytics environment for analysis.

[0694] Data analysis on the server and identification of anniversaries

[0695] server

[0696] Using a server-based natural language processing (NLP) engine and image recognition algorithms, messages and search history are analyzed for keywords and context related to "birthdays" and "wedding anniversaries." Specific examples include Python's NLTK and SpaCy. The analyzed information is stored in a database and reflected in the user's calendar using the Google Calendar API.

[0697] Generate and send reminder notifications

[0698] server

[0699] When a specific anniversary approaches, the server's scheduler (e.g., Apache Airflow) generates reminder notifications one month, one week, and one day before. The notification messages include messages such as "Your birthday is in one month."

[0700] Terminal

[0701] Reminder notifications are sent to the user's device using Firebase Cloud Messaging (FCM) and displayed to the user. The notifications may include gift suggestions or restaurant links.

[0702] Handling gift suggestions and restaurant reservations

[0703] server

[0704] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. Specific examples include recommendation engines such as Amazon Personalize and Google Recommendations AI. Furthermore, the system uses the Google Places API to search for nearby restaurant information based on the user's location, generates a list of suggestions, and sends them to the user's device.

[0705] Terminal

[0706] The suggested gifts and restaurant information are displayed to the user, who then checks the suggestions and selects the gift or restaurant they like.

[0707] User

[0708] The user confirms the selected gift or restaurant reservation and, if necessary, issues instructions for purchase or reservation.

[0709] Restaurant reservation completed

[0710] server

[0711] Based on the user's selection, the restaurant reservation is made via a partner reservation system. For example, OpenTable API or Gurunavi API is used. Reservation completion information is generated and sent to the user's device.

[0712] Terminal

[0713] A reservation completion notification is displayed to the user, allowing the user to enjoy their important anniversary with peace of mind.

[0714] Specific examples

[0715] Example 1: Wedding anniversary reminders and support

[0716] User

[0717] You can use a messaging app to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[0718] Terminal

[0719] The app collects this communication, encrypts it with AES-256, and sends it to a server.

[0720] server

[0721] An NLP engine (NLTK or SpaCy) analyzes the conversation and identifies that the wedding anniversary is approaching. A reminder notification is generated one month in advance and sent to the device with restaurant suggestions using the Google Places API.

[0722] Terminal

[0723] The notification is received and displayed to the user, who can then view the suggested restaurants and make reservations at the ones they like.

[0724] Example 2: Gift suggestions for a friend's birthday

[0725] User

[0726] Send a message to a friend using a messaging app, such as "Happy Birthday!"

[0727] Terminal

[0728] The message is collected, encrypted with AES-256, and sent to the server.

[0729] server

[0730] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generates a reminder one week in advance, and uses a recommendation engine to suggest gift ideas.

[0731] Terminal

[0732] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[0733] Prompt Sentence Examples

[0734] "I sent a message to celebrate my friend's birthday and I'd like some gift suggestions."

[0735] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[0736] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0737] Step 1:

[0738] Terminal

[0739] This data is collected on your device every time you take a photo, record a video, or send a message in a messaging app.

[0740] Specifically, the smartphone's camera app automatically collects photographic information (e.g., photo metadata), and the messaging app records sent and received text messages. The collected data is encrypted with AES-256 and then sent to a server using HTTPS at regular intervals.

[0741] input

[0742] User-generated photo, video, message, and search history data.

[0743] output

[0744] AES-256 encrypted data.

[0745] Step 2:

[0746] server

[0747] The encrypted data is received by the server and temporarily stored in Amazon Web Services (AWS) S3 storage, where it is then transferred via Apache Kafka to a secure analysis environment (e.g., AWS Glue or an Apache Spark cluster) for analysis.

[0748] input

[0749] Encrypted data sent from the device.

[0750] output

[0751] Data transmitted to a secure analysis environment.

[0752] Step 3:

[0753] server

[0754] A natural language processing (NLP) engine (e.g., NLTK or SpaCy) and image recognition algorithms analyze incoming data to identify anniversaries. Keywords and context in messages and search history are used to detect "birthdays" and "wedding anniversaries," and relevant date information is extracted. Algorithms are used to identify events and dates in images and videos. The analyzed information is stored in a database (e.g., MySQL or PostgreSQL) and updated in the user's calendar using the Google Calendar API.

[0755] input

[0756] Temporarily stored data.

[0757] output

[0758] Parsed anniversary information and events reflected in the calendar.

[0759] Step 4:

[0760] server

[0761] Generate reminder notifications one month, one week, or the day before a specified anniversary. A scheduler like Apache Airflow can manage this and generate the reminder message (e.g., "Your birthday is in one month").

[0762] input

[0763] Parsed anniversary information.

[0764] output

[0765] The generated reminder notification.

[0766] Step 5:

[0767] server

[0768] The generated reminder notification is sent to the user's device using Firebase Cloud Messaging (FCM), and once sent, the notification is recorded in the notification log.

[0769] input

[0770] The generated reminder notification.

[0771] output

[0772] Reminder message sent to user device.

[0773] Step 6:

[0774] Terminal

[0775] When a reminder notification is received, it pops up on the user's device. For example, the notification may contain gift suggestions or restaurant links. The user checks the notification and decides what to do from the options displayed.

[0776] input

[0777] Reminder notifications sent by FCM.

[0778] output

[0779] A reminder message that will be displayed on the user's device.

[0780] Step 7:

[0781] server

[0782] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. It uses recommendation engines such as Amazon Personalize and Google Recommendations AI, and also uses the Google Places API to search for restaurant information based on the user's location to generate a list of suggestions. The generated information is sent to the user's device.

[0783] input

[0784] Parsed anniversary information and user history data.

[0785] output

[0786] Generated gift suggestions and restaurant information.

[0787] Step 8:

[0788] Terminal

[0789] The suggested gifts and restaurant information are displayed on the user's device. The user can review the suggestions and select the gift or restaurant they like. They can also purchase the selected gift at an online shop or make a reservation at the restaurant.

[0790] input

[0791] Gift suggestions and restaurant information sent from the server.

[0792] output

[0793] Purchase a gift or make a restaurant reservation at the user's option.

[0794] Step 9:

[0795] server

[0796] Based on the user's selection, a restaurant reservation is made via a partner reservation system, such as the OpenTable API or Gurunavi API. Reservation completion information is generated and sent to the user's device.

[0797] input

[0798] Restaurant reservation information selected by the user.

[0799] output

[0800] The generated reservation completion information is sent to the user terminal.

[0801] Step 10:

[0802] Terminal

[0803] A reservation completion notification is received on the user's device and displayed as a pop-up. The user can check the reservation details and enjoy the anniversary with peace of mind.

[0804] input

[0805] Reservation completion information sent from the server.

[0806] output

[0807] A reservation completion notification will be displayed on the user's device.

[0808] In this way, the present invention allows the user to make efficient preparations for important anniversaries without forgetting them.

[0809] (Application example 1)

[0810] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0811] In modern factories, robot maintenance and part replacement are important, but there is a lack of systems that efficiently manage this and provide timely reminders. In particular, in environments where many robots are in operation, it is difficult for personnel to keep track of each individual maintenance schedule, resulting in maintenance delays and unnecessary costs. Furthermore, in order to carry out scheduled maintenance at the appropriate time, a system that not only provides reminders but also allows for the smooth preparation and ordering of necessary parts is needed.

[0812] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0813] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries and regular maintenance dates, means for sending reminder notifications to the user terminal based on the identified anniversaries and regular maintenance dates, and means for suggesting gifts suitable for anniversaries, making restaurant reservations, and suggesting and ordering maintenance parts. This allows for appropriate management of robot maintenance times and efficient reminders and preparation and ordering of maintenance parts.

[0814] "User terminal" means an electronic device utilized by a user to collect and display data.

[0815] A "server" is a computer system that receives and analyzes data sent from a user terminal and provides the necessary information.

[0816] "Photos, videos, messages, and search history" are the types of information that users generate and store on a daily basis, and from these, the data to be analyzed is identified.

[0817] "Data" refers to various types of information collected from User Devices.

[0818] "Means of collection" refers to the methods and technologies used to obtain data from user devices.

[0819] "Means of transmission" refers to the method or technology used to send the collected data to the server.

[0820] "Means for analyzing" refers to techniques for analyzing received data and extracting useful information.

[0821] An "anniversary" refers to an important date or event for a user, such as a birthday or wedding anniversary.

[0822] "Scheduled maintenance days" refer to days scheduled for maintenance or part replacement of factory robots.

[0823] "Reminder Notification" means a notification that contains information informing the User that a particular event or maintenance is approaching.

[0824] A "present" refers to a gift given on a particular anniversary.

[0825] "Restaurant" refers to a food and beverage establishment whose primary business is serving food.

[0826] "Reservation" means a reservation made in advance to use the Service at a specific date and time.

[0827] "Means of suggestion" refers to methods and technologies for providing users with appropriate options and information.

[0828] "Parts proposal and ordering" refers to the selection and purchase procedures for parts necessary for maintaining factory robots.

[0829] The present invention includes a user terminal, a server, and a system for exchanging data between them. The purpose of the system is to efficiently manage users' anniversaries and scheduled maintenance of factory robots, and to send reminder notifications at appropriate times. An embodiment of the present invention will be described in detail below.

[0830] 1. System Configuration

[0831] The system includes a user terminal, a server, a natural language processing engine, an image recognition algorithm, and a reminder notification function.

[0832] 1.1 User Device

[0833] Users use devices such as smartphones and tablets that have the ability to take photos and videos, send messages, and collect search history from web browsers and applications.

[0834] 1.2 Server

[0835] The server receives and analyzes the data sent from the user terminal. The following hardware and software are used:

[0836] Database: Stores and manages received data

[0837] Natural language processing engine: Used to analyze messages and search history

[0838] Image recognition algorithms: used to analyze photos and videos

[0839] The data is encrypted and sent to a server, where it is analyzed to identify anniversaries and scheduled maintenance dates.

[0840] 2. Data Analysis

[0841] The server uses a natural language processing engine to analyze keywords and context related to anniversaries from users' messages and search history. It also uses image recognition algorithms to identify important events and dates from photos and videos. For example, it extracts date information from messages such as "birthday" or "wedding anniversary" and stores it in a database.

[0842] 3. Reminder function

[0843] Reminder notifications are generated and sent to the user's device one month, one week, or the day before a specified anniversary or maintenance date. These notifications include information about the upcoming anniversary, suggested gifts, restaurant information, maintenance parts, etc. This allows users and staff to remember important events and maintenance and make appropriate preparations.

[0844] 4. Proposal and reservation of gifts and maintenance parts

[0845] The server uses past history and trend information to suggest ideal gifts, restaurants, and maintenance parts for anniversaries. Users and staff can review the suggestions and select and reserve their favorite gifts, restaurants, and parts. This allows for smooth preparation and execution.

[0846] Specific examples

[0847] For example, suppose factory robot 001 last underwent maintenance on October 1, 2023, and the next maintenance is scheduled for every 90 days. The next maintenance is scheduled for January 1, 2024, and a reminder notification is sent on November 1. The notification includes a list of needed parts and a link to purchase them.

[0848] Example prompts to input to the generative AI model:

[0849] We are developing a system to manage the maintenance schedule of factory robots and send reminder notifications. We would like you to create a specific program in Python that will show how the robots collect data, calculate the next scheduled maintenance date, and send reminder emails when the date approaches.

[0850] In this way, the present invention efficiently manages anniversaries and maintenance, helping users and staff to ensure they do not miss important dates and times and to make appropriate preparations.

[0851] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0852] Step 1: Collect data

[0853] It collects data such as photos, videos, messages, and search history taken on user devices. There are various data generated by users as inputs, and these data are collected as outputs. Specifically, users use their smartphones to record their daily activities and store them in a database through a specific application.

[0854] Step 2: Sending data

[0855] The collected data is sent to the server. The input is the data stored on the user's device, and the output is the state in which that data has been sent to the server. Specifically, the collected data is encrypted and then sent to the server at regular time intervals or based on certain conditions.

[0856] Step 3: Analyze the data

[0857] The data received by the server is analyzed to identify anniversaries and scheduled maintenance dates. The input is data stored on the server, and the output is the analysis results. Specifically, a natural language processing engine is used to analyze keywords and context from messages and search history, and an image recognition algorithm is used to identify events and dates from photos and videos.

[0858] Step 4: Generate a reminder notification

[0859] Reminder notifications are generated based on the identified anniversaries and maintenance dates. The input is the analysis results, and the output is the reminder notification. Specifically, reminder notifications are prepared and generated in message format one month, one week, and the day before the identified date.

[0860] Step 5: Send reminders

[0861] The generated reminder notification is sent to the user's device. The input is the reminder notification, and the output is the notification that arrives on the user's device. Specifically, the notification is sent from the server to the user's device and displayed on the user's device.

[0862] Step 6: Proposal for gifts and maintenance parts

[0863] It suggests the best gifts and parts for anniversaries and maintenance. The input is past history, trend information, and data on the user's preferences, and the output is a list of suggested items and parts. Specifically, the server generates the optimal options based on trend information and history and sends them to the user's device.

[0864] Step 7: Facilitate bookings

[0865] Users or staff can check the suggested gifts, restaurants, and maintenance parts, and make reservations or orders. The input is a list of suggested items and parts, and the output is the user's selection. Specifically, the user can check the suggestions on their device, select their favorite items or services, and make a reservation.

[0866] Step 8: Complete your reservation or order

[0867] This notifies the user that a reservation or order has been completed. The input is the user's selection, and the output is a reservation completion notification. Specifically, the reservation or order is confirmed through cooperation with a reservation system or online shop, and the confirmation information is sent to the user's device.

[0868] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0869] The present invention can be implemented in a variety of ways, including a user terminal, a server, an emotion engine, and a method for exchanging data among these. This system recognizes the user's emotions to further personalize anniversary reminders, gift suggestions, and restaurant reservations. The specific processing steps are described below.

[0870] Collection and transmission of user information

[0871] Terminal

[0872] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and then sent to a server.

[0873] server

[0874] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[0875] Data analysis and anniversary identification

[0876] server

[0877] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[0878] Emotion Analysis

[0879] server

[0880] The emotion engine analyzes the user's emotions from their message content, voice, and images. For example, it uses tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to tailor reminders and suggestions.

[0881] Execute the reminder function

[0882] server

[0883] The system generates and schedules reminders one month, one week, or the day before the specified anniversary. The tone and content of the reminder message are adjusted based on the analyzed emotional information. For example, if the user is feeling stressed, the reminder will contain more encouraging words.

[0884] Terminal

[0885] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[0886] Gift suggestions and reservation support

[0887] server

[0888] In addition to past gift history and trend information, the system generates ideal gift suggestions for anniversaries based on emotional information. The system makes suggestions that reflect the user's emotions and recent interests. Furthermore, the system searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[0889] Terminal

[0890] The app displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0891] User

[0892] The user can select a gift or restaurant from the suggestions and make a purchase or reservation. The suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that suit their own and the other person's feelings.

[0893] Restaurant reservation completed

[0894] server

[0895] Based on the user's selection, restaurant reservations are made via affiliated reservation systems.

[0896] server

[0897] Reservation completion information is generated and sent to the user's terminal.

[0898] Terminal

[0899] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[0900] Specific examples

[0901] Example 1: Wedding anniversary reminders and support with sentiment analysis

[0902] User

[0903] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[0904] Terminal

[0905] The app collects this interaction and sends it to a server.

[0906] server

[0907] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[0908] Terminal

[0909] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[0910] Example 2: Gift suggestions for a friend's birthday

[0911] User

[0912] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[0913] Terminal

[0914] Collects messages and sends them to the server.

[0915] server

[0916] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[0917] Terminal

[0918] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[0919] In this way, the present invention is a system that combines an emotion engine to provide a personalized experience according to the user's emotions, making important anniversaries more fulfilling.

[0920] The processing flow will be explained below.

[0921] Step 1:

[0922] Users take photos and videos with their smartphones, send messages on LINE, and conduct daily searches using web browsers and applications.

[0923] Step 2:

[0924] The device periodically collects data such as photos, videos, LINE messages, and search history. This data is stored in local storage each time it is collected.

[0925] Step 3:

[0926] The data collected by the device is encrypted before being sent to the server, in order to protect the user's privacy.

[0927] Step 4:

[0928] The server stores the received data in temporary storage, where it is prepared for analysis.

[0929] Step 5:

[0930] The server runs a natural language processing engine to analyze anniversary-related keywords and context from LINE messages and search history. For example, keywords such as "birthday" and "wedding anniversary" are detected.

[0931] Step 6:

[0932] The server uses image recognition algorithms to extract events or specific dates from photos or videos, for example, recognizing a calendar date or an image of a cake in a photo.

[0933] Step 7:

[0934] The server combines the results of analysis from a natural language processing engine and image recognition algorithms to identify important dates such as birthdays and wedding anniversaries, which are then stored in a database and reflected in the user's calendar.

[0935] Step 8:

[0936] The server generates reminder notifications based on the identified anniversary, with multiple reminders for one month, one week, and one day in advance.

[0937] Step 9:

[0938] The server runs an emotion engine to tailor the message content included in the reminder notification, for example, adding encouraging words to the reminder notification if the user is feeling stressed.

[0939] Step 10:

[0940] The device receives a reminder notification from the server and displays it to the user, with a message such as "Your wedding anniversary is in one month."

[0941] Step 11:

[0942] The server generates the best gift candidates for the anniversary based on the past gift history and trend information.

[0943] Step 12:

[0944] The server uses the emotion engine to further personalize gift suggestions based on the user's recent emotional data: for example, if the user has recently been feeling happy, tickets to a fun event might be suggested.

[0945] Step 13:

[0946] The server searches for nearby restaurant information based on the user's location information and generates a list of suggestions, including options that reflect the user's emotional information.

[0947] Step 14:

[0948] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0949] Step 15:

[0950] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[0951] Step 16:

[0952] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[0953] Step 17:

[0954] The server generates reservation completion information and sends it to the user's terminal.

[0955] Step 18:

[0956] The device will display a reservation completion notification to the user, for example, "Reservation completed."

[0957] In this way, the system collects information from daily activities and provides timely reminders and support to help users remember important anniversaries. It also uses an emotion engine to provide a personalized experience based on the user's emotional state.

[0958] Example 2

[0959] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] Conventional reminder notification systems only notify users based on anniversaries and events individually set by the user, and do not take into account emotions or personalized suggestions, which has the drawback of not sufficiently improving the user experience. Furthermore, privacy protection is also insufficient when collected data is transmitted unencrypted.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0962] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, means for suggesting gifts suitable for anniversaries and reserving sophisticated services, and means for analyzing emotion information using an emotion engine and adjusting the reminder notifications and suggestion content based on the results. This provides a personalized experience according to the user's emotions and enables efficient reminder notifications and suggestions while protecting privacy.

[0963] A "user terminal" is a device operated by a user, such as a smartphone or tablet.

[0964] "Photos, videos, messages, search history, and other data" refers to various digital information that users generate and save on a daily basis, and this information is used for reminders and suggestions.

[0965] "Means of sending to the server" refers to the process of delivering data collected from the user's terminal to the server via communication means such as the Internet.

[0966] "Means of analysis" refers to the process of extracting and understanding information from received data using natural language processing engines and image recognition algorithms.

[0967] "Means for identifying anniversaries" refers to the process of identifying specific significant date information from the analyzed data.

[0968] "Means for sending reminder notifications" refers to a process for delivering a notification message to a user device based on the identified anniversary.

[0969] "Means for suggesting gifts and reserving sophisticated services" refers to the process of proposing products and services related to anniversaries and supporting the reservation and purchase procedures.

[0970] An "emotion engine" refers to software or technology that analyzes a user's emotions, extracting emotional information from the content of the user's messages, behavior, facial expressions, etc.

[0971] "Emotional information" refers to emotional data that can be read from a user's statements and behavior, and includes information such as joy, sadness, and stress.

[0972] "Adjustment" refers to the process of optimizing reminder notifications and suggestions based on analyzed emotional information.

[0973] "Encryption" refers to the process of converting data collected from users so that it cannot be deciphered by third parties.

[0974] The present invention is embodied in a form that uses a user terminal, a server, an emotion engine, and software for operating these in cooperation with each other.

[0975] Collection and transmission of user information

[0976] Terminal

[0977] Data is automatically collected whenever a user takes a photo or video on a smartphone, tablet, or other device, or sends a message on LINE or similar applications. Search history on web browsers and applications is also collected. The collected data is encrypted using an encryption algorithm such as AES256 and transmitted to a server via a secure communication protocol such as HTTPS.

[0978] Data analysis and anniversary identification

[0979] server

[0980] The server temporarily stores the received encrypted data. Next, the server launches an NLP engine (e.g., BERT) to analyze messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary" and extracts date information. It then uses an image recognition algorithm (e.g., an image analysis model using TensorFlow) to identify events and dates from photos and videos. The analyzed information is then stored in a database and reflected in the user's calendar application (e.g., Google Calendar).

[0981] Emotion Analysis

[0982] server

[0983] The server launches an emotion engine (such as IBM Watson's Tone Analyzer) to analyze the user's emotions from the content, voice, and image of the message. Specifically, it uses the tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[0984] Execute the reminder function

[0985] server

[0986] The server generates reminder notifications one month, one week, and one day before the specified anniversary. The tone and content of the generated notification message are adjusted based on the emotional information analyzed by the emotion engine. For example, if the user is feeling stressed, the reminder notification will contain more encouraging words.

[0987] Terminal

[0988] The device will then display the received reminder notification to the user, which may include gift suggestions or restaurant links based on the analyzed emotional information.

[0989] Gift suggestions and reservation support

[0990] server

[0991] The server generates gift suggestions for an anniversary based on past gift history, trend information, and emotional information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions, helping users make reservations for sophisticated services.

[0992] Terminal

[0993] The device displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[0994] User

[0995] Users can review the suggestions and select a gift or restaurant they like, then make a purchase or reservation via their device. These suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that match their own and the other person's emotions.

[0996] Restaurant reservation completed

[0997] server

[0998] When a user requests a restaurant reservation, the server makes the reservation via a partner reservation system (e.g., OpenTable). Once the reservation is complete, the server generates reservation completion information and sends it to the user's terminal.

[0999] Terminal

[1000] The device will then display a notification to the user that the reservation is complete, allowing them to enjoy their special day with peace of mind.

[1001] Specific examples

[1002] Example 1: Wedding anniversary reminders and support with sentiment analysis

[1003] User

[1004] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[1005] Terminal

[1006] The app collects this interaction and sends it to a server.

[1007] server

[1008] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[1009] Terminal

[1010] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[1011] Example 2: Gift suggestions for a friend's birthday

[1012] User

[1013] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[1014] Terminal

[1015] Collects messages and sends them to the server.

[1016] server

[1017] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[1018] Terminal

[1019] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[1020] Example prompt sentence:

[1021] "A user sends a message to a friend on LINE saying 'Happy Birthday!' Analyze this message to determine that the friend's birthday is coming up and generate a reminder notification one week in advance. Also, use an emotion engine to suggest gift ideas that match the positive emotion."

[1022] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1023] Step 1:

[1024] Collection of User Information

[1025] Terminal

[1026] When a user takes a photo or video on their smartphone or sends a message on LINE, that data is collected on the device. Additionally, the user's search history on their web browser or app is also periodically collected. This data is encrypted using the AES256 encryption algorithm. The input data is photos, videos, messages, and search history, and the output data is encrypted data.

[1027] Step 2:

[1028] Data transmission

[1029] Terminal

[1030] The encrypted data is sent securely to the server using HTTPS. The server transmission protocol is used here as HTTPS. The input data is the encrypted user data, and the output data is the encrypted data sent to the server.

[1031] Step 3:

[1032] Data storage and analysis

[1033] server

[1034] The server temporarily stores the received encrypted data. Next, it launches a natural language processing engine (NLP engine, e.g., BERT) to analyze messages and search history. It identifies messages containing specific keywords or contexts (e.g., birthdays, wedding anniversaries) and extracts date information. It then uses an image recognition algorithm (e.g., TensorFlow) to identify events and dates from photos and videos. The input data is the encrypted data sent to the server, and the output data is the analyzed events and date information.

[1035] Step 4:

[1036] Save analysis results and display them on the calendar

[1037] server

[1038] The server saves the analysis results in a database and reflects the information in the user's calendar application (e.g., Google Calendar). The input data is the analyzed event and date information, and the output data is the information reflected in the user's calendar.

[1039] Step 5:

[1040] Emotion analysis

[1041] server

[1042] The server launches an emotion engine (e.g., IBM Watson Tone Analyzer) and analyzes emotions from the message content, voice, and images. Specifically, it determines the user's emotions (e.g., joy, sadness, stress) using the tone of the message and facial recognition technology. The input data is the message content, voice, and images, and the output data is the analyzed emotional information.

[1043] Step 6:

[1044] Generate reminder notifications

[1045] server

[1046] The server generates reminder notifications one month, one week, and the day before the specified anniversary, adjusting the tone and content of the notification message based on the emotional information analyzed by the emotion engine. The input data are the analyzed emotional information and anniversary information, and the output data is the adjusted reminder notification.

[1047] Step 7:

[1048] Sending reminders

[1049] Terminal

[1050] The server sends the generated reminder notification to the user's device. The device displays the received reminder notification to the user. The notification includes gift suggestions and restaurant links based on the analyzed emotional information. The input data is the reminder notification, and the output data is the notification displayed to the user.

[1051] Step 8:

[1052] Gift suggestions and reservation support

[1053] server

[1054] The server generates gift suggestions for the perfect anniversary based on past gift history, trend information, and emotion information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. The input data are gift history, trend information, emotion information, and location information, and the output data is the suggestion list.

[1055] Terminal

[1056] The terminal displays suggested gifts and restaurant information to the user. The list includes links for selecting gifts and making restaurant reservations. The input data is the suggested list, and the output data is the list displayed to the user.

[1057] user

[1058] The user checks the suggestions, selects a gift or restaurant they like, and issues a purchase or reservation instruction. The input data is the suggested gift or restaurant information, and the output data is the user's purchase or reservation instruction.

[1059] Step 9:

[1060] Restaurant reservation completed

[1061] server

[1062] When a user requests a restaurant reservation, the server makes the reservation via an affiliated reservation system. After the reservation is completed, the server generates reservation completion information and sends it to the terminal. The input data is the user's reservation request, and the output data is the reservation completion information.

[1063] Terminal

[1064] The terminal displays a reservation completion notification to the user, allowing the user to celebrate their important anniversary with peace of mind. The input data is the reservation completion information, and the output data is the reservation completion notification displayed to the user.

[1065] (Application example 2)

[1066] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1067] Conventional reminder notification systems and gift suggestion systems provide uniform notifications and suggestions without considering the user's emotions, which means they are unable to provide appropriate and effective suggestions.In addition, there are no systems that understand the user's current emotional state and suggest personalized content based on that, making it difficult to provide the experience the user desires.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user information collection means, a data transmission means, a data analysis means, a reminder notification means, a gift suggestion means, an emotion analysis means, and a content personalization means. This makes it possible to provide personalized reminder notifications and gift suggestions based on the user's emotions, and further to suggest content according to the emotions, thereby improving user satisfaction.

[1069] "User terminal" refers to a portable information terminal or computer device used by a user.

[1070] A "photo" is a still or moving image taken by a user terminal.

[1071] A "message" is text data that a user sends and receives through a communication application.

[1072] "Search history" refers to historical data of searches a user has conducted on the Internet.

[1073] A "server" is a computer system connected to a network that receives and analyzes data from user terminals.

[1074] "Data analysis" is the process of converting data collected on a server into required information.

[1075] An "anniversary" is a special event that occurs on a specific date for a user.

[1076] A "reminder" is a notification message that notifies the user of a specific date, time, or event.

[1077] "Gift Suggestion" is a function that recommends suitable gifts to users on special occasions such as anniversaries.

[1078] "Restaurant Reservation" means the process of making a reservation at a restaurant designated by the User.

[1079] "Sentiment analysis" is the process of determining a user's emotions from their message content, images, and audio data.

[1080] "Content personalization" is a feature that individually suggests content such as movies, music, and articles based on the user's emotions and preferences.

[1081] The following system is an embodiment of the present invention. The system is composed of a user terminal, a server, an emotion analysis engine, and a content personalizer. The user terminal is a mobile information terminal or a computer device, and collects data such as photos and videos taken by the user, message transmissions and receptions, and search history. The collected data is sent to the server via the Internet.

[1082] Hardware and software used

[1083] Hardware: Smartphones (iPhone, Android), network-connected servers

[1084] Software: Emotion analysis library (EmotionEngine), content recommendation system (ContentRecommender)

[1085] Data collection and transmission

[1086] The user's device automatically collects data such as photos taken by the user, videos, messages sent, and search history, which is then encrypted and sent to a server using a secure protocol.

[1087] Data analysis and anniversary identification

[1088] The server runs a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. This information is then stored in a database and updated in the user's calendar.

[1089] Emotion analysis

[1090] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the content of the user's messages, images, and voice data. For example, the analysis identifies the user's emotional state, such as tired, happy, or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[1091] Reminders and gift suggestions

[1092] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the analyzed emotional information. If the user is feeling stressed, the message will contain more encouraging words. The notification is sent to the user's device and displayed to the user.

[1093] Similarly, the server also has the ability to suggest ideal gift ideas to users for anniversaries based on the results of sentiment analysis, past gift history, and current trend information.

[1094] Content Personalization

[1095] The server then recommends the most suitable content to the user based on the results of the sentiment analysis. This function uses a content recommendation system (ContentRecommender). For example, if the server determines that the user is tired, it will suggest relaxing music, movies, articles, etc. This allows the server to provide content that is appropriate for the user at that time.

[1096] Examples of specific examples and prompts

[1097] For example, if a user sends a message on LINE saying, "I'm tired today, I want to watch a relaxing movie," the emotion analysis engine will identify the emotion "tired, wanting to be relaxed" and suggest a relaxing movie based on that emotion.

[1098] Example prompt sentence:

[1099] A user sends a message saying, "I'm tired today and want to watch a relaxing movie." Use a sentiment analysis engine to identify the emotion "tired and wanting to relax" and suggest movies based on that emotion.

[1100] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1101] Step 1: Data collection

[1102] User devices collect data such as photos and videos taken by users, messages sent, and search history. These data are collected in real time or periodically based on user behavior. The input data are photos, videos, messages, and search history, and the output data is the collected user data.

[1103] Step 2: Send data

[1104] The user device encrypts the collected data and sends it to the server using a secure protocol. The input data is the collected user data, and the output data is the encrypted transmission data. Specifically, a secure communication protocol such as TLS / SSL is used.

[1105] Step 3: Data analysis

[1106] The server invokes a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. The input data is the received user data, and the output data is the identified anniversary information. The identified information is stored in a database.

[1107] Step 4: Sentiment Analysis

[1108] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the user's message content, image data, and voice data. The input data is the message content, image data, and voice data, and the output data is the user's emotional information. Specifically, it uses algorithms that determine emotions through text analysis and facial recognition technology.

[1109] Step 5: Generate reminder notifications

[1110] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the results of emotion analysis. The input data is the specified anniversary information and the user's emotion information, and the output data is the generated reminder notification. For example, if the user is feeling stressed, a notification message containing encouraging words is generated.

[1111] Step 6: Send reminder notifications

[1112] The server sends the generated reminder notification to the user terminal. The input data is the generated reminder notification, and the output data is the notification sent to the user terminal. The user terminal displays the notification and notifies the user.

[1113] Step 7: Gift Suggestion

[1114] The server proposes ideal gift candidates for the user's anniversary based on the emotion analysis results, past gift history, and current trend information. The input data is the user's emotion information, gift history, and trend information, and the output data is a list of gift candidates.

[1115] Step 8: Content proposal

[1116] The server uses a content recommendation system (ContentRecommender) to suggest optimal content to users based on the results of emotion analysis. The input data is the user's emotion information, and the output data is suggested content (movies, music, articles, etc.). For example, if the analysis finds that the user is tired, it will suggest relaxing music, movies, articles, etc.

[1117] Step 9: User Notification

[1118] The user terminal displays the gift suggestions and content suggestions sent from the server to the user. The input data is the suggestion information sent from the server, and the output data is the displayed suggestion information. The user can make purchases or reservations based on this suggestion information.

[1119] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1120] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1121] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1122] [Third embodiment]

[1123] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1124] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1125] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1126] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1127] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1128] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1129] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1130] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1131] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1132] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1133] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1134] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1135] The present invention can be implemented in the form of a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations. The specific processing steps are described below.

[1136] Collection and transmission of user information

[1137] Terminal

[1138] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and sent to a server at regular intervals.

[1139] server

[1140] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[1141] Data analysis and anniversary identification

[1142] server

[1143] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[1144] Execute the reminder function

[1145] server

[1146] Generate and schedule reminders one month, one week, or the day before a specific anniversary. Reminders include messages such as "Your birthday is in one month."

[1147] Terminal

[1148] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[1149] Gift suggestions and reservation support

[1150] server

[1151] Based on past gift history and trend information, the system generates gift suggestions that are perfect for anniversaries. It also takes into consideration the user's purchasing history and preferences. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[1152] Terminal

[1153] The suggested gifts and restaurant information are displayed to the user, who can then review the suggestions and select the gift or restaurant they like.

[1154] User

[1155] Confirm selected gifts and restaurant reservations and, in some cases, provide purchase or reservation instructions.

[1156] Restaurant reservation completed

[1157] server

[1158] Based on the user's selection, a restaurant reservation is made via the affiliated reservation system. Reservation completion information is generated and sent to the user's device.

[1159] Terminal

[1160] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[1161] Specific examples

[1162] Example 1: Wedding anniversary reminders and support

[1163] User

[1164] You can use LINE to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[1165] Terminal

[1166] The app collects this interaction and sends it to a server.

[1167] server

[1168] An NLP engine analyzes the conversation, identifies an upcoming wedding anniversary, and generates a reminder notification one month in advance, sent to the device with restaurant suggestions.

[1169] Terminal

[1170] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[1171] Example 2: Gift suggestions for a friend's birthday

[1172] User

[1173] Send a message to a friend on LINE such as "Happy Birthday!"

[1174] Terminal

[1175] Collects messages and sends them to the server.

[1176] server

[1177] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generating a reminder one week in advance and suggesting gift ideas.

[1178] Terminal

[1179] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[1180] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[1181] The processing flow will be explained below.

[1182] Step 1:

[1183] Users take photos and videos with their smartphones, send messages on LINE, and their search history on web browsers and applications is also accumulated on a daily basis.

[1184] Step 2:

[1185] The device periodically collects data such as photos, videos, LINE messages, and search history, which is then used for analysis.

[1186] Step 3:

[1187] The data collected by the device is encrypted before being sent to the server. The encryption process protects the user's privacy.

[1188] Step 4:

[1189] The server stores the received data in temporary storage, making it available for analysis.

[1190] Step 5:

[1191] The server runs a natural language processing engine to extract anniversary-related keywords from LINE messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary."

[1192] Step 6:

[1193] The server uses image recognition algorithms to analyze photos and videos for events and specific dates and times, for example recognizing features such as a calendar or cake in a photo.

[1194] Step 7:

[1195] The server uses the analysis results to identify important dates such as birthdays and wedding anniversaries and reflects them on the user's calendar.

[1196] Step 8:

[1197] The server generates reminder notifications based on the identified anniversaries, with multiple reminder notifications occurring one month in advance, one week in advance, and one day in advance.

[1198] Step 9:

[1199] The device displays reminder notifications received from the server to the user, such as "Your wedding anniversary is in one month."

[1200] Step 10:

[1201] The server generates the best gift candidates for the anniversary based on past gift history and current trend information, and creates a list of suitable gifts.

[1202] Step 11:

[1203] The server searches for nearby restaurants based on the user's location information and creates a list of restaurants that match the user's preferences.

[1204] Step 12:

[1205] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[1206] Step 13:

[1207] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[1208] Step 14:

[1209] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[1210] Step 15:

[1211] The server generates reservation completion information and sends it to the user's terminal.

[1212] Step 16:

[1213] The device will display a reservation completion notification to the user, such as "Reservation completed."

[1214] In this way, the system collects information from users' daily activities and provides reminders and support at the appropriate times to help them remember important anniversaries.

[1215] Example 1

[1216] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1217] In today's society, it is extremely important for users to remember important anniversaries and events and to efficiently prepare for them. However, due to busy daily lives and an abundance of information, people tend to forget anniversaries, which can lead to issues such as not having time to prepare the right gift or make restaurant reservations. Collecting information while protecting user privacy and providing timely reminders are also important challenges.

[1218] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1219] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for encrypting and transmitting the collected data, means for temporarily storing the data received by the server and transferring it to a secure analysis environment, means for analyzing the received data and using a natural language processing engine and an image recognition algorithm to identify anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, and means for suggesting gifts suitable for anniversaries and searching for information on available restaurants and making reservations. This allows users to efficiently prepare for important anniversaries without forgetting them.

[1220] "User device" refers to a mobile information device such as a smartphone, tablet, or PC used by a user.

[1221] "Data" includes information such as photos, videos, messages, and search history.

[1222] "Encryption" refers to the technology of transforming collected data using specific algorithms to protect it from unauthorized access.

[1223] "Transmission" refers to the act of transferring collected data from a user terminal to a server via a network.

[1224] "Server" refers to a computer system that receives, stores, analyzes, notifies, and processes data.

[1225] "Temporary storage" refers to the short-term storage of data received by the server in preparation for analysis.

[1226] "Analysis" refers to the act of analyzing received data using specific algorithms or engines to extract meaning and information.

[1227] A "natural language processing engine" refers to a software engine that analyzes text data and extracts meaning and related information.

[1228] "Image recognition algorithm" refers to an algorithm that identifies specific objects, events, dates, etc. from photos and videos.

[1229] A "reminder notification" refers to a message that notifies the user when a specific anniversary is approaching.

[1230] "Gift suggestion" refers to the act of presenting the user with gift suggestions suitable for an anniversary.

[1231] "Restaurant reservation" refers to the act of making a reservation at a restaurant selected by the user and notifying the user of the results.

[1232] "Linked reservation system" refers to an external reservation system that works in conjunction with the server to automatically make restaurant reservations.

[1233] The present invention can be implemented in a variety of ways, including a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations.

[1234] Collection and encryption of user information

[1235] Terminal

[1236] Data is collected every time a user takes a photo or video with their smartphone or sends a message using a messaging app. Search history from web browsers and applications is also collected periodically. This data is encrypted using AES-256 at regular intervals and then sent to a server using the HTTPS protocol. Specific examples of systems that use smartphone camera apps and messaging apps are:

[1237] Receiving data on the server and preparing for analysis

[1238] server

[1239] The server receives and temporarily stores the encrypted data using cloud storage, specifically Amazon Web Services' (AWS) S3 storage service. To ensure data security, access is restricted immediately after reception. The data is then transferred via Apache Kafka to a secure analytics environment for analysis.

[1240] Data analysis on the server and identification of anniversaries

[1241] server

[1242] Using a server-based natural language processing (NLP) engine and image recognition algorithms, messages and search history are analyzed for keywords and context related to "birthdays" and "wedding anniversaries." Specific examples include Python's NLTK and SpaCy. The analyzed information is stored in a database and reflected in the user's calendar using the Google Calendar API.

[1243] Generate and send reminder notifications

[1244] server

[1245] When a specific anniversary approaches, the server's scheduler (e.g., Apache Airflow) generates reminder notifications one month, one week, and one day before. The notification messages include messages such as "Your birthday is in one month."

[1246] Terminal

[1247] Reminder notifications are sent to the user's device using Firebase Cloud Messaging (FCM) and displayed to the user. The notifications may include gift suggestions or restaurant links.

[1248] Handling gift suggestions and restaurant reservations

[1249] server

[1250] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. Specific examples include recommendation engines such as Amazon Personalize and Google Recommendations AI. Furthermore, the system uses the Google Places API to search for nearby restaurant information based on the user's location, generates a list of suggestions, and sends them to the user's device.

[1251] Terminal

[1252] The suggested gifts and restaurant information are displayed to the user, who then checks the suggestions and selects the gift or restaurant they like.

[1253] User

[1254] The user confirms the selected gift or restaurant reservation and, if necessary, issues instructions for purchase or reservation.

[1255] Restaurant reservation completed

[1256] server

[1257] Based on the user's selection, the restaurant reservation is made via a partner reservation system. For example, OpenTable API or Gurunavi API is used. Reservation completion information is generated and sent to the user's device.

[1258] Terminal

[1259] A reservation completion notification is displayed to the user, allowing the user to enjoy their important anniversary with peace of mind.

[1260] Specific examples

[1261] Example 1: Wedding anniversary reminders and support

[1262] User

[1263] You can use a messaging app to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[1264] Terminal

[1265] The app collects this communication, encrypts it with AES-256, and sends it to a server.

[1266] server

[1267] An NLP engine (NLTK or SpaCy) analyzes the conversation and identifies that the wedding anniversary is approaching. A reminder notification is generated one month in advance and sent to the device with restaurant suggestions using the Google Places API.

[1268] Terminal

[1269] The notification is received and displayed to the user, who can then view the suggested restaurants and make reservations at the ones they like.

[1270] Example 2: Gift suggestions for a friend's birthday

[1271] User

[1272] Send a message to a friend using a messaging app, such as "Happy Birthday!"

[1273] Terminal

[1274] The message is collected, encrypted with AES-256, and sent to the server.

[1275] server

[1276] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generates a reminder one week in advance, and uses a recommendation engine to suggest gift ideas.

[1277] Terminal

[1278] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[1279] Prompt Sentence Examples

[1280] "I sent a message to celebrate my friend's birthday and I'd like some gift suggestions."

[1281] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[1282] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1283] Step 1:

[1284] Terminal

[1285] This data is collected on your device every time you take a photo, record a video, or send a message in a messaging app.

[1286] Specifically, the smartphone's camera app automatically collects photographic information (e.g., photo metadata), and the messaging app records sent and received text messages. The collected data is encrypted with AES-256 and then sent to a server using HTTPS at regular intervals.

[1287] input

[1288] User-generated photo, video, message, and search history data.

[1289] output

[1290] AES-256 encrypted data.

[1291] Step 2:

[1292] server

[1293] The encrypted data is received by the server and temporarily stored in Amazon Web Services (AWS) S3 storage, where it is then transferred via Apache Kafka to a secure analysis environment (e.g., AWS Glue or an Apache Spark cluster) for analysis.

[1294] input

[1295] Encrypted data sent from the device.

[1296] output

[1297] Data transmitted to a secure analysis environment.

[1298] Step 3:

[1299] server

[1300] A natural language processing (NLP) engine (e.g., NLTK or SpaCy) and image recognition algorithms analyze incoming data to identify anniversaries. Keywords and context in messages and search history are used to detect "birthdays" and "wedding anniversaries," and relevant date information is extracted. Algorithms are used to identify events and dates in images and videos. The analyzed information is stored in a database (e.g., MySQL or PostgreSQL) and updated in the user's calendar using the Google Calendar API.

[1301] input

[1302] Temporarily stored data.

[1303] output

[1304] Parsed anniversary information and events reflected in the calendar.

[1305] Step 4:

[1306] server

[1307] Generate reminder notifications one month, one week, or the day before a specified anniversary. A scheduler like Apache Airflow can manage this and generate the reminder message (e.g., "Your birthday is in one month").

[1308] input

[1309] Parsed anniversary information.

[1310] output

[1311] The generated reminder notification.

[1312] Step 5:

[1313] server

[1314] The generated reminder notification is sent to the user's device using Firebase Cloud Messaging (FCM), and once sent, the notification is recorded in the notification log.

[1315] input

[1316] The generated reminder notification.

[1317] output

[1318] Reminder message sent to user device.

[1319] Step 6:

[1320] Terminal

[1321] When a reminder notification is received, it pops up on the user's device. For example, the notification may contain gift suggestions or restaurant links. The user checks the notification and decides what to do from the options displayed.

[1322] input

[1323] Reminder notifications sent by FCM.

[1324] output

[1325] A reminder message that will be displayed on the user's device.

[1326] Step 7:

[1327] server

[1328] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. It uses recommendation engines such as Amazon Personalize and Google Recommendations AI, and also uses the Google Places API to search for restaurant information based on the user's location to generate a list of suggestions. The generated information is sent to the user's device.

[1329] input

[1330] Parsed anniversary information and user history data.

[1331] output

[1332] Generated gift suggestions and restaurant information.

[1333] Step 8:

[1334] Terminal

[1335] The suggested gifts and restaurant information are displayed on the user's device. The user can review the suggestions and select the gift or restaurant they like. They can also purchase the selected gift at an online shop or make a reservation at the restaurant.

[1336] input

[1337] Gift suggestions and restaurant information sent from the server.

[1338] output

[1339] Purchase a gift or make a restaurant reservation at the user's option.

[1340] Step 9:

[1341] server

[1342] Based on the user's selection, a restaurant reservation is made via a partner reservation system, such as the OpenTable API or Gurunavi API. Reservation completion information is generated and sent to the user's device.

[1343] input

[1344] Restaurant reservation information selected by the user.

[1345] output

[1346] The generated reservation completion information is sent to the user terminal.

[1347] Step 10:

[1348] Terminal

[1349] A reservation completion notification is received on the user's device and displayed as a pop-up. The user can check the reservation details and enjoy the anniversary with peace of mind.

[1350] input

[1351] Reservation completion information sent from the server.

[1352] output

[1353] A reservation completion notification will be displayed on the user's device.

[1354] In this way, the present invention allows the user to make efficient preparations for important anniversaries without forgetting them.

[1355] (Application example 1)

[1356] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1357] In modern factories, robot maintenance and part replacement are important, but there is a lack of systems that efficiently manage this and provide timely reminders. In particular, in environments where many robots are in operation, it is difficult for personnel to keep track of each individual maintenance schedule, resulting in maintenance delays and unnecessary costs. Furthermore, in order to carry out scheduled maintenance at the appropriate time, a system that not only provides reminders but also allows for the smooth preparation and ordering of necessary parts is needed.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1359] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries and regular maintenance dates, means for sending reminder notifications to the user terminal based on the identified anniversaries and regular maintenance dates, and means for suggesting gifts suitable for anniversaries, making restaurant reservations, and suggesting and ordering maintenance parts. This allows for appropriate management of robot maintenance times and efficient reminders and preparation and ordering of maintenance parts.

[1360] "User terminal" means an electronic device utilized by a user to collect and display data.

[1361] A "server" is a computer system that receives and analyzes data sent from a user terminal and provides the necessary information.

[1362] "Photos, videos, messages, and search history" are the types of information that users generate and store on a daily basis, and from these, the data to be analyzed is identified.

[1363] "Data" refers to various types of information collected from User Devices.

[1364] "Means of collection" refers to the methods and technologies used to obtain data from user devices.

[1365] "Means of transmission" refers to the method or technology used to send the collected data to the server.

[1366] "Means for analyzing" refers to techniques for analyzing received data and extracting useful information.

[1367] An "anniversary" refers to an important date or event for a user, such as a birthday or wedding anniversary.

[1368] "Scheduled maintenance days" refer to days scheduled for maintenance or part replacement of factory robots.

[1369] "Reminder Notification" means a notification that contains information informing the User that a particular event or maintenance is approaching.

[1370] A "present" refers to a gift given on a particular anniversary.

[1371] "Restaurant" refers to a food and beverage establishment whose primary business is serving food.

[1372] "Reservation" means a reservation made in advance to use the Service at a specific date and time.

[1373] "Means of suggestion" refers to methods and technologies for providing users with appropriate options and information.

[1374] "Parts proposal and ordering" refers to the selection and purchase procedures for parts necessary for maintaining factory robots.

[1375] The present invention includes a user terminal, a server, and a system for exchanging data between them. The purpose of the system is to efficiently manage users' anniversaries and scheduled maintenance of factory robots, and to send reminder notifications at appropriate times. An embodiment of the present invention will be described in detail below.

[1376] 1. System Configuration

[1377] The system includes a user terminal, a server, a natural language processing engine, an image recognition algorithm, and a reminder notification function.

[1378] 1.1 User Device

[1379] Users use devices such as smartphones and tablets that have the ability to take photos and videos, send messages, and collect search history from web browsers and applications.

[1380] 1.2 Server

[1381] The server receives and analyzes the data sent from the user terminal. The following hardware and software are used:

[1382] Database: Stores and manages received data

[1383] Natural language processing engine: Used to analyze messages and search history

[1384] Image recognition algorithms: used to analyze photos and videos

[1385] The data is encrypted and sent to a server, where it is analyzed to identify anniversaries and scheduled maintenance dates.

[1386] 2. Data Analysis

[1387] The server uses a natural language processing engine to analyze keywords and context related to anniversaries from users' messages and search history. It also uses image recognition algorithms to identify important events and dates from photos and videos. For example, it extracts date information from messages such as "birthday" or "wedding anniversary" and stores it in a database.

[1388] 3. Reminder function

[1389] Reminder notifications are generated and sent to the user's device one month, one week, or the day before a specified anniversary or maintenance date. These notifications include information about the upcoming anniversary, suggested gifts, restaurant information, maintenance parts, etc. This allows users and staff to remember important events and maintenance and make appropriate preparations.

[1390] 4. Proposal and reservation of gifts and maintenance parts

[1391] The server uses past history and trend information to suggest ideal gifts, restaurants, and maintenance parts for anniversaries. Users and staff can review the suggestions and select and reserve their favorite gifts, restaurants, and parts. This allows for smooth preparation and execution.

[1392] Specific examples

[1393] For example, suppose factory robot 001 last underwent maintenance on October 1, 2023, and the next maintenance is scheduled for every 90 days. The next maintenance is scheduled for January 1, 2024, and a reminder notification is sent on November 1. The notification includes a list of needed parts and a link to purchase them.

[1394] Example prompts to input to the generative AI model:

[1395] We are developing a system to manage the maintenance schedule of factory robots and send reminder notifications. We would like you to create a specific program in Python that will show how the robots collect data, calculate the next scheduled maintenance date, and send reminder emails when the date approaches.

[1396] In this way, the present invention efficiently manages anniversaries and maintenance, helping users and staff to ensure they do not miss important dates and times and to make appropriate preparations.

[1397] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1398] Step 1: Collect data

[1399] It collects data such as photos, videos, messages, and search history taken on user devices. There are various data generated by users as inputs, and these data are collected as outputs. Specifically, users use their smartphones to record their daily activities and store them in a database through a specific application.

[1400] Step 2: Sending data

[1401] The collected data is sent to the server. The input is the data stored on the user's device, and the output is the state in which that data has been sent to the server. Specifically, the collected data is encrypted and then sent to the server at regular time intervals or based on certain conditions.

[1402] Step 3: Analyze the data

[1403] The data received by the server is analyzed to identify anniversaries and scheduled maintenance dates. The input is data stored on the server, and the output is the analysis results. Specifically, a natural language processing engine is used to analyze keywords and context from messages and search history, and an image recognition algorithm is used to identify events and dates from photos and videos.

[1404] Step 4: Generate a reminder notification

[1405] Reminder notifications are generated based on the identified anniversaries and maintenance dates. The input is the analysis results, and the output is the reminder notification. Specifically, reminder notifications are prepared and generated in message format one month, one week, and the day before the identified date.

[1406] Step 5: Send reminders

[1407] The generated reminder notification is sent to the user's device. The input is the reminder notification, and the output is the notification that arrives on the user's device. Specifically, the notification is sent from the server to the user's device and displayed on the user's device.

[1408] Step 6: Proposal for gifts and maintenance parts

[1409] It suggests the best gifts and parts for anniversaries and maintenance. The input is past history, trend information, and data on the user's preferences, and the output is a list of suggested items and parts. Specifically, the server generates the optimal options based on trend information and history and sends them to the user's device.

[1410] Step 7: Facilitate bookings

[1411] Users or staff can check the suggested gifts, restaurants, and maintenance parts, and make reservations or orders. The input is a list of suggested items and parts, and the output is the user's selection. Specifically, the user can check the suggestions on their device, select their favorite items or services, and make a reservation.

[1412] Step 8: Complete your reservation or order

[1413] This notifies the user that a reservation or order has been completed. The input is the user's selection, and the output is a reservation completion notification. Specifically, the reservation or order is confirmed through cooperation with a reservation system or online shop, and the confirmation information is sent to the user's device.

[1414] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1415] The present invention can be implemented in a variety of ways, including a user terminal, a server, an emotion engine, and a method for exchanging data among these. This system recognizes the user's emotions to further personalize anniversary reminders, gift suggestions, and restaurant reservations. The specific processing steps are described below.

[1416] Collection and transmission of user information

[1417] Terminal

[1418] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and then sent to a server.

[1419] server

[1420] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[1421] Data analysis and anniversary identification

[1422] server

[1423] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[1424] Emotion Analysis

[1425] server

[1426] The emotion engine analyzes the user's emotions from their message content, voice, and images. For example, it uses tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to tailor reminders and suggestions.

[1427] Execute the reminder function

[1428] server

[1429] The system generates and schedules reminders one month, one week, or the day before the specified anniversary. The tone and content of the reminder message are adjusted based on the analyzed emotional information. For example, if the user is feeling stressed, the reminder will contain more encouraging words.

[1430] Terminal

[1431] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[1432] Gift suggestions and reservation support

[1433] server

[1434] In addition to past gift history and trend information, the system generates ideal gift suggestions for anniversaries based on emotional information. The system makes suggestions that reflect the user's emotions and recent interests. Furthermore, the system searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[1435] Terminal

[1436] The app displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[1437] User

[1438] The user can select a gift or restaurant from the suggestions and make a purchase or reservation. The suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that suit their own and the other person's feelings.

[1439] Restaurant reservation completed

[1440] server

[1441] Based on the user's selection, restaurant reservations are made via affiliated reservation systems.

[1442] server

[1443] Reservation completion information is generated and sent to the user's terminal.

[1444] Terminal

[1445] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[1446] Specific examples

[1447] Example 1: Wedding anniversary reminders and support with sentiment analysis

[1448] User

[1449] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[1450] Terminal

[1451] The app collects this interaction and sends it to a server.

[1452] server

[1453] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[1454] Terminal

[1455] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[1456] Example 2: Gift suggestions for a friend's birthday

[1457] User

[1458] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[1459] Terminal

[1460] Collects messages and sends them to the server.

[1461] server

[1462] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[1463] Terminal

[1464] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[1465] In this way, the present invention is a system that combines an emotion engine to provide a personalized experience according to the user's emotions, making important anniversaries more fulfilling.

[1466] The processing flow will be explained below.

[1467] Step 1:

[1468] Users take photos and videos with their smartphones, send messages on LINE, and conduct daily searches using web browsers and applications.

[1469] Step 2:

[1470] The device periodically collects data such as photos, videos, LINE messages, and search history. This data is stored in local storage each time it is collected.

[1471] Step 3:

[1472] The data collected by the device is encrypted before being sent to the server, in order to protect the user's privacy.

[1473] Step 4:

[1474] The server stores the received data in temporary storage, where it is prepared for analysis.

[1475] Step 5:

[1476] The server runs a natural language processing engine to analyze anniversary-related keywords and context from LINE messages and search history. For example, keywords such as "birthday" and "wedding anniversary" are detected.

[1477] Step 6:

[1478] The server uses image recognition algorithms to extract events or specific dates from photos or videos, for example, recognizing a calendar date or an image of a cake in a photo.

[1479] Step 7:

[1480] The server combines the results of analysis from a natural language processing engine and image recognition algorithms to identify important dates such as birthdays and wedding anniversaries, which are then stored in a database and reflected in the user's calendar.

[1481] Step 8:

[1482] The server generates reminder notifications based on the identified anniversary, with multiple reminders for one month, one week, and one day in advance.

[1483] Step 9:

[1484] The server runs an emotion engine to tailor the message content included in the reminder notification, for example, adding encouraging words to the reminder notification if the user is feeling stressed.

[1485] Step 10:

[1486] The device receives a reminder notification from the server and displays it to the user, with a message such as "Your wedding anniversary is in one month."

[1487] Step 11:

[1488] The server generates the best gift candidates for the anniversary based on the past gift history and trend information.

[1489] Step 12:

[1490] The server uses the emotion engine to further personalize gift suggestions based on the user's recent emotional data: for example, if the user has recently been feeling happy, tickets to a fun event might be suggested.

[1491] Step 13:

[1492] The server searches for nearby restaurant information based on the user's location information and generates a list of suggestions, including options that reflect the user's emotional information.

[1493] Step 14:

[1494] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[1495] Step 15:

[1496] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[1497] Step 16:

[1498] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[1499] Step 17:

[1500] The server generates reservation completion information and sends it to the user's terminal.

[1501] Step 18:

[1502] The device will display a reservation completion notification to the user, for example, "Reservation completed."

[1503] In this way, the system collects information from daily activities and provides timely reminders and support to help users remember important anniversaries. It also uses an emotion engine to provide a personalized experience based on the user's emotional state.

[1504] Example 2

[1505] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1506] Conventional reminder notification systems only notify users based on anniversaries and events individually set by the user, and do not take into account emotions or personalized suggestions, which has the drawback of not sufficiently improving the user experience. Furthermore, privacy protection is also insufficient when collected data is transmitted unencrypted.

[1507] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1508] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, means for suggesting gifts suitable for anniversaries and reserving sophisticated services, and means for analyzing emotion information using an emotion engine and adjusting the reminder notifications and suggestion content based on the results. This provides a personalized experience according to the user's emotions and enables efficient reminder notifications and suggestions while protecting privacy.

[1509] A "user terminal" is a device operated by a user, such as a smartphone or tablet.

[1510] "Photos, videos, messages, search history, and other data" refers to various digital information that users generate and save on a daily basis, and this information is used for reminders and suggestions.

[1511] "Means of sending to the server" refers to the process of delivering data collected from the user's terminal to the server via communication means such as the Internet.

[1512] "Means of analysis" refers to the process of extracting and understanding information from received data using natural language processing engines and image recognition algorithms.

[1513] "Means for identifying anniversaries" refers to the process of identifying specific significant date information from the analyzed data.

[1514] "Means for sending reminder notifications" refers to a process for delivering a notification message to a user device based on the identified anniversary.

[1515] "Means for suggesting gifts and reserving sophisticated services" refers to the process of proposing products and services related to anniversaries and supporting the reservation and purchase procedures.

[1516] An "emotion engine" refers to software or technology that analyzes a user's emotions, extracting emotional information from the content of the user's messages, behavior, facial expressions, etc.

[1517] "Emotional information" refers to emotional data that can be read from a user's statements and behavior, and includes information such as joy, sadness, and stress.

[1518] "Adjustment" refers to the process of optimizing reminder notifications and suggestions based on analyzed emotional information.

[1519] "Encryption" refers to the process of converting data collected from users so that it cannot be deciphered by third parties.

[1520] The present invention is embodied in a form that uses a user terminal, a server, an emotion engine, and software for operating these in cooperation with each other.

[1521] Collection and transmission of user information

[1522] Terminal

[1523] Data is automatically collected whenever a user takes a photo or video on a smartphone, tablet, or other device, or sends a message on LINE or similar applications. Search history on web browsers and applications is also collected. The collected data is encrypted using an encryption algorithm such as AES256 and transmitted to a server via a secure communication protocol such as HTTPS.

[1524] Data analysis and anniversary identification

[1525] server

[1526] The server temporarily stores the received encrypted data. Next, the server launches an NLP engine (e.g., BERT) to analyze messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary" and extracts date information. It then uses an image recognition algorithm (e.g., an image analysis model using TensorFlow) to identify events and dates from photos and videos. The analyzed information is then stored in a database and reflected in the user's calendar application (e.g., Google Calendar).

[1527] Emotion Analysis

[1528] server

[1529] The server launches an emotion engine (such as IBM Watson's Tone Analyzer) to analyze the user's emotions from the content, voice, and image of the message. Specifically, it uses the tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[1530] Execute the reminder function

[1531] server

[1532] The server generates reminder notifications one month, one week, and one day before the specified anniversary. The tone and content of the generated notification message are adjusted based on the emotional information analyzed by the emotion engine. For example, if the user is feeling stressed, the reminder notification will contain more encouraging words.

[1533] Terminal

[1534] The device will then display the received reminder notification to the user, which may include gift suggestions or restaurant links based on the analyzed emotional information.

[1535] Gift suggestions and reservation support

[1536] server

[1537] The server generates gift suggestions for an anniversary based on past gift history, trend information, and emotional information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions, helping users make reservations for sophisticated services.

[1538] Terminal

[1539] The device displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[1540] User

[1541] Users can review the suggestions and select a gift or restaurant they like, then make a purchase or reservation via their device. These suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that match their own and the other person's emotions.

[1542] Restaurant reservation completed

[1543] server

[1544] When a user requests a restaurant reservation, the server makes the reservation via a partner reservation system (e.g., OpenTable). Once the reservation is complete, the server generates reservation completion information and sends it to the user's terminal.

[1545] Terminal

[1546] The device will then display a notification to the user that the reservation is complete, allowing them to enjoy their special day with peace of mind.

[1547] Specific examples

[1548] Example 1: Wedding anniversary reminders and support with sentiment analysis

[1549] User

[1550] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[1551] Terminal

[1552] The app collects this interaction and sends it to a server.

[1553] server

[1554] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[1555] Terminal

[1556] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[1557] Example 2: Gift suggestions for a friend's birthday

[1558] User

[1559] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[1560] Terminal

[1561] Collects messages and sends them to the server.

[1562] server

[1563] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[1564] Terminal

[1565] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[1566] Example prompt sentence:

[1567] "A user sends a message to a friend on LINE saying 'Happy Birthday!' Analyze this message to determine that the friend's birthday is coming up and generate a reminder notification one week in advance. Also, use an emotion engine to suggest gift ideas that match the positive emotion."

[1568] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1569] Step 1:

[1570] Collection of User Information

[1571] Terminal

[1572] When a user takes a photo or video on their smartphone or sends a message on LINE, that data is collected on the device. Additionally, the user's search history on their web browser or app is also periodically collected. This data is encrypted using the AES256 encryption algorithm. The input data is photos, videos, messages, and search history, and the output data is encrypted data.

[1573] Step 2:

[1574] Data transmission

[1575] Terminal

[1576] The encrypted data is sent securely to the server using HTTPS. The server transmission protocol is used here as HTTPS. The input data is the encrypted user data, and the output data is the encrypted data sent to the server.

[1577] Step 3:

[1578] Data storage and analysis

[1579] server

[1580] The server temporarily stores the received encrypted data. Next, it launches a natural language processing engine (NLP engine, e.g., BERT) to analyze messages and search history. It identifies messages containing specific keywords or contexts (e.g., birthdays, wedding anniversaries) and extracts date information. It then uses an image recognition algorithm (e.g., TensorFlow) to identify events and dates from photos and videos. The input data is the encrypted data sent to the server, and the output data is the analyzed events and date information.

[1581] Step 4:

[1582] Save analysis results and display them on the calendar

[1583] server

[1584] The server saves the analysis results in a database and reflects the information in the user's calendar application (e.g., Google Calendar). The input data is the analyzed event and date information, and the output data is the information reflected in the user's calendar.

[1585] Step 5:

[1586] Emotion analysis

[1587] server

[1588] The server launches an emotion engine (e.g., IBM Watson Tone Analyzer) and analyzes emotions from the message content, voice, and images. Specifically, it determines the user's emotions (e.g., joy, sadness, stress) using the tone of the message and facial recognition technology. The input data is the message content, voice, and images, and the output data is the analyzed emotional information.

[1589] Step 6:

[1590] Generate reminder notifications

[1591] server

[1592] The server generates reminder notifications one month, one week, and the day before the specified anniversary, adjusting the tone and content of the notification message based on the emotional information analyzed by the emotion engine. The input data are the analyzed emotional information and anniversary information, and the output data is the adjusted reminder notification.

[1593] Step 7:

[1594] Sending reminders

[1595] Terminal

[1596] The server sends the generated reminder notification to the user's device. The device displays the received reminder notification to the user. The notification includes gift suggestions and restaurant links based on the analyzed emotional information. The input data is the reminder notification, and the output data is the notification displayed to the user.

[1597] Step 8:

[1598] Gift suggestions and reservation support

[1599] server

[1600] The server generates gift suggestions for the perfect anniversary based on past gift history, trend information, and emotion information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. The input data are gift history, trend information, emotion information, and location information, and the output data is the suggestion list.

[1601] Terminal

[1602] The terminal displays suggested gifts and restaurant information to the user. The list includes links for selecting gifts and making restaurant reservations. The input data is the suggested list, and the output data is the list displayed to the user.

[1603] user

[1604] The user checks the suggestions, selects a gift or restaurant they like, and issues a purchase or reservation instruction. The input data is the suggested gift or restaurant information, and the output data is the user's purchase or reservation instruction.

[1605] Step 9:

[1606] Restaurant reservation completed

[1607] server

[1608] When a user requests a restaurant reservation, the server makes the reservation via an affiliated reservation system. After the reservation is completed, the server generates reservation completion information and sends it to the terminal. The input data is the user's reservation request, and the output data is the reservation completion information.

[1609] Terminal

[1610] The terminal displays a reservation completion notification to the user, allowing the user to celebrate their important anniversary with peace of mind. The input data is the reservation completion information, and the output data is the reservation completion notification displayed to the user.

[1611] (Application example 2)

[1612] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1613] Conventional reminder notification systems and gift suggestion systems provide uniform notifications and suggestions without considering the user's emotions, which means they are unable to provide appropriate and effective suggestions.In addition, there are no systems that understand the user's current emotional state and suggest personalized content based on that, making it difficult to provide the experience the user desires.

[1614] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user information collection means, a data transmission means, a data analysis means, a reminder notification means, a gift suggestion means, an emotion analysis means, and a content personalization means. This makes it possible to provide personalized reminder notifications and gift suggestions based on the user's emotions, and further to suggest content according to the emotions, thereby improving user satisfaction.

[1615] "User terminal" refers to a portable information terminal or computer device used by a user.

[1616] A "photo" is a still or moving image taken by a user terminal.

[1617] A "message" is text data that a user sends and receives through a communication application.

[1618] "Search history" refers to historical data of searches a user has conducted on the Internet.

[1619] A "server" is a computer system connected to a network that receives and analyzes data from user terminals.

[1620] "Data analysis" is the process of converting data collected on a server into required information.

[1621] An "anniversary" is a special event that occurs on a specific date for a user.

[1622] A "reminder" is a notification message that notifies the user of a specific date, time, or event.

[1623] "Gift Suggestion" is a function that recommends suitable gifts to users on special occasions such as anniversaries.

[1624] "Restaurant Reservation" means the process of making a reservation at a restaurant designated by the User.

[1625] "Sentiment analysis" is the process of determining a user's emotions from their message content, images, and audio data.

[1626] "Content personalization" is a feature that individually suggests content such as movies, music, and articles based on the user's emotions and preferences.

[1627] The following system is an embodiment of the present invention. The system is composed of a user terminal, a server, an emotion analysis engine, and a content personalizer. The user terminal is a mobile information terminal or a computer device, and collects data such as photos and videos taken by the user, message transmissions and receptions, and search history. The collected data is sent to the server via the Internet.

[1628] Hardware and software used

[1629] Hardware: Smartphones (iPhone, Android), network-connected servers

[1630] Software: Emotion analysis library (EmotionEngine), content recommendation system (ContentRecommender)

[1631] Data collection and transmission

[1632] The user's device automatically collects data such as photos taken by the user, videos, messages sent, and search history, which is then encrypted and sent to a server using a secure protocol.

[1633] Data analysis and anniversary identification

[1634] The server runs a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. This information is then stored in a database and updated in the user's calendar.

[1635] Emotion analysis

[1636] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the content of the user's messages, images, and voice data. For example, the analysis identifies the user's emotional state, such as tired, happy, or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[1637] Reminders and gift suggestions

[1638] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the analyzed emotional information. If the user is feeling stressed, the message will contain more encouraging words. The notification is sent to the user's device and displayed to the user.

[1639] Similarly, the server also has the ability to suggest ideal gift ideas to users for anniversaries based on the results of sentiment analysis, past gift history, and current trend information.

[1640] Content Personalization

[1641] The server then recommends the most suitable content to the user based on the results of the sentiment analysis. This function uses a content recommendation system (ContentRecommender). For example, if the server determines that the user is tired, it will suggest relaxing music, movies, articles, etc. This allows the server to provide content that is appropriate for the user at that time.

[1642] Examples of specific examples and prompts

[1643] For example, if a user sends a message on LINE saying, "I'm tired today, I want to watch a relaxing movie," the emotion analysis engine will identify the emotion "tired, wanting to be relaxed" and suggest a relaxing movie based on that emotion.

[1644] Example prompt sentence:

[1645] A user sends a message saying, "I'm tired today and want to watch a relaxing movie." Use a sentiment analysis engine to identify the emotion "tired and wanting to relax" and suggest movies based on that emotion.

[1646] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1647] Step 1: Data collection

[1648] User devices collect data such as photos and videos taken by users, messages sent, and search history. These data are collected in real time or periodically based on user behavior. The input data are photos, videos, messages, and search history, and the output data is the collected user data.

[1649] Step 2: Send data

[1650] The user device encrypts the collected data and sends it to the server using a secure protocol. The input data is the collected user data, and the output data is the encrypted transmission data. Specifically, a secure communication protocol such as TLS / SSL is used.

[1651] Step 3: Data analysis

[1652] The server invokes a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. The input data is the received user data, and the output data is the identified anniversary information. The identified information is stored in a database.

[1653] Step 4: Sentiment Analysis

[1654] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the user's message content, image data, and voice data. The input data is the message content, image data, and voice data, and the output data is the user's emotional information. Specifically, it uses algorithms that determine emotions through text analysis and facial recognition technology.

[1655] Step 5: Generate reminder notifications

[1656] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the results of emotion analysis. The input data is the specified anniversary information and the user's emotion information, and the output data is the generated reminder notification. For example, if the user is feeling stressed, a notification message containing encouraging words is generated.

[1657] Step 6: Send reminder notifications

[1658] The server sends the generated reminder notification to the user terminal. The input data is the generated reminder notification, and the output data is the notification sent to the user terminal. The user terminal displays the notification and notifies the user.

[1659] Step 7: Gift Suggestion

[1660] The server proposes ideal gift candidates for the user's anniversary based on the emotion analysis results, past gift history, and current trend information. The input data is the user's emotion information, gift history, and trend information, and the output data is a list of gift candidates.

[1661] Step 8: Content proposal

[1662] The server uses a content recommendation system (ContentRecommender) to suggest optimal content to users based on the results of emotion analysis. The input data is the user's emotion information, and the output data is suggested content (movies, music, articles, etc.). For example, if the analysis finds that the user is tired, it will suggest relaxing music, movies, articles, etc.

[1663] Step 9: User Notification

[1664] The user terminal displays the gift suggestions and content suggestions sent from the server to the user. The input data is the suggestion information sent from the server, and the output data is the displayed suggestion information. The user can make purchases or reservations based on this suggestion information.

[1665] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1666] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1667] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1668] [Fourth embodiment]

[1669] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1670] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1671] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1672] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1673] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1674] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1675] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1676] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1677] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1678] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1679] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1680] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1681] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1682] The present invention can be implemented in the form of a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations. The specific processing steps are described below.

[1683] Collection and transmission of user information

[1684] Terminal

[1685] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and sent to a server at regular intervals.

[1686] server

[1687] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[1688] Data analysis and anniversary identification

[1689] server

[1690] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[1691] Execute the reminder function

[1692] server

[1693] Generate and schedule reminders one month, one week, or the day before a specific anniversary. Reminders include messages such as "Your birthday is in one month."

[1694] Terminal

[1695] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[1696] Gift suggestions and reservation support

[1697] server

[1698] Based on past gift history and trend information, the system generates gift suggestions that are perfect for anniversaries. It also takes into consideration the user's purchasing history and preferences. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[1699] Terminal

[1700] The suggested gifts and restaurant information are displayed to the user, who can then review the suggestions and select the gift or restaurant they like.

[1701] User

[1702] Confirm selected gifts and restaurant reservations and, in some cases, provide purchase or reservation instructions.

[1703] Restaurant reservation completed

[1704] server

[1705] Based on the user's selection, a restaurant reservation is made via the affiliated reservation system. Reservation completion information is generated and sent to the user's device.

[1706] Terminal

[1707] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[1708] Specific examples

[1709] Example 1: Wedding anniversary reminders and support

[1710] User

[1711] You can use LINE to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[1712] Terminal

[1713] The app collects this interaction and sends it to a server.

[1714] server

[1715] An NLP engine analyzes the conversation, identifies an upcoming wedding anniversary, and generates a reminder notification one month in advance, sent to the device with restaurant suggestions.

[1716] Terminal

[1717] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[1718] Example 2: Gift suggestions for a friend's birthday

[1719] User

[1720] Send a message to a friend on LINE such as "Happy Birthday!"

[1721] Terminal

[1722] Collects messages and sends them to the server.

[1723] server

[1724] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generating a reminder one week in advance and suggesting gift ideas.

[1725] Terminal

[1726] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[1727] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[1728] The processing flow will be explained below.

[1729] Step 1:

[1730] Users take photos and videos with their smartphones, send messages on LINE, and their search history on web browsers and applications is also accumulated on a daily basis.

[1731] Step 2:

[1732] The device periodically collects data such as photos, videos, LINE messages, and search history, which is then used for analysis.

[1733] Step 3:

[1734] The data collected by the device is encrypted before being sent to the server. The encryption process protects the user's privacy.

[1735] Step 4:

[1736] The server stores the received data in temporary storage, making it available for analysis.

[1737] Step 5:

[1738] The server runs a natural language processing engine to extract anniversary-related keywords from LINE messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary."

[1739] Step 6:

[1740] The server uses image recognition algorithms to analyze photos and videos for events and specific dates and times, for example recognizing features such as a calendar or cake in a photo.

[1741] Step 7:

[1742] The server uses the analysis results to identify important dates such as birthdays and wedding anniversaries and reflects them on the user's calendar.

[1743] Step 8:

[1744] The server generates reminder notifications based on the identified anniversaries, with multiple reminder notifications occurring one month in advance, one week in advance, and one day in advance.

[1745] Step 9:

[1746] The device displays reminder notifications received from the server to the user, such as "Your wedding anniversary is in one month."

[1747] Step 10:

[1748] The server generates the best gift candidates for the anniversary based on past gift history and current trend information, and creates a list of suitable gifts.

[1749] Step 11:

[1750] The server searches for nearby restaurants based on the user's location information and creates a list of restaurants that match the user's preferences.

[1751] Step 12:

[1752] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[1753] Step 13:

[1754] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[1755] Step 14:

[1756] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[1757] Step 15:

[1758] The server generates reservation completion information and sends it to the user's terminal.

[1759] Step 16:

[1760] The device will display a reservation completion notification to the user, such as "Reservation completed."

[1761] In this way, the system collects information from users' daily activities and provides reminders and support at the appropriate times to help them remember important anniversaries.

[1762] Example 1

[1763] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1764] In today's society, it is extremely important for users to remember important anniversaries and events and to efficiently prepare for them. However, due to busy daily lives and an abundance of information, people tend to forget anniversaries, which can lead to issues such as not having time to prepare the right gift or make restaurant reservations. Collecting information while protecting user privacy and providing timely reminders are also important challenges.

[1765] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1766] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for encrypting and transmitting the collected data, means for temporarily storing the data received by the server and transferring it to a secure analysis environment, means for analyzing the received data and using a natural language processing engine and an image recognition algorithm to identify anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, and means for suggesting gifts suitable for anniversaries and searching for information on available restaurants and making reservations. This allows users to efficiently prepare for important anniversaries without forgetting them.

[1767] "User device" refers to a mobile information device such as a smartphone, tablet, or PC used by a user.

[1768] "Data" includes information such as photos, videos, messages, and search history.

[1769] "Encryption" refers to the technology of transforming collected data using specific algorithms to protect it from unauthorized access.

[1770] "Transmission" refers to the act of transferring collected data from a user terminal to a server via a network.

[1771] "Server" refers to a computer system that receives, stores, analyzes, notifies, and processes data.

[1772] "Temporary storage" refers to the short-term storage of data received by the server in preparation for analysis.

[1773] "Analysis" refers to the act of analyzing received data using specific algorithms or engines to extract meaning and information.

[1774] A "natural language processing engine" refers to a software engine that analyzes text data and extracts meaning and related information.

[1775] "Image recognition algorithm" refers to an algorithm that identifies specific objects, events, dates, etc. from photos and videos.

[1776] A "reminder notification" refers to a message that notifies the user when a specific anniversary is approaching.

[1777] "Gift suggestion" refers to the act of presenting the user with gift suggestions suitable for an anniversary.

[1778] "Restaurant reservation" refers to the act of making a reservation at a restaurant selected by the user and notifying the user of the results.

[1779] "Linked reservation system" refers to an external reservation system that works in conjunction with the server to automatically make restaurant reservations.

[1780] The present invention can be implemented in a variety of ways, including a user terminal, a server, and a method for exchanging data between them. This system protects the user's privacy while reminding them of important anniversaries, providing timely reminders, suggesting gifts, and assisting with restaurant reservations.

[1781] Collection and encryption of user information

[1782] Terminal

[1783] Data is collected every time a user takes a photo or video with their smartphone or sends a message using a messaging app. Search history from web browsers and applications is also collected periodically. This data is encrypted using AES-256 at regular intervals and then sent to a server using the HTTPS protocol. Specific examples of systems that use smartphone camera apps and messaging apps are:

[1784] Receiving data on the server and preparing for analysis

[1785] server

[1786] The server receives and temporarily stores the encrypted data using cloud storage, specifically Amazon Web Services' (AWS) S3 storage service. To ensure data security, access is restricted immediately after reception. The data is then transferred via Apache Kafka to a secure analytics environment for analysis.

[1787] Data analysis on the server and identification of anniversaries

[1788] server

[1789] Using a server-based natural language processing (NLP) engine and image recognition algorithms, messages and search history are analyzed for keywords and context related to "birthdays" and "wedding anniversaries." Specific examples include Python's NLTK and SpaCy. The analyzed information is stored in a database and reflected in the user's calendar using the Google Calendar API.

[1790] Generate and send reminder notifications

[1791] server

[1792] When a specific anniversary approaches, the server's scheduler (e.g., Apache Airflow) generates reminder notifications one month, one week, and one day before. The notification messages include messages such as "Your birthday is in one month."

[1793] Terminal

[1794] Reminder notifications are sent to the user's device using Firebase Cloud Messaging (FCM) and displayed to the user. The notifications may include gift suggestions or restaurant links.

[1795] Handling gift suggestions and restaurant reservations

[1796] server

[1797] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. Specific examples include recommendation engines such as Amazon Personalize and Google Recommendations AI. Furthermore, the system uses the Google Places API to search for nearby restaurant information based on the user's location, generates a list of suggestions, and sends them to the user's device.

[1798] Terminal

[1799] The suggested gifts and restaurant information are displayed to the user, who then checks the suggestions and selects the gift or restaurant they like.

[1800] User

[1801] The user confirms the selected gift or restaurant reservation and, if necessary, issues instructions for purchase or reservation.

[1802] Restaurant reservation completed

[1803] server

[1804] Based on the user's selection, the restaurant reservation is made via a partner reservation system. For example, OpenTable API or Gurunavi API is used. Reservation completion information is generated and sent to the user's device.

[1805] Terminal

[1806] A reservation completion notification is displayed to the user, allowing the user to enjoy their important anniversary with peace of mind.

[1807] Specific examples

[1808] Example 1: Wedding anniversary reminders and support

[1809] User

[1810] You can use a messaging app to exchange ideas with your spouse, such as "Where should we go for our wedding anniversary?"

[1811] Terminal

[1812] The app collects this communication, encrypts it with AES-256, and sends it to a server.

[1813] server

[1814] An NLP engine (NLTK or SpaCy) analyzes the conversation and identifies that the wedding anniversary is approaching. A reminder notification is generated one month in advance and sent to the device with restaurant suggestions using the Google Places API.

[1815] Terminal

[1816] The notification is received and displayed to the user, who can then view the suggested restaurants and make reservations at the ones they like.

[1817] Example 2: Gift suggestions for a friend's birthday

[1818] User

[1819] Send a message to a friend using a messaging app, such as "Happy Birthday!"

[1820] Terminal

[1821] The message is collected, encrypted with AES-256, and sent to the server.

[1822] server

[1823] An NLP engine analyzes birthday messages to identify when a friend's birthday is coming up, generates a reminder one week in advance, and uses a recommendation engine to suggest gift ideas.

[1824] Terminal

[1825] The notification is received and displayed to the user, who can then choose the gift they like from the list of suggested gifts and purchase them from the online shop.

[1826] Prompt Sentence Examples

[1827] "I sent a message to celebrate my friend's birthday and I'd like some gift suggestions."

[1828] Thus, the present invention is a system that supports users so that they do not forget important anniversaries and can efficiently prepare for them.

[1829] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1830] Step 1:

[1831] Terminal

[1832] This data is collected on your device every time you take a photo, record a video, or send a message in a messaging app.

[1833] Specifically, the smartphone's camera app automatically collects photographic information (e.g., photo metadata), and the messaging app records sent and received text messages. The collected data is encrypted with AES-256 and then sent to a server using HTTPS at regular intervals.

[1834] input

[1835] User-generated photo, video, message, and search history data.

[1836] output

[1837] AES-256 encrypted data.

[1838] Step 2:

[1839] server

[1840] The encrypted data is received by the server and temporarily stored in Amazon Web Services (AWS) S3 storage, where it is then transferred via Apache Kafka to a secure analysis environment (e.g., AWS Glue or an Apache Spark cluster) for analysis.

[1841] input

[1842] Encrypted data sent from the device.

[1843] output

[1844] Data transmitted to a secure analysis environment.

[1845] Step 3:

[1846] server

[1847] A natural language processing (NLP) engine (e.g., NLTK or SpaCy) and image recognition algorithms analyze incoming data to identify anniversaries. Keywords and context in messages and search history are used to detect "birthdays" and "wedding anniversaries," and relevant date information is extracted. Algorithms are used to identify events and dates in images and videos. The analyzed information is stored in a database (e.g., MySQL or PostgreSQL) and updated in the user's calendar using the Google Calendar API.

[1848] input

[1849] Temporarily stored data.

[1850] output

[1851] Parsed anniversary information and events reflected in the calendar.

[1852] Step 4:

[1853] server

[1854] Generate reminder notifications one month, one week, or the day before a specified anniversary. A scheduler like Apache Airflow can manage this and generate the reminder message (e.g., "Your birthday is in one month").

[1855] input

[1856] Parsed anniversary information.

[1857] output

[1858] The generated reminder notification.

[1859] Step 5:

[1860] server

[1861] The generated reminder notification is sent to the user's device using Firebase Cloud Messaging (FCM), and once sent, the notification is recorded in the notification log.

[1862] input

[1863] The generated reminder notification.

[1864] output

[1865] Reminder message sent to user device.

[1866] Step 6:

[1867] Terminal

[1868] When a reminder notification is received, it pops up on the user's device. For example, the notification may contain gift suggestions or restaurant links. The user checks the notification and decides what to do from the options displayed.

[1869] input

[1870] Reminder notifications sent by FCM.

[1871] output

[1872] A reminder message that will be displayed on the user's device.

[1873] Step 7:

[1874] server

[1875] Based on past gift history and trend information, the system generates gift suggestions that are perfect for an anniversary. It uses recommendation engines such as Amazon Personalize and Google Recommendations AI, and also uses the Google Places API to search for restaurant information based on the user's location to generate a list of suggestions. The generated information is sent to the user's device.

[1876] input

[1877] Parsed anniversary information and user history data.

[1878] output

[1879] Generated gift suggestions and restaurant information.

[1880] Step 8:

[1881] Terminal

[1882] The suggested gifts and restaurant information are displayed on the user's device. The user can review the suggestions and select the gift or restaurant they like. They can also purchase the selected gift at an online shop or make a reservation at the restaurant.

[1883] input

[1884] Gift suggestions and restaurant information sent from the server.

[1885] output

[1886] Purchase a gift or make a restaurant reservation at the user's option.

[1887] Step 9:

[1888] server

[1889] Based on the user's selection, a restaurant reservation is made via a partner reservation system, such as the OpenTable API or Gurunavi API. Reservation completion information is generated and sent to the user's device.

[1890] input

[1891] Restaurant reservation information selected by the user.

[1892] output

[1893] The generated reservation completion information is sent to the user terminal.

[1894] Step 10:

[1895] Terminal

[1896] A reservation completion notification is received on the user's device and displayed as a pop-up. The user can check the reservation details and enjoy the anniversary with peace of mind.

[1897] input

[1898] Reservation completion information sent from the server.

[1899] output

[1900] A reservation completion notification will be displayed on the user's device.

[1901] In this way, the present invention allows the user to make efficient preparations for important anniversaries without forgetting them.

[1902] (Application example 1)

[1903] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] In modern factories, robot maintenance and part replacement are important, but there is a lack of systems that efficiently manage this and provide timely reminders. In particular, in environments where many robots are in operation, it is difficult for personnel to keep track of each individual maintenance schedule, resulting in maintenance delays and unnecessary costs. Furthermore, in order to carry out scheduled maintenance at the appropriate time, a system that not only provides reminders but also allows for the smooth preparation and ordering of necessary parts is needed.

[1905] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1906] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries and regular maintenance dates, means for sending reminder notifications to the user terminal based on the identified anniversaries and regular maintenance dates, and means for suggesting gifts suitable for anniversaries, making restaurant reservations, and suggesting and ordering maintenance parts. This allows for appropriate management of robot maintenance times and efficient reminders and preparation and ordering of maintenance parts.

[1907] "User terminal" means an electronic device utilized by a user to collect and display data.

[1908] A "server" is a computer system that receives and analyzes data sent from a user terminal and provides the necessary information.

[1909] "Photos, videos, messages, and search history" are the types of information that users generate and store on a daily basis, and from these, the data to be analyzed is identified.

[1910] "Data" refers to various types of information collected from User Devices.

[1911] "Means of collection" refers to the methods and technologies used to obtain data from user devices.

[1912] "Means of transmission" refers to the method or technology used to send the collected data to the server.

[1913] "Means for analyzing" refers to techniques for analyzing received data and extracting useful information.

[1914] An "anniversary" refers to an important date or event for a user, such as a birthday or wedding anniversary.

[1915] "Scheduled maintenance days" refer to days scheduled for maintenance or part replacement of factory robots.

[1916] "Reminder Notification" means a notification that contains information informing the User that a particular event or maintenance is approaching.

[1917] A "present" refers to a gift given on a particular anniversary.

[1918] "Restaurant" refers to a food and beverage establishment whose primary business is serving food.

[1919] "Reservation" means a reservation made in advance to use the Service at a specific date and time.

[1920] "Means of suggestion" refers to methods and technologies for providing users with appropriate options and information.

[1921] "Parts proposal and ordering" refers to the selection and purchase procedures for parts necessary for maintaining factory robots.

[1922] The present invention includes a user terminal, a server, and a system for exchanging data between them. The purpose of the system is to efficiently manage users' anniversaries and scheduled maintenance of factory robots, and to send reminder notifications at appropriate times. An embodiment of the present invention will be described in detail below.

[1923] 1. System Configuration

[1924] The system includes a user terminal, a server, a natural language processing engine, an image recognition algorithm, and a reminder notification function.

[1925] 1.1 User Device

[1926] Users use devices such as smartphones and tablets that have the ability to take photos and videos, send messages, and collect search history from web browsers and applications.

[1927] 1.2 Server

[1928] The server receives and analyzes the data sent from the user terminal. The following hardware and software are used:

[1929] Database: Stores and manages received data

[1930] Natural language processing engine: Used to analyze messages and search history

[1931] Image recognition algorithms: used to analyze photos and videos

[1932] The data is encrypted and sent to a server, where it is analyzed to identify anniversaries and scheduled maintenance dates.

[1933] 2. Data Analysis

[1934] The server uses a natural language processing engine to analyze keywords and context related to anniversaries from users' messages and search history. It also uses image recognition algorithms to identify important events and dates from photos and videos. For example, it extracts date information from messages such as "birthday" or "wedding anniversary" and stores it in a database.

[1935] 3. Reminder function

[1936] Reminder notifications are generated and sent to the user's device one month, one week, or the day before a specified anniversary or maintenance date. These notifications include information about the upcoming anniversary, suggested gifts, restaurant information, maintenance parts, etc. This allows users and staff to remember important events and maintenance and make appropriate preparations.

[1937] 4. Proposal and reservation of gifts and maintenance parts

[1938] The server uses past history and trend information to suggest ideal gifts, restaurants, and maintenance parts for anniversaries. Users and staff can review the suggestions and select and reserve their favorite gifts, restaurants, and parts. This allows for smooth preparation and execution.

[1939] Specific examples

[1940] For example, suppose factory robot 001 last underwent maintenance on October 1, 2023, and the next maintenance is scheduled for every 90 days. The next maintenance is scheduled for January 1, 2024, and a reminder notification is sent on November 1. The notification includes a list of needed parts and a link to purchase them.

[1941] Example prompts to input to the generative AI model:

[1942] We are developing a system to manage the maintenance schedule of factory robots and send reminder notifications. We would like you to create a specific program in Python that will show how the robots collect data, calculate the next scheduled maintenance date, and send reminder emails when the date approaches.

[1943] In this way, the present invention efficiently manages anniversaries and maintenance, helping users and staff to ensure they do not miss important dates and times and to make appropriate preparations.

[1944] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1945] Step 1: Collect data

[1946] It collects data such as photos, videos, messages, and search history taken on user devices. There are various data generated by users as inputs, and these data are collected as outputs. Specifically, users use their smartphones to record their daily activities and store them in a database through a specific application.

[1947] Step 2: Sending data

[1948] The collected data is sent to the server. The input is the data stored on the user's device, and the output is the state in which that data has been sent to the server. Specifically, the collected data is encrypted and then sent to the server at regular time intervals or based on certain conditions.

[1949] Step 3: Analyze the data

[1950] The data received by the server is analyzed to identify anniversaries and scheduled maintenance dates. The input is data stored on the server, and the output is the analysis results. Specifically, a natural language processing engine is used to analyze keywords and context from messages and search history, and an image recognition algorithm is used to identify events and dates from photos and videos.

[1951] Step 4: Generate a reminder notification

[1952] Reminder notifications are generated based on the identified anniversaries and maintenance dates. The input is the analysis results, and the output is the reminder notification. Specifically, reminder notifications are prepared and generated in message format one month, one week, and the day before the identified date.

[1953] Step 5: Send reminders

[1954] The generated reminder notification is sent to the user's device. The input is the reminder notification, and the output is the notification that arrives on the user's device. Specifically, the notification is sent from the server to the user's device and displayed on the user's device.

[1955] Step 6: Proposal for gifts and maintenance parts

[1956] It suggests the best gifts and parts for anniversaries and maintenance. The input is past history, trend information, and data on the user's preferences, and the output is a list of suggested items and parts. Specifically, the server generates the optimal options based on trend information and history and sends them to the user's device.

[1957] Step 7: Facilitate bookings

[1958] Users or staff can check the suggested gifts, restaurants, and maintenance parts, and make reservations or orders. The input is a list of suggested items and parts, and the output is the user's selection. Specifically, the user can check the suggestions on their device, select their favorite items or services, and make a reservation.

[1959] Step 8: Complete your reservation or order

[1960] This notifies the user that a reservation or order has been completed. The input is the user's selection, and the output is a reservation completion notification. Specifically, the reservation or order is confirmed through cooperation with a reservation system or online shop, and the confirmation information is sent to the user's device.

[1961] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1962] The present invention can be implemented in a variety of ways, including a user terminal, a server, an emotion engine, and a method for exchanging data among these. This system recognizes the user's emotions to further personalize anniversary reminders, gift suggestions, and restaurant reservations. The specific processing steps are described below.

[1963] Collection and transmission of user information

[1964] Terminal

[1965] The company said that whenever a user takes a photo or video on their smartphone or sends a message on LINE, relevant data is collected. Search history from web browsers and applications is also periodically collected. This data is encrypted and then sent to a server.

[1966] server

[1967] The server temporarily stores the received data, which is then stored in a secure location for analysis, and the appropriate analysis engine is run.

[1968] Data analysis and anniversary identification

[1969] server

[1970] A natural language processing engine analyzes messages and search history for keywords and context related to anniversaries. For example, it identifies messages containing keywords like "birthday" or "wedding anniversary" and extracts date information from them. Image recognition algorithms similarly identify events and dates in photos and videos. The analyzed information is then stored in a database and updated in the user's calendar.

[1971] Emotion Analysis

[1972] server

[1973] The emotion engine analyzes the user's emotions from their message content, voice, and images. For example, it uses tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to tailor reminders and suggestions.

[1974] Execute the reminder function

[1975] server

[1976] The system generates and schedules reminders one month, one week, or the day before the specified anniversary. The tone and content of the reminder message are adjusted based on the analyzed emotional information. For example, if the user is feeling stressed, the reminder will contain more encouraging words.

[1977] Terminal

[1978] Receive and display reminder notifications to the user, which may include gift suggestions or restaurant links.

[1979] Gift suggestions and reservation support

[1980] server

[1981] In addition to past gift history and trend information, the system generates ideal gift suggestions for anniversaries based on emotional information. The system makes suggestions that reflect the user's emotions and recent interests. Furthermore, the system searches for nearby restaurant information based on the user's location information and generates a list of suggestions. This information is then sent to the user's device.

[1982] Terminal

[1983] The app displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[1984] User

[1985] The user can select a gift or restaurant from the suggestions and make a purchase or reservation. The suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that suit their own and the other person's feelings.

[1986] Restaurant reservation completed

[1987] server

[1988] Based on the user's selection, restaurant reservations are made via affiliated reservation systems.

[1989] server

[1990] Reservation completion information is generated and sent to the user's terminal.

[1991] Terminal

[1992] A reservation completion notification will be displayed to the user, allowing them to enjoy their special day with peace of mind.

[1993] Specific examples

[1994] Example 1: Wedding anniversary reminders and support with sentiment analysis

[1995] User

[1996] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[1997] Terminal

[1998] The app collects this interaction and sends it to a server.

[1999] server

[2000] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[2001] Terminal

[2002] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[2003] Example 2: Gift suggestions for a friend's birthday

[2004] User

[2005] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[2006] Terminal

[2007] Collects messages and sends them to the server.

[2008] server

[2009] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[2010] Terminal

[2011] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[2012] In this way, the present invention is a system that combines an emotion engine to provide a personalized experience according to the user's emotions, making important anniversaries more fulfilling.

[2013] The processing flow will be explained below.

[2014] Step 1:

[2015] Users take photos and videos with their smartphones, send messages on LINE, and conduct daily searches using web browsers and applications.

[2016] Step 2:

[2017] The device periodically collects data such as photos, videos, LINE messages, and search history. This data is stored in local storage each time it is collected.

[2018] Step 3:

[2019] The data collected by the device is encrypted before being sent to the server, in order to protect the user's privacy.

[2020] Step 4:

[2021] The server stores the received data in temporary storage, where it is prepared for analysis.

[2022] Step 5:

[2023] The server runs a natural language processing engine to analyze anniversary-related keywords and context from LINE messages and search history. For example, keywords such as "birthday" and "wedding anniversary" are detected.

[2024] Step 6:

[2025] The server uses image recognition algorithms to extract events or specific dates from photos or videos, for example, recognizing a calendar date or an image of a cake in a photo.

[2026] Step 7:

[2027] The server combines the results of analysis from a natural language processing engine and image recognition algorithms to identify important dates such as birthdays and wedding anniversaries, which are then stored in a database and reflected in the user's calendar.

[2028] Step 8:

[2029] The server generates reminder notifications based on the identified anniversary, with multiple reminders for one month, one week, and one day in advance.

[2030] Step 9:

[2031] The server runs an emotion engine to tailor the message content included in the reminder notification, for example, adding encouraging words to the reminder notification if the user is feeling stressed.

[2032] Step 10:

[2033] The device receives a reminder notification from the server and displays it to the user, with a message such as "Your wedding anniversary is in one month."

[2034] Step 11:

[2035] The server generates the best gift candidates for the anniversary based on the past gift history and trend information.

[2036] Step 12:

[2037] The server uses the emotion engine to further personalize gift suggestions based on the user's recent emotional data: for example, if the user has recently been feeling happy, tickets to a fun event might be suggested.

[2038] Step 13:

[2039] The server searches for nearby restaurant information based on the user's location information and generates a list of suggestions, including options that reflect the user's emotional information.

[2040] Step 14:

[2041] The device displays gift and restaurant information to the user, including links to select gifts and make restaurant reservations.

[2042] Step 15:

[2043] The user reviews the suggested gifts and restaurants, selects the one they like, and then issues instructions to make the purchase or reservation.

[2044] Step 16:

[2045] The server makes a restaurant reservation based on the user's selection via an affiliated reservation system.

[2046] Step 17:

[2047] The server generates reservation completion information and sends it to the user's terminal.

[2048] Step 18:

[2049] The device will display a reservation completion notification to the user, for example, "Reservation completed."

[2050] In this way, the system collects information from daily activities and provides timely reminders and support to help users remember important anniversaries. It also uses an emotion engine to provide a personalized experience based on the user's emotional state.

[2051] Example 2

[2052] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2053] Conventional reminder notification systems only notify users based on anniversaries and events individually set by the user, and do not take into account emotions or personalized suggestions, which has the drawback of not sufficiently improving the user experience. Furthermore, privacy protection is also insufficient when collected data is transmitted unencrypted.

[2054] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2055] In this invention, the server includes means for collecting data such as photos, videos, messages, and search history taken on the user terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server and identifying anniversaries, means for sending reminder notifications to the user terminal based on the identified anniversaries, means for suggesting gifts suitable for anniversaries and reserving sophisticated services, and means for analyzing emotion information using an emotion engine and adjusting the reminder notifications and suggestion content based on the results. This provides a personalized experience according to the user's emotions and enables efficient reminder notifications and suggestions while protecting privacy.

[2056] A "user terminal" is a device operated by a user, such as a smartphone or tablet.

[2057] "Photos, videos, messages, search history, and other data" refers to various digital information that users generate and save on a daily basis, and this information is used for reminders and suggestions.

[2058] "Means of sending to the server" refers to the process of delivering data collected from the user's terminal to the server via communication means such as the Internet.

[2059] "Means of analysis" refers to the process of extracting and understanding information from received data using natural language processing engines and image recognition algorithms.

[2060] "Means for identifying anniversaries" refers to the process of identifying specific significant date information from the analyzed data.

[2061] "Means for sending reminder notifications" refers to a process for delivering a notification message to a user device based on the identified anniversary.

[2062] "Means for suggesting gifts and reserving sophisticated services" refers to the process of proposing products and services related to anniversaries and supporting the reservation and purchase procedures.

[2063] An "emotion engine" refers to software or technology that analyzes a user's emotions, extracting emotional information from the content of the user's messages, behavior, facial expressions, etc.

[2064] "Emotional information" refers to emotional data that can be read from a user's statements and behavior, and includes information such as joy, sadness, and stress.

[2065] "Adjustment" refers to the process of optimizing reminder notifications and suggestions based on analyzed emotional information.

[2066] "Encryption" refers to the process of converting data collected from users so that it cannot be deciphered by third parties.

[2067] The present invention is embodied in a form that uses a user terminal, a server, an emotion engine, and software for operating these in cooperation with each other.

[2068] Collection and transmission of user information

[2069] Terminal

[2070] Data is automatically collected whenever a user takes a photo or video on a smartphone, tablet, or other device, or sends a message on LINE or similar applications. Search history on web browsers and applications is also collected. The collected data is encrypted using an encryption algorithm such as AES256 and transmitted to a server via a secure communication protocol such as HTTPS.

[2071] Data analysis and anniversary identification

[2072] server

[2073] The server temporarily stores the received encrypted data. Next, the server launches an NLP engine (e.g., BERT) to analyze messages and search history. For example, it identifies messages containing keywords such as "birthday" or "wedding anniversary" and extracts date information. It then uses an image recognition algorithm (e.g., an image analysis model using TensorFlow) to identify events and dates from photos and videos. The analyzed information is then stored in a database and reflected in the user's calendar application (e.g., Google Calendar).

[2074] Emotion Analysis

[2075] server

[2076] The server launches an emotion engine (such as IBM Watson's Tone Analyzer) to analyze the user's emotions from the content, voice, and image of the message. Specifically, it uses the tone of the message and facial recognition technology to determine whether the user is happy or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[2077] Execute the reminder function

[2078] server

[2079] The server generates reminder notifications one month, one week, and one day before the specified anniversary. The tone and content of the generated notification message are adjusted based on the emotional information analyzed by the emotion engine. For example, if the user is feeling stressed, the reminder notification will contain more encouraging words.

[2080] Terminal

[2081] The device will then display the received reminder notification to the user, which may include gift suggestions or restaurant links based on the analyzed emotional information.

[2082] Gift suggestions and reservation support

[2083] server

[2084] The server generates gift suggestions for an anniversary based on past gift history, trend information, and emotional information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions, helping users make reservations for sophisticated services.

[2085] Terminal

[2086] The device displays suggested gifts and restaurant information to the user, including links to select gifts and make restaurant reservations.

[2087] User

[2088] Users can review the suggestions and select a gift or restaurant they like, then make a purchase or reservation via their device. These suggestions reflect the analysis results of the emotion engine, making it easier for users to make choices that match their own and the other person's emotions.

[2089] Restaurant reservation completed

[2090] server

[2091] When a user requests a restaurant reservation, the server makes the reservation via a partner reservation system (e.g., OpenTable). Once the reservation is complete, the server generates reservation completion information and sends it to the user's terminal.

[2092] Terminal

[2093] The device will then display a notification to the user that the reservation is complete, allowing them to enjoy their special day with peace of mind.

[2094] Specific examples

[2095] Example 1: Wedding anniversary reminders and support with sentiment analysis

[2096] User

[2097] You can exchange messages with your spouse on LINE, such as "Where should we go for our wedding anniversary?" The emotion engine analyzes positive emotions from this exchange.

[2098] Terminal

[2099] The app collects this interaction and sends it to a server.

[2100] server

[2101] The NLP engine analyzes the conversation, identifies that the wedding anniversary is approaching, and generates a reminder notification one month in advance. Based on the analysis results of the emotion engine, it sends a warm message along with suggested restaurant information.

[2102] Terminal

[2103] The notification is received and displayed to the user, who can then view the restaurant suggestions and make a reservation at the restaurant they like.

[2104] Example 2: Gift suggestions for a friend's birthday

[2105] User

[2106] Send a message to a friend on LINE such as "Happy Birthday!" The emotion engine analyzes the message for birthday wishes and positive emotions.

[2107] Terminal

[2108] Collects messages and sends them to the server.

[2109] server

[2110] The NLP engine analyzes birthday messages to determine that a friend's birthday is coming up, generates a reminder notification one week in advance, and suggests gift ideas that match the emotions analyzed by the emotion engine.

[2111] Terminal

[2112] The notification is received and displayed to the user, who can then select the gift they like from the list of suggested gifts and purchase them from the online shop.

[2113] Example prompt sentence:

[2114] "A user sends a message to a friend on LINE saying 'Happy Birthday!' Analyze this message to determine that the friend's birthday is coming up and generate a reminder notification one week in advance. Also, use an emotion engine to suggest gift ideas that match the positive emotion."

[2115] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2116] Step 1:

[2117] Collection of User Information

[2118] Terminal

[2119] When a user takes a photo or video on their smartphone or sends a message on LINE, that data is collected on the device. Additionally, the user's search history on their web browser or app is also periodically collected. This data is encrypted using the AES256 encryption algorithm. The input data is photos, videos, messages, and search history, and the output data is encrypted data.

[2120] Step 2:

[2121] Data transmission

[2122] Terminal

[2123] The encrypted data is sent securely to the server using HTTPS. The server transmission protocol is used here as HTTPS. The input data is the encrypted user data, and the output data is the encrypted data sent to the server.

[2124] Step 3:

[2125] Data storage and analysis

[2126] server

[2127] The server temporarily stores the received encrypted data. Next, it launches a natural language processing engine (NLP engine, e.g., BERT) to analyze messages and search history. It identifies messages containing specific keywords or contexts (e.g., birthdays, wedding anniversaries) and extracts date information. It then uses an image recognition algorithm (e.g., TensorFlow) to identify events and dates from photos and videos. The input data is the encrypted data sent to the server, and the output data is the analyzed events and date information.

[2128] Step 4:

[2129] Save analysis results and display them on the calendar

[2130] server

[2131] The server saves the analysis results in a database and reflects the information in the user's calendar application (e.g., Google Calendar). The input data is the analyzed event and date information, and the output data is the information reflected in the user's calendar.

[2132] Step 5:

[2133] Emotion analysis

[2134] server

[2135] The server launches an emotion engine (e.g., IBM Watson Tone Analyzer) and analyzes emotions from the message content, voice, and images. Specifically, it determines the user's emotions (e.g., joy, sadness, stress) using the tone of the message and facial recognition technology. The input data is the message content, voice, and images, and the output data is the analyzed emotional information.

[2136] Step 6:

[2137] Generate reminder notifications

[2138] server

[2139] The server generates reminder notifications one month, one week, and the day before the specified anniversary, adjusting the tone and content of the notification message based on the emotional information analyzed by the emotion engine. The input data are the analyzed emotional information and anniversary information, and the output data is the adjusted reminder notification.

[2140] Step 7:

[2141] Sending reminders

[2142] Terminal

[2143] The server sends the generated reminder notification to the user's device. The device displays the received reminder notification to the user. The notification includes gift suggestions and restaurant links based on the analyzed emotional information. The input data is the reminder notification, and the output data is the notification displayed to the user.

[2144] Step 8:

[2145] Gift suggestions and reservation support

[2146] server

[2147] The server generates gift suggestions for the perfect anniversary based on past gift history, trend information, and emotion information. It also searches for nearby restaurant information based on the user's location information and generates a list of suggestions. The input data are gift history, trend information, emotion information, and location information, and the output data is the suggestion list.

[2148] Terminal

[2149] The terminal displays suggested gifts and restaurant information to the user. The list includes links for selecting gifts and making restaurant reservations. The input data is the suggested list, and the output data is the list displayed to the user.

[2150] user

[2151] The user checks the suggestions, selects a gift or restaurant they like, and issues a purchase or reservation instruction. The input data is the suggested gift or restaurant information, and the output data is the user's purchase or reservation instruction.

[2152] Step 9:

[2153] Restaurant reservation completed

[2154] server

[2155] When a user requests a restaurant reservation, the server makes the reservation via an affiliated reservation system. After the reservation is completed, the server generates reservation completion information and sends it to the terminal. The input data is the user's reservation request, and the output data is the reservation completion information.

[2156] Terminal

[2157] The terminal displays a reservation completion notification to the user, allowing the user to celebrate their important anniversary with peace of mind. The input data is the reservation completion information, and the output data is the reservation completion notification displayed to the user.

[2158] (Application example 2)

[2159] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2160] Conventional reminder notification systems and gift suggestion systems provide uniform notifications and suggestions without considering the user's emotions, which means they are unable to provide appropriate and effective suggestions.In addition, there are no systems that understand the user's current emotional state and suggest personalized content based on that, making it difficult to provide the experience the user desires.

[2161] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user information collection means, a data transmission means, a data analysis means, a reminder notification means, a gift suggestion means, an emotion analysis means, and a content personalization means. This makes it possible to provide personalized reminder notifications and gift suggestions based on the user's emotions, and further to suggest content according to the emotions, thereby improving user satisfaction.

[2162] "User terminal" refers to a portable information terminal or computer device used by a user.

[2163] A "photo" is a still or moving image taken by a user terminal.

[2164] A "message" is text data that a user sends and receives through a communication application.

[2165] "Search history" refers to historical data of searches a user has conducted on the Internet.

[2166] A "server" is a computer system connected to a network that receives and analyzes data from user terminals.

[2167] "Data analysis" is the process of converting data collected on a server into required information.

[2168] An "anniversary" is a special event that occurs on a specific date for a user.

[2169] A "reminder" is a notification message that notifies the user of a specific date, time, or event.

[2170] "Gift Suggestion" is a function that recommends suitable gifts to users on special occasions such as anniversaries.

[2171] "Restaurant Reservation" means the process of making a reservation at a restaurant designated by the User.

[2172] "Sentiment analysis" is the process of determining a user's emotions from their message content, images, and audio data.

[2173] "Content personalization" is a feature that individually suggests content such as movies, music, and articles based on the user's emotions and preferences.

[2174] The following system is an embodiment of the present invention. The system is composed of a user terminal, a server, an emotion analysis engine, and a content personalizer. The user terminal is a mobile information terminal or a computer device, and collects data such as photos and videos taken by the user, message transmissions and receptions, and search history. The collected data is sent to the server via the Internet.

[2175] Hardware and software used

[2176] Hardware: Smartphones (iPhone, Android), network-connected servers

[2177] Software: Emotion analysis library (EmotionEngine), content recommendation system (ContentRecommender)

[2178] Data collection and transmission

[2179] The user's device automatically collects data such as photos taken by the user, videos, messages sent, and search history, which is then encrypted and sent to a server using a secure protocol.

[2180] Data analysis and anniversary identification

[2181] The server runs a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. This information is then stored in a database and updated in the user's calendar.

[2182] Emotion analysis

[2183] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the content of the user's messages, images, and voice data. For example, the analysis identifies the user's emotional state, such as tired, happy, or sad. This emotional information is used to adjust subsequent reminder notifications and suggestions.

[2184] Reminders and gift suggestions

[2185] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the analyzed emotional information. If the user is feeling stressed, the message will contain more encouraging words. The notification is sent to the user's device and displayed to the user.

[2186] Similarly, the server also has the ability to suggest ideal gift ideas to users for anniversaries based on the results of sentiment analysis, past gift history, and current trend information.

[2187] Content Personalization

[2188] The server then recommends the most suitable content to the user based on the results of the sentiment analysis. This function uses a content recommendation system (ContentRecommender). For example, if the server determines that the user is tired, it will suggest relaxing music, movies, articles, etc. This allows the server to provide content that is appropriate for the user at that time.

[2189] Examples of specific examples and prompts

[2190] For example, if a user sends a message on LINE saying, "I'm tired today, I want to watch a relaxing movie," the emotion analysis engine will identify the emotion "tired, wanting to be relaxed" and suggest a relaxing movie based on that emotion.

[2191] Example prompt sentence:

[2192] A user sends a message saying, "I'm tired today and want to watch a relaxing movie." Use a sentiment analysis engine to identify the emotion "tired and wanting to relax" and suggest movies based on that emotion.

[2193] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2194] Step 1: Data collection

[2195] User devices collect data such as photos and videos taken by users, messages sent, and search history. These data are collected in real time or periodically based on user behavior. The input data are photos, videos, messages, and search history, and the output data is the collected user data.

[2196] Step 2: Send data

[2197] The user device encrypts the collected data and sends it to the server using a secure protocol. The input data is the collected user data, and the output data is the encrypted transmission data. Specifically, a secure communication protocol such as TLS / SSL is used.

[2198] Step 3: Data analysis

[2199] The server invokes a natural language processing engine (NLP engine) and image recognition algorithms to analyze the received data, extracting anniversary-related information from messages and photos and identifying specific anniversaries. The input data is the received user data, and the output data is the identified anniversary information. The identified information is stored in a database.

[2200] Step 4: Sentiment Analysis

[2201] The server uses an emotion analysis engine (EmotionEngine) to analyze the user's emotions from the user's message content, image data, and voice data. The input data is the message content, image data, and voice data, and the output data is the user's emotional information. Specifically, it uses algorithms that determine emotions through text analysis and facial recognition technology.

[2202] Step 5: Generate reminder notifications

[2203] The server generates reminder notifications one month, one week, and the day before the specified anniversary. The tone and content of the message are adjusted based on the results of emotion analysis. The input data is the specified anniversary information and the user's emotion information, and the output data is the generated reminder notification. For example, if the user is feeling stressed, a notification message containing encouraging words is generated.

[2204] Step 6: Send reminder notifications

[2205] The server sends the generated reminder notification to the user terminal. The input data is the generated reminder notification, and the output data is the notification sent to the user terminal. The user terminal displays the notification and notifies the user.

[2206] Step 7: Gift Suggestion

[2207] The server proposes ideal gift candidates for the user's anniversary based on the emotion analysis results, past gift history, and current trend information. The input data is the user's emotion information, gift history, and trend information, and the output data is a list of gift candidates.

[2208] Step 8: Content proposal

[2209] The server uses a content recommendation system (ContentRecommender) to suggest optimal content to users based on the results of emotion analysis. The input data is the user's emotion information, and the output data is suggested content (movies, music, articles, etc.). For example, if the analysis finds that the user is tired, it will suggest relaxing music, movies, articles, etc.

[2210] Step 9: User Notification

[2211] The user terminal displays the gift suggestions and content suggestions sent from the server to the user. The input data is the suggestion information sent from the server, and the output data is the displayed suggesti...

Claims

1. A means of collecting data such as photos, videos, messages, and search history taken on user devices; means for transmitting the collected data to a server; A means for analyzing the data received by the server and identifying anniversaries; means for sending a reminder notification to a user device based on the identified anniversary; A way to suggest gifts suitable for anniversaries and make restaurant reservations, A system including:

2. 2. The system according to claim 1, wherein the data collected by the user terminal is encrypted before transmission.

3. The system of claim 1, wherein the data analysis uses a natural language processing engine and an image recognition algorithm.

Images