System

Abstract

To provide a system capable of reducing loneliness of a user and improving happiness.SOLUTION: A user authentication unit configured to identify a user, a data linkage unit configured to link the user's existing data system with a system, a past behavior history collection unit configured to collect past behavior history or content of the user, a data analysis unit configured to analyze past data and extract a specific event, an emotion tag, or the like, an emotion state analysis unit configured to analyze emotion information or other data input by the user and evaluate a current emotional state of the user, a reminiscence content generation unit configured to generate an appropriate reminiscence content, and a feedback information collection unit configured to analyze feedback information collected from the user; The system includes a feedback analysis means used for improving the system and an algorithm model update means for updating and optimizing a model of an emotion analysis algorithm and a content proposal algorithm based on a feedback analysis result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's digital society, many people feel lonely. Because this is a subjective issue, it is difficult to resolve. However, it is believed that revisiting past memories can temporarily alleviate loneliness and improve happiness. Conventional systems struggle to provide optimal memory content tailored to the user's emotional state, and lack the ability to continuously improve the system based on user feedback. Therefore, there is a need for a system that can provide optimal memory content tailored to the user's emotional state and that can be continuously improved. [Means for solving the problem]

[0005] The present invention relates to a system that collects and analyzes a user's past behavioral data and generates and provides optimal memory content based on the user's current emotional state. Specifically, the proposed system includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, and an algorithm model update means. This allows the system to provide memory content that best suits the user's emotional state and continuously improve the system based on user feedback. Specifically, by further including an interface means for inputting the user's emotional information and a means for determining the emotional state based on the emotional information, the system can propose content that responds promptly to the user's emotional changes. This can reduce the user's sense of loneliness and improve their sense of happiness.

[0006] "User authentication means" refers to the means used to verify the identity of a user when accessing a system. This includes entering a user ID and password, biometric authentication, etc.

[0007] "Data linking means" refers to a means for linking the system with the user's existing data system, making it possible to access the user's past behavioral data and content.

[0008] "Means for collecting past behavioral data" refers to means for acquiring a user's past behavioral history and content, including email data, photos, and notes.

[0009] "Data analysis means" refers to the means for analyzing collected past data and extracting specific events, emotion tags, etc. Examples include image recognition technology and natural language processing technology.

[0010] "Emotional state analysis means" means a means for analyzing emotional information or other data entered by a user to assess the user's current emotional state, including text analysis and voice analysis.

[0011] The "memory content generation means" is a means for generating appropriate memory content for the user to engage with based on the analyzed data and emotional state. This includes content such as photos, text, and videos.

[0012] "User feedback means" refers to means for collecting feedback information provided by users. For example, it includes a questionnaire format and a rating button.

[0013] "Feedback analysis means" refers to means for analyzing collected feedback information and using it to improve the system, thereby improving the user experience.

[0014] The "algorithm model update means" is a means for updating and optimizing the models of the sentiment analysis algorithm and content suggestion algorithm based on the feedback analysis results.

[0015] "Interface means for inputting emotion information" refers to means for providing an interface for users to input their current emotions or moods. This includes text input forms and voice input interfaces.

[0016] The "means for determining the emotional state based on the emotional information" is a means for analyzing the input emotional information and determining the emotional state of the user, thereby enabling a response according to the user's emotional changes. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system that aims to improve a user's sense of happiness by providing content based on past memories when the user is feeling lonely. The program processing flow of this system will be specifically explained below.

[0039] Overall system operation

[0040] 1. User authentication process

[0041] When a user accesses the system, the terminal displays a form for entering the Yahoo! ID and password.

[0042] The user enters their Yahoo! ID and password and presses the login button.

[0043] The terminal sends the input information to the server.

[0044] The server authenticates the input information using Yahoo!'s authentication API. If authentication is successful, it starts a user session and retrieves user information.

[0045] 2. Data linkage processing

[0046] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[0047] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[0048] 3. Collecting past behavioral data

[0049] The server collects users' past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0050] For example, get emails and photos from the past five years.

[0051] 4. Data Analysis Processing

[0052] Natural language processing algorithms and image recognition technology are used to analyze the data collected by the server.

[0053] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[0054] 5. Emotional state analysis processing

[0055] The terminal displays an interface for the user to input an emotional state.

[0056] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0057] The device transmits this emotion information to the server.

[0058] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[0059] 6. Creating memorable content

[0060] The server generates appropriate memory content from past data based on the user's emotional state.

[0061] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[0062] 7. Content Suggestion and Display

[0063] The server transmits the generated content to the terminal.

[0064] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0065] 8. User Feedback Collection

[0066] The terminal displays a form for the user to input feedback on the proposed content.

[0067] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[0068] The terminal sends feedback information to the server.

[0069] 9. Feedback analysis and algorithmic model updating

[0070] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[0071] This allows the system to continually improve so that next time it suggests content that best suits the user's emotional state.

[0072] In this way, this system provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and increasing feelings of happiness, thereby contributing to improving the user's mental health.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[0076] Step 2:

[0077] The user enters their Yahoo! ID and password and presses the login button.

[0078] Step 3:

[0079] The device sends the entered Yahoo! ID and password to the server.

[0080] Step 4:

[0081] The server authenticates the user information using Yahoo!'s authentication API.

[0082] Step 5:

[0083] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[0084] Step 6:

[0085] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[0086] Step 7:

[0087] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[0088] Step 8:

[0089] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0090] Step 9:

[0091] For example, get emails and photos from the past five years.

[0092] Step 10:

[0093] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[0094] Step 11:

[0095] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[0096] Step 12:

[0097] The terminal displays an interface for the user to input their emotional state.

[0098] Step 13:

[0099] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0100] Step 14:

[0101] The terminal transmits the input emotion information to the server.

[0102] Step 15:

[0103] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[0104] Step 16:

[0105] The server generates appropriate memory content from past data based on the user's emotional state.

[0106] Step 17:

[0107] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[0108] Step 18:

[0109] The server transmits the generated content to the terminal.

[0110] Step 19:

[0111] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0112] Step 20:

[0113] The terminal displays a form for the user to input feedback on the proposed content.

[0114] Step 21:

[0115] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[0116] Step 22:

[0117] The terminal sends feedback information to the server.

[0118] Step 23:

[0119] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[0120] Step 24:

[0121] The server uses an updated algorithm to enable more accurate content suggestions based on the user's emotional state from the next time onwards.

[0122] Example 1

[0123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] In modern society, feelings of loneliness and low happiness among users have become serious problems. To address these issues, a system that can improve users' psychological state by providing content based on past memories is needed. However, existing systems do not adequately provide appropriate content according to emotional states, and they lack a means to effectively utilize user feedback to continuously improve the system.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0126] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, and an algorithm model update means. This makes it possible to generate and display appropriate memory content from past data based on the user's emotional state. Furthermore, by combining an interface means for inputting the user's emotional information with a means for analyzing the emotional information and determining the user's current emotional state, it is possible to grasp the user's psychological state and provide more effective content.

[0127] A "user authentication means" is a mechanism that authenticates a user based on authentication information such as a user name and password that the user inputs when accessing a system.

[0128] "Data linking means" is a mechanism that allows the system to link with external services based on the user's account information and obtain the necessary data.

[0129] The "means for collecting past behavioral data" is used to collect behavioral data from the user's past emails, photos, memo pads, etc.

[0130] "Data analysis means" refers to the function of extracting the meaning and characteristics of collected data using algorithms and techniques to analyze the data.

[0131] The "emotional state analysis means" is a mechanism for analyzing the emotional information input by the user and determining the current emotional state.

[0132] The "memory content generating means" is a function for generating appropriate memory content based on the analyzed data and emotional state.

[0133] The "user feedback means" is an interface for users to input feedback on the provided content.

[0134] The "feedback analysis means" is a mechanism for analyzing feedback information from users and using it to provide content next time.

[0135] The "algorithm model update means" is a function for updating and optimizing the models of the emotion analysis algorithm and content recommendation algorithm based on feedback information.

[0136] The present invention relates to a system for providing content based on past memories in order to reduce the sense of loneliness felt by a user and improve the sense of happiness. An embodiment of this system will be specifically described below.

[0137] This system includes three main elements: a server, a terminal, and a user. First, the user is authenticated using a user authentication method. The terminal displays a login screen for the user, and the user enters authentication information (ID and password). The entered information is sent from the terminal to the server, and the server verifies the information using an authentication API. If authentication is successful, the server starts a session for the user and obtains the necessary user information.

[0138] Next, using the data linking method, the server links with the external service based on the user's account information. At this time, the server prepares to collect past behavioral data such as the user's emails and photo albums. Data linking is performed through the external service's API, and the acquired data is temporarily stored on the server.

[0139] The collected data is analyzed using data analysis tools. Specifically, the server uses natural language processing algorithms (such as NLTK or spaCy) and image recognition technologies (such as OpenCV or TENSORFLOW (registered trademark)) to extract specific events and emotion tags (such as "happiness" or "sadness") from the data. The analysis results are stored in a database.

[0140] Next, the emotional state analysis means is used to analyze the user's current emotional state. The terminal displays an interface for the user to input their emotional state, and the user inputs emotional information. The input emotional information is sent from the terminal to a server, and the server analyzes the emotional information using an emotion analysis API to determine the user's current emotional state.

[0141] The memory content generation means operates based on the user's emotional state. The server generates memory content appropriate to the user's emotional state based on the acquired data and analysis results. For example, if the user inputs "I'm feeling a little down," the server selects past travel photos or conversation records with friends that are related to "happiness." The generated content is sent from the server to the terminal, which displays it to the user.

[0142] After the content is displayed, the user uses the user feedback means to input feedback on the provided content. The user inputs an evaluation such as "it was very good" or "it was average" into the feedback form displayed on the terminal and transmits the input to the server via the form.

[0143] The feedback analysis means receives this feedback information and analyzes it on the server. The analysis results are reflected in the system's learning model through the algorithm model update means. This enables more appropriate suggestions to be made the next time content is provided.

[0144] As a specific example of how it works, if a user enters "I'm feeling a little down" as a prompt, the server will select photos of past family trips and records of fun conversations with friends related to "happiness" and send them to the device. The device will then display these along with a message such as "Look at these family trip photos!" If the user enters feedback such as "It was great," the server will update the system based on that feedback and use it to suggest content next time.

[0145] Examples of prompts: "I'm feeling a little down," "I'm having a good time today."

[0146] In this way, the system of the present invention provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and improving feelings of happiness, thereby contributing to improving the user's mental health.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] User authentication process

[0150] When a user accesses the system, the terminal displays a form for entering a Yahoo! ID and password. After the user provides the input information and presses the login button, the terminal sends the input information to the server. The server verifies the input information using Yahoo!'s authentication API, and if authentication is successful, it starts a user session and obtains the user information.

[0151] Input: Yahoo! ID and password

[0152] Output: Authentication success flag, user information

[0153] Specific operation: The device displays a login screen and the user enters their authentication information. The information is sent from the device to the server, and the server performs authentication via the Yahoo! API.

[0154] Step 2:

[0155] Data linkage processing

[0156] After successful authentication, the server links the user's account to an external service, which prepares the server to collect user behavior data (emails, photos, etc.).

[0157] Input: Authentication success flag, user information

[0158] Output: Data collection ready flag

[0159] Specific operation: The server uses the user information to connect to the API of external services (email, photo album) and prepares to collect data.

[0160] Step 3:

[0161] Collecting past behavioral data

[0162] The server collects the user's past behavior data from emails, photo albums, etc. The server acquires and stores this data for a certain period of time (e.g., the past five years).

[0163] Input: Data collection ready flag

[0164] Output: Collected behavioral data

[0165] Specific operation: The server calls the API and collects email and photo data for the specified period. The acquired data is temporarily stored in storage.

[0166] Step 4:

[0167] Data analysis processing

[0168] The server analyzes the collected data using natural language processing and image recognition technology, extracting emotion tags (such as "happiness" or "sadness") from text data and performing facial and scene recognition from image data.

[0169] Input: Collected behavioral data

[0170] Output: Analysis results (emotion tags, specific events)

[0171] How it works: The server uses Python libraries such as NLTK, spaCy, OpenCV, and TensorFlow to analyze text and image data and store the results in a database.

[0172] Step 5:

[0173] Emotional state analysis processing

[0174] The device provides the user with an interface for inputting their emotional state. After the user inputs their emotional information and presses the send button, the device sends the data to the server. The server then uses an emotion analysis API to analyze the data and determine the user's current emotional state.

[0175] Input: User's emotional information

[0176] Output: Current emotional state

[0177] Specific operation: The device displays the interface, receives input from the user, and sends it to the server, which then analyzes it using the sentiment analysis API.

[0178] Step 6:

[0179] Creating memorable content

[0180] The server generates optimal memory content based on the analysis results, based on the user's emotional state. For example, if the user enters "I'm feeling a little down," the server will select photos of past family trips related to "happiness."

[0181] Input: Current emotional state, analysis results

[0182] Output: Memories content

[0183] What it does: The server queries the database for the appropriate content and sends the generated content to the device in the appropriate format.

[0184] Step 7:

[0185] Content Suggestion and Display

[0186] The server sends the generated memory content to the device, which then displays it to the user, for example, with a message such as "Check out these family vacation photos!"

[0187] Input: Memories content

[0188] Output: The displayed content

[0189] Specific operation: The server sends the generated content to the device via an HTTP response, and the device displays it.

[0190] Step 8:

[0191] User feedback collection

[0192] The device provides the user with a form for entering feedback on the content. The user enters an evaluation such as "very good" or "average," and when the user presses the submit button, the device sends the data to the server.

[0193] Input: User feedback

[0194] Output: Collected feedback

[0195] Specific operation: The device displays a feedback input form, receives input from the user, and sends it to the server.

[0196] Step 9:

[0197] Feedback analysis and algorithmic model updating

[0198] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby improving the quality of the next content recommendation.

[0199] Input: User feedback

[0200] Output: Updated algorithmic model

[0201] What it does: The server stores the feedback information in a database and updates the analysis and generative AI model by retraining it.

[0202] (Application example 1)

[0203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0204] In modern society, there is a demand for technologies that can reduce the feelings of loneliness and stress felt by users and improve their happiness. In particular, in electronic payment services, there is a growing need for systems that not only process payments but also recommend items that will bring happiness to users based on their past purchase history and emotional state. However, conventional technologies have not been able to meet such complex requirements.

[0205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0206] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a means for adding an emotional tag based on past purchase data, and a means for recommending items based on the emotional state, thereby making it possible to provide content and recommended items that are optimal for the user's current emotional state.

[0207] "User authentication means" refers to a means for identifying a user based on authentication information when the user accesses the system.

[0208] "Data linkage means" refers to a means of linking user information with external systems and collecting and managing necessary data.

[0209] The "means for collecting past behavioral data" is a means for collecting data on the past behavior of a user.

[0210] "Data analysis means" is a means for analyzing collected data and extracting useful information and patterns.

[0211] The "emotional state analysis means" is a means for analyzing the current emotional state of the user.

[0212] The "memory content generating means" is a means for generating memory content that matches the emotional state of the user based on past data.

[0213] "User feedback means" refers to a means for collecting feedback from users.

[0214] The "feedback analysis means" is a means for analyzing collected feedback information and using it to improve the system.

[0215] The "algorithm model updating means" is a means for updating and optimizing the algorithm model based on the collected feedback information.

[0216] The "means for adding emotion tags based on past purchase data" is a means for analyzing past purchase data and adding tags related to specific emotional states.

[0217] The "means for recommending items based on emotional state" is a means for recommending appropriate items based on the emotional state of a user.

[0218] This invention is a system that reduces users' feelings of loneliness and improves their happiness, and is particularly aimed at application to electronic payment services. The system is characterized by recommending appropriate content and items based on the user's emotional state.

[0219] Overall system operation

[0220] User authentication method

[0221] When a user accesses the system, the server identifies the user based on the authentication information (ID and password).To do this, the server uses an authentication API to authenticate the entered information, starts a session, and obtains user information.

[0222] Data linking method

[0223] After successful authentication, the server links the user's account to the system, and prepares to collect data such as past emails and photo albums.

[0224] Past behavioral data collection method

[0225] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[0226] Data Analysis Methods

[0227] The server analyzes the collected data, using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[0228] Emotional state analysis means

[0229] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[0230] Means of creating memorable content

[0231] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips and records of fun conversations with friends.

[0232] User feedback tools

[0233] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average" and sends it to the server.

[0234] Feedback Analysis Methods

[0235] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby enabling it to provide more appropriate content in the next recommendation.

[0236] Algorithmic model update method

[0237] Based on the feedback information, the server adjusts the parameters of the AI ​​model used and the proposed algorithm to improve the accuracy of the system.

[0238] Emotion tagging method

[0239] The server analyzes past purchase data and adds tags related to specific emotional states, such as "happy" or "excited," to previously purchased items.

[0240] Item recommendation method

[0241] The server recommends appropriate items based on the user's current emotional state. If the user is currently in a "happy" emotional state, the server creates a list of items that the user has purchased in the past that made them feel happy, sends this list to the device, and displays it.

[0242] Hardware and software used

[0243] Hardware: Smartphone

[0244] software

[0245] Python: Implementation of the entire application

[0246] requests: A library for making API requests

[0247] JSON: Data Format

[0248] Natural Language Processing Algorithms: Analyzing Text Data

[0249] Image recognition technology: Analysis of image data

[0250] Sentiment Analysis API: Emotional information analysis

[0251] Specific examples

[0252] For example, if a user is currently in a "happy" emotional state, the system will recommend items related to "happiness" based on their past purchase history, such as candy or toys that have been purchased in the past and that have brought them happiness.

[0253] Prompt Sentence Examples

[0254] The user is currently in a "happy" emotional state. Recommend items that brought the user happiness based on their past purchase history.

[0255] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0256] Step 1:

[0257] User authentication

[0258] When a user accesses the system, the server displays a form for entering an ID and password. The user enters the ID and password and sends them to the server. The server uses an authentication API to check the input information, and if authentication is successful, starts a session and obtains the user information.

[0259] Input: User ID and password

[0260] Output: Session ID and user information

[0261] Step 2:

[0262] Data Linkage

[0263] After successful authentication, the server performs the procedure to link the user's account with the system, and prepares to collect the user's past data (emails, photo albums, purchase history, etc.).

[0264] Input: Authenticated user information

[0265] Output: Notification that data integration is ready

[0266] Step 3:

[0267] Collecting past behavioral data

[0268] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[0269] Input: Notification that data integration is ready

[0270] Output: Past behavior data

[0271] Step 4:

[0272] Data analysis

[0273] The server analyzes the collected data using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[0274] Input: Past behavior data

[0275] Output: Parsed data and sentiment tags

[0276] Step 5:

[0277] Emotional State Analysis

[0278] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[0279] Input: User's emotional information

[0280] Output: Parsed emotional state

[0281] Step 6:

[0282] Creation of memory content and items

[0283] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos from past family trips or records of fun conversations with friends. It also analyzes past purchase data and adds related emotional tags.

[0284] Input: Analyzed emotional state, past behavioral data

[0285] Output: Generated memory content and recommended items

[0286] Step 7:

[0287] Content Suggestion and Display

[0288] The server sends the generated content to the device and displays it to the user along with a message such as "Look at these family vacation photos!" Recommended items are also displayed at the same time.

[0289] Input: Generated memory content and recommended items

[0290] Output: Content and items displayed to the user

[0291] Step 8:

[0292] Feedback collection

[0293] The device displays a form for the user to enter feedback on the proposed content or recommended items. The user enters feedback such as "It was really good" or "It was average" and sends it to the server.

[0294] Input: User feedback

[0295] Output: Collected feedback information

[0296] Step 9:

[0297] Feedback analysis and algorithm updates

[0298] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and recommendation algorithm, thereby enabling it to provide more appropriate content and items in the next proposal.

[0299] Input: Collected feedback information

[0300] Output: Updated and optimized algorithmic model

[0301] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0302] The present invention is a system that aims to provide optimal memorable content based on the user's emotional state, reducing feelings of loneliness and improving happiness. In particular, by incorporating an emotion engine that recognizes the user's emotions in real time, more accurate emotion analysis and content suggestions are achieved. The processing of the entire system is explained in detail below.

[0303] Overall system operation

[0304] 1. User authentication process

[0305] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[0306] The user enters their Yahoo! ID and password and presses the login button.

[0307] The device sends the entered Yahoo! ID and password to the server.

[0308] The server authenticates the user information using Yahoo!'s authentication API, and if authentication is successful, starts a user session and obtains the user information.

[0309] 2. Data linkage processing

[0310] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[0311] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[0312] 3. Collecting past behavioral data

[0313] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0314] For example, get emails and photos from the past five years.

[0315] 4. Data Analysis Processing

[0316] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[0317] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[0318] 5. Emotional state analysis processing

[0319] The terminal displays an interface for the user to input their emotional state.

[0320] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0321] The device transmits this emotion information to the server.

[0322] The server uses an emotion engine to analyze the input emotion information and determine the user's current emotional state.

[0323] 6. Emotion Recognition by Emotion Engine

[0324] The server uses an emotion engine to recognize emotions by analyzing the user's text, voice, and facial expression data.

[0325] The server updates the user's real-time emotional state based on the recognized emotional information.

[0326] 7. Creating memorable content

[0327] The server generates appropriate memory content from past data based on the user's emotional state.

[0328] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[0329] 8. Content Suggestion and Display

[0330] The server transmits the generated content to the terminal.

[0331] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0332] 9. User Feedback Collection

[0333] The terminal displays a form for the user to input feedback on the proposed content.

[0334] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[0335] The terminal sends feedback information to the server.

[0336] 10. Feedback analysis and algorithm model update

[0337] The server analyzes the feedback information it receives and uses this information to update and optimize the models of the sentiment analysis algorithm and content suggestion algorithm.

[0338] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[0339] In this way, the system constantly monitors the user's real-time emotional state and provides appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness, which can greatly contribute to improving the user's mental health.

[0340] The processing flow will be explained below.

[0341] Step 1:

[0342] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[0343] Step 2:

[0344] The user enters their Yahoo! ID and password and presses the login button.

[0345] Step 3:

[0346] The device sends the entered Yahoo! ID and password to the server.

[0347] Step 4:

[0348] The server authenticates the user information using Yahoo!'s authentication API.

[0349] Step 5:

[0350] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[0351] Step 6:

[0352] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[0353] Step 7:

[0354] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[0355] Step 8:

[0356] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0357] Step 9:

[0358] For example, obtain email and photo data from the past five years.

[0359] Step 10:

[0360] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[0361] Step 11:

[0362] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[0363] Step 12:

[0364] The terminal displays an interface for the user to input an emotional state.

[0365] Step 13:

[0366] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0367] Step 14:

[0368] The terminal transmits the input emotion information to the server.

[0369] Step 15:

[0370] The server uses an emotion engine to analyze the input emotion information to determine the user's current emotional state.

[0371] Step 16:

[0372] The server further displays an interface on the terminal for collecting the user's text, voice, and facial expression data.

[0373] Step 17:

[0374] The user inputs their voice and facial expressions through a camera and microphone, and the device sends this data to the server.

[0375] Step 18:

[0376] The server uses an emotion engine to analyze the collected text, voice, and facial expression data to recognize and update the user's real-time emotional state.

[0377] Step 19:

[0378] The server generates optimal memory content from past data based on the user's emotional state.

[0379] Step 20:

[0380] For example, if the server detects a state of "feeling a little down," it will select photos of past family trips or records of fun conversations with friends that are related to "happiness."

[0381] Step 21:

[0382] The server transmits the generated content to the terminal.

[0383] Step 22:

[0384] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0385] Step 23:

[0386] The terminal displays a form for the user to input feedback on the proposed content.

[0387] Step 24:

[0388] The user enters feedback such as "It was great" or "It was average" and presses the submit button.

[0389] Step 25:

[0390] The terminal sends feedback information to the server.

[0391] Step 26:

[0392] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[0393] Step 27:

[0394] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[0395] Example 2

[0396] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] In modern society, many people suffer from loneliness and stress. In particular, it is difficult to receive appropriate support tailored to each individual's emotional state. Therefore, there is a need for a system that provides support tailored to each user's emotional state, reducing loneliness and improving happiness.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0399] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, and a data analysis means using natural language processing and image recognition technology, which allows the server to efficiently collect and analyze the user's past behavioral data and provide appropriate memory content according to the user's individual emotional state.

[0400] "User authentication means" refers to a means for identifying and authenticating an individual when a user accesses a system.

[0401] The "data linking means" is a means for linking the system with an external data source and collecting data on users' past behavior.

[0402] The "means for collecting past behavioral data" is a means for systematically collecting a user's past activity history and digital content.

[0403] "Data analysis means using natural language processing and image recognition technology" means a means that uses natural language processing (NLP) and image recognition technology to analyze collected data and extract specific event and emotion tags.

[0404] The "interface means for inputting user's emotional information" is a means for providing an interface that allows the user to input his / her own emotional state.

[0405] The "means for analyzing an emotional state using an emotion engine" is a means for analyzing input emotion information, text, voice, and facial expression data to determine the user's current emotional state.

[0406] The "means for generating appropriate memorable content based on the analyzed emotions" is a means for selecting and generating content suitable for a user from past data based on the user's emotional state.

[0407] The "means for presenting the generated content to the user" refers to a means for presenting the generated memory content to the user through visual and auditory means.

[0408] The "user feedback means" is a means for a user to input feedback on the presented content and transmit it to the system.

[0409] The "means for analyzing feedback information and updating algorithm models" refers to a means for analyzing feedback information obtained from users and updating models to improve the accuracy of sentiment analysis algorithms and content suggestion algorithms.

[0410] The present invention is a system that aims to provide optimal memory content based on a user's emotional state, reduce feelings of loneliness, and improve feelings of happiness. The system of the present invention uses the following hardware and software.

[0411] Hardware and software used

[0412] Hardware

[0413] Device: A personal computer, smartphone, or tablet through which a user accesses the device.

[0414] Server: A server computer that processes the entire system.

[0415] Camera and microphone: Devices for collecting facial and voice data from the user.

[0416] software

[0417] Natural Language Processing Algorithms (NLP): Algorithms for analyzing text data, specifically using Google® Cloud Natural Language API or similar technologies.

[0418] Image recognition technology: Technology for analyzing image data. Specifically, Google Cloud Vision API is used.

[0419] Emotion engine: An engine for analyzing the user's emotional state. A specific example is the Emotion API.

[0420] Embodiments of each step

[0421] 1. User authentication process

[0422] When a user accesses the system, they enter their Yahoo! ID and password into the form displayed on the terminal. The terminal then sends this information to the server, which then authenticates the user information using Yahoo!'s authentication API.

[0423] 2. Data linkage processing

[0424] The server will connect the system to the already authenticated user account and prepare to collect the necessary data from Yahoo! Mail and Yahoo! Photo Album.

[0425] 3. Collecting past behavioral data

[0426] The server collects past behavioral data from each Yahoo! service (email, photo album, etc.). For example, it collects emails and photos from the past five years.

[0427] 4. Data Analysis Processing

[0428] The server uses the data collected and applies natural language processing algorithms and image recognition techniques to extract specific events and emotion tags from the data.

[0429] 5. Emotional state analysis processing

[0430] The terminal displays an interface for the user to input their current emotional state. The user inputs emotional information, and this information is sent from the terminal to the server. The server analyzes the emotional state using an emotional engine.

[0431] 6. Emotion Recognition by Emotion Engine

[0432] The server uses an emotion engine to analyze text data, voice data, and facial expression data to recognize the user's emotions in real time.

[0433] 7. Creating memorable content

[0434] The server selects and generates appropriate memory content from past data based on the user's emotional state. For example, if the user is feeling a little down, it will select photos of past family trips and records of fun conversations related to happiness.

[0435] 8. Content Suggestion and Display

[0436] The server sends the generated memory content to the device, which then displays it to the user. For example, it may be displayed with a message such as "Check out these family vacation photos!"

[0437] 9. User Feedback Collection

[0438] The terminal displays a form for the user to input feedback about the content. The user inputs the feedback, which is then sent from the terminal to the server.

[0439] 10. Feedback analysis and algorithm model update

[0440] The server analyzes the received feedback information and updates the algorithm model, enabling more accurate content suggestions based on the user's emotional state from the next time onwards.

[0441] Specific examples

[0442] For example, if the user feels "a little down today," the prompt sentence might be:

[0443] Prompt Sentence Examples

[0444] "Simulate a scenario where a user logs in, inputs their emotional state, and then the system suggests photos from past family trips."

[0445] In this way, the system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[0446] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0447] The flow of this system's program processing

[0448] Step 1: User authentication process

[0449] Input: The user enters their Yahoo! ID and password.

[0450] Processing: The terminal encrypts these input data and sends them to the server.

[0451] Data processing and calculation: The server uses Yahoo!'s authentication API to verify and authenticate user information.

[0452] Output: If authentication is successful, start a user session and get user information.

[0453] Specific behavior:

[0454] A user enters the ID "user@example.com" and password "123" into a form.

[0455] The device encrypts the data and sends it to the server.

[0456] The server calls the Yahoo! authentication API, and if authentication is successful, the message "Welcome, user" is displayed.

[0457] Step 2: Data integration process

[0458] Input: The server maintains the user session after successful authentication.

[0459] Processing: The server prepares to exchange data with each Yahoo! service.

[0460] Data processing and calculation: The server uses the API to ask the user for permission to access past emails and photo album data.

[0461] Output: If the user approves the permission, the data is ready to be collected.

[0462] Specific behavior:

[0463] The server requests access to your Yahoo! account.

[0464] Once the user grants access, the server prepares to collect data.

[0465] Step 3: Collect historical behavioral data

[0466] Input: The server starts collecting data based on the access rights granted by the user.

[0467] Processing: The server collects data from Yahoo! Mail, Photo Album, Notepad, etc.

[0468] Data processing and calculation: The server organizes and stores the collected data in chronological order.

[0469] Output: A complete dataset of emails and photos from the past five years.

[0470] Specific behavior:

[0471] The server collects emails and photos from the period "2018-01-01" to "2023-01-01".

[0472] Collected data is organized and saved in chronological order.

[0473] Step 4: Data analysis and processing

[0474] Input: The dataset collected by the server.

[0475] Processing: Analyzing the data using natural language processing algorithms (NLP) and image recognition techniques.

[0476] Data processing and calculation: The server extracts specific events and emotion tags from the text data and extracts facial expressions from the image data.

[0477] Output: The analysis results in a dataset labeled with event and emotion tags.

[0478] Specific behavior:

[0479] The server uses the Google Cloud Natural Language API to analyze the email content and associate the keyword "family trip" with the tag "happiness."

[0480] It uses Google Cloud Vision API to perform facial recognition on photos and classify photos of "smiling faces" as "happy."

[0481] Step 5: Emotional state analysis processing

[0482] Input: The user inputs their current emotional state.

[0483] Processing: The terminal sends this input data to the server.

[0484] Data processing and calculation: The server uses an emotion engine to analyze the input emotion information and determine the emotional state.

[0485] Output: The user's current emotional state is recognized.

[0486] Specific behavior:

[0487] The user types into the terminal interface, "I'm feeling a little down today."

[0488] This information is sent to the server, which uses its emotion engine to determine that "slightly depressed" means "mild depression."

[0489] Step 6: Emotion Recognition with the Emotion Engine

[0490] Input: User text, voice, and facial expression data.

[0491] Processing: The server analyzes these data using the emotion engine.

[0492] Data processing and calculation: The emotion engine analyzes the data and recognizes the user's real-time emotional state.

[0493] Output: The emotional state is updated as a result of the analysis.

[0494] Specific behavior:

[0495] The user uses the camera of the terminal to capture his / her facial expression and also records voice information.

[0496] The server analyzes this data and recognizes a "smile" as "happiness."

[0497] Step 7: Create memorable content

[0498] Input: User's emotional state and analyzed past behavioral data.

[0499] Processing: The server generates appropriate memory content based on these data.

[0500] Data processing and calculation: Select the most suitable content from past data based on emotional state.

[0501] Output: The generated memories content.

[0502] Specific behavior:

[0503] The server selects photos of past family trips based on the user's "mild depression."

[0504] Pick out a few fun photos from the "Summer Travel 2019" folder.

[0505] Step 8: Content suggestion and display

[0506] Input: Generated memories content.

[0507] Processing: The server sends the content to the terminal.

[0508] Data processing and calculation: Prepares the layout for the device to display the content.

[0509] Output: The memories content displayed to the user.

[0510] Specific behavior:

[0511] The device will display the suggested family vacation photos on a large screen, along with the message, "Check out these family vacation photos!"

[0512] Step 9: Gather user feedback

[0513] Input: User feedback.

[0514] Processing: The terminal displays a form for inputting feedback and sends the information entered by the user to the server.

[0515] Data processing and calculation: The server stores and organizes the feedback information.

[0516] Output: A dataset of feedback information.

[0517] Specific behavior:

[0518] The user enters "It was great" in the feedback form and presses the submit button.

[0519] This information is sent to a server and stored in a database.

[0520] Step 10: Feedback analysis and algorithmic model update

[0521] Input: Feedback information from the user.

[0522] Processing: The server analyzes the feedback information and updates the algorithmic model.

[0523] Data processing and calculation: Optimize the accuracy of sentiment analysis algorithms and content suggestion algorithms based on feedback information.

[0524] Output: The updated algorithm model.

[0525] Specific behavior:

[0526] The server analyzes the feedback such as "It was really good" and changes the parameters of the sentiment analysis algorithm.

[0527] Optimize the algorithm model for the next proposal.

[0528] Through the above process, this system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[0529] (Application example 2)

[0530] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0531] In modern society, people often feel lonely and stressed, which can lead to poor mental health. To address this, a method is needed to provide appropriate memorable content based on the user's emotional state. However, existing systems are inadequate in accurately recognizing a user's emotional state in real time and providing optimal content based on that state. As a result, it is difficult to alleviate the loneliness and stress felt by users. Furthermore, effective feedback for users and optimization of content suggestion algorithms are not adequately implemented.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a real-time emotion recognition means using a head-mounted display, and a means for displaying content based on the real-time emotional state. This makes it possible to accurately recognize the user's emotional state in real time and provide optimal memory content based on that state. By simply wearing the HMD, the user can experience memory content based on their own emotions in real time, effectively reducing feelings of loneliness and stress.

[0533] "User authentication means" is a function that verifies the legitimacy of a user by having the user access the system and enter authentication information.

[0534] "Data linking means" is a function that links an authenticated user's account with the system and shares necessary data.

[0535] "Means for collecting past behavioral data" is a function that collects users' past behavioral data (e.g., emails, photos, etc.).

[0536] "Data analysis means" is a function that analyzes collected data and extracts useful information.

[0537] The "emotional state analysis means" is a function that analyzes the emotional information provided by the user and determines the emotional state of the user.

[0538] The "memory content generating means" is a function that generates appropriate memory content based on the emotional state of the user.

[0539] The "user feedback means" is an interface for users to input feedback on the provided content.

[0540] The "feedback analysis means" is a function that analyzes feedback provided by users and uses it to improve the system.

[0541] The "algorithm model update means" is a function that updates the emotion analysis algorithm and the content suggestion algorithm based on the analysis results.

[0542] "Real-time emotion recognition means using a head-mounted display" is a function that uses sensors built into the head-mounted display to recognize the user's emotions in real time.

[0543] The "means for displaying content based on real-time emotional state" is a function for displaying optimal memory content on the head-mounted display in accordance with the emotional state recognized in real time.

[0544] This invention is a system that provides optimal memory content based on the user's emotional state, reducing feelings of loneliness and stress. It primarily uses a head-mounted display (HMD) to recognize the user's emotions in real time and display memory content accordingly.

[0545] System Configuration

[0546] The system includes the following main components:

[0547] User authentication method

[0548] Data linking method

[0549] Past behavioral data collection method

[0550] Data Analysis Methods

[0551] Emotional state analysis means

[0552] Means of creating memorable content

[0553] User feedback tools

[0554] Feedback Analysis Methods

[0555] Algorithmic model update method

[0556] Real-time emotion recognition using a head-mounted display

[0557] A means of displaying content based on real-time emotional state

[0558] Hardware and Software

[0559] Hardware

[0560] Head-mounted display (HMD): A device worn by the user that displays visual content, with built-in facial recognition cameras and heart rate sensors.

[0561] PC or smartphone: A device that connects to the HMD and processes data.

[0562] Server: Responsible for data collection, analysis, content generation, etc.

[0563] software

[0564] Python: A programming language.

[0565] OpenCV: A library for face recognition.

[0566] Yahoo SDK: A software development kit for working with Yahoo services.

[0567] EmotionEngine: An engine for emotion analysis.

[0568] ContentGenerator: A module that generates and suggests memory content.

[0569] Processing Details

[0570] 1. User Authentication

[0571] The server first authenticates the user based on the credentials provided by the user, which is done using the Yahoo SDK.

[0572] 2. Data Linkage

[0573] After authentication, the server links the user's account to the system and then prepares to collect past behavioral data.

[0574] 3. Collecting past behavioral data

[0575] The server collects data such as users' past emails, photo albums, notepads, etc. The collection period is, for example, the past five years.

[0576] 4. Data Analysis

[0577] The server analyzes the collected data and uses natural language processing algorithms and image recognition technology to extract specific event and emotion tags.

[0578] 5. Emotional State Analysis

[0579] The device (HMD) displays an interface for inputting the user's emotional state and transmits the collected emotional information to the server, which analyzes it and determines the user's current emotional state.

[0580] 6. Real-time emotion recognition

[0581] Using the cameras and sensors built into the HMD, we recognize the user's real-time emotional state using OpenCV and EmotionEngine.

[0582] 7. Creating memorable content

[0583] The server generates appropriate memory content based on the user's real-time emotional state. For example, if the user is feeling a little down, it will select videos of past family trips or photos of happy friends.

[0584] 8. Content Display

[0585] The content selected by the user is displayed through the HMD.

[0586] Specific examples

[0587] The user wears the HMD and the system analyzes the user's emotions in real time. If the user's emotional state is determined to be "a little depressed," the system displays videos of past family trips and happy photos with friends, helping the user to relax and feel happier.

[0588] Prompt Sentence Examples

[0589] "Generate relevant memory content by providing a user's facial image and emotional state as input. The user is in a slightly depressed state."

[0590] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0591] Step 1:

[0592] The user wears a head-mounted display (HMD) and accesses the system. The device displays an interface for the user to enter their Yahoo! ID and password. This input information is the input data for user authentication. The device sends the entered authentication information to the server, which then authenticates the user information using the Yahoo! SDK. If authentication is successful, the server starts a user session and obtains the user information. The authenticated user information is obtained as output data.

[0593] Step 2:

[0594] The server activates the data integration method that links the authenticated user's account with the system. At this point, it is ready to retrieve data from past Yahoo! services (e.g., email and photo albums). The behavioral data to be collected is integrated and becomes the input data for the next step.

[0595] Step 3:

[0596] The server uses a past behavioral data collection means to collect the user's past behavioral data (emails, photo albums, memo pads, etc.). The collection period is, for example, the past five years. The collected data becomes input data for the data analysis means. The collected behavioral data is obtained as output data.

[0597] Step 4:

[0598] The server analyzes the collected data using data analysis means. Specifically, it uses natural language processing algorithms and image recognition technology to extract specific events and emotion tags (e.g., "happiness," "sadness," etc.) from the data. The results of this analysis are input data for the emotion state analysis means. Emotion tags and event information are obtained as output data.

[0599] Step 5:

[0600] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm a little down" or "I'm having a good day," and the device sends the information to the server. This emotional information becomes input data for the emotional state analysis means. The server uses an emotion engine to analyze the input emotional information and determine the user's current emotional state. The user's current emotional state is obtained as output data.

[0601] Step 6:

[0602] Using the camera and sensors built into the HMD, the server recognizes the user's real-time emotions. Using OpenCV and EmotionEngine, the server analyzes the user's facial images and heart rate data and updates their emotional state in real time. This real-time emotional data becomes the input data for the memory content generation means. The real-time emotional state is obtained as output data.

[0603] Step 7:

[0604] The server generates optimal memory content using the memory content generation means based on the user's real-time emotional state. For example, if the user is feeling a little down, the server selects videos of past family trips and photos of friends looking happy. This memory content becomes input data for the content display means. The generated memory content is obtained as output data.

[0605] Step 8:

[0606] The device presents the selected memory content to the user through the HMD. The user experiences this content visually in real time through the HMD. Intuitive messages and guides are displayed as an interface to the user. The output data becomes the content that the user experiences visually.

[0607] Step 9:

[0608] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average," and this information becomes input data for the user feedback means. The terminal sends the feedback information to the server, and the user's feedback information is obtained as output data.

[0609] Step 10:

[0610] The server uses the feedback analysis means to analyze the received feedback information and, based on this, updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm. The algorithm model update means incorporates this feedback information to improve the accuracy of future suggestions. The updated algorithm model is obtained as output data.

[0611] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0613] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0614] [Second embodiment]

[0615] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0616] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0617] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0618] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0619] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0620] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0621] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0622] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0623] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0624] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0625] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0626] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0627] The present invention relates to a system that aims to improve a user's sense of happiness by providing content based on past memories when the user is feeling lonely. The program processing flow of this system will be specifically explained below.

[0628] Overall system operation

[0629] 1. User authentication process

[0630] When a user accesses the system, the terminal displays a form for entering the Yahoo! ID and password.

[0631] The user enters their Yahoo! ID and password and presses the login button.

[0632] The terminal sends the input information to the server.

[0633] The server authenticates the input information using Yahoo!'s authentication API. If authentication is successful, it starts a user session and retrieves user information.

[0634] 2. Data linkage processing

[0635] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[0636] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[0637] 3. Collecting past behavioral data

[0638] The server collects users' past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0639] For example, get emails and photos from the past five years.

[0640] 4. Data Analysis Processing

[0641] Natural language processing algorithms and image recognition technology are used to analyze the data collected by the server.

[0642] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[0643] 5. Emotional state analysis processing

[0644] The terminal displays an interface for the user to input an emotional state.

[0645] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0646] The device transmits this emotion information to the server.

[0647] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[0648] 6. Creating memorable content

[0649] The server generates appropriate memory content from past data based on the user's emotional state.

[0650] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[0651] 7. Content Suggestion and Display

[0652] The server transmits the generated content to the terminal.

[0653] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0654] 8. User Feedback Collection

[0655] The terminal displays a form for the user to input feedback on the proposed content.

[0656] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[0657] The terminal sends feedback information to the server.

[0658] 9. Feedback analysis and algorithmic model updating

[0659] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[0660] This allows the system to continually improve so that next time it suggests content that best suits the user's emotional state.

[0661] In this way, this system provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and increasing feelings of happiness, thereby contributing to improving the user's mental health.

[0662] The processing flow will be explained below.

[0663] Step 1:

[0664] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[0665] Step 2:

[0666] The user enters their Yahoo! ID and password and presses the login button.

[0667] Step 3:

[0668] The device sends the entered Yahoo! ID and password to the server.

[0669] Step 4:

[0670] The server authenticates the user information using Yahoo!'s authentication API.

[0671] Step 5:

[0672] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[0673] Step 6:

[0674] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[0675] Step 7:

[0676] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[0677] Step 8:

[0678] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0679] Step 9:

[0680] For example, get emails and photos from the past five years.

[0681] Step 10:

[0682] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[0683] Step 11:

[0684] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[0685] Step 12:

[0686] The terminal displays an interface for the user to input their emotional state.

[0687] Step 13:

[0688] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0689] Step 14:

[0690] The terminal transmits the input emotion information to the server.

[0691] Step 15:

[0692] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[0693] Step 16:

[0694] The server generates appropriate memory content from past data based on the user's emotional state.

[0695] Step 17:

[0696] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[0697] Step 18:

[0698] The server transmits the generated content to the terminal.

[0699] Step 19:

[0700] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0701] Step 20:

[0702] The terminal displays a form for the user to input feedback on the proposed content.

[0703] Step 21:

[0704] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[0705] Step 22:

[0706] The terminal sends feedback information to the server.

[0707] Step 23:

[0708] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[0709] Step 24:

[0710] The server uses an updated algorithm to enable more accurate content suggestions based on the user's emotional state from the next time onwards.

[0711] Example 1

[0712] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0713] In modern society, feelings of loneliness and low happiness among users have become serious problems. To address these issues, a system that can improve users' psychological state by providing content based on past memories is needed. However, existing systems do not adequately provide appropriate content according to emotional states, and they lack a means to effectively utilize user feedback to continuously improve the system.

[0714] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0715] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, and an algorithm model update means. This makes it possible to generate and display appropriate memory content from past data based on the user's emotional state. Furthermore, by combining an interface means for inputting the user's emotional information with a means for analyzing the emotional information and determining the user's current emotional state, it is possible to grasp the user's psychological state and provide more effective content.

[0716] A "user authentication means" is a mechanism that authenticates a user based on authentication information such as a user name and password that the user inputs when accessing a system.

[0717] "Data linking means" is a mechanism that allows the system to link with external services based on the user's account information and obtain the necessary data.

[0718] The "means for collecting past behavioral data" is used to collect behavioral data from the user's past emails, photos, memo pads, etc.

[0719] "Data analysis means" refers to the function of extracting the meaning and characteristics of collected data using algorithms and techniques to analyze the data.

[0720] The "emotional state analysis means" is a mechanism for analyzing the emotional information input by the user and determining the current emotional state.

[0721] The "memory content generating means" is a function for generating appropriate memory content based on the analyzed data and emotional state.

[0722] The "user feedback means" is an interface for users to input feedback on the provided content.

[0723] The "feedback analysis means" is a mechanism for analyzing feedback information from users and using it to provide content next time.

[0724] The "algorithm model update means" is a function for updating and optimizing the models of the emotion analysis algorithm and content recommendation algorithm based on feedback information.

[0725] The present invention relates to a system for providing content based on past memories in order to reduce the sense of loneliness felt by a user and improve the sense of happiness. An embodiment of this system will be specifically described below.

[0726] This system includes three main elements: a server, a terminal, and a user. First, the user is authenticated using a user authentication method. The terminal displays a login screen for the user, and the user enters authentication information (ID and password). The entered information is sent from the terminal to the server, and the server verifies the information using an authentication API. If authentication is successful, the server starts a session for the user and obtains the necessary user information.

[0727] Next, using the data linking method, the server links with the external service based on the user's account information. At this time, the server prepares to collect past behavioral data such as the user's emails and photo albums. Data linking is performed through the external service's API, and the acquired data is temporarily stored on the server.

[0728] The collected data is analyzed using data analysis tools. Specifically, the server uses natural language processing algorithms (such as NLTK or spaCy) and image recognition technologies (such as OpenCV or TensorFlow) to extract specific events and emotion tags (such as "happiness" or "sadness") from the data. The analysis results are stored in a database.

[0729] Next, the emotional state analysis means is used to analyze the user's current emotional state. The terminal displays an interface for the user to input their emotional state, and the user inputs emotional information. The input emotional information is sent from the terminal to a server, and the server analyzes the emotional information using an emotion analysis API to determine the user's current emotional state.

[0730] The memory content generation means operates based on the user's emotional state. The server generates memory content appropriate to the user's emotional state based on the acquired data and analysis results. For example, if the user inputs "I'm feeling a little down," the server selects past travel photos or conversation records with friends that are related to "happiness." The generated content is sent from the server to the terminal, which displays it to the user.

[0731] After the content is displayed, the user uses the user feedback means to input feedback on the provided content. The user inputs an evaluation such as "it was very good" or "it was average" into the feedback form displayed on the terminal and transmits the input to the server via the form.

[0732] The feedback analysis means receives this feedback information and analyzes it on the server. The analysis results are reflected in the system's learning model through the algorithm model update means. This enables more appropriate suggestions to be made the next time content is provided.

[0733] As a specific example of how it works, if a user enters "I'm feeling a little down" as a prompt, the server will select photos of past family trips and records of fun conversations with friends related to "happiness" and send them to the device. The device will then display these along with a message such as "Look at these family trip photos!" If the user enters feedback such as "It was great," the server will update the system based on that feedback and use it to suggest content next time.

[0734] Examples of prompts: "I'm feeling a little down," "I'm having a good time today."

[0735] In this way, the system of the present invention provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and improving feelings of happiness, thereby contributing to improving the user's mental health.

[0736] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0737] Step 1:

[0738] User authentication process

[0739] When a user accesses the system, the terminal displays a form for entering a Yahoo! ID and password. After the user provides the input information and presses the login button, the terminal sends the input information to the server. The server verifies the input information using Yahoo!'s authentication API, and if authentication is successful, it starts a user session and obtains the user information.

[0740] Input: Yahoo! ID and password

[0741] Output: Authentication success flag, user information

[0742] Specific operation: The device displays a login screen and the user enters their authentication information. The information is sent from the device to the server, and the server performs authentication via the Yahoo! API.

[0743] Step 2:

[0744] Data linkage processing

[0745] After successful authentication, the server links the user's account to an external service, which prepares the server to collect user behavior data (emails, photos, etc.).

[0746] Input: Authentication success flag, user information

[0747] Output: Data collection ready flag

[0748] Specific operation: The server uses the user information to connect to the API of external services (email, photo album) and prepares to collect data.

[0749] Step 3:

[0750] Collecting past behavioral data

[0751] The server collects the user's past behavior data from emails, photo albums, etc. The server acquires and stores this data for a certain period of time (e.g., the past five years).

[0752] Input: Data collection ready flag

[0753] Output: Collected behavioral data

[0754] Specific operation: The server calls the API and collects email and photo data for the specified period. The acquired data is temporarily stored in storage.

[0755] Step 4:

[0756] Data analysis processing

[0757] The server analyzes the collected data using natural language processing and image recognition technology, extracting emotion tags (such as "happiness" or "sadness") from text data and performing facial and scene recognition from image data.

[0758] Input: Collected behavioral data

[0759] Output: Analysis results (emotion tags, specific events)

[0760] How it works: The server uses Python libraries such as NLTK, spaCy, OpenCV, and TensorFlow to analyze text and image data and store the results in a database.

[0761] Step 5:

[0762] Emotional state analysis processing

[0763] The device provides the user with an interface for inputting their emotional state. After the user inputs their emotional information and presses the send button, the device sends the data to the server. The server then uses an emotion analysis API to analyze the data and determine the user's current emotional state.

[0764] Input: User's emotional information

[0765] Output: Current emotional state

[0766] Specific operation: The device displays the interface, receives input from the user, and sends it to the server, which then analyzes it using the sentiment analysis API.

[0767] Step 6:

[0768] Creating memorable content

[0769] The server generates optimal memory content based on the analysis results, based on the user's emotional state. For example, if the user enters "I'm feeling a little down," the server will select photos of past family trips related to "happiness."

[0770] Input: Current emotional state, analysis results

[0771] Output: Memories content

[0772] What it does: The server queries the database for the appropriate content and sends the generated content to the device in the appropriate format.

[0773] Step 7:

[0774] Content Suggestion and Display

[0775] The server sends the generated memory content to the device, which then displays it to the user, for example, with a message such as "Check out these family vacation photos!"

[0776] Input: Memories content

[0777] Output: The displayed content

[0778] Specific operation: The server sends the generated content to the device via an HTTP response, and the device displays it.

[0779] Step 8:

[0780] User feedback collection

[0781] The device provides the user with a form for entering feedback on the content. The user enters an evaluation such as "very good" or "average," and when the user presses the submit button, the device sends the data to the server.

[0782] Input: User feedback

[0783] Output: Collected feedback

[0784] Specific operation: The device displays a feedback input form, receives input from the user, and sends it to the server.

[0785] Step 9:

[0786] Feedback analysis and algorithmic model updating

[0787] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby improving the quality of the next content recommendation.

[0788] Input: User feedback

[0789] Output: Updated algorithmic model

[0790] What it does: The server stores the feedback information in a database and updates the analysis and generative AI model by retraining it.

[0791] (Application example 1)

[0792] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0793] In modern society, there is a demand for technologies that can reduce the feelings of loneliness and stress felt by users and improve their happiness. In particular, in electronic payment services, there is a growing need for systems that not only process payments but also recommend items that will bring happiness to users based on their past purchase history and emotional state. However, conventional technologies have not been able to meet such complex requirements.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0795] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a means for adding an emotional tag based on past purchase data, and a means for recommending items based on the emotional state, thereby making it possible to provide content and recommended items that are optimal for the user's current emotional state.

[0796] "User authentication means" refers to a means for identifying a user based on authentication information when the user accesses the system.

[0797] "Data linkage means" refers to a means of linking user information with external systems and collecting and managing necessary data.

[0798] The "means for collecting past behavioral data" is a means for collecting data on the past behavior of a user.

[0799] "Data analysis means" is a means for analyzing collected data and extracting useful information and patterns.

[0800] The "emotional state analysis means" is a means for analyzing the current emotional state of the user.

[0801] The "memory content generating means" is a means for generating memory content that matches the emotional state of the user based on past data.

[0802] "User feedback means" refers to a means for collecting feedback from users.

[0803] The "feedback analysis means" is a means for analyzing collected feedback information and using it to improve the system.

[0804] The "algorithm model updating means" is a means for updating and optimizing the algorithm model based on the collected feedback information.

[0805] The "means for adding emotion tags based on past purchase data" is a means for analyzing past purchase data and adding tags related to specific emotional states.

[0806] The "means for recommending items based on emotional state" is a means for recommending appropriate items based on the emotional state of a user.

[0807] This invention is a system that reduces users' feelings of loneliness and improves their happiness, and is particularly aimed at application to electronic payment services. The system is characterized by recommending appropriate content and items based on the user's emotional state.

[0808] Overall system operation

[0809] User authentication method

[0810] When a user accesses the system, the server identifies the user based on the authentication information (ID and password).To do this, the server uses an authentication API to authenticate the entered information, starts a session, and obtains user information.

[0811] Data linking method

[0812] After successful authentication, the server links the user's account to the system, and prepares to collect data such as past emails and photo albums.

[0813] Past behavioral data collection method

[0814] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[0815] Data Analysis Methods

[0816] The server analyzes the collected data, using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[0817] Emotional state analysis means

[0818] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[0819] Means of creating memorable content

[0820] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips and records of fun conversations with friends.

[0821] User feedback tools

[0822] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average" and sends it to the server.

[0823] Feedback Analysis Methods

[0824] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby enabling it to provide more appropriate content in the next recommendation.

[0825] Algorithmic model update method

[0826] Based on the feedback information, the server adjusts the parameters of the AI ​​model used and the proposed algorithm to improve the accuracy of the system.

[0827] Emotion tagging method

[0828] The server analyzes past purchase data and adds tags related to specific emotional states, such as "happy" or "excited," to previously purchased items.

[0829] Item recommendation method

[0830] The server recommends appropriate items based on the user's current emotional state. If the user is currently in a "happy" emotional state, the server creates a list of items that the user has purchased in the past that made them feel happy, sends this list to the device, and displays it.

[0831] Hardware and software used

[0832] Hardware: Smartphone

[0833] software

[0834] Python: Implementation of the entire application

[0835] requests: A library for making API requests

[0836] JSON: Data Format

[0837] Natural Language Processing Algorithms: Analyzing Text Data

[0838] Image recognition technology: Analysis of image data

[0839] Sentiment Analysis API: Emotional information analysis

[0840] Specific examples

[0841] For example, if a user is currently in a "happy" emotional state, the system will recommend items related to "happiness" based on their past purchase history, such as candy or toys that have been purchased in the past and that have brought them happiness.

[0842] Prompt Sentence Examples

[0843] The user is currently in a "happy" emotional state. Recommend items that brought the user happiness based on their past purchase history.

[0844] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0845] Step 1:

[0846] User authentication

[0847] When a user accesses the system, the server displays a form for entering an ID and password. The user enters the ID and password and sends them to the server. The server uses an authentication API to check the input information, and if authentication is successful, starts a session and obtains the user information.

[0848] Input: User ID and password

[0849] Output: Session ID and user information

[0850] Step 2:

[0851] Data Linkage

[0852] After successful authentication, the server performs the procedure to link the user's account with the system, and prepares to collect the user's past data (emails, photo albums, purchase history, etc.).

[0853] Input: Authenticated user information

[0854] Output: Notification that data integration is ready

[0855] Step 3:

[0856] Collecting past behavioral data

[0857] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[0858] Input: Notification that data integration is ready

[0859] Output: Past behavior data

[0860] Step 4:

[0861] Data analysis

[0862] The server analyzes the collected data using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[0863] Input: Past behavior data

[0864] Output: Parsed data and sentiment tags

[0865] Step 5:

[0866] Emotional State Analysis

[0867] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[0868] Input: User's emotional information

[0869] Output: Parsed emotional state

[0870] Step 6:

[0871] Creation of memory content and items

[0872] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos from past family trips or records of fun conversations with friends. It also analyzes past purchase data and adds related emotional tags.

[0873] Input: Analyzed emotional state, past behavioral data

[0874] Output: Generated memory content and recommended items

[0875] Step 7:

[0876] Content Suggestion and Display

[0877] The server sends the generated content to the device and displays it to the user along with a message such as "Look at these family vacation photos!" Recommended items are also displayed at the same time.

[0878] Input: Generated memory content and recommended items

[0879] Output: Content and items displayed to the user

[0880] Step 8:

[0881] Feedback collection

[0882] The device displays a form for the user to enter feedback on the proposed content or recommended items. The user enters feedback such as "It was really good" or "It was average" and sends it to the server.

[0883] Input: User feedback

[0884] Output: Collected feedback information

[0885] Step 9:

[0886] Feedback analysis and algorithm updates

[0887] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and recommendation algorithm, thereby enabling it to provide more appropriate content and items in the next proposal.

[0888] Input: Collected feedback information

[0889] Output: Updated and optimized algorithmic model

[0890] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0891] The present invention is a system that aims to provide optimal memorable content based on the user's emotional state, reducing feelings of loneliness and improving happiness. In particular, by incorporating an emotion engine that recognizes the user's emotions in real time, more accurate emotion analysis and content suggestions are achieved. The processing of the entire system is explained in detail below.

[0892] Overall system operation

[0893] 1. User authentication process

[0894] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[0895] The user enters their Yahoo! ID and password and presses the login button.

[0896] The device sends the entered Yahoo! ID and password to the server.

[0897] The server authenticates the user information using Yahoo!'s authentication API, and if authentication is successful, starts a user session and obtains the user information.

[0898] 2. Data linkage processing

[0899] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[0900] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[0901] 3. Collecting past behavioral data

[0902] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0903] For example, get emails and photos from the past five years.

[0904] 4. Data Analysis Processing

[0905] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[0906] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[0907] 5. Emotional state analysis processing

[0908] The terminal displays an interface for the user to input their emotional state.

[0909] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0910] The device transmits this emotion information to the server.

[0911] The server uses an emotion engine to analyze the input emotion information and determine the user's current emotional state.

[0912] 6. Emotion Recognition by Emotion Engine

[0913] The server uses an emotion engine to recognize emotions by analyzing the user's text, voice, and facial expression data.

[0914] The server updates the user's real-time emotional state based on the recognized emotional information.

[0915] 7. Creating memorable content

[0916] The server generates appropriate memory content from past data based on the user's emotional state.

[0917] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[0918] 8. Content Suggestion and Display

[0919] The server transmits the generated content to the terminal.

[0920] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0921] 9. User Feedback Collection

[0922] The terminal displays a form for the user to input feedback on the proposed content.

[0923] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[0924] The terminal sends feedback information to the server.

[0925] 10. Feedback analysis and algorithm model update

[0926] The server analyzes the feedback information it receives and uses this information to update and optimize the models of the sentiment analysis algorithm and content suggestion algorithm.

[0927] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[0928] In this way, the system constantly monitors the user's real-time emotional state and provides appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness, which can greatly contribute to improving the user's mental health.

[0929] The processing flow will be explained below.

[0930] Step 1:

[0931] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[0932] Step 2:

[0933] The user enters their Yahoo! ID and password and presses the login button.

[0934] Step 3:

[0935] The device sends the entered Yahoo! ID and password to the server.

[0936] Step 4:

[0937] The server authenticates the user information using Yahoo!'s authentication API.

[0938] Step 5:

[0939] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[0940] Step 6:

[0941] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[0942] Step 7:

[0943] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[0944] Step 8:

[0945] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[0946] Step 9:

[0947] For example, obtain email and photo data from the past five years.

[0948] Step 10:

[0949] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[0950] Step 11:

[0951] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[0952] Step 12:

[0953] The terminal displays an interface for the user to input an emotional state.

[0954] Step 13:

[0955] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[0956] Step 14:

[0957] The terminal transmits the input emotion information to the server.

[0958] Step 15:

[0959] The server uses an emotion engine to analyze the input emotion information to determine the user's current emotional state.

[0960] Step 16:

[0961] The server further displays an interface on the terminal for collecting the user's text, voice, and facial expression data.

[0962] Step 17:

[0963] The user inputs their voice and facial expressions through a camera and microphone, and the device sends this data to the server.

[0964] Step 18:

[0965] The server uses an emotion engine to analyze the collected text, voice, and facial expression data to recognize and update the user's real-time emotional state.

[0966] Step 19:

[0967] The server generates optimal memory content from past data based on the user's emotional state.

[0968] Step 20:

[0969] For example, if the server detects a state of "feeling a little down," it will select photos of past family trips or records of fun conversations with friends that are related to "happiness."

[0970] Step 21:

[0971] The server transmits the generated content to the terminal.

[0972] Step 22:

[0973] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[0974] Step 23:

[0975] The terminal displays a form for the user to input feedback on the proposed content.

[0976] Step 24:

[0977] The user enters feedback such as "It was great" or "It was average" and presses the submit button.

[0978] Step 25:

[0979] The terminal sends feedback information to the server.

[0980] Step 26:

[0981] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[0982] Step 27:

[0983] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[0984] Example 2

[0985] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0986] In modern society, many people suffer from loneliness and stress. In particular, it is difficult to receive appropriate support tailored to each individual's emotional state. Therefore, there is a need for a system that provides support tailored to each user's emotional state, reducing loneliness and improving happiness.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0988] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, and a data analysis means using natural language processing and image recognition technology, which allows the server to efficiently collect and analyze the user's past behavioral data and provide appropriate memory content according to the user's individual emotional state.

[0989] "User authentication means" refers to a means for identifying and authenticating an individual when a user accesses a system.

[0990] The "data linking means" is a means for linking the system with an external data source and collecting data on users' past behavior.

[0991] The "means for collecting past behavioral data" is a means for systematically collecting a user's past activity history and digital content.

[0992] "Data analysis means using natural language processing and image recognition technology" means a means that uses natural language processing (NLP) and image recognition technology to analyze collected data and extract specific event and emotion tags.

[0993] The "interface means for inputting user's emotional information" is a means for providing an interface that allows the user to input his / her own emotional state.

[0994] The "means for analyzing an emotional state using an emotion engine" is a means for analyzing input emotion information, text, voice, and facial expression data to determine the user's current emotional state.

[0995] The "means for generating appropriate memorable content based on the analyzed emotions" is a means for selecting and generating content suitable for a user from past data based on the user's emotional state.

[0996] The "means for presenting the generated content to the user" refers to a means for presenting the generated memory content to the user through visual and auditory means.

[0997] The "user feedback means" is a means for a user to input feedback on the presented content and transmit it to the system.

[0998] The "means for analyzing feedback information and updating algorithm models" refers to a means for analyzing feedback information obtained from users and updating models to improve the accuracy of sentiment analysis algorithms and content suggestion algorithms.

[0999] The present invention is a system that aims to provide optimal memory content based on a user's emotional state, reduce feelings of loneliness, and improve feelings of happiness. The system of the present invention uses the following hardware and software.

[1000] Hardware and software used

[1001] Hardware

[1002] Device: A personal computer, smartphone, or tablet through which a user accesses the device.

[1003] Server: A server computer that processes the entire system.

[1004] Camera and microphone: Devices for collecting facial and voice data from the user.

[1005] software

[1006] Natural Language Processing Algorithms (NLP): Algorithms for analyzing text data, specifically using Google Cloud Natural Language API or similar technologies.

[1007] Image recognition technology: Technology for analyzing image data. Specifically, Google Cloud Vision API is used.

[1008] Emotion engine: An engine for analyzing the user's emotional state. A specific example is the Emotion API.

[1009] Embodiments of each step

[1010] 1. User authentication process

[1011] When a user accesses the system, they enter their Yahoo! ID and password into the form displayed on the terminal. The terminal then sends this information to the server, which then authenticates the user information using Yahoo!'s authentication API.

[1012] 2. Data linkage processing

[1013] The server will connect the system to the already authenticated user account and prepare to collect the necessary data from Yahoo! Mail and Yahoo! Photo Album.

[1014] 3. Collecting past behavioral data

[1015] The server collects past behavioral data from each Yahoo! service (email, photo album, etc.). For example, it collects emails and photos from the past five years.

[1016] 4. Data Analysis Processing

[1017] The server uses the data collected and applies natural language processing algorithms and image recognition techniques to extract specific events and emotion tags from the data.

[1018] 5. Emotional state analysis processing

[1019] The terminal displays an interface for the user to input their current emotional state. The user inputs emotional information, and this information is sent from the terminal to the server. The server analyzes the emotional state using an emotional engine.

[1020] 6. Emotion Recognition by Emotion Engine

[1021] The server uses an emotion engine to analyze text data, voice data, and facial expression data to recognize the user's emotions in real time.

[1022] 7. Creating memorable content

[1023] The server selects and generates appropriate memory content from past data based on the user's emotional state. For example, if the user is feeling a little down, it will select photos of past family trips and records of fun conversations related to happiness.

[1024] 8. Content Suggestion and Display

[1025] The server sends the generated memory content to the device, which then displays it to the user. For example, it may be displayed with a message such as "Check out these family vacation photos!"

[1026] 9. User Feedback Collection

[1027] The terminal displays a form for the user to input feedback about the content. The user inputs the feedback, which is then sent from the terminal to the server.

[1028] 10. Feedback analysis and algorithm model update

[1029] The server analyzes the received feedback information and updates the algorithm model, enabling more accurate content suggestions based on the user's emotional state from the next time onwards.

[1030] Specific examples

[1031] For example, if the user feels "a little down today," the prompt sentence might be:

[1032] Prompt Sentence Examples

[1033] "Simulate a scenario where a user logs in, inputs their emotional state, and then the system suggests photos from past family trips."

[1034] In this way, the system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[1035] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1036] The flow of this system's program processing

[1037] Step 1: User authentication process

[1038] Input: The user enters their Yahoo! ID and password.

[1039] Processing: The terminal encrypts these input data and sends them to the server.

[1040] Data processing and calculation: The server uses Yahoo!'s authentication API to verify and authenticate user information.

[1041] Output: If authentication is successful, start a user session and get user information.

[1042] Specific behavior:

[1043] A user enters the ID "user@example.com" and password "123" into a form.

[1044] The device encrypts the data and sends it to the server.

[1045] The server calls the Yahoo! authentication API, and if authentication is successful, the message "Welcome, user" is displayed.

[1046] Step 2: Data integration process

[1047] Input: The server maintains the user session after successful authentication.

[1048] Processing: The server prepares to exchange data with each Yahoo! service.

[1049] Data processing and calculation: The server uses the API to ask the user for permission to access past emails and photo album data.

[1050] Output: If the user approves the permission, the data is ready to be collected.

[1051] Specific behavior:

[1052] The server requests access to your Yahoo! account.

[1053] Once the user grants access, the server prepares to collect data.

[1054] Step 3: Collect historical behavioral data

[1055] Input: The server starts collecting data based on the access rights granted by the user.

[1056] Processing: The server collects data from Yahoo! Mail, Photo Album, Notepad, etc.

[1057] Data processing and calculation: The server organizes and stores the collected data in chronological order.

[1058] Output: A complete dataset of emails and photos from the past five years.

[1059] Specific behavior:

[1060] The server collects emails and photos from the period "2018-01-01" to "2023-01-01".

[1061] Collected data is organized and saved in chronological order.

[1062] Step 4: Data analysis and processing

[1063] Input: The dataset collected by the server.

[1064] Processing: Analyzing the data using natural language processing algorithms (NLP) and image recognition techniques.

[1065] Data processing and calculation: The server extracts specific events and emotion tags from the text data and extracts facial expressions from the image data.

[1066] Output: The analysis results in a dataset labeled with event and emotion tags.

[1067] Specific behavior:

[1068] The server uses the Google Cloud Natural Language API to analyze the email content and associate the keyword "family trip" with the tag "happiness."

[1069] It uses Google Cloud Vision API to perform facial recognition on photos and classify photos of "smiling faces" as "happy."

[1070] Step 5: Emotional state analysis processing

[1071] Input: The user inputs their current emotional state.

[1072] Processing: The terminal sends this input data to the server.

[1073] Data processing and calculation: The server uses an emotion engine to analyze the input emotion information and determine the emotional state.

[1074] Output: The user's current emotional state is recognized.

[1075] Specific behavior:

[1076] The user types into the terminal interface, "I'm feeling a little down today."

[1077] This information is sent to the server, which uses its emotion engine to determine that "slightly depressed" means "mild depression."

[1078] Step 6: Emotion Recognition with the Emotion Engine

[1079] Input: User text, voice, and facial expression data.

[1080] Processing: The server analyzes these data using the emotion engine.

[1081] Data processing and calculation: The emotion engine analyzes the data and recognizes the user's real-time emotional state.

[1082] Output: The emotional state is updated as a result of the analysis.

[1083] Specific behavior:

[1084] The user uses the camera of the terminal to capture his / her facial expression and also records voice information.

[1085] The server analyzes this data and recognizes a "smile" as "happiness."

[1086] Step 7: Create memorable content

[1087] Input: User's emotional state and analyzed past behavioral data.

[1088] Processing: The server generates appropriate memory content based on these data.

[1089] Data processing and calculation: Select the most suitable content from past data based on emotional state.

[1090] Output: The generated memories content.

[1091] Specific behavior:

[1092] The server selects photos of past family trips based on the user's "mild depression."

[1093] Pick out a few fun photos from the "Summer Travel 2019" folder.

[1094] Step 8: Content suggestion and display

[1095] Input: Generated memories content.

[1096] Processing: The server sends the content to the terminal.

[1097] Data processing and calculation: Prepares the layout for the device to display the content.

[1098] Output: The memories content displayed to the user.

[1099] Specific behavior:

[1100] The device will display the suggested family vacation photos on a large screen, along with the message, "Check out these family vacation photos!"

[1101] Step 9: Gather user feedback

[1102] Input: User feedback.

[1103] Processing: The terminal displays a form for inputting feedback and sends the information entered by the user to the server.

[1104] Data processing and calculation: The server stores and organizes the feedback information.

[1105] Output: A dataset of feedback information.

[1106] Specific behavior:

[1107] The user enters "It was great" in the feedback form and presses the submit button.

[1108] This information is sent to a server and stored in a database.

[1109] Step 10: Feedback analysis and algorithmic model update

[1110] Input: Feedback information from the user.

[1111] Processing: The server analyzes the feedback information and updates the algorithmic model.

[1112] Data processing and calculation: Optimize the accuracy of sentiment analysis algorithms and content suggestion algorithms based on feedback information.

[1113] Output: The updated algorithm model.

[1114] Specific behavior:

[1115] The server analyzes the feedback such as "It was really good" and changes the parameters of the sentiment analysis algorithm.

[1116] Optimize the algorithm model for the next proposal.

[1117] Through the above process, this system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[1118] (Application example 2)

[1119] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1120] In modern society, people often feel lonely and stressed, which can lead to poor mental health. To address this, a method is needed to provide appropriate memorable content based on the user's emotional state. However, existing systems are inadequate in accurately recognizing a user's emotional state in real time and providing optimal content based on that state. As a result, it is difficult to alleviate the loneliness and stress felt by users. Furthermore, effective feedback for users and optimization of content suggestion algorithms are not adequately implemented.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a real-time emotion recognition means using a head-mounted display, and a means for displaying content based on the real-time emotional state. This makes it possible to accurately recognize the user's emotional state in real time and provide optimal memory content based on that state. By simply wearing the HMD, the user can experience memory content based on their own emotions in real time, effectively reducing feelings of loneliness and stress.

[1122] "User authentication means" is a function that verifies the legitimacy of a user by having the user access the system and enter authentication information.

[1123] "Data linking means" is a function that links an authenticated user's account with the system and shares necessary data.

[1124] "Means for collecting past behavioral data" is a function that collects users' past behavioral data (e.g., emails, photos, etc.).

[1125] "Data analysis means" is a function that analyzes collected data and extracts useful information.

[1126] The "emotional state analysis means" is a function that analyzes the emotional information provided by the user and determines the emotional state of the user.

[1127] The "memory content generating means" is a function that generates appropriate memory content based on the emotional state of the user.

[1128] The "user feedback means" is an interface for users to input feedback on the provided content.

[1129] The "feedback analysis means" is a function that analyzes feedback provided by users and uses it to improve the system.

[1130] The "algorithm model update means" is a function that updates the emotion analysis algorithm and the content suggestion algorithm based on the analysis results.

[1131] "Real-time emotion recognition means using a head-mounted display" is a function that uses sensors built into the head-mounted display to recognize the user's emotions in real time.

[1132] The "means for displaying content based on real-time emotional state" is a function for displaying optimal memory content on the head-mounted display in accordance with the emotional state recognized in real time.

[1133] This invention is a system that provides optimal memory content based on the user's emotional state, reducing feelings of loneliness and stress. It primarily uses a head-mounted display (HMD) to recognize the user's emotions in real time and display memory content accordingly.

[1134] System Configuration

[1135] The system includes the following main components:

[1136] User authentication method

[1137] Data linking method

[1138] Past behavioral data collection method

[1139] Data Analysis Methods

[1140] Emotional state analysis means

[1141] Means of creating memorable content

[1142] User feedback tools

[1143] Feedback Analysis Methods

[1144] Algorithmic model update method

[1145] Real-time emotion recognition using a head-mounted display

[1146] A means of displaying content based on real-time emotional state

[1147] Hardware and Software

[1148] Hardware

[1149] Head-mounted display (HMD): A device worn by the user that displays visual content, with built-in facial recognition cameras and heart rate sensors.

[1150] PC or smartphone: A device that connects to the HMD and processes data.

[1151] Server: Responsible for data collection, analysis, content generation, etc.

[1152] software

[1153] Python: A programming language.

[1154] OpenCV: A library for face recognition.

[1155] Yahoo SDK: A software development kit for working with Yahoo services.

[1156] EmotionEngine: An engine for emotion analysis.

[1157] ContentGenerator: A module that generates and suggests memory content.

[1158] Processing Details

[1159] 1. User Authentication

[1160] The server first authenticates the user based on the credentials provided by the user, which is done using the Yahoo SDK.

[1161] 2. Data Linkage

[1162] After authentication, the server links the user's account to the system and then prepares to collect past behavioral data.

[1163] 3. Collecting past behavioral data

[1164] The server collects data such as users' past emails, photo albums, notepads, etc. The collection period is, for example, the past five years.

[1165] 4. Data Analysis

[1166] The server analyzes the collected data and uses natural language processing algorithms and image recognition technology to extract specific event and emotion tags.

[1167] 5. Emotional State Analysis

[1168] The device (HMD) displays an interface for inputting the user's emotional state and transmits the collected emotional information to the server, which analyzes it and determines the user's current emotional state.

[1169] 6. Real-time emotion recognition

[1170] Using the cameras and sensors built into the HMD, we recognize the user's real-time emotional state using OpenCV and EmotionEngine.

[1171] 7. Creating memorable content

[1172] The server generates appropriate memory content based on the user's real-time emotional state. For example, if the user is feeling a little down, it will select videos of past family trips or photos of happy friends.

[1173] 8. Content Display

[1174] The content selected by the user is displayed through the HMD.

[1175] Specific examples

[1176] The user wears the HMD and the system analyzes the user's emotions in real time. If the user's emotional state is determined to be "a little depressed," the system displays videos of past family trips and happy photos with friends, helping the user to relax and feel happier.

[1177] Prompt Sentence Examples

[1178] "Generate relevant memory content by providing a user's facial image and emotional state as input. The user is in a slightly depressed state."

[1179] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1180] Step 1:

[1181] The user wears a head-mounted display (HMD) and accesses the system. The device displays an interface for the user to enter their Yahoo! ID and password. This input information is the input data for user authentication. The device sends the entered authentication information to the server, which then authenticates the user information using the Yahoo! SDK. If authentication is successful, the server starts a user session and obtains the user information. The authenticated user information is obtained as output data.

[1182] Step 2:

[1183] The server activates the data integration method that links the authenticated user's account with the system. At this point, it is ready to retrieve data from past Yahoo! services (e.g., email and photo albums). The behavioral data to be collected is integrated and becomes the input data for the next step.

[1184] Step 3:

[1185] The server uses a past behavioral data collection means to collect the user's past behavioral data (emails, photo albums, memo pads, etc.). The collection period is, for example, the past five years. The collected data becomes input data for the data analysis means. The collected behavioral data is obtained as output data.

[1186] Step 4:

[1187] The server analyzes the collected data using data analysis means. Specifically, it uses natural language processing algorithms and image recognition technology to extract specific events and emotion tags (e.g., "happiness," "sadness," etc.) from the data. The results of this analysis are input data for the emotion state analysis means. Emotion tags and event information are obtained as output data.

[1188] Step 5:

[1189] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm a little down" or "I'm having a good day," and the device sends the information to the server. This emotional information becomes input data for the emotional state analysis means. The server uses an emotion engine to analyze the input emotional information and determine the user's current emotional state. The user's current emotional state is obtained as output data.

[1190] Step 6:

[1191] Using the camera and sensors built into the HMD, the server recognizes the user's real-time emotions. Using OpenCV and EmotionEngine, the server analyzes the user's facial images and heart rate data and updates their emotional state in real time. This real-time emotional data becomes the input data for the memory content generation means. The real-time emotional state is obtained as output data.

[1192] Step 7:

[1193] The server generates optimal memory content using the memory content generation means based on the user's real-time emotional state. For example, if the user is feeling a little down, the server selects videos of past family trips and photos of friends looking happy. This memory content becomes input data for the content display means. The generated memory content is obtained as output data.

[1194] Step 8:

[1195] The device presents the selected memory content to the user through the HMD. The user experiences this content visually in real time through the HMD. Intuitive messages and guides are displayed as an interface to the user. The output data becomes the content that the user experiences visually.

[1196] Step 9:

[1197] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average," and this information becomes input data for the user feedback means. The terminal sends the feedback information to the server, and the user's feedback information is obtained as output data.

[1198] Step 10:

[1199] The server uses the feedback analysis means to analyze the received feedback information and, based on this, updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm. The algorithm model update means incorporates this feedback information to improve the accuracy of future suggestions. The updated algorithm model is obtained as output data.

[1200] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1201] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1202] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1203] [Third embodiment]

[1204] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1205] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1206] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1207] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1208] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1209] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1210] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1211] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1212] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1213] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1214] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1215] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1216] The present invention relates to a system that aims to improve a user's sense of happiness by providing content based on past memories when the user is feeling lonely. The program processing flow of this system will be specifically explained below.

[1217] Overall system operation

[1218] 1. User authentication process

[1219] When a user accesses the system, the terminal displays a form for entering the Yahoo! ID and password.

[1220] The user enters their Yahoo! ID and password and presses the login button.

[1221] The terminal sends the input information to the server.

[1222] The server authenticates the input information using Yahoo!'s authentication API. If authentication is successful, it starts a user session and retrieves user information.

[1223] 2. Data linkage processing

[1224] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[1225] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[1226] 3. Collecting past behavioral data

[1227] The server collects users' past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[1228] For example, get emails and photos from the past five years.

[1229] 4. Data Analysis Processing

[1230] Natural language processing algorithms and image recognition technology are used to analyze the data collected by the server.

[1231] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[1232] 5. Emotional state analysis processing

[1233] The terminal displays an interface for the user to input an emotional state.

[1234] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[1235] The device transmits this emotion information to the server.

[1236] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[1237] 6. Creating memorable content

[1238] The server generates appropriate memory content from past data based on the user's emotional state.

[1239] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[1240] 7. Content Suggestion and Display

[1241] The server transmits the generated content to the terminal.

[1242] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[1243] 8. User Feedback Collection

[1244] The terminal displays a form for the user to input feedback on the proposed content.

[1245] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[1246] The terminal sends feedback information to the server.

[1247] 9. Feedback analysis and algorithmic model updating

[1248] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[1249] This allows the system to continually improve so that next time it suggests content that best suits the user's emotional state.

[1250] In this way, this system provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and increasing feelings of happiness, thereby contributing to improving the user's mental health.

[1251] The processing flow will be explained below.

[1252] Step 1:

[1253] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[1254] Step 2:

[1255] The user enters their Yahoo! ID and password and presses the login button.

[1256] Step 3:

[1257] The device sends the entered Yahoo! ID and password to the server.

[1258] Step 4:

[1259] The server authenticates the user information using Yahoo!'s authentication API.

[1260] Step 5:

[1261] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[1262] Step 6:

[1263] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[1264] Step 7:

[1265] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[1266] Step 8:

[1267] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[1268] Step 9:

[1269] For example, get emails and photos from the past five years.

[1270] Step 10:

[1271] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[1272] Step 11:

[1273] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[1274] Step 12:

[1275] The terminal displays an interface for the user to input their emotional state.

[1276] Step 13:

[1277] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[1278] Step 14:

[1279] The terminal transmits the input emotion information to the server.

[1280] Step 15:

[1281] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[1282] Step 16:

[1283] The server generates appropriate memory content from past data based on the user's emotional state.

[1284] Step 17:

[1285] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[1286] Step 18:

[1287] The server transmits the generated content to the terminal.

[1288] Step 19:

[1289] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[1290] Step 20:

[1291] The terminal displays a form for the user to input feedback on the proposed content.

[1292] Step 21:

[1293] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[1294] Step 22:

[1295] The terminal sends feedback information to the server.

[1296] Step 23:

[1297] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[1298] Step 24:

[1299] The server uses an updated algorithm to enable more accurate content suggestions based on the user's emotional state from the next time onwards.

[1300] Example 1

[1301] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1302] In modern society, feelings of loneliness and low happiness among users have become serious problems. To address these issues, a system that can improve users' psychological state by providing content based on past memories is needed. However, existing systems do not adequately provide appropriate content according to emotional states, and they lack a means to effectively utilize user feedback to continuously improve the system.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1304] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, and an algorithm model update means. This makes it possible to generate and display appropriate memory content from past data based on the user's emotional state. Furthermore, by combining an interface means for inputting the user's emotional information with a means for analyzing the emotional information and determining the user's current emotional state, it is possible to grasp the user's psychological state and provide more effective content.

[1305] A "user authentication means" is a mechanism that authenticates a user based on authentication information such as a user name and password that the user inputs when accessing a system.

[1306] "Data linking means" is a mechanism that allows the system to link with external services based on the user's account information and obtain the necessary data.

[1307] The "means for collecting past behavioral data" is used to collect behavioral data from the user's past emails, photos, memo pads, etc.

[1308] "Data analysis means" refers to the function of extracting the meaning and characteristics of collected data using algorithms and techniques to analyze the data.

[1309] The "emotional state analysis means" is a mechanism for analyzing the emotional information input by the user and determining the current emotional state.

[1310] The "memory content generating means" is a function for generating appropriate memory content based on the analyzed data and emotional state.

[1311] The "user feedback means" is an interface for users to input feedback on the provided content.

[1312] The "feedback analysis means" is a mechanism for analyzing feedback information from users and using it to provide content next time.

[1313] The "algorithm model update means" is a function for updating and optimizing the models of the emotion analysis algorithm and content recommendation algorithm based on feedback information.

[1314] The present invention relates to a system for providing content based on past memories in order to reduce the sense of loneliness felt by a user and improve the sense of happiness. An embodiment of this system will be specifically described below.

[1315] This system includes three main elements: a server, a terminal, and a user. First, the user is authenticated using a user authentication method. The terminal displays a login screen for the user, and the user enters authentication information (ID and password). The entered information is sent from the terminal to the server, and the server verifies the information using an authentication API. If authentication is successful, the server starts a session for the user and obtains the necessary user information.

[1316] Next, using the data linking method, the server links with the external service based on the user's account information. At this time, the server prepares to collect past behavioral data such as the user's emails and photo albums. Data linking is performed through the external service's API, and the acquired data is temporarily stored on the server.

[1317] The collected data is analyzed using data analysis tools. Specifically, the server uses natural language processing algorithms (such as NLTK or spaCy) and image recognition technologies (such as OpenCV or TensorFlow) to extract specific events and emotion tags (such as "happiness" or "sadness") from the data. The analysis results are stored in a database.

[1318] Next, the emotional state analysis means is used to analyze the user's current emotional state. The terminal displays an interface for the user to input their emotional state, and the user inputs emotional information. The input emotional information is sent from the terminal to a server, and the server analyzes the emotional information using an emotion analysis API to determine the user's current emotional state.

[1319] The memory content generation means operates based on the user's emotional state. The server generates memory content appropriate to the user's emotional state based on the acquired data and analysis results. For example, if the user inputs "I'm feeling a little down," the server selects past travel photos or conversation records with friends that are related to "happiness." The generated content is sent from the server to the terminal, which displays it to the user.

[1320] After the content is displayed, the user uses the user feedback means to input feedback on the provided content. The user inputs an evaluation such as "it was very good" or "it was average" into the feedback form displayed on the terminal and transmits the input to the server via the form.

[1321] The feedback analysis means receives this feedback information and analyzes it on the server. The analysis results are reflected in the system's learning model through the algorithm model update means. This enables more appropriate suggestions to be made the next time content is provided.

[1322] As a specific example of how it works, if a user enters "I'm feeling a little down" as a prompt, the server will select photos of past family trips and records of fun conversations with friends related to "happiness" and send them to the device. The device will then display these along with a message such as "Look at these family trip photos!" If the user enters feedback such as "It was great," the server will update the system based on that feedback and use it to suggest content next time.

[1323] Examples of prompts: "I'm feeling a little down," "I'm having a good time today."

[1324] In this way, the system of the present invention provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and improving feelings of happiness, thereby contributing to improving the user's mental health.

[1325] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1326] Step 1:

[1327] User authentication process

[1328] When a user accesses the system, the terminal displays a form for entering a Yahoo! ID and password. After the user provides the input information and presses the login button, the terminal sends the input information to the server. The server verifies the input information using Yahoo!'s authentication API, and if authentication is successful, it starts a user session and obtains the user information.

[1329] Input: Yahoo! ID and password

[1330] Output: Authentication success flag, user information

[1331] Specific operation: The device displays a login screen and the user enters their authentication information. The information is sent from the device to the server, and the server performs authentication via the Yahoo! API.

[1332] Step 2:

[1333] Data linkage processing

[1334] After successful authentication, the server links the user's account to an external service, which prepares the server to collect user behavior data (emails, photos, etc.).

[1335] Input: Authentication success flag, user information

[1336] Output: Data collection ready flag

[1337] Specific operation: The server uses the user information to connect to the API of external services (email, photo album) and prepares to collect data.

[1338] Step 3:

[1339] Collecting past behavioral data

[1340] The server collects the user's past behavior data from emails, photo albums, etc. The server acquires and stores this data for a certain period of time (e.g., the past five years).

[1341] Input: Data collection ready flag

[1342] Output: Collected behavioral data

[1343] Specific operation: The server calls the API and collects email and photo data for the specified period. The acquired data is temporarily stored in storage.

[1344] Step 4:

[1345] Data analysis processing

[1346] The server analyzes the collected data using natural language processing and image recognition technology, extracting emotion tags (such as "happiness" or "sadness") from text data and performing facial and scene recognition from image data.

[1347] Input: Collected behavioral data

[1348] Output: Analysis results (emotion tags, specific events)

[1349] How it works: The server uses Python libraries such as NLTK, spaCy, OpenCV, and TensorFlow to analyze text and image data and store the results in a database.

[1350] Step 5:

[1351] Emotional state analysis processing

[1352] The device provides the user with an interface for inputting their emotional state. After the user inputs their emotional information and presses the send button, the device sends the data to the server. The server then uses an emotion analysis API to analyze the data and determine the user's current emotional state.

[1353] Input: User's emotional information

[1354] Output: Current emotional state

[1355] Specific operation: The device displays the interface, receives input from the user, and sends it to the server, which then analyzes it using the sentiment analysis API.

[1356] Step 6:

[1357] Creating memorable content

[1358] The server generates optimal memory content based on the analysis results, based on the user's emotional state. For example, if the user enters "I'm feeling a little down," the server will select photos of past family trips related to "happiness."

[1359] Input: Current emotional state, analysis results

[1360] Output: Memories content

[1361] What it does: The server queries the database for the appropriate content and sends the generated content to the device in the appropriate format.

[1362] Step 7:

[1363] Content Suggestion and Display

[1364] The server sends the generated memory content to the device, which then displays it to the user, for example, with a message such as "Check out these family vacation photos!"

[1365] Input: Memories content

[1366] Output: The displayed content

[1367] Specific operation: The server sends the generated content to the device via an HTTP response, and the device displays it.

[1368] Step 8:

[1369] User feedback collection

[1370] The device provides the user with a form for entering feedback on the content. The user enters an evaluation such as "very good" or "average," and when the user presses the submit button, the device sends the data to the server.

[1371] Input: User feedback

[1372] Output: Collected feedback

[1373] Specific operation: The device displays a feedback input form, receives input from the user, and sends it to the server.

[1374] Step 9:

[1375] Feedback analysis and algorithmic model updating

[1376] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby improving the quality of the next content recommendation.

[1377] Input: User feedback

[1378] Output: Updated algorithmic model

[1379] What it does: The server stores the feedback information in a database and updates the analysis and generative AI model by retraining it.

[1380] (Application example 1)

[1381] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1382] In modern society, there is a demand for technologies that can reduce the feelings of loneliness and stress felt by users and improve their happiness. In particular, in electronic payment services, there is a growing need for systems that not only process payments but also recommend items that will bring happiness to users based on their past purchase history and emotional state. However, conventional technologies have not been able to meet such complex requirements.

[1383] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1384] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a means for adding an emotional tag based on past purchase data, and a means for recommending items based on the emotional state, thereby making it possible to provide content and recommended items that are optimal for the user's current emotional state.

[1385] "User authentication means" refers to a means for identifying a user based on authentication information when the user accesses the system.

[1386] "Data linkage means" refers to a means of linking user information with external systems and collecting and managing necessary data.

[1387] The "means for collecting past behavioral data" is a means for collecting data on the past behavior of a user.

[1388] "Data analysis means" is a means for analyzing collected data and extracting useful information and patterns.

[1389] The "emotional state analysis means" is a means for analyzing the current emotional state of the user.

[1390] The "memory content generating means" is a means for generating memory content that matches the emotional state of the user based on past data.

[1391] "User feedback means" refers to a means for collecting feedback from users.

[1392] The "feedback analysis means" is a means for analyzing collected feedback information and using it to improve the system.

[1393] The "algorithm model updating means" is a means for updating and optimizing the algorithm model based on the collected feedback information.

[1394] The "means for adding emotion tags based on past purchase data" is a means for analyzing past purchase data and adding tags related to specific emotional states.

[1395] The "means for recommending items based on emotional state" is a means for recommending appropriate items based on the emotional state of a user.

[1396] This invention is a system that reduces users' feelings of loneliness and improves their happiness, and is particularly aimed at application to electronic payment services. The system is characterized by recommending appropriate content and items based on the user's emotional state.

[1397] Overall system operation

[1398] User authentication method

[1399] When a user accesses the system, the server identifies the user based on the authentication information (ID and password).To do this, the server uses an authentication API to authenticate the entered information, starts a session, and obtains user information.

[1400] Data linking method

[1401] After successful authentication, the server links the user's account to the system, and prepares to collect data such as past emails and photo albums.

[1402] Past behavioral data collection method

[1403] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[1404] Data Analysis Methods

[1405] The server analyzes the collected data, using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[1406] Emotional state analysis means

[1407] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[1408] Means of creating memorable content

[1409] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips and records of fun conversations with friends.

[1410] User feedback tools

[1411] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average" and sends it to the server.

[1412] Feedback Analysis Methods

[1413] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby enabling it to provide more appropriate content in the next recommendation.

[1414] Algorithmic model update method

[1415] Based on the feedback information, the server adjusts the parameters of the AI ​​model used and the proposed algorithm to improve the accuracy of the system.

[1416] Emotion tagging method

[1417] The server analyzes past purchase data and adds tags related to specific emotional states, such as "happy" or "excited," to previously purchased items.

[1418] Item recommendation method

[1419] The server recommends appropriate items based on the user's current emotional state. If the user is currently in a "happy" emotional state, the server creates a list of items that the user has purchased in the past that made them feel happy, sends this list to the device, and displays it.

[1420] Hardware and software used

[1421] Hardware: Smartphone

[1422] software

[1423] Python: Implementation of the entire application

[1424] requests: A library for making API requests

[1425] JSON: Data Format

[1426] Natural Language Processing Algorithms: Analyzing Text Data

[1427] Image recognition technology: Analysis of image data

[1428] Sentiment Analysis API: Emotional information analysis

[1429] Specific examples

[1430] For example, if a user is currently in a "happy" emotional state, the system will recommend items related to "happiness" based on their past purchase history, such as candy or toys that have been purchased in the past and that have brought them happiness.

[1431] Prompt Sentence Examples

[1432] The user is currently in a "happy" emotional state. Recommend items that brought the user happiness based on their past purchase history.

[1433] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1434] Step 1:

[1435] User authentication

[1436] When a user accesses the system, the server displays a form for entering an ID and password. The user enters the ID and password and sends them to the server. The server uses an authentication API to check the input information, and if authentication is successful, starts a session and obtains the user information.

[1437] Input: User ID and password

[1438] Output: Session ID and user information

[1439] Step 2:

[1440] Data Linkage

[1441] After successful authentication, the server performs the procedure to link the user's account with the system, and prepares to collect the user's past data (emails, photo albums, purchase history, etc.).

[1442] Input: Authenticated user information

[1443] Output: Notification that data integration is ready

[1444] Step 3:

[1445] Collecting past behavioral data

[1446] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[1447] Input: Notification that data integration is ready

[1448] Output: Past behavior data

[1449] Step 4:

[1450] Data analysis

[1451] The server analyzes the collected data using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[1452] Input: Past behavior data

[1453] Output: Parsed data and sentiment tags

[1454] Step 5:

[1455] Emotional State Analysis

[1456] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[1457] Input: User's emotional information

[1458] Output: Parsed emotional state

[1459] Step 6:

[1460] Creation of memory content and items

[1461] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos from past family trips or records of fun conversations with friends. It also analyzes past purchase data and adds related emotional tags.

[1462] Input: Analyzed emotional state, past behavioral data

[1463] Output: Generated memory content and recommended items

[1464] Step 7:

[1465] Content Suggestion and Display

[1466] The server sends the generated content to the device and displays it to the user along with a message such as "Look at these family vacation photos!" Recommended items are also displayed at the same time.

[1467] Input: Generated memory content and recommended items

[1468] Output: Content and items displayed to the user

[1469] Step 8:

[1470] Feedback collection

[1471] The device displays a form for the user to enter feedback on the proposed content or recommended items. The user enters feedback such as "It was really good" or "It was average" and sends it to the server.

[1472] Input: User feedback

[1473] Output: Collected feedback information

[1474] Step 9:

[1475] Feedback analysis and algorithm updates

[1476] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and recommendation algorithm, thereby enabling it to provide more appropriate content and items in the next proposal.

[1477] Input: Collected feedback information

[1478] Output: Updated and optimized algorithmic model

[1479] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1480] The present invention is a system that aims to provide optimal memorable content based on the user's emotional state, reducing feelings of loneliness and improving happiness. In particular, by incorporating an emotion engine that recognizes the user's emotions in real time, more accurate emotion analysis and content suggestions are achieved. The processing of the entire system is explained in detail below.

[1481] Overall system operation

[1482] 1. User authentication process

[1483] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[1484] The user enters their Yahoo! ID and password and presses the login button.

[1485] The device sends the entered Yahoo! ID and password to the server.

[1486] The server authenticates the user information using Yahoo!'s authentication API, and if authentication is successful, starts a user session and obtains the user information.

[1487] 2. Data linkage processing

[1488] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[1489] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[1490] 3. Collecting past behavioral data

[1491] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[1492] For example, get emails and photos from the past five years.

[1493] 4. Data Analysis Processing

[1494] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[1495] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[1496] 5. Emotional state analysis processing

[1497] The terminal displays an interface for the user to input their emotional state.

[1498] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[1499] The device transmits this emotion information to the server.

[1500] The server uses an emotion engine to analyze the input emotion information and determine the user's current emotional state.

[1501] 6. Emotion Recognition by Emotion Engine

[1502] The server uses an emotion engine to recognize emotions by analyzing the user's text, voice, and facial expression data.

[1503] The server updates the user's real-time emotional state based on the recognized emotional information.

[1504] 7. Creating memorable content

[1505] The server generates appropriate memory content from past data based on the user's emotional state.

[1506] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[1507] 8. Content Suggestion and Display

[1508] The server transmits the generated content to the terminal.

[1509] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[1510] 9. User Feedback Collection

[1511] The terminal displays a form for the user to input feedback on the proposed content.

[1512] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[1513] The terminal sends feedback information to the server.

[1514] 10. Feedback analysis and algorithm model update

[1515] The server analyzes the feedback information it receives and uses this information to update and optimize the models of the sentiment analysis algorithm and content suggestion algorithm.

[1516] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[1517] In this way, the system constantly monitors the user's real-time emotional state and provides appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness, which can greatly contribute to improving the user's mental health.

[1518] The processing flow will be explained below.

[1519] Step 1:

[1520] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[1521] Step 2:

[1522] The user enters their Yahoo! ID and password and presses the login button.

[1523] Step 3:

[1524] The device sends the entered Yahoo! ID and password to the server.

[1525] Step 4:

[1526] The server authenticates the user information using Yahoo!'s authentication API.

[1527] Step 5:

[1528] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[1529] Step 6:

[1530] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[1531] Step 7:

[1532] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[1533] Step 8:

[1534] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[1535] Step 9:

[1536] For example, obtain email and photo data from the past five years.

[1537] Step 10:

[1538] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[1539] Step 11:

[1540] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[1541] Step 12:

[1542] The terminal displays an interface for the user to input an emotional state.

[1543] Step 13:

[1544] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[1545] Step 14:

[1546] The terminal transmits the input emotion information to the server.

[1547] Step 15:

[1548] The server uses an emotion engine to analyze the input emotion information to determine the user's current emotional state.

[1549] Step 16:

[1550] The server further displays an interface on the terminal for collecting the user's text, voice, and facial expression data.

[1551] Step 17:

[1552] The user inputs their voice and facial expressions through a camera and microphone, and the device sends this data to the server.

[1553] Step 18:

[1554] The server uses an emotion engine to analyze the collected text, voice, and facial expression data to recognize and update the user's real-time emotional state.

[1555] Step 19:

[1556] The server generates optimal memory content from past data based on the user's emotional state.

[1557] Step 20:

[1558] For example, if the server detects a state of "feeling a little down," it will select photos of past family trips or records of fun conversations with friends that are related to "happiness."

[1559] Step 21:

[1560] The server transmits the generated content to the terminal.

[1561] Step 22:

[1562] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[1563] Step 23:

[1564] The terminal displays a form for the user to input feedback on the proposed content.

[1565] Step 24:

[1566] The user enters feedback such as "It was great" or "It was average" and presses the submit button.

[1567] Step 25:

[1568] The terminal sends feedback information to the server.

[1569] Step 26:

[1570] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[1571] Step 27:

[1572] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[1573] Example 2

[1574] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1575] In modern society, many people suffer from loneliness and stress. In particular, it is difficult to receive appropriate support tailored to each individual's emotional state. Therefore, there is a need for a system that provides support tailored to each user's emotional state, reducing loneliness and improving happiness.

[1576] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1577] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, and a data analysis means using natural language processing and image recognition technology, which allows the server to efficiently collect and analyze the user's past behavioral data and provide appropriate memory content according to the user's individual emotional state.

[1578] "User authentication means" refers to a means for identifying and authenticating an individual when a user accesses a system.

[1579] The "data linking means" is a means for linking the system with an external data source and collecting data on users' past behavior.

[1580] The "means for collecting past behavioral data" is a means for systematically collecting a user's past activity history and digital content.

[1581] "Data analysis means using natural language processing and image recognition technology" means a means that uses natural language processing (NLP) and image recognition technology to analyze collected data and extract specific event and emotion tags.

[1582] The "interface means for inputting user's emotional information" is a means for providing an interface that allows the user to input his / her own emotional state.

[1583] The "means for analyzing an emotional state using an emotion engine" is a means for analyzing input emotion information, text, voice, and facial expression data to determine the user's current emotional state.

[1584] The "means for generating appropriate memorable content based on the analyzed emotions" is a means for selecting and generating content suitable for a user from past data based on the user's emotional state.

[1585] The "means for presenting the generated content to the user" refers to a means for presenting the generated memory content to the user through visual and auditory means.

[1586] The "user feedback means" is a means for a user to input feedback on the presented content and transmit it to the system.

[1587] The "means for analyzing feedback information and updating algorithm models" refers to a means for analyzing feedback information obtained from users and updating models to improve the accuracy of sentiment analysis algorithms and content suggestion algorithms.

[1588] The present invention is a system that aims to provide optimal memory content based on a user's emotional state, reduce feelings of loneliness, and improve feelings of happiness. The system of the present invention uses the following hardware and software.

[1589] Hardware and software used

[1590] Hardware

[1591] Device: A personal computer, smartphone, or tablet through which a user accesses the device.

[1592] Server: A server computer that processes the entire system.

[1593] Camera and microphone: Devices for collecting facial and voice data from the user.

[1594] software

[1595] Natural Language Processing Algorithms (NLP): Algorithms for analyzing text data, specifically using Google Cloud Natural Language API or similar technologies.

[1596] Image recognition technology: Technology for analyzing image data. Specifically, Google Cloud Vision API is used.

[1597] Emotion engine: An engine for analyzing the user's emotional state. A specific example is the Emotion API.

[1598] Embodiments of each step

[1599] 1. User authentication process

[1600] When a user accesses the system, they enter their Yahoo! ID and password into the form displayed on the terminal. The terminal then sends this information to the server, which then authenticates the user information using Yahoo!'s authentication API.

[1601] 2. Data linkage processing

[1602] The server will connect the system to the already authenticated user account and prepare to collect the necessary data from Yahoo! Mail and Yahoo! Photo Album.

[1603] 3. Collecting past behavioral data

[1604] The server collects past behavioral data from each Yahoo! service (email, photo album, etc.). For example, it collects emails and photos from the past five years.

[1605] 4. Data Analysis Processing

[1606] The server uses the data collected and applies natural language processing algorithms and image recognition techniques to extract specific events and emotion tags from the data.

[1607] 5. Emotional state analysis processing

[1608] The terminal displays an interface for the user to input their current emotional state. The user inputs emotional information, and this information is sent from the terminal to the server. The server analyzes the emotional state using an emotional engine.

[1609] 6. Emotion Recognition by Emotion Engine

[1610] The server uses an emotion engine to analyze text data, voice data, and facial expression data to recognize the user's emotions in real time.

[1611] 7. Creating memorable content

[1612] The server selects and generates appropriate memory content from past data based on the user's emotional state. For example, if the user is feeling a little down, it will select photos of past family trips and records of fun conversations related to happiness.

[1613] 8. Content Suggestion and Display

[1614] The server sends the generated memory content to the device, which then displays it to the user. For example, it may be displayed with a message such as "Check out these family vacation photos!"

[1615] 9. User Feedback Collection

[1616] The terminal displays a form for the user to input feedback about the content. The user inputs the feedback, which is then sent from the terminal to the server.

[1617] 10. Feedback analysis and algorithm model update

[1618] The server analyzes the received feedback information and updates the algorithm model, enabling more accurate content suggestions based on the user's emotional state from the next time onwards.

[1619] Specific examples

[1620] For example, if the user feels "a little down today," the prompt sentence might be:

[1621] Prompt Sentence Examples

[1622] "Simulate a scenario where a user logs in, inputs their emotional state, and then the system suggests photos from past family trips."

[1623] In this way, the system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[1624] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1625] The flow of this system's program processing

[1626] Step 1: User authentication process

[1627] Input: The user enters their Yahoo! ID and password.

[1628] Processing: The terminal encrypts these input data and sends them to the server.

[1629] Data processing and calculation: The server uses Yahoo!'s authentication API to verify and authenticate user information.

[1630] Output: If authentication is successful, start a user session and get user information.

[1631] Specific behavior:

[1632] A user enters the ID "user@example.com" and password "123" into a form.

[1633] The device encrypts the data and sends it to the server.

[1634] The server calls the Yahoo! authentication API, and if authentication is successful, the message "Welcome, user" is displayed.

[1635] Step 2: Data integration process

[1636] Input: The server maintains the user session after successful authentication.

[1637] Processing: The server prepares to exchange data with each Yahoo! service.

[1638] Data processing and calculation: The server uses the API to ask the user for permission to access past emails and photo album data.

[1639] Output: If the user approves the permission, the data is ready to be collected.

[1640] Specific behavior:

[1641] The server requests access to your Yahoo! account.

[1642] Once the user grants access, the server prepares to collect data.

[1643] Step 3: Collect historical behavioral data

[1644] Input: The server starts collecting data based on the access rights granted by the user.

[1645] Processing: The server collects data from Yahoo! Mail, Photo Album, Notepad, etc.

[1646] Data processing and calculation: The server organizes and stores the collected data in chronological order.

[1647] Output: A complete dataset of emails and photos from the past five years.

[1648] Specific behavior:

[1649] The server collects emails and photos from the period "2018-01-01" to "2023-01-01".

[1650] Collected data is organized and saved in chronological order.

[1651] Step 4: Data analysis and processing

[1652] Input: The dataset collected by the server.

[1653] Processing: Analyzing the data using natural language processing algorithms (NLP) and image recognition techniques.

[1654] Data processing and calculation: The server extracts specific events and emotion tags from the text data and extracts facial expressions from the image data.

[1655] Output: The analysis results in a dataset labeled with event and emotion tags.

[1656] Specific behavior:

[1657] The server uses the Google Cloud Natural Language API to analyze the email content and associate the keyword "family trip" with the tag "happiness."

[1658] It uses Google Cloud Vision API to perform facial recognition on photos and classify photos of "smiling faces" as "happy."

[1659] Step 5: Emotional state analysis processing

[1660] Input: The user inputs their current emotional state.

[1661] Processing: The terminal sends this input data to the server.

[1662] Data processing and calculation: The server uses an emotion engine to analyze the input emotion information and determine the emotional state.

[1663] Output: The user's current emotional state is recognized.

[1664] Specific behavior:

[1665] The user types into the terminal interface, "I'm feeling a little down today."

[1666] This information is sent to the server, which uses its emotion engine to determine that "slightly depressed" means "mild depression."

[1667] Step 6: Emotion Recognition with the Emotion Engine

[1668] Input: User text, voice, and facial expression data.

[1669] Processing: The server analyzes these data using the emotion engine.

[1670] Data processing and calculation: The emotion engine analyzes the data and recognizes the user's real-time emotional state.

[1671] Output: The emotional state is updated as a result of the analysis.

[1672] Specific behavior:

[1673] The user uses the camera of the terminal to capture his / her facial expression and also records voice information.

[1674] The server analyzes this data and recognizes a "smile" as "happiness."

[1675] Step 7: Create memorable content

[1676] Input: User's emotional state and analyzed past behavioral data.

[1677] Processing: The server generates appropriate memory content based on these data.

[1678] Data processing and calculation: Select the most suitable content from past data based on emotional state.

[1679] Output: The generated memories content.

[1680] Specific behavior:

[1681] The server selects photos of past family trips based on the user's "mild depression."

[1682] Pick out a few fun photos from the "Summer Travel 2019" folder.

[1683] Step 8: Content suggestion and display

[1684] Input: Generated memories content.

[1685] Processing: The server sends the content to the terminal.

[1686] Data processing and calculation: Prepares the layout for the device to display the content.

[1687] Output: The memories content displayed to the user.

[1688] Specific behavior:

[1689] The device will display the suggested family vacation photos on a large screen, along with the message, "Check out these family vacation photos!"

[1690] Step 9: Gather user feedback

[1691] Input: User feedback.

[1692] Processing: The terminal displays a form for inputting feedback and sends the information entered by the user to the server.

[1693] Data processing and calculation: The server stores and organizes the feedback information.

[1694] Output: A dataset of feedback information.

[1695] Specific behavior:

[1696] The user enters "It was great" in the feedback form and presses the submit button.

[1697] This information is sent to a server and stored in a database.

[1698] Step 10: Feedback analysis and algorithmic model update

[1699] Input: Feedback information from the user.

[1700] Processing: The server analyzes the feedback information and updates the algorithmic model.

[1701] Data processing and calculation: Optimize the accuracy of sentiment analysis algorithms and content suggestion algorithms based on feedback information.

[1702] Output: The updated algorithm model.

[1703] Specific behavior:

[1704] The server analyzes the feedback such as "It was really good" and changes the parameters of the sentiment analysis algorithm.

[1705] Optimize the algorithm model for the next proposal.

[1706] Through the above process, this system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[1707] (Application example 2)

[1708] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1709] In modern society, people often feel lonely and stressed, which can lead to poor mental health. To address this, a method is needed to provide appropriate memorable content based on the user's emotional state. However, existing systems are inadequate in accurately recognizing a user's emotional state in real time and providing optimal content based on that state. As a result, it is difficult to alleviate the loneliness and stress felt by users. Furthermore, effective feedback for users and optimization of content suggestion algorithms are not adequately implemented.

[1710] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a real-time emotion recognition means using a head-mounted display, and a means for displaying content based on the real-time emotional state. This makes it possible to accurately recognize the user's emotional state in real time and provide optimal memory content based on that state. By simply wearing the HMD, the user can experience memory content based on their own emotions in real time, effectively reducing feelings of loneliness and stress.

[1711] "User authentication means" is a function that verifies the legitimacy of a user by having the user access the system and enter authentication information.

[1712] "Data linking means" is a function that links an authenticated user's account with the system and shares necessary data.

[1713] "Means for collecting past behavioral data" is a function that collects users' past behavioral data (e.g., emails, photos, etc.).

[1714] "Data analysis means" is a function that analyzes collected data and extracts useful information.

[1715] The "emotional state analysis means" is a function that analyzes the emotional information provided by the user and determines the emotional state of the user.

[1716] The "memory content generating means" is a function that generates appropriate memory content based on the emotional state of the user.

[1717] The "user feedback means" is an interface for users to input feedback on the provided content.

[1718] The "feedback analysis means" is a function that analyzes feedback provided by users and uses it to improve the system.

[1719] The "algorithm model update means" is a function that updates the emotion analysis algorithm and the content suggestion algorithm based on the analysis results.

[1720] "Real-time emotion recognition means using a head-mounted display" is a function that uses sensors built into the head-mounted display to recognize the user's emotions in real time.

[1721] The "means for displaying content based on real-time emotional state" is a function for displaying optimal memory content on the head-mounted display in accordance with the emotional state recognized in real time.

[1722] This invention is a system that provides optimal memory content based on the user's emotional state, reducing feelings of loneliness and stress. It primarily uses a head-mounted display (HMD) to recognize the user's emotions in real time and display memory content accordingly.

[1723] System Configuration

[1724] The system includes the following main components:

[1725] User authentication method

[1726] Data linking method

[1727] Past behavioral data collection method

[1728] Data Analysis Methods

[1729] Emotional state analysis means

[1730] Means of creating memorable content

[1731] User feedback tools

[1732] Feedback Analysis Methods

[1733] Algorithmic model update method

[1734] Real-time emotion recognition using a head-mounted display

[1735] A means of displaying content based on real-time emotional state

[1736] Hardware and Software

[1737] Hardware

[1738] Head-mounted display (HMD): A device worn by the user that displays visual content, with built-in facial recognition cameras and heart rate sensors.

[1739] PC or smartphone: A device that connects to the HMD and processes data.

[1740] Server: Responsible for data collection, analysis, content generation, etc.

[1741] software

[1742] Python: A programming language.

[1743] OpenCV: A library for face recognition.

[1744] Yahoo SDK: A software development kit for working with Yahoo services.

[1745] EmotionEngine: An engine for emotion analysis.

[1746] ContentGenerator: A module that generates and suggests memory content.

[1747] Processing Details

[1748] 1. User Authentication

[1749] The server first authenticates the user based on the credentials provided by the user, which is done using the Yahoo SDK.

[1750] 2. Data Linkage

[1751] After authentication, the server links the user's account to the system and then prepares to collect past behavioral data.

[1752] 3. Collecting past behavioral data

[1753] The server collects data such as users' past emails, photo albums, notepads, etc. The collection period is, for example, the past five years.

[1754] 4. Data Analysis

[1755] The server analyzes the collected data and uses natural language processing algorithms and image recognition technology to extract specific event and emotion tags.

[1756] 5. Emotional State Analysis

[1757] The device (HMD) displays an interface for inputting the user's emotional state and transmits the collected emotional information to the server, which analyzes it and determines the user's current emotional state.

[1758] 6. Real-time emotion recognition

[1759] Using the cameras and sensors built into the HMD, we recognize the user's real-time emotional state using OpenCV and EmotionEngine.

[1760] 7. Creating memorable content

[1761] The server generates appropriate memory content based on the user's real-time emotional state. For example, if the user is feeling a little down, it will select videos of past family trips or photos of happy friends.

[1762] 8. Content Display

[1763] The content selected by the user is displayed through the HMD.

[1764] Specific examples

[1765] The user wears the HMD and the system analyzes the user's emotions in real time. If the user's emotional state is determined to be "a little depressed," the system displays videos of past family trips and happy photos with friends, helping the user to relax and feel happier.

[1766] Prompt Sentence Examples

[1767] "Generate relevant memory content by providing a user's facial image and emotional state as input. The user is in a slightly depressed state."

[1768] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1769] Step 1:

[1770] The user wears a head-mounted display (HMD) and accesses the system. The device displays an interface for the user to enter their Yahoo! ID and password. This input information is the input data for user authentication. The device sends the entered authentication information to the server, which then authenticates the user information using the Yahoo! SDK. If authentication is successful, the server starts a user session and obtains the user information. The authenticated user information is obtained as output data.

[1771] Step 2:

[1772] The server activates the data integration method that links the authenticated user's account with the system. At this point, it is ready to retrieve data from past Yahoo! services (e.g., email and photo albums). The behavioral data to be collected is integrated and becomes the input data for the next step.

[1773] Step 3:

[1774] The server uses a past behavioral data collection means to collect the user's past behavioral data (emails, photo albums, memo pads, etc.). The collection period is, for example, the past five years. The collected data becomes input data for the data analysis means. The collected behavioral data is obtained as output data.

[1775] Step 4:

[1776] The server analyzes the collected data using data analysis means. Specifically, it uses natural language processing algorithms and image recognition technology to extract specific events and emotion tags (e.g., "happiness," "sadness," etc.) from the data. The results of this analysis are input data for the emotion state analysis means. Emotion tags and event information are obtained as output data.

[1777] Step 5:

[1778] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm a little down" or "I'm having a good day," and the device sends the information to the server. This emotional information becomes input data for the emotional state analysis means. The server uses an emotion engine to analyze the input emotional information and determine the user's current emotional state. The user's current emotional state is obtained as output data.

[1779] Step 6:

[1780] Using the camera and sensors built into the HMD, the server recognizes the user's real-time emotions. Using OpenCV and EmotionEngine, the server analyzes the user's facial images and heart rate data and updates their emotional state in real time. This real-time emotional data becomes the input data for the memory content generation means. The real-time emotional state is obtained as output data.

[1781] Step 7:

[1782] The server generates optimal memory content using the memory content generation means based on the user's real-time emotional state. For example, if the user is feeling a little down, the server selects videos of past family trips and photos of friends looking happy. This memory content becomes input data for the content display means. The generated memory content is obtained as output data.

[1783] Step 8:

[1784] The device presents the selected memory content to the user through the HMD. The user experiences this content visually in real time through the HMD. Intuitive messages and guides are displayed as an interface to the user. The output data becomes the content that the user experiences visually.

[1785] Step 9:

[1786] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average," and this information becomes input data for the user feedback means. The terminal sends the feedback information to the server, and the user's feedback information is obtained as output data.

[1787] Step 10:

[1788] The server uses the feedback analysis means to analyze the received feedback information and, based on this, updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm. The algorithm model update means incorporates this feedback information to improve the accuracy of future suggestions. The updated algorithm model is obtained as output data.

[1789] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1790] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1791] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1792] [Fourth embodiment]

[1793] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1794] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1795] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1796] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1797] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1798] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1799] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1800] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1801] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1802] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1803] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1804] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1805] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1806] The present invention relates to a system that aims to improve a user's sense of happiness by providing content based on past memories when the user is feeling lonely. The program processing flow of this system will be specifically explained below.

[1807] Overall system operation

[1808] 1. User authentication process

[1809] When a user accesses the system, the terminal displays a form for entering the Yahoo! ID and password.

[1810] The user enters their Yahoo! ID and password and presses the login button.

[1811] The terminal sends the input information to the server.

[1812] The server authenticates the input information using Yahoo!'s authentication API. If authentication is successful, it starts a user session and retrieves user information.

[1813] 2. Data linkage processing

[1814] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[1815] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[1816] 3. Collecting past behavioral data

[1817] The server collects users' past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[1818] For example, get emails and photos from the past five years.

[1819] 4. Data Analysis Processing

[1820] Natural language processing algorithms and image recognition technology are used to analyze the data collected by the server.

[1821] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[1822] 5. Emotional state analysis processing

[1823] The terminal displays an interface for the user to input an emotional state.

[1824] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[1825] The device transmits this emotion information to the server.

[1826] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[1827] 6. Creating memorable content

[1828] The server generates appropriate memory content from past data based on the user's emotional state.

[1829] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[1830] 7. Content Suggestion and Display

[1831] The server transmits the generated content to the terminal.

[1832] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[1833] 8. User Feedback Collection

[1834] The terminal displays a form for the user to input feedback on the proposed content.

[1835] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[1836] The terminal sends feedback information to the server.

[1837] 9. Feedback analysis and algorithmic model updating

[1838] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[1839] This allows the system to continually improve so that next time it suggests content that best suits the user's emotional state.

[1840] In this way, this system provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and increasing feelings of happiness, thereby contributing to improving the user's mental health.

[1841] The processing flow will be explained below.

[1842] Step 1:

[1843] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[1844] Step 2:

[1845] The user enters their Yahoo! ID and password and presses the login button.

[1846] Step 3:

[1847] The device sends the entered Yahoo! ID and password to the server.

[1848] Step 4:

[1849] The server authenticates the user information using Yahoo!'s authentication API.

[1850] Step 5:

[1851] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[1852] Step 6:

[1853] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[1854] Step 7:

[1855] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[1856] Step 8:

[1857] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[1858] Step 9:

[1859] For example, get emails and photos from the past five years.

[1860] Step 10:

[1861] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[1862] Step 11:

[1863] The server extracts specific event and emotion tags (e.g., "happy," "sad," etc.) from the data.

[1864] Step 12:

[1865] The terminal displays an interface for the user to input their emotional state.

[1866] Step 13:

[1867] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[1868] Step 14:

[1869] The terminal transmits the input emotion information to the server.

[1870] Step 15:

[1871] The server uses an emotion analysis API to analyze the input emotion information and determine the user's current emotional state.

[1872] Step 16:

[1873] The server generates appropriate memory content from past data based on the user's emotional state.

[1874] Step 17:

[1875] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[1876] Step 18:

[1877] The server transmits the generated content to the terminal.

[1878] Step 19:

[1879] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[1880] Step 20:

[1881] The terminal displays a form for the user to input feedback on the proposed content.

[1882] Step 21:

[1883] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[1884] Step 22:

[1885] The terminal sends feedback information to the server.

[1886] Step 23:

[1887] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[1888] Step 24:

[1889] The server uses an updated algorithm to enable more accurate content suggestions based on the user's emotional state from the next time onwards.

[1890] Example 1

[1891] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1892] In modern society, feelings of loneliness and low happiness among users have become serious problems. To address these issues, a system that can improve users' psychological state by providing content based on past memories is needed. However, existing systems do not adequately provide appropriate content according to emotional states, and they lack a means to effectively utilize user feedback to continuously improve the system.

[1893] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1894] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, and an algorithm model update means. This makes it possible to generate and display appropriate memory content from past data based on the user's emotional state. Furthermore, by combining an interface means for inputting the user's emotional information with a means for analyzing the emotional information and determining the user's current emotional state, it is possible to grasp the user's psychological state and provide more effective content.

[1895] A "user authentication means" is a mechanism that authenticates a user based on authentication information such as a user name and password that the user inputs when accessing a system.

[1896] "Data linking means" is a mechanism that allows the system to link with external services based on the user's account information and obtain the necessary data.

[1897] The "means for collecting past behavioral data" is used to collect behavioral data from the user's past emails, photos, memo pads, etc.

[1898] "Data analysis means" refers to the function of extracting the meaning and characteristics of collected data using algorithms and techniques to analyze the data.

[1899] The "emotional state analysis means" is a mechanism for analyzing the emotional information input by the user and determining the current emotional state.

[1900] The "memory content generating means" is a function for generating appropriate memory content based on the analyzed data and emotional state.

[1901] The "user feedback means" is an interface for users to input feedback on the provided content.

[1902] The "feedback analysis means" is a mechanism for analyzing feedback information from users and using it to provide content next time.

[1903] The "algorithm model update means" is a function for updating and optimizing the models of the emotion analysis algorithm and content recommendation algorithm based on feedback information.

[1904] The present invention relates to a system for providing content based on past memories in order to reduce the sense of loneliness felt by a user and improve the sense of happiness. An embodiment of this system will be specifically described below.

[1905] This system includes three main elements: a server, a terminal, and a user. First, the user is authenticated using a user authentication method. The terminal displays a login screen for the user, and the user enters authentication information (ID and password). The entered information is sent from the terminal to the server, and the server verifies the information using an authentication API. If authentication is successful, the server starts a session for the user and obtains the necessary user information.

[1906] Next, using the data linking method, the server links with the external service based on the user's account information. At this time, the server prepares to collect past behavioral data such as the user's emails and photo albums. Data linking is performed through the external service's API, and the acquired data is temporarily stored on the server.

[1907] The collected data is analyzed using data analysis tools. Specifically, the server uses natural language processing algorithms (such as NLTK or spaCy) and image recognition technologies (such as OpenCV or TensorFlow) to extract specific events and emotion tags (such as "happiness" or "sadness") from the data. The analysis results are stored in a database.

[1908] Next, the emotional state analysis means is used to analyze the user's current emotional state. The terminal displays an interface for the user to input their emotional state, and the user inputs emotional information. The input emotional information is sent from the terminal to a server, and the server analyzes the emotional information using an emotion analysis API to determine the user's current emotional state.

[1909] The memory content generation means operates based on the user's emotional state. The server generates memory content appropriate to the user's emotional state based on the acquired data and analysis results. For example, if the user inputs "I'm feeling a little down," the server selects past travel photos or conversation records with friends that are related to "happiness." The generated content is sent from the server to the terminal, which displays it to the user.

[1910] After the content is displayed, the user uses the user feedback means to input feedback on the provided content. The user inputs an evaluation such as "it was very good" or "it was average" into the feedback form displayed on the terminal and transmits the input to the server via the form.

[1911] The feedback analysis means receives this feedback information and analyzes it on the server. The analysis results are reflected in the system's learning model through the algorithm model update means. This enables more appropriate suggestions to be made the next time content is provided.

[1912] As a specific example of how it works, if a user enters "I'm feeling a little down" as a prompt, the server will select photos of past family trips and records of fun conversations with friends related to "happiness" and send them to the device. The device will then display these along with a message such as "Look at these family trip photos!" If the user enters feedback such as "It was great," the server will update the system based on that feedback and use it to suggest content next time.

[1913] Examples of prompts: "I'm feeling a little down," "I'm having a good time today."

[1914] In this way, the system of the present invention provides optimal memory content according to the user's current emotional state, reducing feelings of loneliness and improving feelings of happiness, thereby contributing to improving the user's mental health.

[1915] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1916] Step 1:

[1917] User authentication process

[1918] When a user accesses the system, the terminal displays a form for entering a Yahoo! ID and password. After the user provides the input information and presses the login button, the terminal sends the input information to the server. The server verifies the input information using Yahoo!'s authentication API, and if authentication is successful, it starts a user session and obtains the user information.

[1919] Input: Yahoo! ID and password

[1920] Output: Authentication success flag, user information

[1921] Specific operation: The device displays a login screen and the user enters their authentication information. The information is sent from the device to the server, and the server performs authentication via the Yahoo! API.

[1922] Step 2:

[1923] Data linkage processing

[1924] After successful authentication, the server links the user's account to an external service, which prepares the server to collect user behavior data (emails, photos, etc.).

[1925] Input: Authentication success flag, user information

[1926] Output: Data collection ready flag

[1927] Specific operation: The server uses the user information to connect to the API of external services (email, photo album) and prepares to collect data.

[1928] Step 3:

[1929] Collecting past behavioral data

[1930] The server collects the user's past behavior data from emails, photo albums, etc. The server acquires and stores this data for a certain period of time (e.g., the past five years).

[1931] Input: Data collection ready flag

[1932] Output: Collected behavioral data

[1933] Specific operation: The server calls the API and collects email and photo data for the specified period. The acquired data is temporarily stored in storage.

[1934] Step 4:

[1935] Data analysis processing

[1936] The server analyzes the collected data using natural language processing and image recognition technology, extracting emotion tags (such as "happiness" or "sadness") from text data and performing facial and scene recognition from image data.

[1937] Input: Collected behavioral data

[1938] Output: Analysis results (emotion tags, specific events)

[1939] How it works: The server uses Python libraries such as NLTK, spaCy, OpenCV, and TensorFlow to analyze text and image data and store the results in a database.

[1940] Step 5:

[1941] Emotional state analysis processing

[1942] The device provides the user with an interface for inputting their emotional state. After the user inputs their emotional information and presses the send button, the device sends the data to the server. The server then uses an emotion analysis API to analyze the data and determine the user's current emotional state.

[1943] Input: User's emotional information

[1944] Output: Current emotional state

[1945] Specific operation: The device displays the interface, receives input from the user, and sends it to the server, which then analyzes it using the sentiment analysis API.

[1946] Step 6:

[1947] Creating memorable content

[1948] The server generates optimal memory content based on the analysis results, based on the user's emotional state. For example, if the user enters "I'm feeling a little down," the server will select photos of past family trips related to "happiness."

[1949] Input: Current emotional state, analysis results

[1950] Output: Memories content

[1951] What it does: The server queries the database for the appropriate content and sends the generated content to the device in the appropriate format.

[1952] Step 7:

[1953] Content Suggestion and Display

[1954] The server sends the generated memory content to the device, which then displays it to the user, for example, with a message such as "Check out these family vacation photos!"

[1955] Input: Memories content

[1956] Output: The displayed content

[1957] Specific operation: The server sends the generated content to the device via an HTTP response, and the device displays it.

[1958] Step 8:

[1959] User feedback collection

[1960] The device provides the user with a form for entering feedback on the content. The user enters an evaluation such as "very good" or "average," and when the user presses the submit button, the device sends the data to the server.

[1961] Input: User feedback

[1962] Output: Collected feedback

[1963] Specific operation: The device displays a feedback input form, receives input from the user, and sends it to the server.

[1964] Step 9:

[1965] Feedback analysis and algorithmic model updating

[1966] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby improving the quality of the next content recommendation.

[1967] Input: User feedback

[1968] Output: Updated algorithmic model

[1969] What it does: The server stores the feedback information in a database and updates the analysis and generative AI model by retraining it.

[1970] (Application example 1)

[1971] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1972] In modern society, there is a demand for technologies that can reduce the feelings of loneliness and stress felt by users and improve their happiness. In particular, in electronic payment services, there is a growing need for systems that not only process payments but also recommend items that will bring happiness to users based on their past purchase history and emotional state. However, conventional technologies have not been able to meet such complex requirements.

[1973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1974] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a means for adding an emotional tag based on past purchase data, and a means for recommending items based on the emotional state, thereby making it possible to provide content and recommended items that are optimal for the user's current emotional state.

[1975] "User authentication means" refers to a means for identifying a user based on authentication information when the user accesses the system.

[1976] "Data linkage means" refers to a means of linking user information with external systems and collecting and managing necessary data.

[1977] The "means for collecting past behavioral data" is a means for collecting data on the past behavior of a user.

[1978] "Data analysis means" is a means for analyzing collected data and extracting useful information and patterns.

[1979] The "emotional state analysis means" is a means for analyzing the current emotional state of the user.

[1980] The "memory content generating means" is a means for generating memory content that matches the emotional state of the user based on past data.

[1981] "User feedback means" refers to a means for collecting feedback from users.

[1982] The "feedback analysis means" is a means for analyzing collected feedback information and using it to improve the system.

[1983] The "algorithm model updating means" is a means for updating and optimizing the algorithm model based on the collected feedback information.

[1984] The "means for adding emotion tags based on past purchase data" is a means for analyzing past purchase data and adding tags related to specific emotional states.

[1985] The "means for recommending items based on emotional state" is a means for recommending appropriate items based on the emotional state of a user.

[1986] This invention is a system that reduces users' feelings of loneliness and improves their happiness, and is particularly aimed at application to electronic payment services. The system is characterized by recommending appropriate content and items based on the user's emotional state.

[1987] Overall system operation

[1988] User authentication method

[1989] When a user accesses the system, the server identifies the user based on the authentication information (ID and password).To do this, the server uses an authentication API to authenticate the entered information, starts a session, and obtains user information.

[1990] Data linking method

[1991] After successful authentication, the server links the user's account to the system, and prepares to collect data such as past emails and photo albums.

[1992] Past behavioral data collection method

[1993] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[1994] Data Analysis Methods

[1995] The server analyzes the collected data, using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[1996] Emotional state analysis means

[1997] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[1998] Means of creating memorable content

[1999] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips and records of fun conversations with friends.

[2000] User feedback tools

[2001] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average" and sends it to the server.

[2002] Feedback Analysis Methods

[2003] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm, thereby enabling it to provide more appropriate content in the next recommendation.

[2004] Algorithmic model update method

[2005] Based on the feedback information, the server adjusts the parameters of the AI ​​model used and the proposed algorithm to improve the accuracy of the system.

[2006] Emotion tagging method

[2007] The server analyzes past purchase data and adds tags related to specific emotional states, such as "happy" or "excited," to previously purchased items.

[2008] Item recommendation method

[2009] The server recommends appropriate items based on the user's current emotional state. If the user is currently in a "happy" emotional state, the server creates a list of items that the user has purchased in the past that made them feel happy, sends this list to the device, and displays it.

[2010] Hardware and software used

[2011] Hardware: Smartphone

[2012] software

[2013] Python: Implementation of the entire application

[2014] requests: A library for making API requests

[2015] JSON: Data Format

[2016] Natural Language Processing Algorithms: Analyzing Text Data

[2017] Image recognition technology: Analysis of image data

[2018] Sentiment Analysis API: Emotional information analysis

[2019] Specific examples

[2020] For example, if a user is currently in a "happy" emotional state, the system will recommend items related to "happiness" based on their past purchase history, such as candy or toys that have been purchased in the past and that have brought them happiness.

[2021] Prompt Sentence Examples

[2022] The user is currently in a "happy" emotional state. Recommend items that brought the user happiness based on their past purchase history.

[2023] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2024] Step 1:

[2025] User authentication

[2026] When a user accesses the system, the server displays a form for entering an ID and password. The user enters the ID and password and sends them to the server. The server uses an authentication API to check the input information, and if authentication is successful, starts a session and obtains the user information.

[2027] Input: User ID and password

[2028] Output: Session ID and user information

[2029] Step 2:

[2030] Data Linkage

[2031] After successful authentication, the server performs the procedure to link the user's account with the system, and prepares to collect the user's past data (emails, photo albums, purchase history, etc.).

[2032] Input: Authenticated user information

[2033] Output: Notification that data integration is ready

[2034] Step 3:

[2035] Collecting past behavioral data

[2036] The server collects the user's past behavioral data (emails, photo albums, purchase history, etc.). For example, it obtains emails, photos, and purchase history from the past five years.

[2037] Input: Notification that data integration is ready

[2038] Output: Past behavior data

[2039] Step 4:

[2040] Data analysis

[2041] The server analyzes the collected data using natural language processing algorithms for text data and image recognition technology for image data, thereby extracting specific event and emotion tags from the data.

[2042] Input: Past behavior data

[2043] Output: Parsed data and sentiment tags

[2044] Step 5:

[2045] Emotional State Analysis

[2046] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and sends it to the server. The server uses an emotion analysis API to analyze this emotional information and determine the user's current emotional state.

[2047] Input: User's emotional information

[2048] Output: Parsed emotional state

[2049] Step 6:

[2050] Creation of memory content and items

[2051] The server generates appropriate memory content and items from past data based on the user's emotional state. For example, if a user inputs "I'm feeling a little down," the server will select photos from past family trips or records of fun conversations with friends. It also analyzes past purchase data and adds related emotional tags.

[2052] Input: Analyzed emotional state, past behavioral data

[2053] Output: Generated memory content and recommended items

[2054] Step 7:

[2055] Content Suggestion and Display

[2056] The server sends the generated content to the device and displays it to the user along with a message such as "Look at these family vacation photos!" Recommended items are also displayed at the same time.

[2057] Input: Generated memory content and recommended items

[2058] Output: Content and items displayed to the user

[2059] Step 8:

[2060] Feedback collection

[2061] The device displays a form for the user to enter feedback on the proposed content or recommended items. The user enters feedback such as "It was really good" or "It was average" and sends it to the server.

[2062] Input: User feedback

[2063] Output: Collected feedback information

[2064] Step 9:

[2065] Feedback analysis and algorithm updates

[2066] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and recommendation algorithm, thereby enabling it to provide more appropriate content and items in the next proposal.

[2067] Input: Collected feedback information

[2068] Output: Updated and optimized algorithmic model

[2069] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2070] The present invention is a system that aims to provide optimal memorable content based on the user's emotional state, reducing feelings of loneliness and improving happiness. In particular, by incorporating an emotion engine that recognizes the user's emotions in real time, more accurate emotion analysis and content suggestions are achieved. The processing of the entire system is explained in detail below.

[2071] Overall system operation

[2072] 1. User authentication process

[2073] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[2074] The user enters their Yahoo! ID and password and presses the login button.

[2075] The device sends the entered Yahoo! ID and password to the server.

[2076] The server authenticates the user information using Yahoo!'s authentication API, and if authentication is successful, starts a user session and obtains the user information.

[2077] 2. Data linkage processing

[2078] After the server successfully authenticates, it carries out the procedure to link the user's Yahoo! account with the system.

[2079] The server prepares to collect past Yahoo! Mail and Yahoo! Photo Album data.

[2080] 3. Collecting past behavioral data

[2081] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[2082] For example, get emails and photos from the past five years.

[2083] 4. Data Analysis Processing

[2084] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[2085] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[2086] 5. Emotional state analysis processing

[2087] The terminal displays an interface for the user to input their emotional state.

[2088] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[2089] The device transmits this emotion information to the server.

[2090] The server uses an emotion engine to analyze the input emotion information and determine the user's current emotional state.

[2091] 6. Emotion Recognition by Emotion Engine

[2092] The server uses an emotion engine to recognize emotions by analyzing the user's text, voice, and facial expression data.

[2093] The server updates the user's real-time emotional state based on the recognized emotional information.

[2094] 7. Creating memorable content

[2095] The server generates appropriate memory content from past data based on the user's emotional state.

[2096] For example, if a user inputs "I'm feeling a little down," the server will select photos of past family trips or records of fun conversations with friends that relate to "happiness."

[2097] 8. Content Suggestion and Display

[2098] The server transmits the generated content to the terminal.

[2099] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[2100] 9. User Feedback Collection

[2101] The terminal displays a form for the user to input feedback on the proposed content.

[2102] The user enters feedback such as "It was great" or "It was average" and presses the send button.

[2103] The terminal sends feedback information to the server.

[2104] 10. Feedback analysis and algorithm model update

[2105] The server analyzes the feedback information it receives and uses this information to update and optimize the models of the sentiment analysis algorithm and content suggestion algorithm.

[2106] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[2107] In this way, the system constantly monitors the user's real-time emotional state and provides appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness, which can greatly contribute to improving the user's mental health.

[2108] The processing flow will be explained below.

[2109] Step 1:

[2110] The user accesses the system and the terminal displays a form for entering the Yahoo! ID and password.

[2111] Step 2:

[2112] The user enters their Yahoo! ID and password and presses the login button.

[2113] Step 3:

[2114] The device sends the entered Yahoo! ID and password to the server.

[2115] Step 4:

[2116] The server authenticates the user information using Yahoo!'s authentication API.

[2117] Step 5:

[2118] The server receives the authentication result, and if the authentication is successful, starts a user session and acquires user information.

[2119] Step 6:

[2120] After the server has successfully authenticated, it uses a data linking method to link the user's Yahoo! account with the system.

[2121] Step 7:

[2122] The server notifies the terminal that the link setting is complete, and the terminal is now ready to collect past data.

[2123] Step 8:

[2124] The server collects past behavioral data from Yahoo! Mail, Yahoo! Photo Album, Notepad, etc.

[2125] Step 9:

[2126] For example, obtain email and photo data from the past five years.

[2127] Step 10:

[2128] The server uses natural language processing algorithms and image recognition technology to analyze the collected data.

[2129] Step 11:

[2130] The server extracts specific event and emotion tags (e.g., "happiness," "sadness," etc.) from the data.

[2131] Step 12:

[2132] The terminal displays an interface for the user to input an emotional state.

[2133] Step 13:

[2134] The user inputs emotional information such as "I'm feeling a little down" or "I'm having fun today" and presses the send button.

[2135] Step 14:

[2136] The terminal transmits the input emotion information to the server.

[2137] Step 15:

[2138] The server uses an emotion engine to analyze the input emotion information to determine the user's current emotional state.

[2139] Step 16:

[2140] The server further displays an interface on the terminal for collecting the user's text, voice, and facial expression data.

[2141] Step 17:

[2142] The user inputs their voice and facial expressions through a camera and microphone, and the device sends this data to the server.

[2143] Step 18:

[2144] The server uses an emotion engine to analyze the collected text, voice, and facial expression data to recognize and update the user's real-time emotional state.

[2145] Step 19:

[2146] The server generates optimal memory content from past data based on the user's emotional state.

[2147] Step 20:

[2148] For example, if the server detects a state of "feeling a little down," it will select photos of past family trips or records of fun conversations with friends that are related to "happiness."

[2149] Step 21:

[2150] The server transmits the generated content to the terminal.

[2151] Step 22:

[2152] The device displays the suggested content to the user, showing a photo with a message such as "Check out these family vacation photos!"

[2153] Step 23:

[2154] The terminal displays a form for the user to input feedback on the proposed content.

[2155] Step 24:

[2156] The user enters feedback such as "It was great" or "It was average" and presses the submit button.

[2157] Step 25:

[2158] The terminal sends feedback information to the server.

[2159] Step 26:

[2160] The server analyzes the received feedback information and updates and optimizes the models of the sentiment analysis algorithm and content suggestion algorithm based on this information.

[2161] Step 27:

[2162] The server uses the updated algorithm to suggest more accurate content based on the user's emotional state from the next time onwards.

[2163] Example 2

[2164] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2165] In modern society, many people suffer from loneliness and stress. In particular, it is difficult to receive appropriate support tailored to each individual's emotional state. Therefore, there is a need for a system that provides support tailored to each user's emotional state, reducing loneliness and improving happiness.

[2166] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2167] In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, and a data analysis means using natural language processing and image recognition technology, which allows the server to efficiently collect and analyze the user's past behavioral data and provide appropriate memory content according to the user's individual emotional state.

[2168] "User authentication means" refers to a means for identifying and authenticating an individual when a user accesses a system.

[2169] The "data linking means" is a means for linking the system with an external data source and collecting data on users' past behavior.

[2170] The "means for collecting past behavioral data" is a means for systematically collecting a user's past activity history and digital content.

[2171] "Data analysis means using natural language processing and image recognition technology" means a means that uses natural language processing (NLP) and image recognition technology to analyze collected data and extract specific event and emotion tags.

[2172] The "interface means for inputting user's emotional information" is a means for providing an interface that allows the user to input his / her own emotional state.

[2173] The "means for analyzing an emotional state using an emotion engine" is a means for analyzing input emotion information, text, voice, and facial expression data to determine the user's current emotional state.

[2174] The "means for generating appropriate memorable content based on the analyzed emotions" is a means for selecting and generating content suitable for a user from past data based on the user's emotional state.

[2175] The "means for presenting the generated content to the user" refers to a means for presenting the generated memory content to the user through visual and auditory means.

[2176] The "user feedback means" is a means for a user to input feedback on the presented content and transmit it to the system.

[2177] The "means for analyzing feedback information and updating algorithm models" refers to a means for analyzing feedback information obtained from users and updating models to improve the accuracy of sentiment analysis algorithms and content suggestion algorithms.

[2178] The present invention is a system that aims to provide optimal memory content based on a user's emotional state, reduce feelings of loneliness, and improve feelings of happiness. The system of the present invention uses the following hardware and software.

[2179] Hardware and software used

[2180] Hardware

[2181] Device: A personal computer, smartphone, or tablet through which a user accesses the device.

[2182] Server: A server computer that processes the entire system.

[2183] Camera and microphone: Devices for collecting facial and voice data from the user.

[2184] software

[2185] Natural Language Processing Algorithms (NLP): Algorithms for analyzing text data, specifically using Google Cloud Natural Language API or similar technologies.

[2186] Image recognition technology: Technology for analyzing image data. Specifically, Google Cloud Vision API is used.

[2187] Emotion engine: An engine for analyzing the user's emotional state. A specific example is the Emotion API.

[2188] Embodiments of each step

[2189] 1. User authentication process

[2190] When a user accesses the system, they enter their Yahoo! ID and password into the form displayed on the terminal. The terminal then sends this information to the server, which then authenticates the user information using Yahoo!'s authentication API.

[2191] 2. Data linkage processing

[2192] The server will connect the system to the already authenticated user account and prepare to collect the necessary data from Yahoo! Mail and Yahoo! Photo Album.

[2193] 3. Collecting past behavioral data

[2194] The server collects past behavioral data from each Yahoo! service (email, photo album, etc.). For example, it collects emails and photos from the past five years.

[2195] 4. Data Analysis Processing

[2196] The server uses the data collected and applies natural language processing algorithms and image recognition techniques to extract specific events and emotion tags from the data.

[2197] 5. Emotional state analysis processing

[2198] The terminal displays an interface for the user to input their current emotional state. The user inputs emotional information, and this information is sent from the terminal to the server. The server analyzes the emotional state using an emotional engine.

[2199] 6. Emotion Recognition by Emotion Engine

[2200] The server uses an emotion engine to analyze text data, voice data, and facial expression data to recognize the user's emotions in real time.

[2201] 7. Creating memorable content

[2202] The server selects and generates appropriate memory content from past data based on the user's emotional state. For example, if the user is feeling a little down, it will select photos of past family trips and records of fun conversations related to happiness.

[2203] 8. Content Suggestion and Display

[2204] The server sends the generated memory content to the device, which then displays it to the user. For example, it may be displayed with a message such as "Check out these family vacation photos!"

[2205] 9. User Feedback Collection

[2206] The terminal displays a form for the user to input feedback about the content. The user inputs the feedback, which is then sent from the terminal to the server.

[2207] 10. Feedback analysis and algorithm model update

[2208] The server analyzes the received feedback information and updates the algorithm model, enabling more accurate content suggestions based on the user's emotional state from the next time onwards.

[2209] Specific examples

[2210] For example, if the user feels "a little down today," the prompt sentence might be:

[2211] Prompt Sentence Examples

[2212] "Simulate a scenario where a user logs in, inputs their emotional state, and then the system suggests photos from past family trips."

[2213] In this way, the system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[2214] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2215] The flow of this system's program processing

[2216] Step 1: User authentication process

[2217] Input: The user enters their Yahoo! ID and password.

[2218] Processing: The terminal encrypts these input data and sends them to the server.

[2219] Data processing and calculation: The server uses Yahoo!'s authentication API to verify and authenticate user information.

[2220] Output: If authentication is successful, start a user session and get user information.

[2221] Specific behavior:

[2222] A user enters the ID "user@example.com" and password "123" into a form.

[2223] The device encrypts the data and sends it to the server.

[2224] The server calls the Yahoo! authentication API, and if authentication is successful, the message "Welcome, user" is displayed.

[2225] Step 2: Data integration process

[2226] Input: The server maintains the user session after successful authentication.

[2227] Processing: The server prepares to exchange data with each Yahoo! service.

[2228] Data processing and calculation: The server uses the API to ask the user for permission to access past emails and photo album data.

[2229] Output: If the user approves the permission, the data is ready to be collected.

[2230] Specific behavior:

[2231] The server requests access to your Yahoo! account.

[2232] Once the user grants access, the server prepares to collect data.

[2233] Step 3: Collect historical behavioral data

[2234] Input: The server starts collecting data based on the access rights granted by the user.

[2235] Processing: The server collects data from Yahoo! Mail, Photo Album, Notepad, etc.

[2236] Data processing and calculation: The server organizes and stores the collected data in chronological order.

[2237] Output: A complete dataset of emails and photos from the past five years.

[2238] Specific behavior:

[2239] The server collects emails and photos from the period "2018-01-01" to "2023-01-01".

[2240] Collected data is organized and saved in chronological order.

[2241] Step 4: Data analysis and processing

[2242] Input: The dataset collected by the server.

[2243] Processing: Analyzing the data using natural language processing algorithms (NLP) and image recognition techniques.

[2244] Data processing and calculation: The server extracts specific events and emotion tags from the text data and extracts facial expressions from the image data.

[2245] Output: The analysis results in a dataset labeled with event and emotion tags.

[2246] Specific behavior:

[2247] The server uses the Google Cloud Natural Language API to analyze the email content and associate the keyword "family trip" with the tag "happiness."

[2248] It uses Google Cloud Vision API to perform facial recognition on photos and classify photos of "smiling faces" as "happy."

[2249] Step 5: Emotional state analysis processing

[2250] Input: The user inputs their current emotional state.

[2251] Processing: The terminal sends this input data to the server.

[2252] Data processing and calculation: The server uses an emotion engine to analyze the input emotion information and determine the emotional state.

[2253] Output: The user's current emotional state is recognized.

[2254] Specific behavior:

[2255] The user types into the terminal interface, "I'm feeling a little down today."

[2256] This information is sent to the server, which uses its emotion engine to determine that "slightly depressed" means "mild depression."

[2257] Step 6: Emotion Recognition with the Emotion Engine

[2258] Input: User text, voice, and facial expression data.

[2259] Processing: The server analyzes these data using the emotion engine.

[2260] Data processing and calculation: The emotion engine analyzes the data and recognizes the user's real-time emotional state.

[2261] Output: The emotional state is updated as a result of the analysis.

[2262] Specific behavior:

[2263] The user uses the camera of the terminal to capture his / her facial expression and also records voice information.

[2264] The server analyzes this data and recognizes a "smile" as "happiness."

[2265] Step 7: Create memorable content

[2266] Input: User's emotional state and analyzed past behavioral data.

[2267] Processing: The server generates appropriate memory content based on these data.

[2268] Data processing and calculation: Select the most suitable content from past data based on emotional state.

[2269] Output: The generated memories content.

[2270] Specific behavior:

[2271] The server selects photos of past family trips based on the user's "mild depression."

[2272] Pick out a few fun photos from the "Summer Travel 2019" folder.

[2273] Step 8: Content suggestion and display

[2274] Input: Generated memories content.

[2275] Processing: The server sends the content to the terminal.

[2276] Data processing and calculation: Prepares the layout for the device to display the content.

[2277] Output: The memories content displayed to the user.

[2278] Specific behavior:

[2279] The device will display the suggested family vacation photos on a large screen, along with the message, "Check out these family vacation photos!"

[2280] Step 9: Gather user feedback

[2281] Input: User feedback.

[2282] Processing: The terminal displays a form for inputting feedback and sends the information entered by the user to the server.

[2283] Data processing and calculation: The server stores and organizes the feedback information.

[2284] Output: A dataset of feedback information.

[2285] Specific behavior:

[2286] The user enters "It was great" in the feedback form and presses the submit button.

[2287] This information is sent to a server and stored in a database.

[2288] Step 10: Feedback analysis and algorithmic model update

[2289] Input: Feedback information from the user.

[2290] Processing: The server analyzes the feedback information and updates the algorithmic model.

[2291] Data processing and calculation: Optimize the accuracy of sentiment analysis algorithms and content suggestion algorithms based on feedback information.

[2292] Output: The updated algorithm model.

[2293] Specific behavior:

[2294] The server analyzes the feedback such as "It was really good" and changes the parameters of the sentiment analysis algorithm.

[2295] Optimize the algorithm model for the next proposal.

[2296] Through the above process, this system can monitor the user's real-time emotional state and provide appropriate memorable content, thereby reducing the user's sense of loneliness and improving their sense of happiness.

[2297] (Application example 2)

[2298] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2299] In modern society, people often feel lonely and stressed, which can lead to poor mental health. To address this, a method is needed to provide appropriate memorable content based on the user's emotional state. However, existing systems are inadequate in accurately recognizing a user's emotional state in real time and providing optimal content based on that state. As a result, it is difficult to alleviate the loneliness and stress felt by users. Furthermore, effective feedback for users and optimization of content suggestion algorithms are not adequately implemented.

[2300] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a user authentication means, a data linking means, a past behavioral data collection means, a data analysis means, an emotional state analysis means, a memory content generation means, a user feedback means, a feedback analysis means, an algorithm model update means, a real-time emotion recognition means using a head-mounted display, and a means for displaying content based on the real-time emotional state. This makes it possible to accurately recognize the user's emotional state in real time and provide optimal memory content based on that state. By simply wearing the HMD, the user can experience memory content based on their own emotions in real time, effectively reducing feelings of loneliness and stress.

[2301] "User authentication means" is a function that verifies the legitimacy of a user by having the user access the system and enter authentication information.

[2302] "Data linking means" is a function that links an authenticated user's account with the system and shares necessary data.

[2303] "Means for collecting past behavioral data" is a function that collects users' past behavioral data (e.g., emails, photos, etc.).

[2304] "Data analysis means" is a function that analyzes collected data and extracts useful information.

[2305] The "emotional state analysis means" is a function that analyzes the emotional information provided by the user and determines the emotional state of the user.

[2306] The "memory content generating means" is a function that generates appropriate memory content based on the emotional state of the user.

[2307] The "user feedback means" is an interface for users to input feedback on the provided content.

[2308] The "feedback analysis means" is a function that analyzes feedback provided by users and uses it to improve the system.

[2309] The "algorithm model update means" is a function that updates the emotion analysis algorithm and the content suggestion algorithm based on the analysis results.

[2310] "Real-time emotion recognition means using a head-mounted display" is a function that uses sensors built into the head-mounted display to recognize the user's emotions in real time.

[2311] The "means for displaying content based on real-time emotional state" is a function for displaying optimal memory content on the head-mounted display in accordance with the emotional state recognized in real time.

[2312] This invention is a system that provides optimal memory content based on the user's emotional state, reducing feelings of loneliness and stress. It primarily uses a head-mounted display (HMD) to recognize the user's emotions in real time and display memory content accordingly.

[2313] System Configuration

[2314] The system includes the following main components:

[2315] User authentication method

[2316] Data linking method

[2317] Past behavioral data collection method

[2318] Data Analysis Methods

[2319] Emotional state analysis means

[2320] Means of creating memorable content

[2321] User feedback tools

[2322] Feedback Analysis Methods

[2323] Algorithmic model update method

[2324] Real-time emotion recognition using a head-mounted display

[2325] A means of displaying content based on real-time emotional state

[2326] Hardware and Software

[2327] Hardware

[2328] Head-mounted display (HMD): A device worn by the user that displays visual content, with built-in facial recognition cameras and heart rate sensors.

[2329] PC or smartphone: A device that connects to the HMD and processes data.

[2330] Server: Responsible for data collection, analysis, content generation, etc.

[2331] software

[2332] Python: A programming language.

[2333] OpenCV: A library for face recognition.

[2334] Yahoo SDK: A software development kit for working with Yahoo services.

[2335] EmotionEngine: An engine for emotion analysis.

[2336] ContentGenerator: A module that generates and suggests memory content.

[2337] Processing Details

[2338] 1. User Authentication

[2339] The server first authenticates the user based on the credentials provided by the user, which is done using the Yahoo SDK.

[2340] 2. Data Linkage

[2341] After authentication, the server links the user's account to the system and then prepares to collect past behavioral data.

[2342] 3. Collecting past behavioral data

[2343] The server collects data such as users' past emails, photo albums, notepads, etc. The collection period is, for example, the past five years.

[2344] 4. Data Analysis

[2345] The server analyzes the collected data and uses natural language processing algorithms and image recognition technology to extract specific event and emotion tags.

[2346] 5. Emotional State Analysis

[2347] The device (HMD) displays an interface for inputting the user's emotional state and transmits the collected emotional information to the server, which analyzes it and determines the user's current emotional state.

[2348] 6. Real-time emotion recognition

[2349] Using the cameras and sensors built into the HMD, we recognize the user's real-time emotional state using OpenCV and EmotionEngine.

[2350] 7. Creating memorable content

[2351] The server generates appropriate memory content based on the user's real-time emotional state. For example, if the user is feeling a little down, it will select videos of past family trips or photos of happy friends.

[2352] 8. Content Display

[2353] The content selected by the user is displayed through the HMD.

[2354] Specific examples

[2355] The user wears the HMD and the system analyzes the user's emotions in real time. If the user's emotional state is determined to be "a little depressed," the system displays videos of past family trips and happy photos with friends, helping the user to relax and feel happier.

[2356] Prompt Sentence Examples

[2357] "Generate relevant memory content by providing a user's facial image and emotional state as input. The user is in a slightly depressed state."

[2358] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2359] Step 1:

[2360] The user wears a head-mounted display (HMD) and accesses the system. The device displays an interface for the user to enter their Yahoo! ID and password. This input information is the input data for user authentication. The device sends the entered authentication information to the server, which then authenticates the user information using the Yahoo! SDK. If authentication is successful, the server starts a user session and obtains the user information. The authenticated user information is obtained as output data.

[2361] Step 2:

[2362] The server activates the data integration method that links the authenticated user's account with the system. At this point, it is ready to retrieve data from past Yahoo! services (e.g., email and photo albums). The behavioral data to be collected is integrated and becomes the input data for the next step.

[2363] Step 3:

[2364] The server uses a past behavioral data collection means to collect the user's past behavioral data (emails, photo albums, memo pads, etc.). The collection period is, for example, the past five years. The collected data becomes input data for the data analysis means. The collected behavioral data is obtained as output data.

[2365] Step 4:

[2366] The server analyzes the collected data using data analysis means. Specifically, it uses natural language processing algorithms and image recognition technology to extract specific events and emotion tags (e.g., "happiness," "sadness," etc.) from the data. The results of this analysis are input data for the emotion state analysis means. Emotion tags and event information are obtained as output data.

[2367] Step 5:

[2368] The device displays an interface for the user to input their emotional state. The user inputs emotional information such as "I'm a little down" or "I'm having a good day," and the device sends the information to the server. This emotional information becomes input data for the emotional state analysis means. The server uses an emotion engine to analyze the input emotional information and determine the user's current emotional state. The user's current emotional state is obtained as output data.

[2369] Step 6:

[2370] Using the camera and sensors built into the HMD, the server recognizes the user's real-time emotions. Using OpenCV and EmotionEngine, the server analyzes the user's facial images and heart rate data and updates their emotional state in real time. This real-time emotional data becomes the input data for the memory content generation means. The real-time emotional state is obtained as output data.

[2371] Step 7:

[2372] The server generates optimal memory content using the memory content generation means based on the user's real-time emotional state. For example, if the user is feeling a little down, the server selects videos of past family trips and photos of friends looking happy. This memory content becomes input data for the content display means. The generated memory content is obtained as output data.

[2373] Step 8:

[2374] The device presents the selected memory content to the user through the HMD. The user experiences this content visually in real time through the HMD. Intuitive messages and guides are displayed as an interface to the user. The output data becomes the content that the user experiences visually.

[2375] Step 9:

[2376] The terminal displays a form for the user to input feedback on the proposed content. The user inputs feedback such as "It was really good" or "It was average," and this information becomes input data for the user feedback means. The terminal sends the feedback information to the server, and the user's feedback information is obtained as output data.

[2377] Step 10:

[2378] The server uses the feedback analysis means to analyze the received feedback information and, based on this, updates and optimizes the models of the sentiment analysis algorithm and content recommendation algorithm. The algorithm model update means incorporates this feedback information to improve the accuracy of future suggestions. The updated algorithm model is obtained as output data.

[2379] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2380] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2381] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2382] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2383] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2384] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2385] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2386] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfor...

Claims

1. a user authentication means for verifying the identity of a user when the user accesses the system; A data linking means for linking the system with the user's existing data system; A past behavior data collection means for acquiring a user's past behavior history and content; A data analysis method that analyzes collected past data and extracts specific events and emotion tags, etc. emotional state analysis means for analyzing the emotional information and other data input by the user and evaluating the user's current emotional state; a memory content generating means for generating appropriate memory content for the user to view based on the analyzed data and the user's emotional state; a user feedback means for collecting feedback information provided by users; a feedback analysis means for analyzing the collected feedback information and using it to improve the system; an algorithm model update means for updating and optimizing the models of the sentiment analysis algorithm and the content suggestion algorithm based on the feedback analysis results; A system including:

2. 2. The system according to claim 1, further comprising means for selecting appropriate memory content according to the emotional state of the user.

3. an interface means for inputting user emotion information; 10. The system of claim 1, further comprising means for determining an emotional state based on the emotional information.

Images