System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting historical data of a user; means for analyzing the collected historical data to identify emotional moments or key events; means for generating a life story of the user based on the analyzed data; and means for presenting the generated life story to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional methods make it difficult for individuals to compile their past memories and experiences into rich stories, preventing people from easily creating moving mementos of special occasions and important life events. It's also difficult to extract meaningful content from vast amounts of digital data (photos, text, social media posts, etc.) to highlight emotional moments and significant events. This creates a need for a way to visually represent memories and present them in a format that's easy to share with family and friends. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for collecting a user's past data, a means for analyzing the collected past data and identifying emotional moments and important events, a means for generating a user's life story based on the analyzed data, and a means for presenting the generated life story to the user. Specifically, the system analyzes image data from the collected data and performs facial and object recognition, and also analyzes text data and performs sentiment analysis using natural language processing technology. Furthermore, the system incorporates a means for organizing the data chronologically based on timestamps and creating a timeline of important events. This mechanism allows a user's past memories and stories to be visually represented and preserved in a form that is easy to share across generations.

[0006] "User" refers to any individual or organization that uses this system.

[0007] "Historical data" refers to digital content such as photos, text messages, and social media posts that users have created, saved, or shared in the past.

[0008] "Means of collection" refers to the methods and technologies used to import historical data into the system with the user's permission.

[0009] "Means of analysis" refers to techniques used to analyze collected historical data and identify emotional moments and important events.

[0010] "Emotional moments" refer to moments or events in a user's past data that express particularly strong emotions.

[0011] "Missing moments" refer to notable events or experiences in a user's life.

[0012] "Means of generation" refers to methods and techniques for constructing a user's life story based on the analyzed data.

[0013] "Life Story" refers to narrative content generated based on a user's past data.

[0014] "Presentation means" refers to the methods and techniques used to visually display the generated life story to the user.

[0015] "Image data" refers to photos and video files taken and saved by users.

[0016] "Facial recognition" refers to the technology of identifying a person's face from image data and identifying that person.

[0017] "Object recognition" refers to the technology of identifying specific objects or scenes from image data.

[0018] "Text data" refers to data containing text information created, sent, or saved by a user.

[0019] "Natural language processing technology" refers to artificial intelligence technology that analyzes text data and understands meaning and emotions.

[0020] "Sentiment analysis" refers to the technology of analyzing user emotions from text data and image data.

[0021] A "timestamp" refers to information that indicates the date and time when data was created.

[0022] "Chronological organization" refers to methods or techniques for sorting data chronologically based on timestamps.

[0023] A "timeline" is a format that displays important events along a chronological order. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] overview

[0046] The present invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. Specific embodiments are described below.

[0047] Program Description

[0048] Data collection

[0049] 1. Obtaining User Consent

[0050] The user grants permission for data collection through the privacy settings screen.

[0051] The device sends the user's local data (photos and text messages) to the server.

[0052] 2. Collecting social media data

[0053] The server collects past posting data from users' social media accounts via API.

[0054] Data analysis

[0055] 3. Data Classification

[0056] The server categorizes the collected data into photos, text messages, and social media posts.

[0057] 4. Image Analysis

[0058] The server uses image analysis algorithms to perform facial and object recognition from the photos.

[0059] Identifying emotional moments or specific events (e.g., birthdays, weddings).

[0060] 5. Text Analysis

[0061] The server analyzes text data and social media posts using natural language processing (NLP).

[0062] Conduct sentiment analysis to assess the emotional weight of each post.

[0063] 6. Time Series Analysis

[0064] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[0065] Story Generation

[0066] 7. Story draft creation

[0067] The server creates a draft life story based on the analysis results and time-series data.

[0068] 8. Content Integration

[0069] The server combines photos, text, and social media posts, adding narration and text descriptions to complete the draft.

[0070] 9. Presentation to Users

[0071] The server transmits the generated draft to the user's terminal so that it can be displayed.

[0072] Users can review the draft and make edits or additions as needed.

[0073] Visual Representation

[0074] 10. Final story generation

[0075] The server generates the final life story that reflects the user's edits.

[0076] 11. Content Optimization

[0077] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device.

[0078] 12. Data Transmission and Playback

[0079] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[0080] Specific examples

[0081] Creating a family album

[0082] Here is an example of user C creating a family album.

[0083] 1. Data Collection

[0084] User C launches the application and consents to the collection of photos and text messages of his or her family over a specific period of time.

[0085] The terminal transmits the selected photos and messages to the server.

[0086] The server collects family-related posts from user C's SNS account.

[0087] 2. Data analysis and story generation

[0088] The server uses facial recognition to identify key family members and organize specific events (daily life, travel, birthdays, etc.).

[0089] The server drafts a life story, highlighting emotional events appropriate for a family album.

[0090] User C checks the draft and adds or edits photos or messages as necessary.

[0091] 3. Visual representation and sharing

[0092] The server converts the final story into a video format and transmits it to the terminal.

[0093] The device displays a slideshow for the whole family to enjoy.

[0094] The system allows users to create inspiring and personal family albums without any hassle.

[0095] The processing flow will be explained below.

[0096] Step 1:

[0097] Users launch the application and grant the necessary permissions through a privacy settings screen regarding data collection, including access to their photo library, collection of text messages, and linking to social media accounts.

[0098] Step 2:

[0099] The device searches for specified photos and text messages from its internal storage and sends them to a server via a secure protocol. It also connects with the user's social media accounts to collect past posting data.

[0100] Step 3:

[0101] The server receives the data and categorizes it into photos, text messages, social media posts, etc. During the categorization process, metadata (timestamps, tags, etc.) is added to each piece of data.

[0102] Step 4:

[0103] The server applies image analysis algorithms to the photo data to perform facial and object recognition, identify emotional expressions, and identify specific events (e.g., birthdays, weddings).

[0104] Step 5:

[0105] The server performs sentiment analysis on text messages and social media posts using natural language processing (NLP) techniques to identify emotions such as joy, sadness, and surprise, and rate the intensity of those emotions.

[0106] Step 6:

[0107] The server organizes the collected data chronologically based on timestamps, creating a timeline of significant events that provides a consistent view of the user's past events.

[0108] Step 7:

[0109] Based on the analysis and the timeline, the server generates a draft of the user's life story, highlighting emotional moments and important events.

[0110] Step 8:

[0111] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen to the user and provides an interface for editing and adding to the life story.

[0112] Step 9:

[0113] Users can review the draft and add or edit photos or messages as needed. Once editing is complete, users can save the final story.

[0114] Step 10:

[0115] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[0116] Step 11:

[0117] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device, which displays the final story and allows the user to play it.

[0118] Step 12:

[0119] The device launches a viewer to play the completed life story and visually displays it to the user, allowing the user to enjoy the moving life story.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Until now, there has been no way to automatically generate emotional and unique life stories based on a user's past data (photos, text messages, social media posts, etc.) and provide them as visually enjoyable content. This has meant that users have had to manually organize and edit large amounts of data, which has been a huge burden.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for collecting a user's past data, means for classifying the collected past data into photos, text messages, and social media posts, means for performing facial and object recognition on the classified data using an image analysis algorithm, means for performing sentiment analysis on the classified data using natural language processing technology, means for chronologically organizing the classified data based on the timestamps of each data and creating a timeline of important events, means for generating a draft of the user's life story based on the analyzed data, means for combining the generated draft of the life story with photos, text, and social media posts and adding narration and text descriptions to complete the generated draft of the life story, means for presenting the generated life story to the user and for editing and additions, means for converting the final life story into a visually appealing format (such as a slideshow or video) and transmitting it to a terminal, and means for visually displaying the generated life story to the user. This allows users to effortlessly create and enjoy moving and unique life stories.

[0125] "User" refers to the entity that uses this system to provide historical data and generate a life story.

[0126] "Past data" refers to information such as photos, text messages, and social media posts that users have created and saved in the past.

[0127] "Collection means" refers to the function for sending past data to a server with the user's consent.

[0128] "Classification means" refers to the function of classifying collected historical data into photos, text messages, social media posts, etc.

[0129] "Image analysis algorithm" refers to a computational method for performing face and object recognition on photographic data.

[0130] "Natural language processing technology" refers to technology for performing sentiment analysis and other linguistic analysis on text data.

[0131] A "timestamp" refers to information indicating the time at which each piece of data was generated.

[0132] A "timeline" is a chronological arrangement of important events based on timestamps.

[0133] "Story draft" refers to the outline of an initial life story generated based on the analyzed data.

[0134] "Content integration tools" refers to the ability to complete a story draft by adding photos, text, social media posts, narration, and text descriptions.

[0135] "Final Story" refers to the final version of your life story that reflects the user's edits.

[0136] "Visual format" refers to the format (slideshow or video) used to present the final story in an appealing way.

[0137] "Presentation means" refers to the function of displaying the generated life story to the user, allowing them to check, edit, and play it back.

[0138] overview

[0139] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. This system is implemented through the following three-way collaboration between a server, a device, and the user.

[0140] Data collection

[0141] Obtaining User Consent

[0142] The user consents to data collection on the application's privacy settings screen. For example, a confirmation screen asking, "Do you allow photos and messages to be collected?" is displayed. The device then sends the collected data (photos, text messages) from the user's local storage to the server. The data is securely transmitted using an encrypted communication protocol (e.g., HTTPS).

[0143] Social media data collection

[0144] The server obtains the user's social media account information and uses an API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The server then classifies and stores the collected social media data by user ID.

[0145] Data analysis

[0146] Data Classification

[0147] The server categorizes the received past data into photos, text messages, and social media posts, for example, based on the data's meta information (file format and extension).

[0148] Image analysis

[0149] The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to perform facial and object recognition on the photo data, tagging identified emotional moments and specific events (e.g., birthdays, weddings).

[0150] Text analytics

[0151] The server analyzes text data and social media posts using natural language processing (NLP) technology (e.g., Google Cloud Natural Language API) to perform sentiment analysis. It scores the sentiment (positive, negative, neutral) of each post and extracts sentiment trends.

[0152] Time Series Analysis

[0153] The server organizes the data chronologically based on the timestamp of each piece of data, creating a timeline of significant events, which can then be used to extract important events and sentiment trends.

[0154] Story Generation

[0155] Story draft creation

[0156] The server automatically generates a draft of the user's life story based on the data analysis results and time series data, creating a timeline such as "June 2022: The date I went on a family trip."

[0157] Content Integration

[0158] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft.

[0159] Presenting to the user

[0160] The server sends the generated draft of the life story to the user's device, allowing the user to review and edit the draft. The user can review the draft of the life story displayed on the device and add or edit photos or messages as necessary.

[0161] Visual Representation

[0162] Final story generation

[0163] The server generates a final life story that reflects the user's edits, recreating the user's most moving memories.

[0164] Content Optimization

[0165] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers the final story to the user's device.

[0166] Data transmission and playback

[0167] The device launches a viewer to play the received life story, allowing the user to visually check and play the final story.

[0168] Prompt Sentence Examples

[0169] Analyze users' social media posts, photos, and text messages to generate the perfect life story for their family album. Identify notable events (weddings, birthdays, trips, etc.) and reflect key emotional moments.

[0170] The present invention allows users to effortlessly create moving and unique life stories that are visually enjoyable.

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Step 1: Obtain user consent

[0173] The user consents to data collection on the application's privacy settings screen. This refers to the action of displaying a confirmation screen asking, "Do you allow photos and messages to be collected?" The input of this step is the user's consent information, and the output is the data collection permission settings.

[0174] Step 2: Collect local data

[0175] The device collects the consented data from local storage. In this case, the device retrieves photos and text messages from local folders. The input to this step is the user's data collection permissions and local data location information, and the output is the collected data itself.

[0176] Step 3: Send the data to the server

[0177] The terminal sends the collected data to the server using an encrypted communication protocol (e.g., HTTPS). The input of this step is the collected data, and the output is the data sent to the server.

[0178] Step 4: Collect social media data

[0179] The server obtains the user's social media account information and uses the API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The input for this step is the user's social media account information and API key, and the output is the collected social media data.

[0180] Step 5: Classify your data

[0181] The server categorizes the received historical data into photos, text messages, and social media posts. For example, it categorizes the data based on its meta information (file format and extension). The input of this step is the collected data, and the output is the categorized data.

[0182] Step 6: Analyze the photo data

[0183] The server uses image analysis algorithms to perform face and object recognition on the photo data, for example, using OpenCV or TensorFlow. The input of this step is the photo data, and the output is the resulting recognition information.

[0184] Step 7: Parse the text data

[0185] The server analyzes the text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis. The Google Cloud Natural Language API is used here. The input for this step is text data, and the output is the results of the sentiment analysis.

[0186] Step 8: Organize your data chronologically

[0187] The server organizes the data chronologically based on the timestamp of each piece of data and creates a timeline of important events. The input of this step is the time-stamped data, and the output is the timeline.

[0188] Step 9: Create a story draft

[0189] The server automatically generates a draft of the user's life story based on the data analysis results and time-series data. For example, it creates a timeline in the form of "June 2022: The date I went on a family trip." The input of this step is the analyzed data and timeline, and the output is a story draft.

[0190] Step 10: Integrate your content

[0191] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft. The input for this step is the story draft and related content, and the output is the completed draft.

[0192] Step 11: Present to the user

[0193] The server sends the generated life story draft to the user's device, allowing the user to review and edit the draft. The input of this step is the completed draft, and the output is the user's review result.

[0194] Step 12: Generate the final story

[0195] The server generates the final life story that reflects the user's edits, e.g., photos added by the user or text modified by the user. The input of this step is the user's edits, and the output is the final life story.

[0196] Step 13: Optimize your content

[0197] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers it. The input for this step is the final story, and the output is a visually appealing format.

[0198] Step 14: Send and play data

[0199] The device launches a viewer to play the received life story, allowing the user to visually confirm and play the final story. The input of this step is the transmitted life story, and the output is the played life story.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Conventional life story generation systems could collect and analyze users' past data and generate emotional stories, but they lacked the functionality to convert the data into a visually appealing format or upload the generated stories to content distribution services, resulting in output that was less appealing to users and making it difficult to share with other viewers.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story for the user based on the analyzed data, means for presenting the generated life story to the user, means for converting the generated life story into a visually appealing format, and means for uploading the generated life story to a content distribution service, thereby enabling high-quality and moving life stories to be automatically generated and easily shared with other viewers.

[0205] "Historical Data" refers to information that a user has generated or stored in the past, including photos, text messages, social media posts, etc.

[0206] "Collection means" refers to a device or program that includes technical procedures for transmitting a user's past data from the terminal to the server.

[0207] "Means of analysis" refers to techniques that analyze collected data and identify emotional moments and significant events within photos, text messages, and social media posts.

[0208] A "life story" is a narrative about a user's life and emotional moments, generated based on their past data.

[0209] The "presentation means" is a device or program for visually presenting the generated life story to the user.

[0210] "Visually appealing format" refers to a technique for presenting the generated life story in a visually appealing format (e.g., a slideshow or video).

[0211] A "content distribution service" is a service for providing the generated life story to other users via the Internet.

[0212] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) to generate and present an emotional and engaging life story. To implement this invention, a program that executes the following steps is required.

[0213] Hardware and Software Use

[0214] The server and terminal use the following hardware and software, respectively.

[0215] 1. Smartphone: A device used to collect user data.

[0216] 2. Server: A computer that analyzes data and generates life stories.

[0217] 3. Python: It is the primary programming language used for data analysis and video generation.

[0218] 4. PIL (Python Imaging Library): Software used for image processing.

[0219] 5. MoviePy: Software used for video editing.

[0220] 6. API: This is the interface used to collect social media data.

[0221] Data collection

[0222] The user launches the application and gives permission to collect data. This causes the smartphone (device) to send the user's local data (photos and text messages) to the server. At the same time, the server uses the SNS API to collect past posting data from the user's SNS account.

[0223] Data analysis

[0224] The server categorizes the collected data into photos, text messages, and social media posts. It uses image analysis algorithms (using PIL) to perform facial and object recognition in photos, and identifies emotional moments and specific events (e.g., birthdays, weddings). It also analyzes text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis and evaluate the emotional weight of each post.

[0225] Time Series Analysis

[0226] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[0227] Story Generation

[0228] The server creates a draft life story based on the analysis results and time-series data. It combines photos, text, and social media posts, and adds narration and text descriptions to complete the draft. It then presents the draft life story to the user, who can review it and make edits or additions as needed.

[0229] Visual Representation and Delivery

[0230] The server generates the final life story that reflects the user's edits. It converts the final story into a visually appealing format (slideshow or video) and sends it to the device. The device launches a viewer to play the generated life story and visually displays it to the user. It also connects to a content distribution service and uploads the generated life story, making it available to other viewers.

[0231] Specific examples

[0232] For example, if a user uses the system to think about family memories, the application will collect and analyze past photos, text messages, and social media posts related to the family. As a result, it will identify photos of family trips and birthdays as key events and add emotional text messages to the timeline, creating a moving life story. This life story is presented in video format, and users can play it on their smartphones or share it with others via content distribution services.

[0233] Prompt Sentence Examples

[0234] For this system, an example prompt for the generative AI model is as follows:

[0235] "Create a moving life story video using photos, text messages, and social media posts based on family memories."

[0236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0237] Step 1:

[0238] The user launches the application and gives permission to collect data. This causes the user's smartphone (device) to collect locally stored photos and text messages and send them to the server. The server uses the SNS API to retrieve the user's past posting data. The input is the user's consent, and the output is the transmission of data to the server.

[0239] Step 2:

[0240] The server categorizes the received historical data into photos, text messages, and social media posts. This categorization process stores each data in a corresponding folder or database. The input is the collected data, and the output is a categorized dataset.

[0241] Step 3:

[0242] The server uses PIL to perform image analysis on the photo data, identifying emotional moments and specific events through facial and object recognition. The input is the classified photo data, and the output is a list of analyzed events.

[0243] Step 4:

[0244] The server uses natural language processing (NLP) techniques to analyze text data and social media posts, performing sentiment analysis and assessing the emotional weight of each post. The input is the text data and social media posts, and the output is a set of emotional ratings.

[0245] Step 5:

[0246] The server organizes the analyzed photos, text messages, and social media posts chronologically to create a timeline of important events. The input is the analyzed dataset, and the output is the timeline.

[0247] Step 6:

[0248] The server creates a draft life story based on the analysis results and time series data. It combines photos, text, and social media posts, and adds narration and text descriptions. The input is a timeline, and the output is a draft life story.

[0249] Step 7:

[0250] The server presents the generated draft of the life story to the user's smartphone. The user reviews the draft and makes edits or additions as necessary. The input is the draft of the life story, and the output is the user's edited draft.

[0251] Step 8:

[0252] The server generates the final life story, reflecting the user's edits. This final story is then converted into a visually appealing format (slideshow or video). The input is the edited draft, and the output is the final life story video.

[0253] Step 9:

[0254] The server sends the generated final life story to the user's smartphone, where the user plays it. It then connects to a content distribution service and uploads the generated life story. The input is the final life story video, and the output is displaying it to the user and uploading it to the distribution service.

[0255] This allows users to auto-generate high-quality, inspiring life stories and easily share them with other viewers.

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] overview

[0258] This invention is a life story generation system that combines a life story engine that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) and recognizes the user's emotions. The system generates a life story that highlights emotional moments and important events and presents it visually.

[0259] Program Description

[0260] Data collection

[0261] 1. Obtaining User Consent

[0262] Users launch the application and grant the necessary permissions through a privacy settings screen for data collection and emotion recognition.

[0263] 2. Collecting local and social media data

[0264] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[0265] The server collects past posting data from users' social media accounts via API.

[0266] Data analysis

[0267] 3. Data Classification and Image Analysis

[0268] The server categorizes the collected data into photos, text messages, and social media posts.

[0269] Facial and object recognition is performed on the collected image data to identify emotional expressions and specific events.

[0270] 4. Analysis using text analysis and emotion engine

[0271] The server analyzes text data and social media posts using natural language processing (NLP) technology and performs sentiment analysis using an emotion engine.

[0272] The emotion engine identifies the emotional state of the text data and rates the intensity of the emotion.

[0273] 5. Time series analysis and assessment of emotional states

[0274] The server organizes the collected data chronologically based on timestamps, creating a timeline of important events.

[0275] The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of the life story.

[0276] Story Generation

[0277] 6. Draft the story and present it to users

[0278] The server generates a draft life story based on the analysis results and time series data, highlighting emotional moments and important events.

[0279] The terminal displays a preview screen to present the draft to the user and provides an interface that allows editing and additions.

[0280] 7. User editing and final story generation

[0281] Users can review the draft and add or edit photos or messages as needed.

[0282] The server generates the final life story, incorporating the user's edits, in a visually appealing format (slideshow or video) and incorporating emotional information into narration and sound effects.

[0283] Visual Representation

[0284] 8. Displaying the final story and real-time emotion recognition

[0285] The server sends the final story to the terminal.

[0286] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[0287] The emotion engine recognizes the user's real-time emotional responses as the life story plays and dynamically adjusts the story based on those responses.

[0288] Specific examples

[0289] User D's birthday album

[0290] Here is an example of user D creating a birthday album.

[0291] 1. Data Collection

[0292] User D allows the collection of birthday photos and messages.

[0293] The terminal transmits data for the specified period to the server.

[0294] 2. Data analysis and emotion evaluation

[0295] The server performs face recognition and object recognition on the photo data.

[0296] The server analyzes text data and social media posts and evaluates emotions using an emotion engine.

[0297] 3. Story Generation

[0298] The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[0299] User D checks the draft and edits the content as necessary.

[0300] 4. Visual representation and real-time adjustment

[0301] The server converts the final story into a video format and transmits it to the terminal.

[0302] The device plays the album at the birthday party.

[0303] The emotion engine recognizes user D's emotional reactions during playback and adjusts the story accordingly.

[0304] With this system, User D can create a touching and unique birthday commemorative album and share the moment with many people.

[0305] The processing flow will be explained below.

[0306] Step 1:

[0307] The user launches the application and grants the necessary permissions through a privacy settings screen regarding data collection and emotion recognition. The user approves access to the photo library, collection of text messages, and linking to social media accounts.

[0308] Step 2:

[0309] The device searches for the specified photos and text messages from its internal storage and transmits them to the server over a secure protocol.

[0310] Step 3:

[0311] The server collects past posting data from users' social media accounts via API, and the collected data is stored in a centralized database.

[0312] Step 4:

[0313] The server categorizes the data it receives, separating photos, text messages, and social media posts into their respective categories, and adds metadata (such as timestamps and tags) to each piece of data.

[0314] Step 5:

[0315] The server applies image analysis algorithms to the photo data to perform facial and object recognition, a process that identifies emotional expressions and specific events (such as birthdays and weddings).

[0316] Step 6:

[0317] The server analyzes text data and social media posts using natural language processing (NLP) technology, and then uses an emotion engine to perform sentiment analysis, identifying emotions such as joy, sadness, and surprise, and assessing their intensity.

[0318] Step 7:

[0319] The server organizes the data chronologically based on the timestamps of the collected data, then associates identified events with the emotion data to create a timeline of significant events.

[0320] Step 8:

[0321] The server generates a draft of the user's life story based on the analysis and time series data, designed to highlight emotional moments and important events.

[0322] Step 9:

[0323] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen so that the user can check the draft and provides an interface for editing and adding to it.

[0324] Step 10:

[0325] The user reviews the draft, adds or edits photos or messages as needed, and once the user has completed editing, requests the creation of the final story.

[0326] Step 11:

[0327] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[0328] Step 12:

[0329] The server converts the final story into a visually appealing format (e.g., slideshow or video) and sends it to the device, which launches a viewer to play the completed life story and visually display it to the user.

[0330] Step 13:

[0331] The emotion engine recognizes users' real-time emotional reactions while playing the life story, and dynamically adjusts the story accordingly based on the user's reactions, providing a more moving experience.

[0332] Example 2

[0333] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] In recent years, the widespread availability of digital data has made it easy to record individual users' life events, but methods for utilizing this data effectively and emotionally remain limited. In particular, there are no systems that can generate life stories that take the user's emotions into account and present them in a visually appealing manner. Furthermore, conventional systems have difficulty dynamically adjusting the story to reflect the user's real-time emotional reactions while the story is being played. This leads to the issue of a degraded user experience.

[0335] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for recognizing emotional responses in real time during playback of the user's life story and dynamically adjusting the story based on the responses. This allows the user to replay past memories while placing emphasis on emotions, and further enables a more personalized experience by dynamically adjusting the story while reflecting emotional responses in real time during playback of the life story.

[0336] "User's past data" refers to digital data such as photos, text messages, and social media posts that a user has created and saved in the past.

[0337] "Means of collection" refers to the technology and functionality used to obtain historical user data from designated data sources.

[0338] "Means of analysis" refers to the algorithms and techniques used to sort through collected historical data and identify emotional moments and significant events.

[0339] "Emotional moments" refer to points or events in the collected data that are presumed to have a strong emotional reaction from the user.

[0340] "Significant moments" are notable events or happenings in a user's life story.

[0341] "Life Story" refers to a continuous narrative reenactment generated from a user's past data and emotional moments and significant events.

[0342] "Presentation means" refers to the technology and methods for visually and audibly displaying and playing the generated life story to the user.

[0343] "Means for recognizing emotional responses in real time" refers to technology for capturing and analyzing users' emotional responses in real time while playing a life story.

[0344] "Means for dynamically adjusting the story based on the response" refers to technologies and algorithms that allow the content of a life story to be instantly changed or adjusted in response to emotional responses recognized in real time.

[0345] overview

[0346] This invention is a system that recognizes a user's emotions by collecting and analyzing their past data (photos, text messages, social media posts, etc.) and generates a life story. Using an emotion engine, the system generates a life story that highlights emotional moments and important events and visually presents it to the user.

[0347] System Configuration

[0348] This system mainly consists of a server, a terminal, and a user.

[0349] Hardware and software used

[0350] Hardware: high-performance computer servers, user devices (smartphones, tablets, PCs), web cameras

[0351] Software: Natural Language Processing (NLP) libraries (e.g., BERT model), facial recognition libraries (e.g., OpenCV), data collection APIs, emotion engines

[0352] Processing Details

[0353] 1. Obtaining User Consent

[0354] Users launch the system application and grant the necessary permissions for data collection and emotion recognition through the privacy settings screen. Based on this permission, the system collects the user's past data (photos, text messages, social media posts, etc.).

[0355] 2. Data Collection

[0356] The device collects photos and text messages stored on the device and sends them to a server, which then collects past posting data from the user's social media accounts (e.g., Facebook, Twitter) via API.

[0357] 3. Data Analysis

[0358] The server categorizes the collected data into images, text messages, and social media posts, and performs facial and object recognition on the image data to identify emotional expressions and events.

[0359] The server analyzes text data and social media posts using natural language processing (NLP) and performs sentiment analysis using an emotion engine, specifically using the BERT model to evaluate the emotional state of the text.

[0360] 4. Time series analysis and emotional state assessment

[0361] The server organizes the collected data chronologically based on timestamps to create a timeline of important events. The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of their life story.

[0362] 5. Story Generation

[0363] The server generates a draft life story based on the analysis and time series data, highlighting emotional moments and important events.

[0364] The device presents the draft to the user and provides a preview screen where edits and content can be added.

[0365] 6. Editing a User

[0366] The user can review the draft and add or edit photos or text messages as needed. The device provides an interface to support these edits.

[0367] 7. Final story generation

[0368] The server generates the final life story, incorporating the user's edits, in a visually appealing format (e.g., slideshow or video), incorporating emotional information into narration and sound effects.

[0369] 8. Visual representation and real-time emotion recognition

[0370] The server transmits the final story to the terminal, and the terminal starts a viewer for playing the received life story.

[0371] The device uses a webcam and sensors to capture the user's real-time emotional responses.

[0372] The server performs real-time emotion recognition and dynamically adjusts the story according to the user's emotional response.

[0373] Specific examples

[0374] The following is a specific example of user D creating a birthday commemorative album.

[0375] 1. Data collection: User D allows the collection of photos and messages taken on his birthday. The device sends the data for a specified period to the server.

[0376] 2. Data analysis: The server performs facial and object recognition on the photo data and analyzes the text data using an emotion engine.

[0377] 3. Story generation: The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[0378] 4. User Edit: User D previews the draft and edits the content.

[0379] 5. Visual representation and real-time adjustment: The server converts the final story into a video format and sends it to the device. The album is played at the birthday party, and emotional responses are recognized in real time, adjusting the story accordingly.

[0380] The system allows users to replay past memories in an emotionally sensitive way, and even enjoy a personalized life story while reflecting their emotional responses in real time.

[0381] keyword

[0382] Generative AI model, prompt sentence

[0383] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0384] Step 1:

[0385] Obtaining User Consent

[0386] The user launches the application and provides permission for data collection and emotion recognition on the privacy settings screen.

[0387] Input: User consent information.

[0388] Output: Data collection and analysis permission flags.

[0389] What happens: The application displays a privacy settings screen to the user, asking for permission to collect data, and if consent is given, sets the permission flag.

[0390] Step 2:

[0391] Local and social media data collection

[0392] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[0393] The server uses an API to collect past posting data from the user's social media account.

[0394] Input: Photos, text messages, social media account information.

[0395] Output: Historical data sent to the server.

[0396] Specific operation: Scans specified folders and directories on the device and extracts relevant data. Social media data is uploaded to the server along with other collected data via an API call.

[0397] Step 3:

[0398] Data Classification

[0399] The server categorizes the received data into photos, text messages, and social media posts.

[0400] Input: Collected historical data.

[0401] Output: Categorized data (photos, text messages, social media posts).

[0402] Specific behavior: Classifies data into different categories based on file format and metadata.

[0403] Step 4:

[0404] Image analysis

[0405] The server performs facial and object recognition to identify emotional expressions and specific events from the image data.

[0406] Input: Classified photo data.

[0407] Output: Recognized emotional facial expressions and event information.

[0408] Specific operation: Uses face recognition libraries such as OpenCV to identify and classify faces and objects in photos.

[0409] Step 5:

[0410] Text analysis and sentiment engine analysis

[0411] The server uses natural language processing technology to analyze text data and social media posts, and performs sentiment analysis using an emotion engine.

[0412] Input: Text data, social media posts.

[0413] Output: Sentiment analysis results.

[0414] What it does: It uses NLP libraries such as the BERT model to identify the sentiment of a sentence and assess its emotional state.

[0415] Step 6:

[0416] Time series analysis and emotional state assessment

[0417] The server organizes the data chronologically based on its timestamps, creating a timeline of significant events.

[0418] Input: Classified data, timestamp.

[0419] Output: A timeline of important events.

[0420] Specific operation: Sorts data based on timestamps to generate a time series of life events.

[0421] Step 7:

[0422] Story draft creation

[0423] The server generates a draft life story based on the analysis results and time-series data.

[0424] Input: Analysis results, timeline.

[0425] Output: Draft life story.

[0426] Specific operation: Based on the timeline and sentiment analysis results, the necessary text and images are placed and an initial draft is generated.

[0427] Step 8:

[0428] Presenting and editing a draft

[0429] The terminal presents the draft to the user and provides an interface where edits and additions can be made.

[0430] The user reviews the draft and makes edits or additions as necessary.

[0431] Input: Draft life story.

[0432] Output: User edited draft.

[0433] What it does: Displays a preview screen and provides an interface that allows the user to review and edit the draft.

[0434] Step 9:

[0435] Final story generation

[0436] The server generates the final life story that reflects the user's edits.

[0437] Input: Edited draft.

[0438] Output: Final life story.

[0439] What it does: Reflects user edits on the draft and converts it into the final format (slideshow or video).

[0440] Step 10:

[0441] Visual representation and real-time emotion recognition

[0442] The device visually displays the final story and uses a webcam and sensors to capture the user's real-time emotional responses.

[0443] The server performs real-time emotion recognition and dynamically adjusts the story.

[0444] Input: Final life story, real-time emotional responses of users.

[0445] Output: A tailored life story.

[0446] Specific behavior: Detects emotional changes during playback in real time and dynamically changes the content and effects of the story accordingly.

[0447] (Application example 2)

[0448] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0449] In physical stores, it is difficult to provide a personalized shopping experience that reflects a customer's past experiences and emotions. Furthermore, there is a lack of systems that can analyze a customer's past data and suggest products based on emotional moments and important events. Traditional physical stores lack established methods for making personalized product suggestions, making it difficult to improve customer satisfaction and stimulate purchasing motivation.

[0450] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for making personalized product suggestions using the collected data to individualize the user's shopping experience in a physical store. This makes it possible to provide a personalized shopping experience based on the customer's past experiences and emotions.

[0451] "User" refers to an individual who uses the System.

[0452] "Historical data" refers to information previously generated by a user, such as photos, text messages, social media posts, and purchase history.

[0453] "Means of collection" refers to software and hardware used to acquire and store users' historical data.

[0454] "Analysis tools" refers to algorithms and technologies used to analyze collected historical data and identify emotional moments and significant events.

[0455] "Emotional moments" are moments in user data that express particularly strong emotions.

[0456] "Significant events" refer to events that are particularly meaningful in the user's life story.

[0457] A "life story" is a narrative that depicts a series of events and emotions in a user's life, generated based on the user's past data.

[0458] "Presentation means" refers to devices or software for visually or audibly presenting the generated life story to the user.

[0459] "Brick and mortar store" refers to a physical store where customers can actually visit and purchase products.

[0460] "Shopping experience" refers to the series of actions and emotions that a user goes through when selecting and purchasing a product in a physical store.

[0461] "Personalized product suggestions" refers to suggesting the best products for individual users based on their past data and sentiment analysis.

[0462] The invention is based on a system that collects and analyzes a user's past data, identifies emotional moments and important events, and generates a life story based on that data. The system also has the ability to personalize the shopping experience in physical stores and provide personalized product recommendations to users.

[0463] Hardware and software used

[0464] This system uses the following hardware and software:

[0465] Hardware: Smartphones, servers

[0466] Software: Natural Language Processing (NLP) API, Face recognition API, Emotion recognition API

[0467] Data collection and analysis

[0468] 1. Data Collection

[0469] The device (smartphone) collects the user's past data (photos, text messages, social media posts), and after the user launches the application and gives permission for the handling of personal information, this data is sent to the server.

[0470] 2. Data Analysis

[0471] The server analyzes the collected data, performing facial and object recognition on the image data and sentiment analysis on the text data using natural language processing (NLP).

[0472] 3. Generating a life story

[0473] The server generates a user's life story based on the analysis results, identifying emotional moments and important events, organizing them chronologically, and creating a story draft, which the user can review and is given the option to edit or add to.

[0474] Personalized shopping experience

[0475] 4. Personalized product recommendations

[0476] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[0477] The device (smartphone) presents the product list to the user in the physical store, allowing the user to have a shopping experience that is linked to past memories.

[0478] Example Programs and Data

[0479] A concrete example will be given to explain how this system works.

[0480] Specific examples

[0481] User A visits a general store and launches an app on their smartphone. The app identifies emotionally significant moments based on past social media posts, photos, and text messages, and then makes personalized product recommendations to User A. For example, the product list might include a candle used on User A's birthday a year ago or a photo frame from their honeymoon.

[0482] Example prompt for generative AI model:

[0483] Collect users' past photos, text messages, and social media posts to generate personalized product listings based on:

[0484] 1. Products related to photos and messages of moments of joy

[0485] 2. Products related to happy events

[0486] 3. Products related to specific events (birthdays, holidays, etc.)

[0487] In this way, the system provides a personalized shopping experience that is relevant to the customer's past experiences and emotions, and customers can receive emotionally relevant product suggestions in the physical store, which can increase satisfaction and boost purchasing intent.

[0488] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0489] Step 1:

[0490] Data collection (user manual permission + device behavior)

[0491] The user launches the application and grants permission to process their personal information.

[0492] The device (smartphone) collects the user's past data (photos, text messages, social media posts) from its internal storage and social media accounts.

[0493] Input: User permission information, digital data stored on the device, data from SNS APIs

[0494] Output: Collected historical data (photos, messages, social media posts)

[0495] Step 2:

[0496] Sending data (terminal operation)

[0497] The terminal sends the collected past data to a server on the cloud.

[0498] Input: Historical data collected in the device

[0499] Output: Historical data sent to the server

[0500] Step 3:

[0501] Image data analysis (server operation)

[0502] The server analyzes the transmitted image data and performs facial and object recognition, thereby recognizing emotional moments and specific events.

[0503] Input: Collected image data

[0504] Output: Analysis of emotional moments and events

[0505] Step 4:

[0506] Analysis of text data (server operation)

[0507] The server analyzes the text data using natural language processing (NLP) and performs sentiment analysis, thereby identifying the emotional state and intensity of the text data.

[0508] Input: Collected text data

[0509] Output: Analysis results including emotional state and its intensity

[0510] Step 5:

[0511] Life story generation (server behavior)

[0512] The server organizes the analysis results chronologically and generates a draft life story that weights emotional moments and important events.

[0513] Input: Analyzed image data and text data results

[0514] Output: Draft life story

[0515] Step 6:

[0516] Presenting and editing life stories (device and user actions)

[0517] The device presents the draft to the user and displays a preview screen to give the user editing options.

[0518] Users can review the draft and add edits to the photo or message.

[0519] Input: Life Story Draft

[0520] Output: User edited life story

[0521] Step 7:

[0522] Generate final life story (server behavior)

[0523] The server converts the final life story, incorporating the user's edits, into a visually appealing format (slideshow or video).

[0524] Input: User-edited life story

[0525] Output: Final life story (slideshow and video)

[0526] Step 8:

[0527] Personalized product proposals (server and device operations)

[0528] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[0529] The terminal presents the product list to the user in the physical store, providing a shopping experience linked to past memories.

[0530] Input: Data for product recommendations based on sentiment analysis results and past experiences

[0531] Output: Personalized product list

[0532] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0533] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0534] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0535] [Second embodiment]

[0536] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0537] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0538] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0539] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0540] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0541] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0542] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0543] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0544] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0545] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0546] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0547] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0548] overview

[0549] The present invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. Specific embodiments are described below.

[0550] Program Description

[0551] Data collection

[0552] 1. Obtaining User Consent

[0553] The user grants permission for data collection through the privacy settings screen.

[0554] The device sends the user's local data (photos and text messages) to the server.

[0555] 2. Collecting social media data

[0556] The server collects past posting data from users' social media accounts via API.

[0557] Data analysis

[0558] 3. Data Classification

[0559] The server categorizes the collected data into photos, text messages, and social media posts.

[0560] 4. Image Analysis

[0561] The server uses image analysis algorithms to perform facial and object recognition from the photos.

[0562] Identifying emotional moments or specific events (e.g., birthdays, weddings).

[0563] 5. Text Analysis

[0564] The server analyzes text data and social media posts using natural language processing (NLP).

[0565] Conduct sentiment analysis to assess the emotional weight of each post.

[0566] 6. Time Series Analysis

[0567] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[0568] Story Generation

[0569] 7. Story draft creation

[0570] The server creates a draft life story based on the analysis results and time-series data.

[0571] 8. Content Integration

[0572] The server combines photos, text, and social media posts, adding narration and text descriptions to complete the draft.

[0573] 9. Presentation to Users

[0574] The server transmits the generated draft to the user's terminal so that it can be displayed.

[0575] Users can review the draft and make edits or additions as needed.

[0576] Visual Representation

[0577] 10. Final story generation

[0578] The server generates the final life story that reflects the user's edits.

[0579] 11. Content Optimization

[0580] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device.

[0581] 12. Data Transmission and Playback

[0582] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[0583] Specific examples

[0584] Creating a family album

[0585] Here is an example of user C creating a family album.

[0586] 1. Data Collection

[0587] User C launches the application and consents to the collection of photos and text messages of his or her family over a specific period of time.

[0588] The terminal transmits the selected photos and messages to the server.

[0589] The server collects family-related posts from user C's SNS account.

[0590] 2. Data analysis and story generation

[0591] The server uses facial recognition to identify key family members and organize specific events (daily life, travel, birthdays, etc.).

[0592] The server drafts a life story, highlighting emotional events appropriate for a family album.

[0593] User C checks the draft and adds or edits photos or messages as necessary.

[0594] 3. Visual representation and sharing

[0595] The server converts the final story into a video format and transmits it to the terminal.

[0596] The device displays a slideshow for the whole family to enjoy.

[0597] The system allows users to create inspiring and personal family albums without any hassle.

[0598] The processing flow will be explained below.

[0599] Step 1:

[0600] Users launch the application and grant the necessary permissions through a privacy settings screen regarding data collection, including access to their photo library, collection of text messages, and linking to social media accounts.

[0601] Step 2:

[0602] The device searches for specified photos and text messages from its internal storage and sends them to a server via a secure protocol. It also connects with the user's social media accounts to collect past posting data.

[0603] Step 3:

[0604] The server receives the data and categorizes it into photos, text messages, social media posts, etc. During the categorization process, metadata (timestamps, tags, etc.) is added to each piece of data.

[0605] Step 4:

[0606] The server applies image analysis algorithms to the photo data to perform facial and object recognition, identify emotional expressions, and identify specific events (e.g., birthdays, weddings).

[0607] Step 5:

[0608] The server performs sentiment analysis on text messages and social media posts using natural language processing (NLP) techniques to identify emotions such as joy, sadness, and surprise, and rate the intensity of those emotions.

[0609] Step 6:

[0610] The server organizes the collected data chronologically based on timestamps, creating a timeline of significant events that provides a consistent view of the user's past events.

[0611] Step 7:

[0612] Based on the analysis and the timeline, the server generates a draft of the user's life story, highlighting emotional moments and important events.

[0613] Step 8:

[0614] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen to the user and provides an interface for editing and adding to the life story.

[0615] Step 9:

[0616] Users can review the draft and add or edit photos or messages as needed. Once editing is complete, users can save the final story.

[0617] Step 10:

[0618] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[0619] Step 11:

[0620] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device, which displays the final story and allows the user to play it.

[0621] Step 12:

[0622] The device launches a viewer to play the completed life story and visually displays it to the user, allowing the user to enjoy the moving life story.

[0623] Example 1

[0624] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0625] Until now, there has been no way to automatically generate emotional and unique life stories based on a user's past data (photos, text messages, social media posts, etc.) and provide them as visually enjoyable content. This has meant that users have had to manually organize and edit large amounts of data, which has been a huge burden.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0627] In this invention, the server includes means for collecting a user's past data, means for classifying the collected past data into photos, text messages, and social media posts, means for performing facial and object recognition on the classified data using an image analysis algorithm, means for performing sentiment analysis on the classified data using natural language processing technology, means for chronologically organizing the classified data based on the timestamps of each data and creating a timeline of important events, means for generating a draft of the user's life story based on the analyzed data, means for combining the generated draft of the life story with photos, text, and social media posts and adding narration and text descriptions to complete the generated draft of the life story, means for presenting the generated life story to the user and for editing and additions, means for converting the final life story into a visually appealing format (such as a slideshow or video) and transmitting it to a terminal, and means for visually displaying the generated life story to the user. This allows users to effortlessly create and enjoy moving and unique life stories.

[0628] "User" refers to the entity that uses this system to provide historical data and generate a life story.

[0629] "Past data" refers to information such as photos, text messages, and social media posts that users have created and saved in the past.

[0630] "Collection means" refers to the function for sending past data to a server with the user's consent.

[0631] "Classification means" refers to the function of classifying collected historical data into photos, text messages, social media posts, etc.

[0632] "Image analysis algorithm" refers to a computational method for performing face and object recognition on photographic data.

[0633] "Natural language processing technology" refers to technology for performing sentiment analysis and other linguistic analysis on text data.

[0634] A "timestamp" refers to information indicating the time at which each piece of data was generated.

[0635] A "timeline" is a chronological arrangement of important events based on timestamps.

[0636] "Story draft" refers to the outline of an initial life story generated based on the analyzed data.

[0637] "Content integration tools" refers to the ability to complete a story draft by adding photos, text, social media posts, narration, and text descriptions.

[0638] "Final Story" refers to the final version of your life story that reflects the user's edits.

[0639] "Visual format" refers to the format (slideshow or video) used to present the final story in an appealing way.

[0640] "Presentation means" refers to the function of displaying the generated life story to the user, allowing them to check, edit, and play it back.

[0641] overview

[0642] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. This system is implemented through the following three-way collaboration between a server, a device, and the user.

[0643] Data collection

[0644] Obtaining User Consent

[0645] The user consents to data collection on the application's privacy settings screen. For example, a confirmation screen asking, "Do you allow photos and messages to be collected?" is displayed. The device then sends the collected data (photos, text messages) from the user's local storage to the server. The data is securely transmitted using an encrypted communication protocol (e.g., HTTPS).

[0646] Social media data collection

[0647] The server obtains the user's social media account information and uses an API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The server then classifies and stores the collected social media data by user ID.

[0648] Data analysis

[0649] Data Classification

[0650] The server categorizes the received past data into photos, text messages, and social media posts, for example, based on the data's meta information (file format and extension).

[0651] Image analysis

[0652] The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to perform facial and object recognition on the photo data, tagging identified emotional moments and specific events (e.g., birthdays, weddings).

[0653] Text analytics

[0654] The server analyzes text data and social media posts using natural language processing (NLP) technology (e.g., Google Cloud Natural Language API) to perform sentiment analysis. It scores the sentiment (positive, negative, neutral) of each post and extracts sentiment trends.

[0655] Time Series Analysis

[0656] The server organizes the data chronologically based on the timestamp of each piece of data, creating a timeline of significant events, which can then be used to extract important events and sentiment trends.

[0657] Story Generation

[0658] Story draft creation

[0659] The server automatically generates a draft of the user's life story based on the data analysis results and time series data, creating a timeline such as "June 2022: The date I went on a family trip."

[0660] Content Integration

[0661] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft.

[0662] Presenting to the user

[0663] The server sends the generated draft of the life story to the user's device, allowing the user to review and edit the draft. The user can review the draft of the life story displayed on the device and add or edit photos or messages as necessary.

[0664] Visual Representation

[0665] Final story generation

[0666] The server generates a final life story that reflects the user's edits, recreating the user's most moving memories.

[0667] Content Optimization

[0668] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers the final story to the user's device.

[0669] Data transmission and playback

[0670] The device launches a viewer to play the received life story, allowing the user to visually check and play the final story.

[0671] Prompt Sentence Examples

[0672] Analyze users' social media posts, photos, and text messages to generate the perfect life story for their family album. Identify notable events (weddings, birthdays, trips, etc.) and reflect key emotional moments.

[0673] The present invention allows users to effortlessly create moving and unique life stories that are visually enjoyable.

[0674] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0675] Step 1: Obtain user consent

[0676] The user consents to data collection on the application's privacy settings screen. This refers to the action of displaying a confirmation screen asking, "Do you allow photos and messages to be collected?" The input of this step is the user's consent information, and the output is the data collection permission settings.

[0677] Step 2: Collect local data

[0678] The device collects the consented data from local storage. In this case, the device retrieves photos and text messages from local folders. The input to this step is the user's data collection permissions and local data location information, and the output is the collected data itself.

[0679] Step 3: Send the data to the server

[0680] The terminal sends the collected data to the server using an encrypted communication protocol (e.g., HTTPS). The input of this step is the collected data, and the output is the data sent to the server.

[0681] Step 4: Collect social media data

[0682] The server obtains the user's social media account information and uses the API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The input for this step is the user's social media account information and API key, and the output is the collected social media data.

[0683] Step 5: Classify your data

[0684] The server categorizes the received historical data into photos, text messages, and social media posts. For example, it categorizes the data based on its meta information (file format and extension). The input of this step is the collected data, and the output is the categorized data.

[0685] Step 6: Analyze the photo data

[0686] The server uses image analysis algorithms to perform face and object recognition on the photo data, for example, using OpenCV or TensorFlow. The input of this step is the photo data, and the output is the resulting recognition information.

[0687] Step 7: Parse the text data

[0688] The server analyzes the text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis. The Google Cloud Natural Language API is used here. The input for this step is text data, and the output is the results of the sentiment analysis.

[0689] Step 8: Organize your data chronologically

[0690] The server organizes the data chronologically based on the timestamp of each piece of data and creates a timeline of important events. The input of this step is the time-stamped data, and the output is the timeline.

[0691] Step 9: Create a story draft

[0692] The server automatically generates a draft of the user's life story based on the data analysis results and time-series data. For example, it creates a timeline in the form of "June 2022: The date I went on a family trip." The input of this step is the analyzed data and timeline, and the output is a story draft.

[0693] Step 10: Integrate your content

[0694] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft. The input for this step is the story draft and related content, and the output is the completed draft.

[0695] Step 11: Present to the user

[0696] The server sends the generated life story draft to the user's device, allowing the user to review and edit the draft. The input of this step is the completed draft, and the output is the user's review result.

[0697] Step 12: Generate the final story

[0698] The server generates the final life story that reflects the user's edits, e.g., photos added by the user or text modified by the user. The input of this step is the user's edits, and the output is the final life story.

[0699] Step 13: Optimize your content

[0700] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers it. The input for this step is the final story, and the output is a visually appealing format.

[0701] Step 14: Send and play data

[0702] The device launches a viewer to play the received life story, allowing the user to visually confirm and play the final story. The input of this step is the transmitted life story, and the output is the played life story.

[0703] (Application example 1)

[0704] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] Conventional life story generation systems could collect and analyze users' past data and generate emotional stories, but they lacked the functionality to convert the data into a visually appealing format or upload the generated stories to content distribution services, resulting in output that was less appealing to users and making it difficult to share with other viewers.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0707] In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story for the user based on the analyzed data, means for presenting the generated life story to the user, means for converting the generated life story into a visually appealing format, and means for uploading the generated life story to a content distribution service, thereby enabling high-quality and moving life stories to be automatically generated and easily shared with other viewers.

[0708] "Historical Data" refers to information that a user has generated or stored in the past, including photos, text messages, social media posts, etc.

[0709] "Collection means" refers to a device or program that includes technical procedures for transmitting a user's past data from the terminal to the server.

[0710] "Means of analysis" refers to techniques that analyze collected data and identify emotional moments and significant events within photos, text messages, and social media posts.

[0711] A "life story" is a narrative about a user's life and emotional moments, generated based on their past data.

[0712] The "presentation means" is a device or program for visually presenting the generated life story to the user.

[0713] "Visually appealing format" refers to a technique for presenting the generated life story in a visually appealing format (e.g., a slideshow or video).

[0714] A "content distribution service" is a service for providing the generated life story to other users via the Internet.

[0715] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) to generate and present an emotional and engaging life story. To implement this invention, a program that executes the following steps is required.

[0716] Hardware and Software Use

[0717] The server and terminal use the following hardware and software, respectively.

[0718] 1. Smartphone: A device used to collect user data.

[0719] 2. Server: A computer that analyzes data and generates life stories.

[0720] 3. Python: It is the primary programming language used for data analysis and video generation.

[0721] 4. PIL (Python Imaging Library): Software used for image processing.

[0722] 5. MoviePy: Software used for video editing.

[0723] 6. API: This is the interface used to collect social media data.

[0724] Data collection

[0725] The user launches the application and gives permission to collect data. This causes the smartphone (device) to send the user's local data (photos and text messages) to the server. At the same time, the server uses the SNS API to collect past posting data from the user's SNS account.

[0726] Data analysis

[0727] The server categorizes the collected data into photos, text messages, and social media posts. It uses image analysis algorithms (using PIL) to perform facial and object recognition in photos, and identifies emotional moments and specific events (e.g., birthdays, weddings). It also analyzes text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis and evaluate the emotional weight of each post.

[0728] Time Series Analysis

[0729] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[0730] Story Generation

[0731] The server creates a draft life story based on the analysis results and time-series data. It combines photos, text, and social media posts, and adds narration and text descriptions to complete the draft. It then presents the draft life story to the user, who can review it and make edits or additions as needed.

[0732] Visual Representation and Delivery

[0733] The server generates the final life story that reflects the user's edits. It converts the final story into a visually appealing format (slideshow or video) and sends it to the device. The device launches a viewer to play the generated life story and visually displays it to the user. It also connects to a content distribution service and uploads the generated life story, making it available to other viewers.

[0734] Specific examples

[0735] For example, if a user uses the system to think about family memories, the application will collect and analyze past photos, text messages, and social media posts related to the family. As a result, it will identify photos of family trips and birthdays as key events and add emotional text messages to the timeline, creating a moving life story. This life story is presented in video format, and users can play it on their smartphones or share it with others via content distribution services.

[0736] Prompt Sentence Examples

[0737] For this system, an example prompt for the generative AI model is as follows:

[0738] "Create a moving life story video using photos, text messages, and social media posts based on family memories."

[0739] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0740] Step 1:

[0741] The user launches the application and gives permission to collect data. This causes the user's smartphone (device) to collect locally stored photos and text messages and send them to the server. The server uses the SNS API to retrieve the user's past posting data. The input is the user's consent, and the output is the transmission of data to the server.

[0742] Step 2:

[0743] The server categorizes the received historical data into photos, text messages, and social media posts. This categorization process stores each data in a corresponding folder or database. The input is the collected data, and the output is a categorized dataset.

[0744] Step 3:

[0745] The server uses PIL to perform image analysis on the photo data, identifying emotional moments and specific events through facial and object recognition. The input is the classified photo data, and the output is a list of analyzed events.

[0746] Step 4:

[0747] The server uses natural language processing (NLP) techniques to analyze text data and social media posts, performing sentiment analysis and assessing the emotional weight of each post. The input is the text data and social media posts, and the output is a set of emotional ratings.

[0748] Step 5:

[0749] The server organizes the analyzed photos, text messages, and social media posts chronologically to create a timeline of important events. The input is the analyzed dataset, and the output is the timeline.

[0750] Step 6:

[0751] The server creates a draft life story based on the analysis results and time series data. It combines photos, text, and social media posts, and adds narration and text descriptions. The input is a timeline, and the output is a draft life story.

[0752] Step 7:

[0753] The server presents the generated draft of the life story to the user's smartphone. The user reviews the draft and makes edits or additions as necessary. The input is the draft of the life story, and the output is the user's edited draft.

[0754] Step 8:

[0755] The server generates the final life story, reflecting the user's edits. This final story is then converted into a visually appealing format (slideshow or video). The input is the edited draft, and the output is the final life story video.

[0756] Step 9:

[0757] The server sends the generated final life story to the user's smartphone, where the user plays it. It then connects to a content distribution service and uploads the generated life story. The input is the final life story video, and the output is displaying it to the user and uploading it to the distribution service.

[0758] This allows users to auto-generate high-quality, inspiring life stories and easily share them with other viewers.

[0759] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0760] overview

[0761] This invention is a life story generation system that combines a life story engine that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) and recognizes the user's emotions. The system generates a life story that highlights emotional moments and important events and presents it visually.

[0762] Program Description

[0763] Data collection

[0764] 1. Obtaining User Consent

[0765] Users launch the application and grant the necessary permissions through a privacy settings screen for data collection and emotion recognition.

[0766] 2. Collecting local and social media data

[0767] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[0768] The server collects past posting data from users' social media accounts via API.

[0769] Data analysis

[0770] 3. Data Classification and Image Analysis

[0771] The server categorizes the collected data into photos, text messages, and social media posts.

[0772] Facial and object recognition is performed on the collected image data to identify emotional expressions and specific events.

[0773] 4. Analysis using text analysis and emotion engine

[0774] The server analyzes text data and social media posts using natural language processing (NLP) technology and performs sentiment analysis using an emotion engine.

[0775] The emotion engine identifies the emotional state of the text data and rates the intensity of the emotion.

[0776] 5. Time series analysis and assessment of emotional states

[0777] The server organizes the collected data chronologically based on timestamps, creating a timeline of important events.

[0778] The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of the life story.

[0779] Story Generation

[0780] 6. Draft the story and present it to users

[0781] The server generates a draft life story based on the analysis results and time series data, highlighting emotional moments and important events.

[0782] The terminal displays a preview screen to present the draft to the user and provides an interface that allows editing and additions.

[0783] 7. User editing and final story generation

[0784] Users can review the draft and add or edit photos or messages as needed.

[0785] The server generates the final life story, incorporating the user's edits, in a visually appealing format (slideshow or video) and incorporating emotional information into narration and sound effects.

[0786] Visual Representation

[0787] 8. Displaying the final story and real-time emotion recognition

[0788] The server sends the final story to the terminal.

[0789] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[0790] The emotion engine recognizes the user's real-time emotional responses as the life story plays and dynamically adjusts the story based on those responses.

[0791] Specific examples

[0792] User D's birthday album

[0793] Here is an example of user D creating a birthday album.

[0794] 1. Data Collection

[0795] User D allows the collection of birthday photos and messages.

[0796] The terminal transmits data for the specified period to the server.

[0797] 2. Data analysis and emotion evaluation

[0798] The server performs face recognition and object recognition on the photo data.

[0799] The server analyzes text data and social media posts and evaluates emotions using an emotion engine.

[0800] 3. Story Generation

[0801] The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[0802] User D checks the draft and edits the content as necessary.

[0803] 4. Visual representation and real-time adjustment

[0804] The server converts the final story into a video format and transmits it to the terminal.

[0805] The device plays the album at the birthday party.

[0806] The emotion engine recognizes user D's emotional reactions during playback and adjusts the story accordingly.

[0807] With this system, User D can create a touching and unique birthday commemorative album and share the moment with many people.

[0808] The processing flow will be explained below.

[0809] Step 1:

[0810] The user launches the application and grants the necessary permissions through a privacy settings screen regarding data collection and emotion recognition. The user approves access to the photo library, collection of text messages, and linking to social media accounts.

[0811] Step 2:

[0812] The device searches for the specified photos and text messages from its internal storage and transmits them to the server over a secure protocol.

[0813] Step 3:

[0814] The server collects past posting data from users' social media accounts via API, and the collected data is stored in a centralized database.

[0815] Step 4:

[0816] The server categorizes the data it receives, separating photos, text messages, and social media posts into their respective categories, and adds metadata (such as timestamps and tags) to each piece of data.

[0817] Step 5:

[0818] The server applies image analysis algorithms to the photo data to perform facial and object recognition, a process that identifies emotional expressions and specific events (such as birthdays and weddings).

[0819] Step 6:

[0820] The server analyzes text data and social media posts using natural language processing (NLP) technology, and then uses an emotion engine to perform sentiment analysis, identifying emotions such as joy, sadness, and surprise, and assessing their intensity.

[0821] Step 7:

[0822] The server organizes the data chronologically based on the timestamps of the collected data, then associates identified events with the emotion data to create a timeline of significant events.

[0823] Step 8:

[0824] The server generates a draft of the user's life story based on the analysis and time series data, designed to highlight emotional moments and important events.

[0825] Step 9:

[0826] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen so that the user can check the draft and provides an interface for editing and adding to it.

[0827] Step 10:

[0828] The user reviews the draft, adds or edits photos or messages as needed, and once the user has completed editing, requests the creation of the final story.

[0829] Step 11:

[0830] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[0831] Step 12:

[0832] The server converts the final story into a visually appealing format (e.g., slideshow or video) and sends it to the device, which launches a viewer to play the completed life story and visually display it to the user.

[0833] Step 13:

[0834] The emotion engine recognizes users' real-time emotional reactions while playing the life story, and dynamically adjusts the story accordingly based on the user's reactions, providing a more moving experience.

[0835] Example 2

[0836] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0837] In recent years, the widespread availability of digital data has made it easy to record individual users' life events, but methods for utilizing this data effectively and emotionally remain limited. In particular, there are no systems that can generate life stories that take the user's emotions into account and present them in a visually appealing manner. Furthermore, conventional systems have difficulty dynamically adjusting the story to reflect the user's real-time emotional reactions while the story is being played. This leads to the issue of a degraded user experience.

[0838] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for recognizing emotional responses in real time during playback of the user's life story and dynamically adjusting the story based on the responses. This allows the user to replay past memories while placing emphasis on emotions, and further enables a more personalized experience by dynamically adjusting the story while reflecting emotional responses in real time during playback of the life story.

[0839] "User's past data" refers to digital data such as photos, text messages, and social media posts that a user has created and saved in the past.

[0840] "Means of collection" refers to the technology and functionality used to obtain historical user data from designated data sources.

[0841] "Means of analysis" refers to the algorithms and techniques used to sort through collected historical data and identify emotional moments and significant events.

[0842] "Emotional moments" refer to points or events in the collected data that are presumed to have a strong emotional reaction from the user.

[0843] "Significant moments" are notable events or happenings in a user's life story.

[0844] "Life Story" refers to a continuous narrative reenactment generated from a user's past data and emotional moments and significant events.

[0845] "Presentation means" refers to the technology and methods for visually and audibly displaying and playing the generated life story to the user.

[0846] "Means for recognizing emotional responses in real time" refers to technology for capturing and analyzing users' emotional responses in real time while playing a life story.

[0847] "Means for dynamically adjusting the story based on the response" refers to technologies and algorithms that allow the content of a life story to be instantly changed or adjusted in response to emotional responses recognized in real time.

[0848] overview

[0849] This invention is a system that recognizes a user's emotions by collecting and analyzing their past data (photos, text messages, social media posts, etc.) and generates a life story. Using an emotion engine, the system generates a life story that highlights emotional moments and important events and visually presents it to the user.

[0850] System Configuration

[0851] This system mainly consists of a server, a terminal, and a user.

[0852] Hardware and software used

[0853] Hardware: high-performance computer servers, user devices (smartphones, tablets, PCs), web cameras

[0854] Software: Natural Language Processing (NLP) libraries (e.g., BERT model), facial recognition libraries (e.g., OpenCV), data collection APIs, emotion engines

[0855] Processing Details

[0856] 1. Obtaining User Consent

[0857] Users launch the system application and grant the necessary permissions for data collection and emotion recognition through the privacy settings screen. Based on this permission, the system collects the user's past data (photos, text messages, social media posts, etc.).

[0858] 2. Data Collection

[0859] The device collects photos and text messages stored on the device and sends them to a server, which then collects past posting data from the user's social media accounts (e.g., Facebook, Twitter) via API.

[0860] 3. Data Analysis

[0861] The server categorizes the collected data into images, text messages, and social media posts, and performs facial and object recognition on the image data to identify emotional expressions and events.

[0862] The server analyzes text data and social media posts using natural language processing (NLP) and performs sentiment analysis using an emotion engine, specifically using the BERT model to evaluate the emotional state of the text.

[0863] 4. Time series analysis and emotional state assessment

[0864] The server organizes the collected data chronologically based on timestamps to create a timeline of important events. The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of their life story.

[0865] 5. Story Generation

[0866] The server generates a draft life story based on the analysis and time series data, highlighting emotional moments and important events.

[0867] The device presents the draft to the user and provides a preview screen where edits and content can be added.

[0868] 6. Editing a User

[0869] The user can review the draft and add or edit photos or text messages as needed. The device provides an interface to support these edits.

[0870] 7. Final story generation

[0871] The server generates the final life story, incorporating the user's edits, in a visually appealing format (e.g., slideshow or video), incorporating emotional information into narration and sound effects.

[0872] 8. Visual representation and real-time emotion recognition

[0873] The server transmits the final story to the terminal, and the terminal starts a viewer for playing the received life story.

[0874] The device uses a webcam and sensors to capture the user's real-time emotional responses.

[0875] The server performs real-time emotion recognition and dynamically adjusts the story according to the user's emotional response.

[0876] Specific examples

[0877] The following is a specific example of user D creating a birthday commemorative album.

[0878] 1. Data collection: User D allows the collection of photos and messages taken on his birthday. The device sends the data for a specified period to the server.

[0879] 2. Data analysis: The server performs facial and object recognition on the photo data and analyzes the text data using an emotion engine.

[0880] 3. Story generation: The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[0881] 4. User Edit: User D previews the draft and edits the content.

[0882] 5. Visual representation and real-time adjustment: The server converts the final story into a video format and sends it to the device. The album is played at the birthday party, and emotional responses are recognized in real time, adjusting the story accordingly.

[0883] The system allows users to replay past memories in an emotionally sensitive way, and even enjoy a personalized life story while reflecting their emotional responses in real time.

[0884] keyword

[0885] Generative AI model, prompt sentence

[0886] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0887] Step 1:

[0888] Obtaining User Consent

[0889] The user launches the application and provides permission for data collection and emotion recognition on the privacy settings screen.

[0890] Input: User consent information.

[0891] Output: Data collection and analysis permission flags.

[0892] What happens: The application displays a privacy settings screen to the user, asking for permission to collect data, and if consent is given, sets the permission flag.

[0893] Step 2:

[0894] Local and social media data collection

[0895] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[0896] The server uses an API to collect past posting data from the user's social media account.

[0897] Input: Photos, text messages, social media account information.

[0898] Output: Historical data sent to the server.

[0899] Specific operation: Scans specified folders and directories on the device and extracts relevant data. Social media data is uploaded to the server along with other collected data via an API call.

[0900] Step 3:

[0901] Data Classification

[0902] The server categorizes the received data into photos, text messages, and social media posts.

[0903] Input: Collected historical data.

[0904] Output: Categorized data (photos, text messages, social media posts).

[0905] Specific behavior: Classifies data into different categories based on file format and metadata.

[0906] Step 4:

[0907] Image analysis

[0908] The server performs facial and object recognition to identify emotional expressions and specific events from the image data.

[0909] Input: Classified photo data.

[0910] Output: Recognized emotional facial expressions and event information.

[0911] Specific operation: Uses face recognition libraries such as OpenCV to identify and classify faces and objects in photos.

[0912] Step 5:

[0913] Text analysis and sentiment engine analysis

[0914] The server uses natural language processing technology to analyze text data and social media posts, and performs sentiment analysis using an emotion engine.

[0915] Input: Text data, social media posts.

[0916] Output: Sentiment analysis results.

[0917] What it does: It uses NLP libraries such as the BERT model to identify the sentiment of a sentence and assess its emotional state.

[0918] Step 6:

[0919] Time series analysis and emotional state assessment

[0920] The server organizes the data chronologically based on its timestamps, creating a timeline of significant events.

[0921] Input: Classified data, timestamp.

[0922] Output: A timeline of important events.

[0923] Specific operation: Sorts data based on timestamps to generate a time series of life events.

[0924] Step 7:

[0925] Story draft creation

[0926] The server generates a draft life story based on the analysis results and time-series data.

[0927] Input: Analysis results, timeline.

[0928] Output: Draft life story.

[0929] Specific operation: Based on the timeline and sentiment analysis results, the necessary text and images are placed and an initial draft is generated.

[0930] Step 8:

[0931] Presenting and editing a draft

[0932] The terminal presents the draft to the user and provides an interface where edits and additions can be made.

[0933] The user reviews the draft and makes edits or additions as necessary.

[0934] Input: Draft life story.

[0935] Output: User edited draft.

[0936] What it does: Displays a preview screen and provides an interface that allows the user to review and edit the draft.

[0937] Step 9:

[0938] Final story generation

[0939] The server generates the final life story that reflects the user's edits.

[0940] Input: Edited draft.

[0941] Output: Final life story.

[0942] What it does: Reflects user edits on the draft and converts it into the final format (slideshow or video).

[0943] Step 10:

[0944] Visual representation and real-time emotion recognition

[0945] The device visually displays the final story and uses a webcam and sensors to capture the user's real-time emotional responses.

[0946] The server performs real-time emotion recognition and dynamically adjusts the story.

[0947] Input: Final life story, real-time emotional responses of users.

[0948] Output: A tailored life story.

[0949] Specific behavior: Detects emotional changes during playback in real time and dynamically changes the content and effects of the story accordingly.

[0950] (Application example 2)

[0951] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0952] In physical stores, it is difficult to provide a personalized shopping experience that reflects a customer's past experiences and emotions. Furthermore, there is a lack of systems that can analyze a customer's past data and suggest products based on emotional moments and important events. Traditional physical stores lack established methods for making personalized product suggestions, making it difficult to improve customer satisfaction and stimulate purchasing motivation.

[0953] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for making personalized product suggestions using the collected data to individualize the user's shopping experience in a physical store. This makes it possible to provide a personalized shopping experience based on the customer's past experiences and emotions.

[0954] "User" refers to an individual who uses the System.

[0955] "Historical data" refers to information previously generated by a user, such as photos, text messages, social media posts, and purchase history.

[0956] "Means of collection" refers to software and hardware used to acquire and store users' historical data.

[0957] "Analysis tools" refers to algorithms and technologies used to analyze collected historical data and identify emotional moments and significant events.

[0958] "Emotional moments" are moments in user data that express particularly strong emotions.

[0959] "Significant events" refer to events that are particularly meaningful in the user's life story.

[0960] A "life story" is a narrative that depicts a series of events and emotions in a user's life, generated based on the user's past data.

[0961] "Presentation means" refers to devices or software for visually or audibly presenting the generated life story to the user.

[0962] "Brick and mortar store" refers to a physical store where customers can actually visit and purchase products.

[0963] "Shopping experience" refers to the series of actions and emotions that a user goes through when selecting and purchasing a product in a physical store.

[0964] "Personalized product suggestions" refers to suggesting the best products for individual users based on their past data and sentiment analysis.

[0965] The invention is based on a system that collects and analyzes a user's past data, identifies emotional moments and important events, and generates a life story based on that data. The system also has the ability to personalize the shopping experience in physical stores and provide personalized product recommendations to users.

[0966] Hardware and software used

[0967] This system uses the following hardware and software:

[0968] Hardware: Smartphones, servers

[0969] Software: Natural Language Processing (NLP) API, Face recognition API, Emotion recognition API

[0970] Data collection and analysis

[0971] 1. Data Collection

[0972] The device (smartphone) collects the user's past data (photos, text messages, social media posts), and after the user launches the application and gives permission for the handling of personal information, this data is sent to the server.

[0973] 2. Data Analysis

[0974] The server analyzes the collected data, performing facial and object recognition on the image data and sentiment analysis on the text data using natural language processing (NLP).

[0975] 3. Generating a life story

[0976] The server generates a user's life story based on the analysis results, identifying emotional moments and important events, organizing them chronologically, and creating a story draft, which the user can review and is given the option to edit or add to.

[0977] Personalized shopping experience

[0978] 4. Personalized product recommendations

[0979] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[0980] The device (smartphone) presents the product list to the user in the physical store, allowing the user to have a shopping experience that is linked to past memories.

[0981] Example Programs and Data

[0982] A concrete example will be given to explain how this system works.

[0983] Specific examples

[0984] User A visits a general store and launches an app on their smartphone. The app identifies emotionally significant moments based on past social media posts, photos, and text messages, and then makes personalized product recommendations to User A. For example, the product list might include a candle used on User A's birthday a year ago or a photo frame from their honeymoon.

[0985] Example prompt for generative AI model:

[0986] Collect users' past photos, text messages, and social media posts to generate personalized product listings based on:

[0987] 1. Products related to photos and messages of moments of joy

[0988] 2. Products related to happy events

[0989] 3. Products related to specific events (birthdays, holidays, etc.)

[0990] In this way, the system provides a personalized shopping experience that is relevant to the customer's past experiences and emotions, and customers can receive emotionally relevant product suggestions in the physical store, which can increase satisfaction and boost purchasing intent.

[0991] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0992] Step 1:

[0993] Data collection (user manual permission + device behavior)

[0994] The user launches the application and grants permission to process their personal information.

[0995] The device (smartphone) collects the user's past data (photos, text messages, social media posts) from its internal storage and social media accounts.

[0996] Input: User permission information, digital data stored on the device, data from SNS APIs

[0997] Output: Collected historical data (photos, messages, social media posts)

[0998] Step 2:

[0999] Sending data (terminal operation)

[1000] The terminal sends the collected past data to a server on the cloud.

[1001] Input: Historical data collected in the device

[1002] Output: Historical data sent to the server

[1003] Step 3:

[1004] Image data analysis (server operation)

[1005] The server analyzes the transmitted image data and performs facial and object recognition, thereby recognizing emotional moments and specific events.

[1006] Input: Collected image data

[1007] Output: Analysis of emotional moments and events

[1008] Step 4:

[1009] Analysis of text data (server operation)

[1010] The server analyzes the text data using natural language processing (NLP) and performs sentiment analysis, thereby identifying the emotional state and intensity of the text data.

[1011] Input: Collected text data

[1012] Output: Analysis results including emotional state and its intensity

[1013] Step 5:

[1014] Life story generation (server behavior)

[1015] The server organizes the analysis results chronologically and generates a draft life story that weights emotional moments and important events.

[1016] Input: Analyzed image data and text data results

[1017] Output: Draft life story

[1018] Step 6:

[1019] Presenting and editing life stories (device and user actions)

[1020] The device presents the draft to the user and displays a preview screen to give the user editing options.

[1021] Users can review the draft and add edits to the photo or message.

[1022] Input: Life Story Draft

[1023] Output: User edited life story

[1024] Step 7:

[1025] Generate final life story (server behavior)

[1026] The server converts the final life story, incorporating the user's edits, into a visually appealing format (slideshow or video).

[1027] Input: User-edited life story

[1028] Output: Final life story (slideshow and video)

[1029] Step 8:

[1030] Personalized product proposals (server and device operations)

[1031] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[1032] The terminal presents the product list to the user in the physical store, providing a shopping experience linked to past memories.

[1033] Input: Data for product recommendations based on sentiment analysis results and past experiences

[1034] Output: Personalized product list

[1035] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1036] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1037] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1038] [Third embodiment]

[1039] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1040] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1041] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1042] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1043] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1044] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1045] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1046] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1047] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1048] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1049] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1050] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1051] overview

[1052] The present invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. Specific embodiments are described below.

[1053] Program Description

[1054] Data collection

[1055] 1. Obtaining User Consent

[1056] The user grants permission for data collection through the privacy settings screen.

[1057] The device sends the user's local data (photos and text messages) to the server.

[1058] 2. Collecting social media data

[1059] The server collects past posting data from users' social media accounts via API.

[1060] Data analysis

[1061] 3. Data Classification

[1062] The server categorizes the collected data into photos, text messages, and social media posts.

[1063] 4. Image Analysis

[1064] The server uses image analysis algorithms to perform facial and object recognition from the photos.

[1065] Identifying emotional moments or specific events (e.g., birthdays, weddings).

[1066] 5. Text Analysis

[1067] The server analyzes text data and social media posts using natural language processing (NLP).

[1068] Conduct sentiment analysis to assess the emotional weight of each post.

[1069] 6. Time Series Analysis

[1070] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[1071] Story Generation

[1072] 7. Story draft creation

[1073] The server creates a draft life story based on the analysis results and time-series data.

[1074] 8. Content Integration

[1075] The server combines photos, text, and social media posts, adding narration and text descriptions to complete the draft.

[1076] 9. Presentation to Users

[1077] The server transmits the generated draft to the user's terminal so that it can be displayed.

[1078] Users can review the draft and make edits or additions as needed.

[1079] Visual Representation

[1080] 10. Final story generation

[1081] The server generates the final life story that reflects the user's edits.

[1082] 11. Content Optimization

[1083] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device.

[1084] 12. Data Transmission and Playback

[1085] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[1086] Specific examples

[1087] Creating a family album

[1088] Here is an example of user C creating a family album.

[1089] 1. Data Collection

[1090] User C launches the application and consents to the collection of photos and text messages of his or her family over a specific period of time.

[1091] The terminal transmits the selected photos and messages to the server.

[1092] The server collects family-related posts from user C's SNS account.

[1093] 2. Data analysis and story generation

[1094] The server uses facial recognition to identify key family members and organize specific events (daily life, travel, birthdays, etc.).

[1095] The server drafts a life story, highlighting emotional events appropriate for a family album.

[1096] User C checks the draft and adds or edits photos or messages as necessary.

[1097] 3. Visual representation and sharing

[1098] The server converts the final story into a video format and transmits it to the terminal.

[1099] The device displays a slideshow for the whole family to enjoy.

[1100] The system allows users to create inspiring and personal family albums without any hassle.

[1101] The processing flow will be explained below.

[1102] Step 1:

[1103] Users launch the application and grant the necessary permissions through a privacy settings screen regarding data collection, including access to their photo library, collection of text messages, and linking to social media accounts.

[1104] Step 2:

[1105] The device searches for specified photos and text messages from its internal storage and sends them to a server via a secure protocol. It also connects with the user's social media accounts to collect past posting data.

[1106] Step 3:

[1107] The server receives the data and categorizes it into photos, text messages, social media posts, etc. During the categorization process, metadata (timestamps, tags, etc.) is added to each piece of data.

[1108] Step 4:

[1109] The server applies image analysis algorithms to the photo data to perform facial and object recognition, identify emotional expressions, and identify specific events (e.g., birthdays, weddings).

[1110] Step 5:

[1111] The server performs sentiment analysis on text messages and social media posts using natural language processing (NLP) techniques to identify emotions such as joy, sadness, and surprise, and rate the intensity of those emotions.

[1112] Step 6:

[1113] The server organizes the collected data chronologically based on timestamps, creating a timeline of significant events that provides a consistent view of the user's past events.

[1114] Step 7:

[1115] Based on the analysis and the timeline, the server generates a draft of the user's life story, highlighting emotional moments and important events.

[1116] Step 8:

[1117] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen to the user and provides an interface for editing and adding to the life story.

[1118] Step 9:

[1119] Users can review the draft and add or edit photos or messages as needed. Once editing is complete, users can save the final story.

[1120] Step 10:

[1121] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[1122] Step 11:

[1123] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device, which displays the final story and allows the user to play it.

[1124] Step 12:

[1125] The device launches a viewer to play the completed life story and visually displays it to the user, allowing the user to enjoy the moving life story.

[1126] Example 1

[1127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1128] Until now, there has been no way to automatically generate emotional and unique life stories based on a user's past data (photos, text messages, social media posts, etc.) and provide them as visually enjoyable content. This has meant that users have had to manually organize and edit large amounts of data, which has been a huge burden.

[1129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1130] In this invention, the server includes means for collecting a user's past data, means for classifying the collected past data into photos, text messages, and social media posts, means for performing facial and object recognition on the classified data using an image analysis algorithm, means for performing sentiment analysis on the classified data using natural language processing technology, means for chronologically organizing the classified data based on the timestamps of each data and creating a timeline of important events, means for generating a draft of the user's life story based on the analyzed data, means for combining the generated draft of the life story with photos, text, and social media posts and adding narration and text descriptions to complete the generated draft of the life story, means for presenting the generated life story to the user and for editing and additions, means for converting the final life story into a visually appealing format (such as a slideshow or video) and transmitting it to a terminal, and means for visually displaying the generated life story to the user. This allows users to effortlessly create and enjoy moving and unique life stories.

[1131] "User" refers to the entity that uses this system to provide historical data and generate a life story.

[1132] "Past data" refers to information such as photos, text messages, and social media posts that users have created and saved in the past.

[1133] "Collection means" refers to the function for sending past data to a server with the user's consent.

[1134] "Classification means" refers to the function of classifying collected historical data into photos, text messages, social media posts, etc.

[1135] "Image analysis algorithm" refers to a computational method for performing face and object recognition on photographic data.

[1136] "Natural language processing technology" refers to technology for performing sentiment analysis and other linguistic analysis on text data.

[1137] A "timestamp" refers to information indicating the time at which each piece of data was generated.

[1138] A "timeline" is a chronological arrangement of important events based on timestamps.

[1139] "Story draft" refers to the outline of an initial life story generated based on the analyzed data.

[1140] "Content integration tools" refers to the ability to complete a story draft by adding photos, text, social media posts, narration, and text descriptions.

[1141] "Final Story" refers to the final version of your life story that reflects the user's edits.

[1142] "Visual format" refers to the format (slideshow or video) used to present the final story in an appealing way.

[1143] "Presentation means" refers to the function of displaying the generated life story to the user, allowing them to check, edit, and play it back.

[1144] overview

[1145] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. This system is implemented through the following three-way collaboration between a server, a device, and the user.

[1146] Data collection

[1147] Obtaining User Consent

[1148] The user consents to data collection on the application's privacy settings screen. For example, a confirmation screen asking, "Do you allow photos and messages to be collected?" is displayed. The device then sends the collected data (photos, text messages) from the user's local storage to the server. The data is securely transmitted using an encrypted communication protocol (e.g., HTTPS).

[1149] Social media data collection

[1150] The server obtains the user's social media account information and uses an API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The server then classifies and stores the collected social media data by user ID.

[1151] Data analysis

[1152] Data Classification

[1153] The server categorizes the received past data into photos, text messages, and social media posts, for example, based on the data's meta information (file format and extension).

[1154] Image analysis

[1155] The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to perform facial and object recognition on the photo data, tagging identified emotional moments and specific events (e.g., birthdays, weddings).

[1156] Text analytics

[1157] The server analyzes text data and social media posts using natural language processing (NLP) technology (e.g., Google Cloud Natural Language API) to perform sentiment analysis. It scores the sentiment (positive, negative, neutral) of each post and extracts sentiment trends.

[1158] Time Series Analysis

[1159] The server organizes the data chronologically based on the timestamp of each piece of data, creating a timeline of significant events, which can then be used to extract important events and sentiment trends.

[1160] Story Generation

[1161] Story draft creation

[1162] The server automatically generates a draft of the user's life story based on the data analysis results and time series data, creating a timeline such as "June 2022: The date I went on a family trip."

[1163] Content Integration

[1164] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft.

[1165] Presenting to the user

[1166] The server sends the generated draft of the life story to the user's device, allowing the user to review and edit the draft. The user can review the draft of the life story displayed on the device and add or edit photos or messages as necessary.

[1167] Visual Representation

[1168] Final story generation

[1169] The server generates a final life story that reflects the user's edits, recreating the user's most moving memories.

[1170] Content Optimization

[1171] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers the final story to the user's device.

[1172] Data transmission and playback

[1173] The device launches a viewer to play the received life story, allowing the user to visually check and play the final story.

[1174] Prompt Sentence Examples

[1175] Analyze users' social media posts, photos, and text messages to generate the perfect life story for their family album. Identify notable events (weddings, birthdays, trips, etc.) and reflect key emotional moments.

[1176] The present invention allows users to effortlessly create moving and unique life stories that are visually enjoyable.

[1177] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1178] Step 1: Obtain user consent

[1179] The user consents to data collection on the application's privacy settings screen. This refers to the action of displaying a confirmation screen asking, "Do you allow photos and messages to be collected?" The input of this step is the user's consent information, and the output is the data collection permission settings.

[1180] Step 2: Collect local data

[1181] The device collects the consented data from local storage. In this case, the device retrieves photos and text messages from local folders. The input to this step is the user's data collection permissions and local data location information, and the output is the collected data itself.

[1182] Step 3: Send the data to the server

[1183] The terminal sends the collected data to the server using an encrypted communication protocol (e.g., HTTPS). The input of this step is the collected data, and the output is the data sent to the server.

[1184] Step 4: Collect social media data

[1185] The server obtains the user's social media account information and uses the API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The input for this step is the user's social media account information and API key, and the output is the collected social media data.

[1186] Step 5: Classify your data

[1187] The server categorizes the received historical data into photos, text messages, and social media posts. For example, it categorizes the data based on its meta information (file format and extension). The input of this step is the collected data, and the output is the categorized data.

[1188] Step 6: Analyze the photo data

[1189] The server uses image analysis algorithms to perform face and object recognition on the photo data, for example, using OpenCV or TensorFlow. The input of this step is the photo data, and the output is the resulting recognition information.

[1190] Step 7: Parse the text data

[1191] The server analyzes the text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis. The Google Cloud Natural Language API is used here. The input for this step is text data, and the output is the results of the sentiment analysis.

[1192] Step 8: Organize your data chronologically

[1193] The server organizes the data chronologically based on the timestamp of each piece of data and creates a timeline of important events. The input of this step is the time-stamped data, and the output is the timeline.

[1194] Step 9: Create a story draft

[1195] The server automatically generates a draft of the user's life story based on the data analysis results and time-series data. For example, it creates a timeline in the form of "June 2022: The date I went on a family trip." The input of this step is the analyzed data and timeline, and the output is a story draft.

[1196] Step 10: Integrate your content

[1197] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft. The input for this step is the story draft and related content, and the output is the completed draft.

[1198] Step 11: Present to the user

[1199] The server sends the generated life story draft to the user's device, allowing the user to review and edit the draft. The input of this step is the completed draft, and the output is the user's review result.

[1200] Step 12: Generate the final story

[1201] The server generates the final life story that reflects the user's edits, e.g., photos added by the user or text modified by the user. The input of this step is the user's edits, and the output is the final life story.

[1202] Step 13: Optimize your content

[1203] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers it. The input for this step is the final story, and the output is a visually appealing format.

[1204] Step 14: Send and play data

[1205] The device launches a viewer to play the received life story, allowing the user to visually confirm and play the final story. The input of this step is the transmitted life story, and the output is the played life story.

[1206] (Application example 1)

[1207] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1208] Conventional life story generation systems could collect and analyze users' past data and generate emotional stories, but they lacked the functionality to convert the data into a visually appealing format or upload the generated stories to content distribution services, resulting in output that was less appealing to users and making it difficult to share with other viewers.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1210] In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story for the user based on the analyzed data, means for presenting the generated life story to the user, means for converting the generated life story into a visually appealing format, and means for uploading the generated life story to a content distribution service, thereby enabling high-quality and moving life stories to be automatically generated and easily shared with other viewers.

[1211] "Historical Data" refers to information that a user has generated or stored in the past, including photos, text messages, social media posts, etc.

[1212] "Collection means" refers to a device or program that includes technical procedures for transmitting a user's past data from the terminal to the server.

[1213] "Means of analysis" refers to techniques that analyze collected data and identify emotional moments and significant events within photos, text messages, and social media posts.

[1214] A "life story" is a narrative about a user's life and emotional moments, generated based on their past data.

[1215] The "presentation means" is a device or program for visually presenting the generated life story to the user.

[1216] "Visually appealing format" refers to a technique for presenting the generated life story in a visually appealing format (e.g., a slideshow or video).

[1217] A "content distribution service" is a service for providing the generated life story to other users via the Internet.

[1218] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) to generate and present an emotional and engaging life story. To implement this invention, a program that executes the following steps is required.

[1219] Hardware and Software Use

[1220] The server and terminal use the following hardware and software, respectively.

[1221] 1. Smartphone: A device used to collect user data.

[1222] 2. Server: A computer that analyzes data and generates life stories.

[1223] 3. Python: It is the primary programming language used for data analysis and video generation.

[1224] 4. PIL (Python Imaging Library): Software used for image processing.

[1225] 5. MoviePy: Software used for video editing.

[1226] 6. API: This is the interface used to collect social media data.

[1227] Data collection

[1228] The user launches the application and gives permission to collect data. This causes the smartphone (device) to send the user's local data (photos and text messages) to the server. At the same time, the server uses the SNS API to collect past posting data from the user's SNS account.

[1229] Data analysis

[1230] The server categorizes the collected data into photos, text messages, and social media posts. It uses image analysis algorithms (using PIL) to perform facial and object recognition in photos, and identifies emotional moments and specific events (e.g., birthdays, weddings). It also analyzes text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis and evaluate the emotional weight of each post.

[1231] Time Series Analysis

[1232] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[1233] Story Generation

[1234] The server creates a draft life story based on the analysis results and time-series data. It combines photos, text, and social media posts, and adds narration and text descriptions to complete the draft. It then presents the draft life story to the user, who can review it and make edits or additions as needed.

[1235] Visual Representation and Delivery

[1236] The server generates the final life story that reflects the user's edits. It converts the final story into a visually appealing format (slideshow or video) and sends it to the device. The device launches a viewer to play the generated life story and visually displays it to the user. It also connects to a content distribution service and uploads the generated life story, making it available to other viewers.

[1237] Specific examples

[1238] For example, if a user uses the system to think about family memories, the application will collect and analyze past photos, text messages, and social media posts related to the family. As a result, it will identify photos of family trips and birthdays as key events and add emotional text messages to the timeline, creating a moving life story. This life story is presented in video format, and users can play it on their smartphones or share it with others via content distribution services.

[1239] Prompt Sentence Examples

[1240] For this system, an example prompt for the generative AI model is as follows:

[1241] "Create a moving life story video using photos, text messages, and social media posts based on family memories."

[1242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1243] Step 1:

[1244] The user launches the application and gives permission to collect data. This causes the user's smartphone (device) to collect locally stored photos and text messages and send them to the server. The server uses the SNS API to retrieve the user's past posting data. The input is the user's consent, and the output is the transmission of data to the server.

[1245] Step 2:

[1246] The server categorizes the received historical data into photos, text messages, and social media posts. This categorization process stores each data in a corresponding folder or database. The input is the collected data, and the output is a categorized dataset.

[1247] Step 3:

[1248] The server uses PIL to perform image analysis on the photo data, identifying emotional moments and specific events through facial and object recognition. The input is the classified photo data, and the output is a list of analyzed events.

[1249] Step 4:

[1250] The server uses natural language processing (NLP) techniques to analyze text data and social media posts, performing sentiment analysis and assessing the emotional weight of each post. The input is the text data and social media posts, and the output is a set of emotional ratings.

[1251] Step 5:

[1252] The server organizes the analyzed photos, text messages, and social media posts chronologically to create a timeline of important events. The input is the analyzed dataset, and the output is the timeline.

[1253] Step 6:

[1254] The server creates a draft life story based on the analysis results and time series data. It combines photos, text, and social media posts, and adds narration and text descriptions. The input is a timeline, and the output is a draft life story.

[1255] Step 7:

[1256] The server presents the generated draft of the life story to the user's smartphone. The user reviews the draft and makes edits or additions as necessary. The input is the draft of the life story, and the output is the user's edited draft.

[1257] Step 8:

[1258] The server generates the final life story, reflecting the user's edits. This final story is then converted into a visually appealing format (slideshow or video). The input is the edited draft, and the output is the final life story video.

[1259] Step 9:

[1260] The server sends the generated final life story to the user's smartphone, where the user plays it. It then connects to a content distribution service and uploads the generated life story. The input is the final life story video, and the output is displaying it to the user and uploading it to the distribution service.

[1261] This allows users to auto-generate high-quality, inspiring life stories and easily share them with other viewers.

[1262] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1263] overview

[1264] This invention is a life story generation system that combines a life story engine that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) and recognizes the user's emotions. The system generates a life story that highlights emotional moments and important events and presents it visually.

[1265] Program Description

[1266] Data collection

[1267] 1. Obtaining User Consent

[1268] Users launch the application and grant the necessary permissions through a privacy settings screen for data collection and emotion recognition.

[1269] 2. Collecting local and social media data

[1270] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[1271] The server collects past posting data from users' social media accounts via API.

[1272] Data analysis

[1273] 3. Data Classification and Image Analysis

[1274] The server categorizes the collected data into photos, text messages, and social media posts.

[1275] Facial and object recognition is performed on the collected image data to identify emotional expressions and specific events.

[1276] 4. Analysis using text analysis and emotion engine

[1277] The server analyzes text data and social media posts using natural language processing (NLP) technology and performs sentiment analysis using an emotion engine.

[1278] The emotion engine identifies the emotional state of the text data and rates the intensity of the emotion.

[1279] 5. Time series analysis and assessment of emotional states

[1280] The server organizes the collected data chronologically based on timestamps, creating a timeline of important events.

[1281] The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of the life story.

[1282] Story Generation

[1283] 6. Draft the story and present it to users

[1284] The server generates a draft life story based on the analysis results and time series data, highlighting emotional moments and important events.

[1285] The terminal displays a preview screen to present the draft to the user and provides an interface that allows editing and additions.

[1286] 7. User editing and final story generation

[1287] Users can review the draft and add or edit photos or messages as needed.

[1288] The server generates the final life story, incorporating the user's edits, in a visually appealing format (slideshow or video) and incorporating emotional information into narration and sound effects.

[1289] Visual Representation

[1290] 8. Displaying the final story and real-time emotion recognition

[1291] The server sends the final story to the terminal.

[1292] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[1293] The emotion engine recognizes the user's real-time emotional responses as the life story plays and dynamically adjusts the story based on those responses.

[1294] Specific examples

[1295] User D's birthday album

[1296] Here is an example of user D creating a birthday album.

[1297] 1. Data Collection

[1298] User D allows the collection of birthday photos and messages.

[1299] The terminal transmits data for the specified period to the server.

[1300] 2. Data analysis and emotion evaluation

[1301] The server performs face recognition and object recognition on the photo data.

[1302] The server analyzes text data and social media posts and evaluates emotions using an emotion engine.

[1303] 3. Story Generation

[1304] The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[1305] User D checks the draft and edits the content as necessary.

[1306] 4. Visual representation and real-time adjustment

[1307] The server converts the final story into a video format and transmits it to the terminal.

[1308] The device plays the album at the birthday party.

[1309] The emotion engine recognizes user D's emotional reactions during playback and adjusts the story accordingly.

[1310] With this system, User D can create a touching and unique birthday commemorative album and share the moment with many people.

[1311] The processing flow will be explained below.

[1312] Step 1:

[1313] The user launches the application and grants the necessary permissions through a privacy settings screen regarding data collection and emotion recognition. The user approves access to the photo library, collection of text messages, and linking to social media accounts.

[1314] Step 2:

[1315] The device searches for the specified photos and text messages from its internal storage and transmits them to the server over a secure protocol.

[1316] Step 3:

[1317] The server collects past posting data from users' social media accounts via API, and the collected data is stored in a centralized database.

[1318] Step 4:

[1319] The server categorizes the data it receives, separating photos, text messages, and social media posts into their respective categories, and adds metadata (such as timestamps and tags) to each piece of data.

[1320] Step 5:

[1321] The server applies image analysis algorithms to the photo data to perform facial and object recognition, a process that identifies emotional expressions and specific events (such as birthdays and weddings).

[1322] Step 6:

[1323] The server analyzes text data and social media posts using natural language processing (NLP) technology, and then uses an emotion engine to perform sentiment analysis, identifying emotions such as joy, sadness, and surprise, and assessing their intensity.

[1324] Step 7:

[1325] The server organizes the data chronologically based on the timestamps of the collected data, then associates identified events with the emotion data to create a timeline of significant events.

[1326] Step 8:

[1327] The server generates a draft of the user's life story based on the analysis and time series data, designed to highlight emotional moments and important events.

[1328] Step 9:

[1329] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen so that the user can check the draft and provides an interface for editing and adding to it.

[1330] Step 10:

[1331] The user reviews the draft, adds or edits photos or messages as needed, and once the user has completed editing, requests the creation of the final story.

[1332] Step 11:

[1333] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[1334] Step 12:

[1335] The server converts the final story into a visually appealing format (e.g., slideshow or video) and sends it to the device, which launches a viewer to play the completed life story and visually display it to the user.

[1336] Step 13:

[1337] The emotion engine recognizes users' real-time emotional reactions while playing the life story, and dynamically adjusts the story accordingly based on the user's reactions, providing a more moving experience.

[1338] Example 2

[1339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] In recent years, the widespread availability of digital data has made it easy to record individual users' life events, but methods for utilizing this data effectively and emotionally remain limited. In particular, there are no systems that can generate life stories that take the user's emotions into account and present them in a visually appealing manner. Furthermore, conventional systems have difficulty dynamically adjusting the story to reflect the user's real-time emotional reactions while the story is being played. This leads to the issue of a degraded user experience.

[1341] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for recognizing emotional responses in real time during playback of the user's life story and dynamically adjusting the story based on the responses. This allows the user to replay past memories while placing emphasis on emotions, and further enables a more personalized experience by dynamically adjusting the story while reflecting emotional responses in real time during playback of the life story.

[1342] "User's past data" refers to digital data such as photos, text messages, and social media posts that a user has created and saved in the past.

[1343] "Means of collection" refers to the technology and functionality used to obtain historical user data from designated data sources.

[1344] "Means of analysis" refers to the algorithms and techniques used to sort through collected historical data and identify emotional moments and significant events.

[1345] "Emotional moments" refer to points or events in the collected data that are presumed to have a strong emotional reaction from the user.

[1346] "Significant moments" are notable events or happenings in a user's life story.

[1347] "Life Story" refers to a continuous narrative reenactment generated from a user's past data and emotional moments and significant events.

[1348] "Presentation means" refers to the technology and methods for visually and audibly displaying and playing the generated life story to the user.

[1349] "Means for recognizing emotional responses in real time" refers to technology for capturing and analyzing users' emotional responses in real time while playing a life story.

[1350] "Means for dynamically adjusting the story based on the response" refers to technologies and algorithms that allow the content of a life story to be instantly changed or adjusted in response to emotional responses recognized in real time.

[1351] overview

[1352] This invention is a system that recognizes a user's emotions by collecting and analyzing their past data (photos, text messages, social media posts, etc.) and generates a life story. Using an emotion engine, the system generates a life story that highlights emotional moments and important events and visually presents it to the user.

[1353] System Configuration

[1354] This system mainly consists of a server, a terminal, and a user.

[1355] Hardware and software used

[1356] Hardware: high-performance computer servers, user devices (smartphones, tablets, PCs), web cameras

[1357] Software: Natural Language Processing (NLP) libraries (e.g., BERT model), facial recognition libraries (e.g., OpenCV), data collection APIs, emotion engines

[1358] Processing Details

[1359] 1. Obtaining User Consent

[1360] Users launch the system application and grant the necessary permissions for data collection and emotion recognition through the privacy settings screen. Based on this permission, the system collects the user's past data (photos, text messages, social media posts, etc.).

[1361] 2. Data Collection

[1362] The device collects photos and text messages stored on the device and sends them to a server, which then collects past posting data from the user's social media accounts (e.g., Facebook, Twitter) via API.

[1363] 3. Data Analysis

[1364] The server categorizes the collected data into images, text messages, and social media posts, and performs facial and object recognition on the image data to identify emotional expressions and events.

[1365] The server analyzes text data and social media posts using natural language processing (NLP) and performs sentiment analysis using an emotion engine, specifically using the BERT model to evaluate the emotional state of the text.

[1366] 4. Time series analysis and emotional state assessment

[1367] The server organizes the collected data chronologically based on timestamps to create a timeline of important events. The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of their life story.

[1368] 5. Story Generation

[1369] The server generates a draft life story based on the analysis and time series data, highlighting emotional moments and important events.

[1370] The device presents the draft to the user and provides a preview screen where edits and content can be added.

[1371] 6. Editing a User

[1372] The user can review the draft and add or edit photos or text messages as needed. The device provides an interface to support these edits.

[1373] 7. Final story generation

[1374] The server generates the final life story, incorporating the user's edits, in a visually appealing format (e.g., slideshow or video), incorporating emotional information into narration and sound effects.

[1375] 8. Visual representation and real-time emotion recognition

[1376] The server transmits the final story to the terminal, and the terminal starts a viewer for playing the received life story.

[1377] The device uses a webcam and sensors to capture the user's real-time emotional responses.

[1378] The server performs real-time emotion recognition and dynamically adjusts the story according to the user's emotional response.

[1379] Specific examples

[1380] The following is a specific example of user D creating a birthday commemorative album.

[1381] 1. Data collection: User D allows the collection of photos and messages taken on his birthday. The device sends the data for a specified period to the server.

[1382] 2. Data analysis: The server performs facial and object recognition on the photo data and analyzes the text data using an emotion engine.

[1383] 3. Story generation: The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[1384] 4. User Edit: User D previews the draft and edits the content.

[1385] 5. Visual representation and real-time adjustment: The server converts the final story into a video format and sends it to the device. The album is played at the birthday party, and emotional responses are recognized in real time, adjusting the story accordingly.

[1386] The system allows users to replay past memories in an emotionally sensitive way, and even enjoy a personalized life story while reflecting their emotional responses in real time.

[1387] keyword

[1388] Generative AI model, prompt sentence

[1389] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1390] Step 1:

[1391] Obtaining User Consent

[1392] The user launches the application and provides permission for data collection and emotion recognition on the privacy settings screen.

[1393] Input: User consent information.

[1394] Output: Data collection and analysis permission flags.

[1395] What happens: The application displays a privacy settings screen to the user, asking for permission to collect data, and if consent is given, sets the permission flag.

[1396] Step 2:

[1397] Local and social media data collection

[1398] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[1399] The server uses an API to collect past posting data from the user's social media account.

[1400] Input: Photos, text messages, social media account information.

[1401] Output: Historical data sent to the server.

[1402] Specific operation: Scans specified folders and directories on the device and extracts relevant data. Social media data is uploaded to the server along with other collected data via an API call.

[1403] Step 3:

[1404] Data Classification

[1405] The server categorizes the received data into photos, text messages, and social media posts.

[1406] Input: Collected historical data.

[1407] Output: Categorized data (photos, text messages, social media posts).

[1408] Specific behavior: Classifies data into different categories based on file format and metadata.

[1409] Step 4:

[1410] Image analysis

[1411] The server performs facial and object recognition to identify emotional expressions and specific events from the image data.

[1412] Input: Classified photo data.

[1413] Output: Recognized emotional facial expressions and event information.

[1414] Specific operation: Uses face recognition libraries such as OpenCV to identify and classify faces and objects in photos.

[1415] Step 5:

[1416] Text analysis and sentiment engine analysis

[1417] The server uses natural language processing technology to analyze text data and social media posts, and performs sentiment analysis using an emotion engine.

[1418] Input: Text data, social media posts.

[1419] Output: Sentiment analysis results.

[1420] What it does: It uses NLP libraries such as the BERT model to identify the sentiment of a sentence and assess its emotional state.

[1421] Step 6:

[1422] Time series analysis and emotional state assessment

[1423] The server organizes the data chronologically based on its timestamps, creating a timeline of significant events.

[1424] Input: Classified data, timestamp.

[1425] Output: A timeline of important events.

[1426] Specific operation: Sorts data based on timestamps to generate a time series of life events.

[1427] Step 7:

[1428] Story draft creation

[1429] The server generates a draft life story based on the analysis results and time-series data.

[1430] Input: Analysis results, timeline.

[1431] Output: Draft life story.

[1432] Specific operation: Based on the timeline and sentiment analysis results, the necessary text and images are placed and an initial draft is generated.

[1433] Step 8:

[1434] Presenting and editing a draft

[1435] The terminal presents the draft to the user and provides an interface where edits and additions can be made.

[1436] The user reviews the draft and makes edits or additions as necessary.

[1437] Input: Draft life story.

[1438] Output: User edited draft.

[1439] What it does: Displays a preview screen and provides an interface that allows the user to review and edit the draft.

[1440] Step 9:

[1441] Final story generation

[1442] The server generates the final life story that reflects the user's edits.

[1443] Input: Edited draft.

[1444] Output: Final life story.

[1445] What it does: Reflects user edits on the draft and converts it into the final format (slideshow or video).

[1446] Step 10:

[1447] Visual representation and real-time emotion recognition

[1448] The device visually displays the final story and uses a webcam and sensors to capture the user's real-time emotional responses.

[1449] The server performs real-time emotion recognition and dynamically adjusts the story.

[1450] Input: Final life story, real-time emotional responses of users.

[1451] Output: A tailored life story.

[1452] Specific behavior: Detects emotional changes during playback in real time and dynamically changes the content and effects of the story accordingly.

[1453] (Application example 2)

[1454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1455] In physical stores, it is difficult to provide a personalized shopping experience that reflects a customer's past experiences and emotions. Furthermore, there is a lack of systems that can analyze a customer's past data and suggest products based on emotional moments and important events. Traditional physical stores lack established methods for making personalized product suggestions, making it difficult to improve customer satisfaction and stimulate purchasing motivation.

[1456] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for making personalized product suggestions using the collected data to individualize the user's shopping experience in a physical store. This makes it possible to provide a personalized shopping experience based on the customer's past experiences and emotions.

[1457] "User" refers to an individual who uses the System.

[1458] "Historical data" refers to information previously generated by a user, such as photos, text messages, social media posts, and purchase history.

[1459] "Means of collection" refers to software and hardware used to acquire and store users' historical data.

[1460] "Analysis tools" refers to algorithms and technologies used to analyze collected historical data and identify emotional moments and significant events.

[1461] "Emotional moments" are moments in user data that express particularly strong emotions.

[1462] "Significant events" refer to events that are particularly meaningful in the user's life story.

[1463] A "life story" is a narrative that depicts a series of events and emotions in a user's life, generated based on the user's past data.

[1464] "Presentation means" refers to devices or software for visually or audibly presenting the generated life story to the user.

[1465] "Brick and mortar store" refers to a physical store where customers can actually visit and purchase products.

[1466] "Shopping experience" refers to the series of actions and emotions that a user goes through when selecting and purchasing a product in a physical store.

[1467] "Personalized product suggestions" refers to suggesting the best products for individual users based on their past data and sentiment analysis.

[1468] The invention is based on a system that collects and analyzes a user's past data, identifies emotional moments and important events, and generates a life story based on that data. The system also has the ability to personalize the shopping experience in physical stores and provide personalized product recommendations to users.

[1469] Hardware and software used

[1470] This system uses the following hardware and software:

[1471] Hardware: Smartphones, servers

[1472] Software: Natural Language Processing (NLP) API, Face recognition API, Emotion recognition API

[1473] Data collection and analysis

[1474] 1. Data Collection

[1475] The device (smartphone) collects the user's past data (photos, text messages, social media posts), and after the user launches the application and gives permission for the handling of personal information, this data is sent to the server.

[1476] 2. Data Analysis

[1477] The server analyzes the collected data, performing facial and object recognition on the image data and sentiment analysis on the text data using natural language processing (NLP).

[1478] 3. Generating a life story

[1479] The server generates a user's life story based on the analysis results, identifying emotional moments and important events, organizing them chronologically, and creating a story draft, which the user can review and is given the option to edit or add to.

[1480] Personalized shopping experience

[1481] 4. Personalized product recommendations

[1482] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[1483] The device (smartphone) presents the product list to the user in the physical store, allowing the user to have a shopping experience that is linked to past memories.

[1484] Example Programs and Data

[1485] A concrete example will be given to explain how this system works.

[1486] Specific examples

[1487] User A visits a general store and launches an app on their smartphone. The app identifies emotionally significant moments based on past social media posts, photos, and text messages, and then makes personalized product recommendations to User A. For example, the product list might include a candle used on User A's birthday a year ago or a photo frame from their honeymoon.

[1488] Example prompt for generative AI model:

[1489] Collect users' past photos, text messages, and social media posts to generate personalized product listings based on:

[1490] 1. Products related to photos and messages of moments of joy

[1491] 2. Products related to happy events

[1492] 3. Products related to specific events (birthdays, holidays, etc.)

[1493] In this way, the system provides a personalized shopping experience that is relevant to the customer's past experiences and emotions, and customers can receive emotionally relevant product suggestions in the physical store, which can increase satisfaction and boost purchasing intent.

[1494] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1495] Step 1:

[1496] Data collection (user manual permission + device behavior)

[1497] The user launches the application and grants permission to process their personal information.

[1498] The device (smartphone) collects the user's past data (photos, text messages, social media posts) from its internal storage and social media accounts.

[1499] Input: User permission information, digital data stored on the device, data from SNS APIs

[1500] Output: Collected historical data (photos, messages, social media posts)

[1501] Step 2:

[1502] Sending data (terminal operation)

[1503] The terminal sends the collected past data to a server on the cloud.

[1504] Input: Historical data collected in the device

[1505] Output: Historical data sent to the server

[1506] Step 3:

[1507] Image data analysis (server operation)

[1508] The server analyzes the transmitted image data and performs facial and object recognition, thereby recognizing emotional moments and specific events.

[1509] Input: Collected image data

[1510] Output: Analysis of emotional moments and events

[1511] Step 4:

[1512] Analysis of text data (server operation)

[1513] The server analyzes the text data using natural language processing (NLP) and performs sentiment analysis, thereby identifying the emotional state and intensity of the text data.

[1514] Input: Collected text data

[1515] Output: Analysis results including emotional state and its intensity

[1516] Step 5:

[1517] Life story generation (server behavior)

[1518] The server organizes the analysis results chronologically and generates a draft life story that weights emotional moments and important events.

[1519] Input: Analyzed image data and text data results

[1520] Output: Draft life story

[1521] Step 6:

[1522] Presenting and editing life stories (device and user actions)

[1523] The device presents the draft to the user and displays a preview screen to give the user editing options.

[1524] Users can review the draft and add edits to the photo or message.

[1525] Input: Life Story Draft

[1526] Output: User edited life story

[1527] Step 7:

[1528] Generate final life story (server behavior)

[1529] The server converts the final life story, incorporating the user's edits, into a visually appealing format (slideshow or video).

[1530] Input: User-edited life story

[1531] Output: Final life story (slideshow and video)

[1532] Step 8:

[1533] Personalized product proposals (server and device operations)

[1534] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[1535] The terminal presents the product list to the user in the physical store, providing a shopping experience linked to past memories.

[1536] Input: Data for product recommendations based on sentiment analysis results and past experiences

[1537] Output: Personalized product list

[1538] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1540] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1541] [Fourth embodiment]

[1542] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1543] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1544] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1545] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1546] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1547] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1548] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1549] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1550] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1551] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1552] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1553] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1554] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1555] overview

[1556] The present invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. Specific embodiments are described below.

[1557] Program Description

[1558] Data collection

[1559] 1. Obtaining User Consent

[1560] The user grants permission for data collection through the privacy settings screen.

[1561] The device sends the user's local data (photos and text messages) to the server.

[1562] 2. Collecting social media data

[1563] The server collects past posting data from users' social media accounts via API.

[1564] Data analysis

[1565] 3. Data Classification

[1566] The server categorizes the collected data into photos, text messages, and social media posts.

[1567] 4. Image Analysis

[1568] The server uses image analysis algorithms to perform facial and object recognition from the photos.

[1569] Identifying emotional moments or specific events (e.g., birthdays, weddings).

[1570] 5. Text Analysis

[1571] The server analyzes text data and social media posts using natural language processing (NLP).

[1572] Conduct sentiment analysis to assess the emotional weight of each post.

[1573] 6. Time Series Analysis

[1574] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[1575] Story Generation

[1576] 7. Story draft creation

[1577] The server creates a draft life story based on the analysis results and time-series data.

[1578] 8. Content Integration

[1579] The server combines photos, text, and social media posts, adding narration and text descriptions to complete the draft.

[1580] 9. Presentation to Users

[1581] The server transmits the generated draft to the user's terminal so that it can be displayed.

[1582] Users can review the draft and make edits or additions as needed.

[1583] Visual Representation

[1584] 10. Final story generation

[1585] The server generates the final life story that reflects the user's edits.

[1586] 11. Content Optimization

[1587] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device.

[1588] 12. Data Transmission and Playback

[1589] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[1590] Specific examples

[1591] Creating a family album

[1592] Here is an example of user C creating a family album.

[1593] 1. Data Collection

[1594] User C launches the application and consents to the collection of photos and text messages of his or her family over a specific period of time.

[1595] The terminal transmits the selected photos and messages to the server.

[1596] The server collects family-related posts from user C's SNS account.

[1597] 2. Data analysis and story generation

[1598] The server uses facial recognition to identify key family members and organize specific events (daily life, travel, birthdays, etc.).

[1599] The server drafts a life story, highlighting emotional events appropriate for a family album.

[1600] User C checks the draft and adds or edits photos or messages as necessary.

[1601] 3. Visual representation and sharing

[1602] The server converts the final story into a video format and transmits it to the terminal.

[1603] The device displays a slideshow for the whole family to enjoy.

[1604] The system allows users to create inspiring and personal family albums without any hassle.

[1605] The processing flow will be explained below.

[1606] Step 1:

[1607] Users launch the application and grant the necessary permissions through a privacy settings screen regarding data collection, including access to their photo library, collection of text messages, and linking to social media accounts.

[1608] Step 2:

[1609] The device searches for specified photos and text messages from its internal storage and sends them to a server via a secure protocol. It also connects with the user's social media accounts to collect past posting data.

[1610] Step 3:

[1611] The server receives the data and categorizes it into photos, text messages, social media posts, etc. During the categorization process, metadata (timestamps, tags, etc.) is added to each piece of data.

[1612] Step 4:

[1613] The server applies image analysis algorithms to the photo data to perform facial and object recognition, identify emotional expressions, and identify specific events (e.g., birthdays, weddings).

[1614] Step 5:

[1615] The server performs sentiment analysis on text messages and social media posts using natural language processing (NLP) techniques to identify emotions such as joy, sadness, and surprise, and rate the intensity of those emotions.

[1616] Step 6:

[1617] The server organizes the collected data chronologically based on timestamps, creating a timeline of significant events that provides a consistent view of the user's past events.

[1618] Step 7:

[1619] Based on the analysis and the timeline, the server generates a draft of the user's life story, highlighting emotional moments and important events.

[1620] Step 8:

[1621] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen to the user and provides an interface for editing and adding to the life story.

[1622] Step 9:

[1623] Users can review the draft and add or edit photos or messages as needed. Once editing is complete, users can save the final story.

[1624] Step 10:

[1625] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[1626] Step 11:

[1627] The server converts the final story into a visually appealing format (slideshow or video) and sends it to the device, which displays the final story and allows the user to play it.

[1628] Step 12:

[1629] The device launches a viewer to play the completed life story and visually displays it to the user, allowing the user to enjoy the moving life story.

[1630] Example 1

[1631] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1632] Until now, there has been no way to automatically generate emotional and unique life stories based on a user's past data (photos, text messages, social media posts, etc.) and provide them as visually enjoyable content. This has meant that users have had to manually organize and edit large amounts of data, which has been a huge burden.

[1633] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1634] In this invention, the server includes means for collecting a user's past data, means for classifying the collected past data into photos, text messages, and social media posts, means for performing facial and object recognition on the classified data using an image analysis algorithm, means for performing sentiment analysis on the classified data using natural language processing technology, means for chronologically organizing the classified data based on the timestamps of each data and creating a timeline of important events, means for generating a draft of the user's life story based on the analyzed data, means for combining the generated draft of the life story with photos, text, and social media posts and adding narration and text descriptions to complete the generated draft of the life story, means for presenting the generated life story to the user and for editing and additions, means for converting the final life story into a visually appealing format (such as a slideshow or video) and transmitting it to a terminal, and means for visually displaying the generated life story to the user. This allows users to effortlessly create and enjoy moving and unique life stories.

[1635] "User" refers to the entity that uses this system to provide historical data and generate a life story.

[1636] "Past data" refers to information such as photos, text messages, and social media posts that users have created and saved in the past.

[1637] "Collection means" refers to the function for sending past data to a server with the user's consent.

[1638] "Classification means" refers to the function of classifying collected historical data into photos, text messages, social media posts, etc.

[1639] "Image analysis algorithm" refers to a computational method for performing face and object recognition on photographic data.

[1640] "Natural language processing technology" refers to technology for performing sentiment analysis and other linguistic analysis on text data.

[1641] A "timestamp" refers to information indicating the time at which each piece of data was generated.

[1642] A "timeline" is a chronological arrangement of important events based on timestamps.

[1643] "Story draft" refers to the outline of an initial life story generated based on the analyzed data.

[1644] "Content integration tools" refers to the ability to complete a story draft by adding photos, text, social media posts, narration, and text descriptions.

[1645] "Final Story" refers to the final version of your life story that reflects the user's edits.

[1646] "Visual format" refers to the format (slideshow or video) used to present the final story in an appealing way.

[1647] "Presentation means" refers to the function of displaying the generated life story to the user, allowing them to check, edit, and play it back.

[1648] overview

[1649] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.), and generates and presents an emotional life story based on that data. This system is implemented through the following three-way collaboration between a server, a device, and the user.

[1650] Data collection

[1651] Obtaining User Consent

[1652] The user consents to data collection on the application's privacy settings screen. For example, a confirmation screen asking, "Do you allow photos and messages to be collected?" is displayed. The device then sends the collected data (photos, text messages) from the user's local storage to the server. The data is securely transmitted using an encrypted communication protocol (e.g., HTTPS).

[1653] Social media data collection

[1654] The server obtains the user's social media account information and uses an API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The server then classifies and stores the collected social media data by user ID.

[1655] Data analysis

[1656] Data Classification

[1657] The server categorizes the received past data into photos, text messages, and social media posts, for example, based on the data's meta information (file format and extension).

[1658] Image analysis

[1659] The server uses image analysis algorithms (e.g., OpenCV or TensorFlow) to perform facial and object recognition on the photo data, tagging identified emotional moments and specific events (e.g., birthdays, weddings).

[1660] Text analytics

[1661] The server analyzes text data and social media posts using natural language processing (NLP) technology (e.g., Google Cloud Natural Language API) to perform sentiment analysis. It scores the sentiment (positive, negative, neutral) of each post and extracts sentiment trends.

[1662] Time Series Analysis

[1663] The server organizes the data chronologically based on the timestamp of each piece of data, creating a timeline of significant events, which can then be used to extract important events and sentiment trends.

[1664] Story Generation

[1665] Story draft creation

[1666] The server automatically generates a draft of the user's life story based on the data analysis results and time series data, creating a timeline such as "June 2022: The date I went on a family trip."

[1667] Content Integration

[1668] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft.

[1669] Presenting to the user

[1670] The server sends the generated draft of the life story to the user's device, allowing the user to review and edit the draft. The user can review the draft of the life story displayed on the device and add or edit photos or messages as necessary.

[1671] Visual Representation

[1672] Final story generation

[1673] The server generates a final life story that reflects the user's edits, recreating the user's most moving memories.

[1674] Content Optimization

[1675] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers the final story to the user's device.

[1676] Data transmission and playback

[1677] The device launches a viewer to play the received life story, allowing the user to visually check and play the final story.

[1678] Prompt Sentence Examples

[1679] Analyze users' social media posts, photos, and text messages to generate the perfect life story for their family album. Identify notable events (weddings, birthdays, trips, etc.) and reflect key emotional moments.

[1680] The present invention allows users to effortlessly create moving and unique life stories that are visually enjoyable.

[1681] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1682] Step 1: Obtain user consent

[1683] The user consents to data collection on the application's privacy settings screen. This refers to the action of displaying a confirmation screen asking, "Do you allow photos and messages to be collected?" The input of this step is the user's consent information, and the output is the data collection permission settings.

[1684] Step 2: Collect local data

[1685] The device collects the consented data from local storage. In this case, the device retrieves photos and text messages from local folders. The input to this step is the user's data collection permissions and local data location information, and the output is the collected data itself.

[1686] Step 3: Send the data to the server

[1687] The terminal sends the collected data to the server using an encrypted communication protocol (e.g., HTTPS). The input of this step is the collected data, and the output is the data sent to the server.

[1688] Step 4: Collect social media data

[1689] The server obtains the user's social media account information and uses the API to collect past posting data. For example, it uses the Twitter API to obtain tweets from the past year. The input for this step is the user's social media account information and API key, and the output is the collected social media data.

[1690] Step 5: Classify your data

[1691] The server categorizes the received historical data into photos, text messages, and social media posts. For example, it categorizes the data based on its meta information (file format and extension). The input of this step is the collected data, and the output is the categorized data.

[1692] Step 6: Analyze the photo data

[1693] The server uses image analysis algorithms to perform face and object recognition on the photo data, for example, using OpenCV or TensorFlow. The input of this step is the photo data, and the output is the resulting recognition information.

[1694] Step 7: Parse the text data

[1695] The server analyzes the text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis. The Google Cloud Natural Language API is used here. The input for this step is text data, and the output is the results of the sentiment analysis.

[1696] Step 8: Organize your data chronologically

[1697] The server organizes the data chronologically based on the timestamp of each piece of data and creates a timeline of important events. The input of this step is the time-stamped data, and the output is the timeline.

[1698] Step 9: Create a story draft

[1699] The server automatically generates a draft of the user's life story based on the data analysis results and time-series data. For example, it creates a timeline in the form of "June 2022: The date I went on a family trip." The input of this step is the analyzed data and timeline, and the output is a story draft.

[1700] Step 10: Integrate your content

[1701] The server incorporates photos, text messages, and social media posts into the draft, and adds narration and text descriptions to complete the draft. The input for this step is the story draft and related content, and the output is the completed draft.

[1702] Step 11: Present to the user

[1703] The server sends the generated life story draft to the user's device, allowing the user to review and edit the draft. The input of this step is the completed draft, and the output is the user's review result.

[1704] Step 12: Generate the final story

[1705] The server generates the final life story that reflects the user's edits, e.g., photos added by the user or text modified by the user. The input of this step is the user's edits, and the output is the final life story.

[1706] Step 13: Optimize your content

[1707] The server converts the final story into a slideshow or video format, adds appropriate music and effects, and delivers it. The input for this step is the final story, and the output is a visually appealing format.

[1708] Step 14: Send and play data

[1709] The device launches a viewer to play the received life story, allowing the user to visually confirm and play the final story. The input of this step is the transmitted life story, and the output is the played life story.

[1710] (Application example 1)

[1711] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1712] Conventional life story generation systems could collect and analyze users' past data and generate emotional stories, but they lacked the functionality to convert the data into a visually appealing format or upload the generated stories to content distribution services, resulting in output that was less appealing to users and making it difficult to share with other viewers.

[1713] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1714] In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story for the user based on the analyzed data, means for presenting the generated life story to the user, means for converting the generated life story into a visually appealing format, and means for uploading the generated life story to a content distribution service, thereby enabling high-quality and moving life stories to be automatically generated and easily shared with other viewers.

[1715] "Historical Data" refers to information that a user has generated or stored in the past, including photos, text messages, social media posts, etc.

[1716] "Collection means" refers to a device or program that includes technical procedures for transmitting a user's past data from the terminal to the server.

[1717] "Means of analysis" refers to techniques that analyze collected data and identify emotional moments and significant events within photos, text messages, and social media posts.

[1718] A "life story" is a narrative about a user's life and emotional moments, generated based on their past data.

[1719] The "presentation means" is a device or program for visually presenting the generated life story to the user.

[1720] "Visually appealing format" refers to a technique for presenting the generated life story in a visually appealing format (e.g., a slideshow or video).

[1721] A "content distribution service" is a service for providing the generated life story to other users via the Internet.

[1722] This invention is a system that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) to generate and present an emotional and engaging life story. To implement this invention, a program that executes the following steps is required.

[1723] Hardware and Software Use

[1724] The server and terminal use the following hardware and software, respectively.

[1725] 1. Smartphone: A device used to collect user data.

[1726] 2. Server: A computer that analyzes data and generates life stories.

[1727] 3. Python: It is the primary programming language used for data analysis and video generation.

[1728] 4. PIL (Python Imaging Library): Software used for image processing.

[1729] 5. MoviePy: Software used for video editing.

[1730] 6. API: This is the interface used to collect social media data.

[1731] Data collection

[1732] The user launches the application and gives permission to collect data. This causes the smartphone (device) to send the user's local data (photos and text messages) to the server. At the same time, the server uses the SNS API to collect past posting data from the user's SNS account.

[1733] Data analysis

[1734] The server categorizes the collected data into photos, text messages, and social media posts. It uses image analysis algorithms (using PIL) to perform facial and object recognition in photos, and identifies emotional moments and specific events (e.g., birthdays, weddings). It also analyzes text data and social media posts using natural language processing (NLP) techniques to perform sentiment analysis and evaluate the emotional weight of each post.

[1735] Time Series Analysis

[1736] The server organizes each piece of data chronologically based on its timestamp, creating a timeline of important events.

[1737] Story Generation

[1738] The server creates a draft life story based on the analysis results and time-series data. It combines photos, text, and social media posts, and adds narration and text descriptions to complete the draft. It then presents the draft life story to the user, who can review it and make edits or additions as needed.

[1739] Visual Representation and Delivery

[1740] The server generates the final life story that reflects the user's edits. It converts the final story into a visually appealing format (slideshow or video) and sends it to the device. The device launches a viewer to play the generated life story and visually displays it to the user. It also connects to a content distribution service and uploads the generated life story, making it available to other viewers.

[1741] Specific examples

[1742] For example, if a user uses the system to think about family memories, the application will collect and analyze past photos, text messages, and social media posts related to the family. As a result, it will identify photos of family trips and birthdays as key events and add emotional text messages to the timeline, creating a moving life story. This life story is presented in video format, and users can play it on their smartphones or share it with others via content distribution services.

[1743] Prompt Sentence Examples

[1744] For this system, an example prompt for the generative AI model is as follows:

[1745] "Create a moving life story video using photos, text messages, and social media posts based on family memories."

[1746] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1747] Step 1:

[1748] The user launches the application and gives permission to collect data. This causes the user's smartphone (device) to collect locally stored photos and text messages and send them to the server. The server uses the SNS API to retrieve the user's past posting data. The input is the user's consent, and the output is the transmission of data to the server.

[1749] Step 2:

[1750] The server categorizes the received historical data into photos, text messages, and social media posts. This categorization process stores each data in a corresponding folder or database. The input is the collected data, and the output is a categorized dataset.

[1751] Step 3:

[1752] The server uses PIL to perform image analysis on the photo data, identifying emotional moments and specific events through facial and object recognition. The input is the classified photo data, and the output is a list of analyzed events.

[1753] Step 4:

[1754] The server uses natural language processing (NLP) techniques to analyze text data and social media posts, performing sentiment analysis and assessing the emotional weight of each post. The input is the text data and social media posts, and the output is a set of emotional ratings.

[1755] Step 5:

[1756] The server organizes the analyzed photos, text messages, and social media posts chronologically to create a timeline of important events. The input is the analyzed dataset, and the output is the timeline.

[1757] Step 6:

[1758] The server creates a draft life story based on the analysis results and time series data. It combines photos, text, and social media posts, and adds narration and text descriptions. The input is a timeline, and the output is a draft life story.

[1759] Step 7:

[1760] The server presents the generated draft of the life story to the user's smartphone. The user reviews the draft and makes edits or additions as necessary. The input is the draft of the life story, and the output is the user's edited draft.

[1761] Step 8:

[1762] The server generates the final life story, reflecting the user's edits. This final story is then converted into a visually appealing format (slideshow or video). The input is the edited draft, and the output is the final life story video.

[1763] Step 9:

[1764] The server sends the generated final life story to the user's smartphone, where the user plays it. It then connects to a content distribution service and uploads the generated life story. The input is the final life story video, and the output is displaying it to the user and uploading it to the distribution service.

[1765] This allows users to auto-generate high-quality, inspiring life stories and easily share them with other viewers.

[1766] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1767] overview

[1768] This invention is a life story generation system that combines a life story engine that collects and analyzes a user's past data (photos, text messages, social media posts, etc.) and recognizes the user's emotions. The system generates a life story that highlights emotional moments and important events and presents it visually.

[1769] Program Description

[1770] Data collection

[1771] 1. Obtaining User Consent

[1772] Users launch the application and grant the necessary permissions through a privacy settings screen for data collection and emotion recognition.

[1773] 2. Collecting local and social media data

[1774] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[1775] The server collects past posting data from users' social media accounts via API.

[1776] Data analysis

[1777] 3. Data Classification and Image Analysis

[1778] The server categorizes the collected data into photos, text messages, and social media posts.

[1779] Facial and object recognition is performed on the collected image data to identify emotional expressions and specific events.

[1780] 4. Analysis using text analysis and emotion engine

[1781] The server analyzes text data and social media posts using natural language processing (NLP) technology and performs sentiment analysis using an emotion engine.

[1782] The emotion engine identifies the emotional state of the text data and rates the intensity of the emotion.

[1783] 5. Time series analysis and assessment of emotional states

[1784] The server organizes the collected data chronologically based on timestamps, creating a timeline of important events.

[1785] The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of the life story.

[1786] Story Generation

[1787] 6. Draft the story and present it to users

[1788] The server generates a draft life story based on the analysis results and time series data, highlighting emotional moments and important events.

[1789] The terminal displays a preview screen to present the draft to the user and provides an interface that allows editing and additions.

[1790] 7. User editing and final story generation

[1791] Users can review the draft and add or edit photos or messages as needed.

[1792] The server generates the final life story, incorporating the user's edits, in a visually appealing format (slideshow or video) and incorporating emotional information into narration and sound effects.

[1793] Visual Representation

[1794] 8. Displaying the final story and real-time emotion recognition

[1795] The server sends the final story to the terminal.

[1796] The terminal starts a viewer for playing the received life story and visually displays it to the user.

[1797] The emotion engine recognizes the user's real-time emotional responses as the life story plays and dynamically adjusts the story based on those responses.

[1798] Specific examples

[1799] User D's birthday album

[1800] Here is an example of user D creating a birthday album.

[1801] 1. Data Collection

[1802] User D allows the collection of birthday photos and messages.

[1803] The terminal transmits data for the specified period to the server.

[1804] 2. Data analysis and emotion evaluation

[1805] The server performs face recognition and object recognition on the photo data.

[1806] The server analyzes text data and social media posts and evaluates emotions using an emotion engine.

[1807] 3. Story Generation

[1808] The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[1809] User D checks the draft and edits the content as necessary.

[1810] 4. Visual representation and real-time adjustment

[1811] The server converts the final story into a video format and transmits it to the terminal.

[1812] The device plays the album at the birthday party.

[1813] The emotion engine recognizes user D's emotional reactions during playback and adjusts the story accordingly.

[1814] With this system, User D can create a touching and unique birthday commemorative album and share the moment with many people.

[1815] The processing flow will be explained below.

[1816] Step 1:

[1817] The user launches the application and grants the necessary permissions through a privacy settings screen regarding data collection and emotion recognition. The user approves access to the photo library, collection of text messages, and linking to social media accounts.

[1818] Step 2:

[1819] The device searches for the specified photos and text messages from its internal storage and transmits them to the server over a secure protocol.

[1820] Step 3:

[1821] The server collects past posting data from users' social media accounts via API, and the collected data is stored in a centralized database.

[1822] Step 4:

[1823] The server categorizes the data it receives, separating photos, text messages, and social media posts into their respective categories, and adds metadata (such as timestamps and tags) to each piece of data.

[1824] Step 5:

[1825] The server applies image analysis algorithms to the photo data to perform facial and object recognition, a process that identifies emotional expressions and specific events (such as birthdays and weddings).

[1826] Step 6:

[1827] The server analyzes text data and social media posts using natural language processing (NLP) technology, and then uses an emotion engine to perform sentiment analysis, identifying emotions such as joy, sadness, and surprise, and assessing their intensity.

[1828] Step 7:

[1829] The server organizes the data chronologically based on the timestamps of the collected data, then associates identified events with the emotion data to create a timeline of significant events.

[1830] Step 8:

[1831] The server generates a draft of the user's life story based on the analysis and time series data, designed to highlight emotional moments and important events.

[1832] Step 9:

[1833] The server sends the generated draft of the life story to the terminal for presentation to the user, and the terminal displays a preview screen so that the user can check the draft and provides an interface for editing and adding to it.

[1834] Step 10:

[1835] The user reviews the draft, adds or edits photos or messages as needed, and once the user has completed editing, requests the creation of the final story.

[1836] Step 11:

[1837] The server generates the final life story that reflects the user's edits, adding visually appealing effects and music to the final story.

[1838] Step 12:

[1839] The server converts the final story into a visually appealing format (e.g., slideshow or video) and sends it to the device, which launches a viewer to play the completed life story and visually display it to the user.

[1840] Step 13:

[1841] The emotion engine recognizes users' real-time emotional reactions while playing the life story, and dynamically adjusts the story accordingly based on the user's reactions, providing a more moving experience.

[1842] Example 2

[1843] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1844] In recent years, the widespread availability of digital data has made it easy to record individual users' life events, but methods for utilizing this data effectively and emotionally remain limited. In particular, there are no systems that can generate life stories that take the user's emotions into account and present them in a visually appealing manner. Furthermore, conventional systems have difficulty dynamically adjusting the story to reflect the user's real-time emotional reactions while the story is being played. This leads to the issue of a degraded user experience.

[1845] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for recognizing emotional responses in real time during playback of the user's life story and dynamically adjusting the story based on the responses. This allows the user to replay past memories while placing emphasis on emotions, and further enables a more personalized experience by dynamically adjusting the story while reflecting emotional responses in real time during playback of the life story.

[1846] "User's past data" refers to digital data such as photos, text messages, and social media posts that a user has created and saved in the past.

[1847] "Means of collection" refers to the technology and functionality used to obtain historical user data from designated data sources.

[1848] "Means of analysis" refers to the algorithms and techniques used to sort through collected historical data and identify emotional moments and significant events.

[1849] "Emotional moments" refer to points or events in the collected data that are presumed to have a strong emotional reaction from the user.

[1850] "Significant moments" are notable events or happenings in a user's life story.

[1851] "Life Story" refers to a continuous narrative reenactment generated from a user's past data and emotional moments and significant events.

[1852] "Presentation means" refers to the technology and methods for visually and audibly displaying and playing the generated life story to the user.

[1853] "Means for recognizing emotional responses in real time" refers to technology for capturing and analyzing users' emotional responses in real time while playing a life story.

[1854] "Means for dynamically adjusting the story based on the response" refers to technologies and algorithms that allow the content of a life story to be instantly changed or adjusted in response to emotional responses recognized in real time.

[1855] overview

[1856] This invention is a system that recognizes a user's emotions by collecting and analyzing their past data (photos, text messages, social media posts, etc.) and generates a life story. Using an emotion engine, the system generates a life story that highlights emotional moments and important events and visually presents it to the user.

[1857] System Configuration

[1858] This system mainly consists of a server, a terminal, and a user.

[1859] Hardware and software used

[1860] Hardware: high-performance computer servers, user devices (smartphones, tablets, PCs), web cameras

[1861] Software: Natural Language Processing (NLP) libraries (e.g., BERT model), facial recognition libraries (e.g., OpenCV), data collection APIs, emotion engines

[1862] Processing Details

[1863] 1. Obtaining User Consent

[1864] Users launch the system application and grant the necessary permissions for data collection and emotion recognition through the privacy settings screen. Based on this permission, the system collects the user's past data (photos, text messages, social media posts, etc.).

[1865] 2. Data Collection

[1866] The device collects photos and text messages stored on the device and sends them to a server, which then collects past posting data from the user's social media accounts (e.g., Facebook, Twitter) via API.

[1867] 3. Data Analysis

[1868] The server categorizes the collected data into images, text messages, and social media posts, and performs facial and object recognition on the image data to identify emotional expressions and events.

[1869] The server analyzes text data and social media posts using natural language processing (NLP) and performs sentiment analysis using an emotion engine, specifically using the BERT model to evaluate the emotional state of the text.

[1870] 4. Time series analysis and emotional state assessment

[1871] The server organizes the collected data chronologically based on timestamps to create a timeline of important events. The emotion engine evaluates the user's emotional state based on the analysis results and reflects this in the generation of their life story.

[1872] 5. Story Generation

[1873] The server generates a draft life story based on the analysis and time series data, highlighting emotional moments and important events.

[1874] The device presents the draft to the user and provides a preview screen where edits and content can be added.

[1875] 6. Editing a User

[1876] The user can review the draft and add or edit photos or text messages as needed. The device provides an interface to support these edits.

[1877] 7. Final story generation

[1878] The server generates the final life story, incorporating the user's edits, in a visually appealing format (e.g., slideshow or video), incorporating emotional information into narration and sound effects.

[1879] 8. Visual representation and real-time emotion recognition

[1880] The server transmits the final story to the terminal, and the terminal starts a viewer for playing the received life story.

[1881] The device uses a webcam and sensors to capture the user's real-time emotional responses.

[1882] The server performs real-time emotion recognition and dynamically adjusts the story according to the user's emotional response.

[1883] Specific examples

[1884] The following is a specific example of user D creating a birthday commemorative album.

[1885] 1. Data collection: User D allows the collection of photos and messages taken on his birthday. The device sends the data for a specified period to the server.

[1886] 2. Data analysis: The server performs facial and object recognition on the photo data and analyzes the text data using an emotion engine.

[1887] 3. Story generation: The server organizes the birthday events in chronological order and generates a draft life story weighted by emotions.

[1888] 4. User Edit: User D previews the draft and edits the content.

[1889] 5. Visual representation and real-time adjustment: The server converts the final story into a video format and sends it to the device. The album is played at the birthday party, and emotional responses are recognized in real time, adjusting the story accordingly.

[1890] The system allows users to replay past memories in an emotionally sensitive way, and even enjoy a personalized life story while reflecting their emotional responses in real time.

[1891] keyword

[1892] Generative AI model, prompt sentence

[1893] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1894] Step 1:

[1895] Obtaining User Consent

[1896] The user launches the application and provides permission for data collection and emotion recognition on the privacy settings screen.

[1897] Input: User consent information.

[1898] Output: Data collection and analysis permission flags.

[1899] What happens: The application displays a privacy settings screen to the user, asking for permission to collect data, and if consent is given, sets the permission flag.

[1900] Step 2:

[1901] Local and social media data collection

[1902] The device searches for the specified photos and text messages from its internal storage and sends them to the server.

[1903] The server uses an API to collect past posting data from the user's social media account.

[1904] Input: Photos, text messages, social media account information.

[1905] Output: Historical data sent to the server.

[1906] Specific operation: Scans specified folders and directories on the device and extracts relevant data. Social media data is uploaded to the server along with other collected data via an API call.

[1907] Step 3:

[1908] Data Classification

[1909] The server categorizes the received data into photos, text messages, and social media posts.

[1910] Input: Collected historical data.

[1911] Output: Categorized data (photos, text messages, social media posts).

[1912] Specific behavior: Classifies data into different categories based on file format and metadata.

[1913] Step 4:

[1914] Image analysis

[1915] The server performs facial and object recognition to identify emotional expressions and specific events from the image data.

[1916] Input: Classified photo data.

[1917] Output: Recognized emotional facial expressions and event information.

[1918] Specific operation: Uses face recognition libraries such as OpenCV to identify and classify faces and objects in photos.

[1919] Step 5:

[1920] Text analysis and sentiment engine analysis

[1921] The server uses natural language processing technology to analyze text data and social media posts, and performs sentiment analysis using an emotion engine.

[1922] Input: Text data, social media posts.

[1923] Output: Sentiment analysis results.

[1924] What it does: It uses NLP libraries such as the BERT model to identify the sentiment of a sentence and assess its emotional state.

[1925] Step 6:

[1926] Time series analysis and emotional state assessment

[1927] The server organizes the data chronologically based on its timestamps, creating a timeline of significant events.

[1928] Input: Classified data, timestamp.

[1929] Output: A timeline of important events.

[1930] Specific operation: Sorts data based on timestamps to generate a time series of life events.

[1931] Step 7:

[1932] Story draft creation

[1933] The server generates a draft life story based on the analysis results and time-series data.

[1934] Input: Analysis results, timeline.

[1935] Output: Draft life story.

[1936] Specific operation: Based on the timeline and sentiment analysis results, the necessary text and images are placed and an initial draft is generated.

[1937] Step 8:

[1938] Presenting and editing a draft

[1939] The terminal presents the draft to the user and provides an interface where edits and additions can be made.

[1940] The user reviews the draft and makes edits or additions as necessary.

[1941] Input: Draft life story.

[1942] Output: User edited draft.

[1943] What it does: Displays a preview screen and provides an interface that allows the user to review and edit the draft.

[1944] Step 9:

[1945] Final story generation

[1946] The server generates the final life story that reflects the user's edits.

[1947] Input: Edited draft.

[1948] Output: Final life story.

[1949] What it does: Reflects user edits on the draft and converts it into the final format (slideshow or video).

[1950] Step 10:

[1951] Visual representation and real-time emotion recognition

[1952] The device visually displays the final story and uses a webcam and sensors to capture the user's real-time emotional responses.

[1953] The server performs real-time emotion recognition and dynamically adjusts the story.

[1954] Input: Final life story, real-time emotional responses of users.

[1955] Output: A tailored life story.

[1956] Specific behavior: Detects emotional changes during playback in real time and dynamically changes the content and effects of the story accordingly.

[1957] (Application example 2)

[1958] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1959] In physical stores, it is difficult to provide a personalized shopping experience that reflects a customer's past experiences and emotions. Furthermore, there is a lack of systems that can analyze a customer's past data and suggest products based on emotional moments and important events. Traditional physical stores lack established methods for making personalized product suggestions, making it difficult to improve customer satisfaction and stimulate purchasing motivation.

[1960] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting past data of the user, means for analyzing the collected past data and identifying emotional moments and important events, means for generating a life story of the user based on the analyzed data, means for presenting the generated life story to the user, and means for making personalized product suggestions using the collected data to individualize the user's shopping experience in a physical store. This makes it possible to provide a personalized shopping experience based on the customer's past experiences and emotions.

[1961] "User" refers to an individual who uses the System.

[1962] "Historical data" refers to information previously generated by a user, such as photos, text messages, social media posts, and purchase history.

[1963] "Means of collection" refers to software and hardware used to acquire and store users' historical data.

[1964] "Analysis tools" refers to algorithms and technologies used to analyze collected historical data and identify emotional moments and significant events.

[1965] "Emotional moments" are moments in user data that express particularly strong emotions.

[1966] "Significant events" refer to events that are particularly meaningful in the user's life story.

[1967] A "life story" is a narrative that depicts a series of events and emotions in a user's life, generated based on the user's past data.

[1968] "Presentation means" refers to devices or software for visually or audibly presenting the generated life story to the user.

[1969] "Brick and mortar store" refers to a physical store where customers can actually visit and purchase products.

[1970] "Shopping experience" refers to the series of actions and emotions that a user goes through when selecting and purchasing a product in a physical store.

[1971] "Personalized product suggestions" refers to suggesting the best products for individual users based on their past data and sentiment analysis.

[1972] The invention is based on a system that collects and analyzes a user's past data, identifies emotional moments and important events, and generates a life story based on that data. The system also has the ability to personalize the shopping experience in physical stores and provide personalized product recommendations to users.

[1973] Hardware and software used

[1974] This system uses the following hardware and software:

[1975] Hardware: Smartphones, servers

[1976] Software: Natural Language Processing (NLP) API, Face recognition API, Emotion recognition API

[1977] Data collection and analysis

[1978] 1. Data Collection

[1979] The device (smartphone) collects the user's past data (photos, text messages, social media posts), and after the user launches the application and gives permission for the handling of personal information, this data is sent to the server.

[1980] 2. Data Analysis

[1981] The server analyzes the collected data, performing facial and object recognition on the image data and sentiment analysis on the text data using natural language processing (NLP).

[1982] 3. Generating a life story

[1983] The server generates a user's life story based on the analysis results, identifying emotional moments and important events, organizing them chronologically, and creating a story draft, which the user can review and is given the option to edit or add to.

[1984] Personalized shopping experience

[1985] 4. Personalized product recommendations

[1986] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[1987] The device (smartphone) presents the product list to the user in the physical store, allowing the user to have a shopping experience that is linked to past memories.

[1988] Example Programs and Data

[1989] A concrete example will be given to explain how this system works.

[1990] Specific examples

[1991] User A visits a general store and launches an app on their smartphone. The app identifies emotionally significant moments based on past social media posts, photos, and text messages, and then makes personalized product recommendations to User A. For example, the product list might include a candle used on User A's birthday a year ago or a photo frame from their honeymoon.

[1992] Example prompt for generative AI model:

[1993] Collect users' past photos, text messages, and social media posts to generate personalized product listings based on:

[1994] 1. Products related to photos and messages of moments of joy

[1995] 2. Products related to happy events

[1996] 3. Products related to specific events (birthdays, holidays, etc.)

[1997] In this way, the system provides a personalized shopping experience that is relevant to the customer's past experiences and emotions, and customers can receive emotionally relevant product suggestions in the physical store, which can increase satisfaction and boost purchasing intent.

[1998] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1999] Step 1:

[2000] Data collection (user manual permission + device behavior)

[2001] The user launches the application and grants permission to process their personal information.

[2002] The device (smartphone) collects the user's past data (photos, text messages, social media posts) from its internal storage and social media accounts.

[2003] Input: User permission information, digital data stored on the device, data from SNS APIs

[2004] Output: Collected historical data (photos, messages, social media posts)

[2005] Step 2:

[2006] Sending data (terminal operation)

[2007] The terminal sends the collected past data to a server on the cloud.

[2008] Input: Historical data collected in the device

[2009] Output: Historical data sent to the server

[2010] Step 3:

[2011] Image data analysis (server operation)

[2012] The server analyzes the transmitted image data and performs facial and object recognition, thereby recognizing emotional moments and specific events.

[2013] Input: Collected image data

[2014] Output: Analysis of emotional moments and events

[2015] Step 4:

[2016] Analysis of text data (server operation)

[2017] The server analyzes the text data using natural language processing (NLP) and performs sentiment analysis, thereby identifying the emotional state and intensity of the text data.

[2018] Input: Collected text data

[2019] Output: Analysis results including emotional state and its intensity

[2020] Step 5:

[2021] Life story generation (server behavior)

[2022] The server organizes the analysis results chronologically and generates a draft life story that weights emotional moments and important events.

[2023] Input: Analyzed image data and text data results

[2024] Output: Draft life story

[2025] Step 6:

[2026] Presenting and editing life stories (device and user actions)

[2027] The device presents the draft to the user and displays a preview screen to give the user editing options.

[2028] Users can review the draft and add edits to the photo or message.

[2029] Input: Life Story Draft

[2030] Output: User edited life story

[2031] Step 7:

[2032] Generate final life story (server behavior)

[2033] The server converts the final life story, incorporating the user's edits, into a visually appealing format (slideshow or video).

[2034] Input: User-edited life story

[2035] Output: Final life story (slideshow and video)

[2036] Step 8:

[2037] Personalized product proposals (server and device operations)

[2038] Based on the analysis results and sentiment analysis results, the server identifies products related to the user's past experiences and emotions and generates a personalized product list.

[2039] The terminal presents the product list to the user in the physical store, providing a shopping experience linked to past memories.

[2040] Input: Data for product recommendations based on sentiment analysis results and past experiences

[2041] Output: Personalized product list

[2042] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2043] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2044] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2045] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2046] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2047] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2048] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2049] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2050] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2051] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2052] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2053] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2054] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2055] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2056] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2057] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2058] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2059] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2060] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2061] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2062] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2063] The following is further disclosed regarding the above embodiment.

[2064] (Claim 1)

[2065] A means of collecting historical data of users;

[2066] A means of analyzing collected historical data to identify emotional moments and important events;

[2067] A means for generating a user's life story based on the analyzed data; and

[2068] a means for presenting the generated life story to a user;

[2069] A system including:

[2070] (Claim 2)

[2071] The system according to claim 1, wherein image data is analyzed from the collected data and face recognition and object recognition are performed.

[2072] (Claim 3)

[2073] The system according to claim 1, wherein the system analyzes text data from the collected data and performs sentiment analysis using natural language processing technology.

[2074] (Claim 4)

[2075] 10. The system of claim 1, further comprising means for visually representing and displaying the generated life story to the user.

[2076] (Claim 5)

[2077] 10. The system of claim 1, further comprising means for chronologically organizing the collected data based on timestamps to create a timeline of significant events.

[2078] "Example 1"

[2079] (Claim 1)

[2080] A means of collecting historical data of users;

[2081] A means to categorize the collected historical data into photos, text messages, and social media posts;

[2082] a means for performing face recognition and object recognition on the classified data using an image analysis algorithm;

[2083] A means for performing sentiment analysis on the classified data using natural language processing technology;

[2084] A way to organize data chronologically based on the timestamps of each piece of data to create a timeline of important events;

[2085] means for generating a draft of the user's life story based on the analyzed data;

[2086] A way to complete the generated life story draft by combining photos, text, and social media posts, and adding narration and text explanations.

[2087] A means to present the generated life story to the user and allow them to edit or add to it;

[2088] A means to convert the final life story into a visually appealing format (slideshow or video) and send it to your device;

[2089] a means for visually displaying the generated life story to a user;

[2090] A system including:

[2091] (Claim 2)

[2092] The system according to claim 1, wherein image data is analyzed from the collected data and face recognition and object recognition are performed.

[2093] (Claim 3)

[2094] The system according to claim 1, wherein the system analyzes text data from the collected data and performs sentiment analysis using natural language processing technology.

[2095] "Application Example 1"

[2096] (Claim 1)

[2097] A means of collecting historical data of users;

[2098] A means of analyzing collected historical data to identify emotional moments and important events;

[2099] A means for generating a user's life story based on the analyzed data; and

[2100] a means for presenting the generated life story to a user;

[2101] a means of converting the generated life stories into a visually appealing format;

[2102] A means for uploading the generated life story to a content distribution service;

[2103] A system including:

[2104] (Claim 2)

[2105] The system of claim 1 analyzes image data from the collected data, performs facial recognition and object recognition, and identifies emotional events.

[2106] (Claim 3)

[2107] The system of claim 1, wherein the system analyzes text data from the collected data, performs sentiment analysis using natural language processing techniques, and identifies emotional events.

[2108] "Example 2: Combining Emotion Engines"

[2109] (Claim 1)

[2110] A means of collecting historical data of users;

[2111] A means of analyzing collected historical data to identify emotional moments and important events;

[2112] A means for generating a user's life story based on the analyzed data; and

[2113] a means for presenting the generated life story to a user;

[2114] a means for recognizing a user's emotional response in real time during playback of the life story and dynamically adjusting the story based on the response;

[2115] A system including:

[2116] (Claim 2)

[2117] The system according to claim 1, wherein image data is analyzed from the collected data and face recognition and object recognition are performed.

[2118] (Claim 3)

[2119] The system according to claim 1, wherein the system analyzes text data from the collected data and performs sentiment analysis using natural language processing technology.

[2120] "Application example 2 when combining emotion engines"

[2121] (Claim 1)

[2122] A means of collecting historical data of users;

[2123] A means of analyzing collected historical data to identify emotional moments and important events;

[2124] A means for generating a user's life story based on the analyzed data; and

[2125] a means for presenting the generated life story to a user;

[2126] A means of using collected data to make personalized product recommendations to individualize users' shopping experiences in physical stores; and

[2127] A system including:

[2128] (Claim 2)

[2129] The system of claim 1 analyzes image data from the collected data, performs facial recognition and object recognition, and associates the data with product suggestions in a physical store.

[2130] (Claim 3)

[2131] The system according to claim 1, wherein the system analyzes text data from the collected data, performs sentiment analysis using natural language processing technology, and makes product suggestions based on the results. [Explanation of symbols]

[2132] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting historical data of users; A means of analyzing collected historical data to identify emotional moments and important events; A means for generating a user's life story based on the analyzed data; and a means for presenting the generated life story to a user; A system including:

2. 2. The system according to claim 1, wherein image data is analyzed from the collected data to perform face recognition and object recognition.

3. The system according to claim 1, wherein the collected data is analyzed using text data and sentiment analysis is performed using natural language processing technology.

4. 10. The system of claim 1, further comprising means for visually representing and displaying to a user a generated life story.

5. 10. The system of claim 1, further comprising means for organizing the collected data chronologically based on timestamps to create a timeline of significant events.

Images