system

Abstract

Provide a system. An input means for inputting memories from a user; a server means for receiving input stored data; An AI module that analyzes the received memory data and extracts important phrases and keywords; a storage means for converting the analyzed stored data into a digital format and storing the digital data in a database; search and purchase means for users to search and purchase memories; and providing means for providing the purchased stored data to the user.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many people today want to keep certain events and experiences in their memories, but memories tend to fade over time, making it difficult to recall detailed information later. People also want to share their experiences and knowledge with others, and there is a need for a simple and reliable way to do this. Furthermore, the lack of an efficient system for preserving and selling memories makes it difficult to effectively commercialize individual memories. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that allows users to input memories and analyze and store them. Specifically, the system includes an input means for users to input memories, a server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, a storage means for converting the analyzed memory data into a digital format and storing it in a database, a search and purchase means for users to search for and purchase memories, and a provision means for providing the purchased memory data to users. This provides a simple and reliable means for preventing memories from fading and sharing them with others. It also enables memories to be efficiently commercialized and sold.

[0006] "User" means any person or entity that utilizes the System to enter, search, or purchase memories.

[0007] "Input means" refers to a device or interface that allows a user to input a memory into the system. Examples include a voice input device or a text input field.

[0008] "Server means" refers to a centralized computer system that receives input stored data and processes it for analysis and storage.

[0009] The "AI module" is artificial intelligence software that analyzes received stored data and extracts important phrases and keywords.

[0010] "Storage means" refers to a device or system for converting the analyzed stored data into a digital format and storing it in a database.

[0011] "Search and Purchase Method" refers to the interface or process by which a user searches the memories database, selects and purchases memories that interest them.

[0012] "Providing means" refers to a method or system for providing the purchased stored data to the user, specifically in the form of a download link or streaming URL.

[0013] "Memory Data" refers to digital information about human memories and experiences that is entered by a user, analyzed, and stored. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments will be described below.

[0036] Memory input and analysis

[0037] Memory input

[0038] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0039] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[0040] In the case of text input, the same content is written in the text field.

[0041] Sending and Receiving Data

[0042] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[0043] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[0044] Memory analysis

[0045] Natural Language Processing

[0046] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[0047] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[0048] Contextual Analysis

[0049] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[0050] Preserving memories

[0051] Database storage

[0052] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[0053] Server: Stores the stored data in a database after converting it into digital format.

[0054] Searching and Purchasing Memory

[0055] Memory retrieval

[0056] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[0057] Terminal: Sends the entered search query to the server.

[0058] Retrieving and displaying search results

[0059] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0060] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0061] Purchasing and donating memories

[0062] User: Select a memory they like and press the purchase button to purchase it.

[0063] Device: Sends a purchase request to the server.

[0064] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0065] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[0066] This allows users to digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy those memories.

[0067] The processing flow will be explained below.

[0068] Memory input and analysis

[0069] Step 1: Entering memory

[0070] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0071] For example, a user may select voice input and speak into a microphone to record "memories of a high school soccer tournament," or type into a text input field.

[0072] Step 2: Convert audio data to text

[0073] On your device: If you select voice input, your audio recording will be converted to text using transcription software, and the converted text will be temporarily stored in your device's memory.

[0074] Step 3: Sending data

[0075] Terminal: The converted text data or the text data entered directly is sent to the server as an HTTP request.

[0076] Step 4: Receiving the data

[0077] Server: Receives the stored data sent from the device and checks the data format (voice data or text data).

[0078] Memory analysis

[0079] Step 5: Performing Natural Language Processing

[0080] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[0081] Server: Sends data to the NLP module and starts the analysis.

[0082] Step 6: Keyword extraction

[0083] AI module: Analyzes text data using NLP to extract important phrases and keywords.

[0084] For example, important information such as "high school," "soccer tournament," and "championship" is analyzed.

[0085] Step 7: Context Analysis

[0086] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the detailed content.

[0087] Preserving memories

[0088] Step 8: Structuring the database

[0089] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[0090] Server: Prepares structured stored data for conversion into digital format.

[0091] Step 9: Save to Database

[0092] Server: Stores the stored data in a database after converting it into digital format.

[0093] Searching and Purchasing Memory

[0094] Step 10: Searching your memory

[0095] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[0096] Terminal: Sends the entered search query to the server.

[0097] Step 11: Getting search results

[0098] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0099] Step 12: Viewing search results

[0100] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0101] Purchasing and donating memories

[0102] Step 13: Buy Memory

[0103] User: Select a memory they like and press the purchase button to purchase it.

[0104] Device: Sends a purchase request to the server.

[0105] Step 14: Payment Processing

[0106] Server: Receives the purchase request and processes the payment through the payment system.

[0107] Step 15: Confirm your purchase

[0108] Server: Once payment is complete, verify that the purchase was successful.

[0109] Step 16: Providing Stored Data

[0110] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[0111] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[0112] In this way, users can digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy them.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Until now, there has been no system that can handle the entire process of inputting, analyzing, storing, searching, and purchasing memories in a unified manner. Furthermore, converting voice data into text, and tagging and classifying analysis results are time-consuming and difficult, reducing usability. This has made it difficult for users to easily digitize their memories and share or sell them with other users.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes input means for inputting memories from users, conversion means for acquiring the input memory data and converting voice data into text data, transmission means for transmitting the acquired memory data via an HTTP request, analysis means for receiving the transmitted memory data and performing natural language processing and context analysis using an AI module, storage means for classifying, tagging, and storing the analyzed memory data in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data in the form of a download link or streaming URL. This allows users to easily digitize their memories and share or sell them with other users over the network.

[0118] A "user" is someone who uses the system to enter, search, and purchase memories.

[0119] "Input means" refers to a device or function that allows a user to input memories, and provides options for voice input and text input.

[0120] "Conversion means" refers to a device or software for converting voice data into text data, and uses voice recognition technology.

[0121] The "transmission means" refers to a function for transmitting the acquired stored data to the server, and transmits the data via an HTTP request.

[0122] "Analysis means" refers to devices and software that process received stored data using natural language processing (NLP) and contextual analysis.

[0123] "Storage means" refers to the function of converting the analyzed stored data into a digital format, appropriately classifying and tagging it, and storing it in a database.

[0124] "Search means" refers to a function that allows a user to search memories, and searches for relevant stored data from a database through a search query.

[0125] "Purchase means" refers to a function that allows a user to purchase a memory that they like, and executes payment processing.

[0126] The "means for providing" refers to a function for providing the purchased stored data to the user, and is provided in the form of a download link or a streaming URL.

[0127] "Database" refers to an electronic storage device for storing analyzed and archived stored data.

[0128] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments are described below.

[0129] Memory input

[0130] User: Launches the dedicated application and inputs a memory. The user is provided with the option of voice input or text input. For voice input, the user records the memory using the device's microphone. For example, the user speaks, "When I played in the high school soccer tournament," to obtain voice data. For text input, the user inputs the memory using the device's keyboard. For example, the user types, "When I played in the high school soccer tournament," into the text field.

[0131] Device: If the user selects voice input, the device uses speech recognition software to convert the recorded voice data into text data. Specifically, the device uses speech recognition technology such as the Google® Cloud Speech-to-Text API to convert the voice to text data. The converted text data, or text data entered directly by the user, is sent to the server as an HTTP request.

[0132] Sending and Receiving Data

[0133] Terminal: Generates an HTTP request and sends the retrieved text data to the server.

[0134] Server: Receives the HTTP request and processes the text data. If the data is audio, speech recognition is performed again on the server side.

[0135] Memory analysis

[0136] Server: Sends the received text data to the AI ​​module for natural language processing (NLP). Specifically, the text data is analyzed using advanced NLP techniques such as OpenAI's GPT-4 API. Key phrases and keywords are extracted based on the analysis results. For example, information such as "high school," "soccer tournament," and "championship" is extracted.

[0137] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[0138] Preserving memories

[0139] Server: Classifies and tags the stored data based on the analysis results. Tags include category information such as "sports," "school," and "tournament." Analyzed and classified data is saved in a database. Common database technologies such as MySQL (registered trademark) and MongoDB are used for the database.

[0140] Searching and Purchasing Memory

[0141] User: If another user wants to use an application on their device to search for "memories of high school soccer tournaments," they enter a search keyword.

[0142] Terminal: Sends the entered search query to the server.

[0143] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0144] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0145] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[0146] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0147] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[0148] Specific examples (prompt sentence examples)

[0149] Example voice input: "I want to record my memories of playing in a high school soccer tournament."

[0150] Example text entry: "I want to record my time playing in a high school soccer tournament."

[0151] This system allows users to digitize their memories, share them with other users, and sell them, and users who purchase them can enjoy them.

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1:

[0154] User: Launches the dedicated application and starts inputting memories. To input memories, the user can choose between voice input or text input. For example, the user can say "When I played in the high school soccer tournament" or input the same content in text.

[0155] Input: Audio or text data storage

[0156] Output: Continuously updated audio data or consistent text data

[0157] Specific operation: The user opens the application on the device, follows the instructions, speaks the memory aloud using the microphone, presses the voice input button to start recording, and when finished, presses the button again to stop recording.

[0158] Step 2:

[0159] Device: When voice data is acquired, the device converts the voice data into text data using speech recognition technology such as the Google Cloud Speech-to-Text API, and temporarily stores the converted text data and text data entered directly by the user.

[0160] Input: Audio data

[0161] Output: Text data

[0162] Specific operation: The device sends voice data to the speech recognition API, receives the converted text data, and displays it in a text field or stores it in a variable.

[0163] Step 3:

[0164] Terminal: The acquired text data is sent to the server as an HTTP request. The user's memory data is temporarily stored locally at this stage and then sent to the server.

[0165] Input: Text data

[0166] Output: HTTP request

[0167] Specific operation: The device application converts the text data into JSON format and sends it to the specified URL on the server using an HTTP POST request.

[0168] Step 4:

[0169] Server: Receives the HTTP request and prepares to analyze the stored data. If voice data is included, voice recognition is performed again.

[0170] Input: HTTP request

[0171] Output: Text data

[0172] Specific operation: The server side receives the HTTP request, analyzes the payload, extracts text data, and if it contains audio data, calls the speech recognition API on the server side to convert it into text.

[0173] Step 5:

[0174] Server: Sends the received text data to an AI module (e.g., OpenAI GPT-4 API) for natural language processing (NLP), which extracts key phrases and keywords.

[0175] Input: Text data

[0176] Output: Key phrases and keywords

[0177] Specific operation: The server passes the text data to the AI ​​module and sends the prompt "Analyze the following text and extract key phrases and keywords: About playing in a high school soccer tournament." The AI ​​module responds with the extracted keywords.

[0178] Step 6:

[0179] Server: Understands the context of stored data based on extracted keywords, and tags and structures detailed content.

[0180] Input: Extracted keywords

[0181] Output: Tagged text data

[0182] Specific operation: Based on the analysis results from the AI ​​module, the server applies a tagging algorithm to assign category information such as "sports," "school," or "tournament."

[0183] Step 7:

[0184] Server: The tagged and classified data is stored in a database using database technologies such as MySQL or MongoDB.

[0185] Input: Tagged and classified text data

[0186] Output: Memory data stored in the database

[0187] Specific operation: Executes an INSERT statement on the database installed on the server and saves the analysis results.

[0188] Step 8:

[0189] User: Another user enters a search keyword to search for "high school soccer tournament memories" on their device.

[0190] Input: Search keyword

[0191] Output: Search query

[0192] What happens: A user types "high school soccer tournament" into the application's search field and presses the search button.

[0193] Step 9:

[0194] Device: Sends a search query to the server via an HTTP GET request.

[0195] Input: Search keyword

[0196] Output: HTTP GET request

[0197] Specific operation: The device formats the search keywords as query parameters into an HTTP GET request and sends it to the server's search API.

[0198] Step 10:

[0199] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0200] Input: HTTP GET request

[0201] Output: Search result data

[0202] Specific operation: The server searches the database, extracts the corresponding stored data, and returns the extracted data in JSON format as an HTTP response.

[0203] Step 11:

[0204] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0205] Input: Search result data

[0206] Output: Display of search results

[0207] Specific operation: The device parses the received search results and displays them in the user interface, arranging the title, part of the text, tag information, etc. in the display area.

[0208] Step 12:

[0209] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[0210] Input: Purchase Information

[0211] Output: Purchase request

[0212] Specific operation: When the user clicks the purchase button, the device sends an HTTP POST request to the server containing the necessary information for the purchase.

[0213] Step 13:

[0214] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0215] Input: Purchase Request

[0216] Output: Download link or streaming URL

[0217] Specific operation: The server calls a payment API (e.g., Stripe or PayPal) to process the payment. Once the purchase is confirmed, a download link or streaming URL for the corresponding stored data is generated.

[0218] Step 14:

[0219] On the device: The download link or streaming URL received from the server is displayed to the user, allowing them to access the stored data they have purchased.

[0220] Input: Download link or streaming URL

[0221] Output: Display of links and URLs

[0222] Specific operation: The device analyzes the links and URLs received from the server and displays them in the user interface. The user can click on the links to access the stored data.

[0223] This allows users to digitize their memories and share or sell them with other users, who can then enjoy the memories they have purchased.

[0224] (Application example 1)

[0225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0226] Conventional memory data management systems analyze and store memories entered by users, allowing other users to search and purchase them, but lack the means to recreate those memories more specifically and provide them in a form that users can experience. Furthermore, the effective use of memory data is limited, and sharing of experiences is limited to visual and auditory information, so there is a demand for providing deeper emotional experiences.

[0227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0228] In this invention, the server includes an input means for inputting memories from a user, a server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, a natural language processing means for analyzing the context of the memory based on the extracted phrases and keywords, a storage means for converting the analyzed memory data into a digital format and storing it in a database, a search and purchase means for the user to search for and purchase memories, and a virtual reality linkage means for providing the processed data to the user as virtual reality content. This not only enables advanced analysis of the user's memories and storage of their context, but also makes it possible to provide the memory data in a form that other users can experience realistically through virtual reality.

[0229] "User" refers to the entity that uses the system to input, search, and purchase memories.

[0230] "Memories" refer to records of events and emotions experienced by the user, and are input as audio or text data.

[0231] "Input means" refers to a device or interface that allows a user to input memories as voice or text data.

[0232] "Server means" refers to a computer system that receives and processes stored data input by a user.

[0233] "AI Module" refers to artificial intelligence software that analyzes received stored data and extracts important phrases and keywords.

[0234] "Natural language processing means" refers to technology for analyzing memory context based on extracted phrases and keywords.

[0235] "Storage means" refers to a function that converts the analyzed stored data into a digital format and stores it in a database.

[0236] "Search and purchase means" refers to an interface that allows a user to search and purchase saved stored data.

[0237] "Delivery means" refers to a mechanism for providing purchased storage data to a user.

[0238] "Virtual reality linking means" refers to a technology that provides processed stored data to users as virtual reality content.

[0239] This invention relates to a system that allows users to input personal memories, analyze and store them using natural language processing (NLP) technology, and then allows other users to search and purchase them. The purchased memory data can then be reproduced as virtual reality (VR) content.

[0240] Memory input and analysis

[0241] Memory Input:

[0242] User: To input a memory, launch a dedicated application on the device. The application offers the option of voice input or text input.

[0243] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[0244] In the case of text input, the same content is written in the text field.

[0245] Sending and receiving data:

[0246] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[0247] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[0248] Natural Language Processing:

[0249] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[0250] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[0251] Contextual analysis:

[0252] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[0253] Preserving memories

[0254] Database storage

[0255] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[0256] Server: Stores the stored data in a database after converting it into digital format.

[0257] Searching and Purchasing Memory

[0258] Memory Retrieval:

[0259] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[0260] Terminal: Sends the entered search query to the server.

[0261] Retrieving and displaying search results:

[0262] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0263] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0264] Purchasing and donating memory:

[0265] User: Select a memory they like and press the purchase button to purchase it.

[0266] Device: Sends a purchase request to the server.

[0267] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0268] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[0269] Virtual reality reproduction

[0270] Virtual Reality Integration:

[0271] Server: Reconstructs the purchased memory data as virtual reality content and generates links.

[0272] User: Experience virtual reality content using smart glasses or a head-mounted display.

[0273] Specific examples and prompts for the generative AI model

[0274] Examples:

[0275] User A: Inputs a memory of a soccer tournament from high school, and talks about the changes in emotions and detailed episodes that occurred during the event.

[0276] Server: Converts voice data into text and analyzes it using NLP. Stores the analysis results in a database.

[0277] User B: Searches for "high school soccer tournament" in the virtual store and finds User A's memory.

[0278] Server: Returns search results and generates a VR link. User B enjoys the VR experience through the link.

[0279] Prompt statement:

[0280] "Tell me about a time when you participated in a high school soccer tournament. Please describe in detail any particularly memorable episodes or emotions."

[0281] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0282] Step 1:

[0283] User enters memory:

[0284] Input: The user launches a dedicated application on the device and inputs the memory by voice or text.

[0285] Processing: In the case of voice input, the application records the voice data, and in the case of text input, it receives the text data as is.

[0286] Output: Audio or text data.

[0287] Step 2:

[0288] The device converts the data:

[0289] Input: Audio data.

[0290] Processing: When voice input is received, the device converts the voice data into text using transcription software (e.g., Google Speech-to-Text API or IBM Watson®).

[0291] Output: Text data.

[0292] Step 3:

[0293] Sending data:

[0294] Input: Text data.

[0295] Processing: The terminal sends the converted text data or the directly entered text data to the server as an HTTP request.

[0296] Output: HTTP request (text data).

[0297] Step 4:

[0298] Server receives data:

[0299] Input: HTTP request (text data).

[0300] Processing: The server receives the stored data sent from the terminal.

[0301] Output: Received text data.

[0302] Step 5:

[0303] Perform natural language processing:

[0304] Input: Received text data.

[0305] Processing: The server uses AI modules (e.g., Hugging Face Transformers) to perform natural language processing (NLP) and extract key phrases and keywords.

[0306] Output: Extracted phrases and keywords.

[0307] Step 6:

[0308] Contextual analysis:

[0309] Input: Extracted phrases or keywords.

[0310] Processing: The server's AI module analyzes the memory context based on phrases and keywords, and tags and structures the details.

[0311] Output: Tagged data.

[0312] Step 7:

[0313] Save to database:

[0314] Input: Tagged data.

[0315] Processing: The server converts the tagged stored data into a digital format and stores it in a database.

[0316] Output: Memorized data stored in a database.

[0317] Step 8:

[0318] User searches memory:

[0319] Input: Search keyword.

[0320] Process: The user enters search keywords into an application on the device, and the device sends the search query to the server as an HTTP request.

[0321] Output: HTTP request (search query).

[0322] Step 9:

[0323] The server retrieves the search results:

[0324] Input: HTTP request (search query).

[0325] Processing: The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[0326] Output: HTTP response (search results).

[0327] Step 10:

[0328] Display search results:

[0329] Input: HTTP response (search result).

[0330] Processing: The terminal displays the search results received from the server to the user. For example, it displays the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament."

[0331] Output: What is displayed to the user.

[0332] Step 11:

[0333] User purchases memory:

[0334] Input: Purchase request.

[0335] Processing: The user selects a memory they like and presses the purchase button. The device sends a purchase request to the server.

[0336] Output: HTTP request (purchase request).

[0337] Step 12:

[0338] The server processes the payment:

[0339] Input: HTTP request (purchase request).

[0340] Processing: The server processes the payment and generates a download link or streaming URL for the stored data for which the purchase has been confirmed.

[0341] Output: Download link or streaming URL.

[0342] Step 13:

[0343] Providing purchase data:

[0344] Input: Download link or streaming URL.

[0345] Processing: The device will display a download link or streaming URL to the user, allowing them to access the stored data they purchased.

[0346] Output: Provide link to user.

[0347] Step 14:

[0348] Virtual reality reproduction:

[0349] Input: Purchased memory data.

[0350] Processing: Reconstructs purchased memory data into virtual reality content and generates a link.

[0351] Output: Virtual reality experience link.

[0352] Step 15:

[0353] Experience virtual reality content:

[0354] Input: Virtual reality experience link.

[0355] Processing: Users experience virtual reality content using smart glasses or a head-mounted display.

[0356] Output: Experience virtual reality content.

[0357] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0358] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memories input by a user and the emotional state of the memories. Specific embodiments will be described in detail below.

[0359] Memory input and emotion recognition

[0360] Memory input

[0361] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0362] For example, to record "Memories of my college graduation ceremony," a user selects voice input and speaks into the microphone, "The joy I felt at my college graduation ceremony," or writes the memory in the text input field.

[0363] Emotion recognition

[0364] Device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. For example, it determines whether the user is expressing emotions such as joy, sadness, or surprise through voice tone and speed, as well as sentiment analysis of the text.

[0365] Terminal: The recognized emotion data is sent to the server along with the input data.

[0366] Sending and Receiving Data

[0367] Sending data

[0368] Terminal: The converted text data or directly entered text data and emotion data are sent to the server as an HTTP request.

[0369] Receiving data

[0370] Server: Receives emotion-recognized memory data sent from the device.

[0371] Memory analysis

[0372] Performing natural language processing

[0373] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[0374] Server: Sends data to the NLP module and starts the analysis.

[0375] Keyword extraction

[0376] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[0377] Contextual Analysis

[0378] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[0379] Preserving memories

[0380] Database storage

[0381] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[0382] Server: Stores the stored data in a database after converting it into digital format.

[0383] Searching and Purchasing Memory

[0384] Memory retrieval

[0385] User: Another user searches for "joyful memories of college graduation" in an application on their device and enters search keywords and the desired emotional state.

[0386] Terminal: Sends the entered search query to the server.

[0387] Getting search results

[0388] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0389] Displaying search results

[0390] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[0391] Purchasing and donating memories

[0392] Purchasing Memory

[0393] User: Select a memory they like and press the purchase button to purchase it.

[0394] Device: Sends a purchase request to the server.

[0395] Payment Processing

[0396] Server: Receives the purchase request and processes the payment through the payment system.

[0397] Providing stored data

[0398] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[0399] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[0400] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search and purchase them. Furthermore, users who purchase them can enjoy the emotional state of their memories as well.

[0401] The processing flow will be explained below.

[0402] Memory input and emotion recognition

[0403] Step 1: Entering memory

[0404] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0405] For example, if a user wants to record "memories of college graduation," they can select voice input and speak into the microphone, "the joy I felt at college graduation," or write their memories in the text input field.

[0406] Step 2: Recognize emotions

[0407] On the device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. It uses tone and speed of speech and sentiment analysis of text to determine whether the user is expressing emotions such as joy, sadness, or surprise.

[0408] Step 3: Text Conversion (Speech Input)

[0409] Device: If voice input is selected, the recorded voice data is converted to text data using transcription software, and the converted text data is temporarily stored.

[0410] Step 4: Sending data

[0411] Terminal: The converted text data or the directly entered text data and the recognized emotion data are sent to the server as an HTTP request.

[0412] Data reception and analysis

[0413] Step 5: Receiving the data

[0414] Server: Receives emotion-recognized memory data sent from the device.

[0415] Step 6: Performing Natural Language Processing

[0416] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[0417] Server: Sends data to the NLP module and starts the analysis.

[0418] Step 7: Keyword extraction

[0419] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[0420] Step 8: Analyze the sentiment data

[0421] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[0422] Preserving memories

[0423] Step 9: Prepare to save to the database

[0424] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[0425] Step 10: Save to database

[0426] Server: Stores the stored data in a database after converting it into digital format.

[0427] Searching and Purchasing Memory

[0428] Step 11: Memory Retrieval

[0429] User: When another user searches for "joyful memories of college graduation" through an application on a terminal, the user inputs search keywords and a desired emotional state.

[0430] Terminal: Sends the entered search query to the server.

[0431] Step 12: Getting search results

[0432] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0433] Step 13: Viewing search results

[0434] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[0435] Purchasing and donating memories

[0436] Step 14: Buy Memory

[0437] User: Select the memory they like and press the purchase button if they wish to purchase it.

[0438] Device: Sends a purchase request to the server.

[0439] Step 15: Payment Processing

[0440] Server: Receives the purchase request and processes the payment through the payment system.

[0441] Step 16: Confirm your purchase

[0442] Server: Once payment is complete, verify that the purchase was successful.

[0443] Step 17: Provide stored data

[0444] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[0445] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[0446] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search for and purchase them. In addition, users who purchase the memories can enjoy the emotional state of the memories.

[0447] Example 2

[0448] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0449] In existing systems, when a user inputs a memory, it is difficult to recognize the specific emotional state and analyze, store, and provide the memory data. Therefore, there is a need for a system that can accurately recognize the emotional information in memory and analyze, store, search, and provide the data based on that information.

[0450] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0451] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, emotion recognition means for recognizing emotional states from voice and text data, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data to users. This allows users to input, recognize, and store memories more accurately, and enables other users to search, purchase, and share memories, including their emotional information.

[0452] "Input means" refers to the method or device that a user uses to input a memory.

[0453] "Server means" refers to a central computing device for receiving and processing stored data sent by users.

[0454] "AI Module" refers to the artificial intelligence software module used to analyze received stored data and extract key phrases and keywords.

[0455] "Emotion recognizer" refers to a system or method for recognizing emotional states from speech and text data.

[0456] "Storage means" refers to a means or device for converting the analyzed stored data into a digital format and storing it in a database.

[0457] "Search and purchase means" refers to a method or device that allows a user to search for stored data and purchase it if necessary.

[0458] "Providing means" refers to a means or system for providing the purchased storage data to the user in the form of a download link or streaming URL.

[0459] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memory input by a user and the emotional state of that memory.

[0460] The system consists of the following elements:

[0461] 1. Input Method

[0462] User: The user launches a dedicated application on the device and inputs their memories. The application offers the option of voice input or text input.

[0463] Example: To record "Memories of my college graduation ceremony," a user taps the voice input button and speaks "The joy I felt at my college graduation ceremony," or writes "The joy I felt at my college graduation ceremony" in the text input field.

[0464] 2. Server Means

[0465] Terminal: The stored data entered by the user is sent to the server in JSON format, for example, { "text": "The joy felt at the university graduation ceremony", "emotion": "Joy"}.

[0466] Server: The server receives the data sent from the device and prepares it for processing.

[0467] 3. AI Module

[0468] Server: Passes the received text data to an AI module (e.g., TENSORFLOW (registered trademark) or PyTorch) to perform natural language processing (NLP).

[0469] Example: Using NLP to extract important phrases and keywords such as "university," "graduation," and "joy" from text data.

[0470] 4. Emotion recognition means

[0471] On the device: An emotion engine is used to analyze voice and text data, identifying emotions from the tone and rate of voice and the content of text.

[0472] Example: Recognizing emotional states such as happiness, sadness, surprise, etc. For example, identifying "happiness" based on tone and rate of speech and text.

[0473] 5. Preservation means

[0474] Server: Converts the analyzed memory data into a digital format and stores it in a database.

[0475] Example: Save data with tagging that includes category information such as "emotion," "event," "place," etc. For example, set tags like tags: ["university," "graduation ceremony," "joy"].

[0476] 6. Search and Purchase Methods

[0477] User: Another user searches a memory to find information about a specific emotion or event. Enter search keywords and an emotional state.

[0478] Example: A user types "college," "graduation," and "joy" into the app's search bar and clicks the search button.

[0479] Device: Sends a search query to the server.

[0480] 7. Means of provision

[0481] Server: Provides the found stored data to the user in the form of a download link or streaming URL.

[0482] Example: Displaying a title such as "Memories of Mr. / Ms. XX's College Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[0483] User: Selects the memory they like, purchases it, enters payment information and taps the purchase button.

[0484] Server: Receives the purchase request, processes the payment through a payment system (e.g., Stripe or PayPal), and then generates a download link or streaming URL for the purchased stored data and provides it to the user.

[0485] 8. Examples of concrete examples and prompts

[0486] Example: User A inputs the joy of his / her college graduation into a memory, and the emotion is recognized and saved as "joy." User B searches for "college," "graduation ceremony," and "joy," discovers User A's memory, and purchases it, allowing him / her to learn more about the joy.

[0487] Example prompt: "Describe a system that recognizes and stores emotional states based on memories of your college graduation. Explain in detail how the system analyzes and stores the data, using the example memory of 'summer family vacation.'"

[0488] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0489] Step 1:

[0490] Memory input

[0491] User: The user launches a dedicated application on the device and inputs their memories. The application provides the option of voice input or text input.

[0492] Specific actions: To record "Memories of my college graduation ceremony," the user taps the voice input button and speaks "The joy I felt at my college graduation ceremony" into the microphone, or writes "The joy I felt at my college graduation ceremony" in the text input field.

[0493] Input: User voice or text data

[0494] Output: Audio file or text data as stored data

[0495] Step 2:

[0496] Emotion recognition

[0497] Device: The device converts the input voice data into text using a speech recognition function, then passes the text data to an emotion recognition engine to recognize the emotional state.

[0498] What it does: It converts voice data into text using the Google Speech-to-Text API, then passes the text data to IBM Watson Emotion Analysis to identify emotional states such as "joy" or "sadness."

[0499] Input: Audio file or text data as storage data

[0500] Output: Recognized emotion data (e.g., happy, sad)

[0501] Step 3:

[0502] Sending data

[0503] Device: The device sends the recognized emotion data and text data to the server in JSON format.

[0504] Specific operation: The data is sent as an HTTP request in the format { "text": "The joy I felt at my college graduation ceremony", "emotion": "Joy"}.

[0505] Input: Recognized emotion data and text data

[0506] Output: JSON data sent to the server

[0507] Step 4:

[0508] Data reception and analysis

[0509] Server: The server receives the data sent from the device and passes the received text data to the AI ​​module for natural language processing (NLP).

[0510] What it does: It uses AI modules (e.g. TensorFlow and PyTorch) to analyze the content of text and extract important phrases and keywords.

[0511] Input: JSON data sent from the terminal

[0512] Output: Extracted phrases and keywords (e.g., "university," "graduation," "joy")

[0513] Step 5:

[0514] Contextual Analysis

[0515] AI module: Understands the context of memories based on extracted keywords and phrases, along with emotional data.

[0516] Specific operation: The memory content is tagged and structured based on extracted phrases, keywords, and emotional data.

[0517] Input: Extracted phrases and keywords, and sentiment data

[0518] Output: Structured stored data

[0519] Step 6:

[0520] Data storage

[0521] Server: Converts structured stored data into a digital format and stores it in a database.

[0522] Specific operation: Save data with tagging and category information such as "emotion," "event," "location," etc. For example, set tags as follows: tags: ["university," "graduation ceremony," "joy"]

[0523] Input: Structured stored data

[0524] Output: Memory data stored in the database

[0525] Step 7:

[0526] Memory retrieval

[0527] User: Another user uses an application on the device to search for memories, for example, searching for "college graduation joy."

[0528] Specific actions: Enter "college," "graduation," and "joy" in the search bar and click the search button.

[0529] Input: Search keyword

[0530] Output: Data sent to the server as a search query

[0531] Step 8:

[0532] Providing search results

[0533] Server: Searches the database for stored data that matches the search criteria and sends it to the terminal as an HTTP response.

[0534] Specific operation: The found stored data is sent to the device as an API response and converted into a display format.

[0535] Input: search query

[0536] Output: Corresponding stored data

[0537] Step 9:

[0538] Displaying search results

[0539] Device: Displays the received search results to the user. For example, it displays the title such as "Memories of XX's University Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[0540] What it does: It displays search results in a list format and provides a detailed view that displays more information when you click on it.

[0541] Input: Search results received from the server

[0542] Output: The search results that are displayed to the user

[0543] Step 10:

[0544] Purchasing and donating memories

[0545] User: If they want to purchase a memory they like, they press the purchase button and enter their payment information.

[0546] What happens: You tap the buy button, enter your payment information, and confirm your purchase.

[0547] Input: Purchase request and payment information

[0548] Output: Purchase request sent to the server

[0549] Step 11:

[0550] Payment processing and provision

[0551] Server: Receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal). Then, generates a download link or streaming URL for the purchased stored data and provides it to the user.

[0552] Specific operation: After payment confirmation, an access link for the stored data is generated and sent to the user.

[0553] Input: Purchase request and payment information

[0554] Output: A download link or streaming URL provided to the user

[0555] (Application example 2)

[0556] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0557] Conventional memory data management systems have limited user memory input to text or voice data, making it impossible to recognize emotions or sell memory data in virtual stores. This has limited the user experience, making it difficult to share or trade memory data effectively.

[0558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0559] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, provision means for providing purchased memory data to users, virtual store display means for selling memory data in a virtual space, emotion analysis means with emotion recognition capabilities, voice recognition means for converting voice data into text, and purchase management means for processing the purchase procedure for memory data through a payment means. This allows users to record their own memories in detail, analyze them including emotions, and sell, search, and purchase them in a virtual store.

[0560] A "user" is someone who uses this system and performs operations such as memory input, search, and purchase.

[0561] "Memories" refers to information about personal experiences and memories entered by the user.

[0562] "Input means" refers to a device or interface that allows a user to input memories, including voice and text input options.

[0563] The term "server means" refers to computer resources that receive input stored data and perform various processes such as analysis.

[0564] "AI module" refers to the artificial intelligence algorithms and software used to analyze input memory data and extract key phrases and keywords.

[0565] "Storage means" refers to a mechanism for converting the analyzed stored data into a digital format and storing it in a database.

[0566] "Search facility" refers to the interface and associated mechanisms by which a user searches the memory.

[0567] "Purchase vehicle" refers to the interface and associated mechanisms for a user to purchase selected storage.

[0568] "Providing means" refers to a mechanism for providing the purchased stored data to the user, and includes providing the data in the form of a download link or a streaming URL.

[0569] "Virtual store display means" refers to a display interface and related mechanisms for selling stored data in a virtual space.

[0570] "Emotion analysis means" refers to a mechanism for recognizing and analyzing the user's emotions from stored data.

[0571] "Speech recognition means" refers to a mechanism for converting voice data into text data.

[0572] "Purchase management means" refers to a mechanism for processing the purchase procedure for stored data through a payment means.

[0573] "Payment Method" means the electronic payment system or service used to complete a Purchase.

[0574] The present invention relates to a system that allows users to input their own memories, recognize and analyze their emotions, and sell and purchase items in a virtual store. Specific embodiments are described in detail below.

[0575] Memory input and emotion recognition

[0576] Memory input

[0577] Users install and launch a dedicated application for recording memories on their smartphone. This application offers the option of voice input or text input. For example, to record "memories of college graduation," a user can select voice input and speak into the microphone, "the joy I felt at my college graduation ceremony." Alternatively, they can write their memories in the text input field.

[0578] Emotion recognition

[0579] The application analyzes the voice or text data entered by the user in real time and uses an emotion analysis engine to recognize the user's emotional state. For voice data, it uses the Google Cloud Speech-to-Text API to convert the speech to text. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotion of text data. For example, it determines whether the user is expressing emotions such as "happiness," "sadness," or "surprise" based on the tone and speed of the voice and the emotion analysis of the text. This emotion data is sent to the server along with the input data.

[0580] Sending and Receiving Data

[0581] Sending data

[0582] The terminal transmits the converted text data or the directly input text data and the emotion data as an HTTP request to the server.

[0583] Receiving data

[0584] The server receives the emotion-recognized stored data transmitted from the terminal.

[0585] Memory analysis

[0586] Performing natural language processing

[0587] The server passes the received text data to a generative AI module, preparing it for natural language processing (NLP). The generative AI model uses, for example, the Transformers library from Hugging Face.

[0588] Keyword extraction

[0589] The AI ​​module uses NLP to analyze the text data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[0590] Contextual Analysis

[0591] The AI ​​module understands the context of the memory based on extracted keywords, phrases, and emotional data, and then tags and structures the detailed content.

[0592] Preserving memories

[0593] Database storage

[0594] The server categorizes and tags the memory data based on the analysis results and emotional data. The tags include category information such as "emotion," "event," and "location." The memory data converted into digital format is stored in an SQL database (MySQL or PostgreSQL).

[0595] Searching and Purchasing Memory

[0596] Memory retrieval

[0597] A user can use the application's search function to search for memories entered by other users. By entering a desired emotional state along with search keywords, the user can find the corresponding memory data. For example, searching for "graduation ceremony" and "joy."

[0598] Getting search results

[0599] The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[0600] Displaying search results

[0601] The terminal displays the search results received from the server to the user, such as a title such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)," along with detailed information and emotional state.

[0602] Purchasing and donating memories

[0603] Purchasing Memory

[0604] When the user selects a memory they like and presses the purchase button to purchase it, the terminal sends a purchase request to the server.

[0605] Payment Processing

[0606] The server receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal).

[0607] Providing stored data

[0608] The server generates a download link or streaming URL for the purchased stored data and provides it to the user. The terminal displays the link or URL provided by the server to the user, allowing the user to access the purchased stored data.

[0609] Specific examples

[0610] Consider a scenario in which a user registers their "joy felt at their university graduation ceremony" by voice input into the system, and other users search for and purchase this item using the keywords "graduation ceremony" and "joy." This system provides a way for individual users to experience and share their memories and emotions in a virtual store.

[0611] Prompt Sentence Examples

[0612] "Tell me about the joy you felt at your college graduation."

[0613] "Please enter your emotional memory of graduation."

[0614] "Search for the keyword 'graduation' and the emotion 'joy'"

[0615] In this way, the system of the present invention inputs the user's memories and recognizes, analyzes, stores, searches, and provides emotions, thereby realizing functions that enrich the user's experience.

[0616] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0617] Step 1:

[0618] The user launches a dedicated application on their smartphone for inputting memories. They can select either voice input or text input as the input method. If voice input is selected, the user speaks their memories into a microphone. If text input is selected, the user inputs their memories using a keyboard. This input data is sent to the application as voice data or text data.

[0619] Step 2:

[0620] The device processes the input data in real time, converting voice data into text using the Google Cloud Speech-to-Text API, and then performs emotion analysis on the converted text data using IBM Watson Tone Analyzer. The analysis results in emotional data such as joy, sadness, and surprise.

[0621] Step 3:

[0622] The device then sends the converted text data and emotion data to the server as an HTTP request, which includes the user's input memory and associated emotion.

[0623] Step 4:

[0624] The server receives the memory data and emotion data sent from the terminal. This received data includes the user's memory information and emotion information.

[0625] Step 5:

[0626] The server prepares the received memory data for natural language processing (NLP) using a generative AI model, sending the data to an NLP module using a library such as Hugging Face's Transformers library, and begins analysis.

[0627] Step 6:

[0628] The AI ​​module uses NLP to analyze the memory data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[0629] Step 7:

[0630] The AI ​​module understands the context of memories based on extracted keywords, phrases, and emotional data, and then tags and structures the details, assigning tags such as "emotion," "event," and "location" to the memory data.

[0631] Step 8:

[0632] The server converts the tagged analysis results into a digital format and stores them in an SQL database such as MySQL or PostgreSQL, including the tagged stored data.

[0633] Step 9:

[0634] A user makes a request to search for a memory from a terminal. The search request includes keywords, such as "graduation ceremony" and "joy," as well as a desired emotional state.

[0635] Step 10:

[0636] The server searches the database based on the received search request, retrieves the corresponding stored data as search results, and sends the search results to the terminal as an HTTP response.

[0637] Step 11:

[0638] The terminal displays the search results received from the server to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)" are displayed.

[0639] Step 12:

[0640] The user goes through the process of purchasing a memory they like. When they press the purchase button, the device sends a purchase request to the server.

[0641] Step 13:

[0642] The server receives the purchase request, processes the payment through a payment system such as Stripe or PayPal, and, if successful, generates a download link or streaming URL for the stored data.

[0643] Step 14:

[0644] The server sends the generated download link or streaming URL to the terminal, which displays it to the user, allowing the user to access the stored data they purchased.

[0645] Prompt Sentence Examples

[0646] "Tell me about the joy you felt at your college graduation."

[0647] "Please enter your emotional memory of graduation."

[0648] "Search for the keyword 'graduation' and the emotion 'joy'"

[0649] Through the above processing steps, a system is provided that allows users to record their own memories in detail, share them with other users, and trade in virtual stores.

[0650] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0651] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0652] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0653] [Second embodiment]

[0654] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0655] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0656] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0657] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0658] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0659] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0660] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0661] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0662] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0663] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0664] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0665] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0666] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments will be described below.

[0667] Memory input and analysis

[0668] Memory input

[0669] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0670] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[0671] In the case of text input, the same content is written in the text field.

[0672] Sending and Receiving Data

[0673] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[0674] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[0675] Memory analysis

[0676] Natural Language Processing

[0677] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[0678] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[0679] Contextual Analysis

[0680] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[0681] Preserving memories

[0682] Database storage

[0683] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[0684] Server: Stores the stored data in a database after converting it into digital format.

[0685] Searching and Purchasing Memory

[0686] Memory retrieval

[0687] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[0688] Terminal: Sends the entered search query to the server.

[0689] Retrieving and displaying search results

[0690] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0691] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0692] Purchasing and donating memories

[0693] User: Select a memory they like and press the purchase button to purchase it.

[0694] Device: Sends a purchase request to the server.

[0695] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0696] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[0697] This allows users to digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy those memories.

[0698] The processing flow will be explained below.

[0699] Memory input and analysis

[0700] Step 1: Entering memory

[0701] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0702] For example, a user may select voice input and speak into a microphone to record "memories of a high school soccer tournament," or type into a text input field.

[0703] Step 2: Convert audio data to text

[0704] On your device: If you select voice input, your audio recording will be converted to text using transcription software, and the converted text will be temporarily stored in your device's memory.

[0705] Step 3: Sending data

[0706] Terminal: The converted text data or the text data entered directly is sent to the server as an HTTP request.

[0707] Step 4: Receiving the data

[0708] Server: Receives the stored data sent from the device and checks the data format (voice data or text data).

[0709] Memory analysis

[0710] Step 5: Performing Natural Language Processing

[0711] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[0712] Server: Sends data to the NLP module and starts the analysis.

[0713] Step 6: Keyword extraction

[0714] AI module: Analyzes text data using NLP to extract important phrases and keywords.

[0715] For example, important information such as "high school," "soccer tournament," and "championship" is analyzed.

[0716] Step 7: Context Analysis

[0717] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the detailed content.

[0718] Preserving memories

[0719] Step 8: Structuring the database

[0720] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[0721] Server: Prepares structured stored data for conversion into digital format.

[0722] Step 9: Save to Database

[0723] Server: Stores the stored data in a database after converting it into digital format.

[0724] Searching and Purchasing Memory

[0725] Step 10: Searching your memory

[0726] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[0727] Terminal: Sends the entered search query to the server.

[0728] Step 11: Getting search results

[0729] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0730] Step 12: Viewing search results

[0731] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0732] Purchasing and donating memories

[0733] Step 13: Buy Memory

[0734] User: Select a memory they like and press the purchase button to purchase it.

[0735] Device: Sends a purchase request to the server.

[0736] Step 14: Payment Processing

[0737] Server: Receives the purchase request and processes the payment through the payment system.

[0738] Step 15: Confirm your purchase

[0739] Server: Once payment is complete, verify that the purchase was successful.

[0740] Step 16: Providing Stored Data

[0741] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[0742] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[0743] In this way, users can digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy them.

[0744] Example 1

[0745] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] Until now, there has been no system that can handle the entire process of inputting, analyzing, storing, searching, and purchasing memories in a unified manner. Furthermore, converting voice data into text, and tagging and classifying analysis results are time-consuming and difficult, reducing usability. This has made it difficult for users to easily digitize their memories and share or sell them with other users.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0748] In this invention, the server includes input means for inputting memories from users, conversion means for acquiring the input memory data and converting voice data into text data, transmission means for transmitting the acquired memory data via an HTTP request, analysis means for receiving the transmitted memory data and performing natural language processing and context analysis using an AI module, storage means for classifying, tagging, and storing the analyzed memory data in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data in the form of a download link or streaming URL. This allows users to easily digitize their memories and share or sell them with other users over the network.

[0749] A "user" is someone who uses the system to enter, search, and purchase memories.

[0750] "Input means" refers to a device or function that allows a user to input memories, and provides options for voice input and text input.

[0751] "Conversion means" refers to a device or software for converting voice data into text data, and uses voice recognition technology.

[0752] The "transmission means" refers to a function for transmitting the acquired stored data to the server, and transmits the data via an HTTP request.

[0753] "Analysis means" refers to devices and software that process received stored data using natural language processing (NLP) and contextual analysis.

[0754] "Storage means" refers to the function of converting the analyzed stored data into a digital format, appropriately classifying and tagging it, and storing it in a database.

[0755] "Search means" refers to a function that allows a user to search memories, and searches for relevant stored data from a database through a search query.

[0756] "Purchase means" refers to a function that allows a user to purchase a memory that they like, and executes payment processing.

[0757] The "means for providing" refers to a function for providing the purchased stored data to the user, and is provided in the form of a download link or a streaming URL.

[0758] "Database" refers to an electronic storage device for storing analyzed and archived stored data.

[0759] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments are described below.

[0760] Memory input

[0761] User: Launches the dedicated application and inputs a memory. The user is provided with the option of voice input or text input. For voice input, the user records the memory using the device's microphone. For example, the user speaks, "When I played in the high school soccer tournament," to obtain voice data. For text input, the user inputs the memory using the device's keyboard. For example, the user types, "When I played in the high school soccer tournament," into the text field.

[0762] Device: If the user selects voice input, the device uses speech recognition software to convert the recorded voice data into text data. Specifically, the device uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the voice data into text data. The converted text data, or text data entered directly by the user, is sent to the server as an HTTP request.

[0763] Sending and Receiving Data

[0764] Terminal: Generates an HTTP request and sends the retrieved text data to the server.

[0765] Server: Receives the HTTP request and processes the text data. If the data is audio, speech recognition is performed again on the server side.

[0766] Memory analysis

[0767] Server: The received text data is sent to the AI ​​module for natural language processing (NLP). Specifically, the text data is analyzed using advanced NLP techniques such as OpenAI's GPT-4 API. Key phrases and keywords are extracted based on the analysis results. For example, information such as "high school," "soccer tournament," and "championship" are extracted.

[0768] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[0769] Preserving memories

[0770] Server: Classifies and tags the stored data based on the analysis results. Tags include category information such as "sports," "school," and "tournament." The analyzed and classified data is saved in a database. Common database technologies such as MySQL and MongoDB are used for the database.

[0771] Searching and Purchasing Memory

[0772] User: If another user wants to use an application on their device to search for "memories of high school soccer tournaments," they enter a search keyword.

[0773] Terminal: Sends the entered search query to the server.

[0774] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0775] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0776] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[0777] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0778] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[0779] Specific examples (prompt sentence examples)

[0780] Example voice input: "I want to record my memories of playing in a high school soccer tournament."

[0781] Example text entry: "I want to record my time playing in a high school soccer tournament."

[0782] This system allows users to digitize their memories, share them with other users, and sell them, and users who purchase them can enjoy them.

[0783] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0784] Step 1:

[0785] User: Launches the dedicated application and starts inputting memories. To input memories, the user can choose between voice input or text input. For example, the user can say "When I played in the high school soccer tournament" or input the same content in text.

[0786] Input: Audio or text data storage

[0787] Output: Continuously updated audio data or consistent text data

[0788] Specific operation: The user opens the application on the device, follows the instructions, speaks the memory aloud using the microphone, presses the voice input button to start recording, and when finished, presses the button again to stop recording.

[0789] Step 2:

[0790] Device: When voice data is acquired, the device converts the voice data into text data using speech recognition technology such as the Google Cloud Speech-to-Text API, and temporarily stores the converted text data and text data entered directly by the user.

[0791] Input: Audio data

[0792] Output: Text data

[0793] Specific operation: The device sends voice data to the speech recognition API, receives the converted text data, and displays it in a text field or stores it in a variable.

[0794] Step 3:

[0795] Terminal: The acquired text data is sent to the server as an HTTP request. The user's memory data is temporarily stored locally at this stage and then sent to the server.

[0796] Input: Text data

[0797] Output: HTTP request

[0798] Specific operation: The device application converts the text data into JSON format and sends it to the specified URL on the server using an HTTP POST request.

[0799] Step 4:

[0800] Server: Receives the HTTP request and prepares to analyze the stored data. If voice data is included, voice recognition is performed again.

[0801] Input: HTTP request

[0802] Output: Text data

[0803] Specific operation: The server side receives the HTTP request, analyzes the payload, extracts text data, and if it contains audio data, calls the speech recognition API on the server side to convert it into text.

[0804] Step 5:

[0805] Server: Sends the received text data to an AI module (e.g., OpenAI GPT-4 API) for natural language processing (NLP), which extracts key phrases and keywords.

[0806] Input: Text data

[0807] Output: Key phrases and keywords

[0808] Specific operation: The server passes the text data to the AI ​​module and sends the prompt "Analyze the following text and extract key phrases and keywords: About playing in a high school soccer tournament." The AI ​​module responds with the extracted keywords.

[0809] Step 6:

[0810] Server: Understands the context of stored data based on extracted keywords, and tags and structures detailed content.

[0811] Input: Extracted keywords

[0812] Output: Tagged text data

[0813] Specific operation: Based on the analysis results from the AI ​​module, the server applies a tagging algorithm to assign category information such as "sports," "school," or "tournament."

[0814] Step 7:

[0815] Server: The tagged and classified data is stored in a database using database technologies such as MySQL or MongoDB.

[0816] Input: Tagged and classified text data

[0817] Output: Memory data stored in the database

[0818] Specific operation: Executes an INSERT statement on the database installed on the server and saves the analysis results.

[0819] Step 8:

[0820] User: Another user enters a search keyword to search for "high school soccer tournament memories" on their device.

[0821] Input: Search keyword

[0822] Output: Search query

[0823] What happens: A user types "high school soccer tournament" into the application's search field and presses the search button.

[0824] Step 9:

[0825] Device: Sends a search query to the server via an HTTP GET request.

[0826] Input: Search keyword

[0827] Output: HTTP GET request

[0828] Specific operation: The device formats the search keywords as query parameters into an HTTP GET request and sends it to the server's search API.

[0829] Step 10:

[0830] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0831] Input: HTTP GET request

[0832] Output: Search result data

[0833] Specific operation: The server searches the database, extracts the corresponding stored data, and returns the extracted data in JSON format as an HTTP response.

[0834] Step 11:

[0835] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0836] Input: Search result data

[0837] Output: Display of search results

[0838] Specific operation: The device parses the received search results and displays them in the user interface, arranging the title, part of the text, tag information, etc. in the display area.

[0839] Step 12:

[0840] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[0841] Input: Purchase Information

[0842] Output: Purchase request

[0843] Specific operation: When the user clicks the purchase button, the device sends an HTTP POST request to the server containing the necessary information for the purchase.

[0844] Step 13:

[0845] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0846] Input: Purchase Request

[0847] Output: Download link or streaming URL

[0848] Specific operation: The server calls a payment API (e.g., Stripe or PayPal) to process the payment. Once the purchase is confirmed, a download link or streaming URL for the corresponding stored data is generated.

[0849] Step 14:

[0850] On the device: The download link or streaming URL received from the server is displayed to the user, allowing them to access the stored data they have purchased.

[0851] Input: Download link or streaming URL

[0852] Output: Display of links and URLs

[0853] Specific operation: The device analyzes the links and URLs received from the server and displays them in the user interface. The user can click on the links to access the stored data.

[0854] This allows users to digitize their memories and share or sell them with other users, who can then enjoy the memories they have purchased.

[0855] (Application example 1)

[0856] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0857] Conventional memory data management systems analyze and store memories entered by users, allowing other users to search and purchase them, but lack the means to recreate those memories more specifically and provide them in a form that users can experience. Furthermore, the effective use of memory data is limited, and sharing of experiences is limited to visual and auditory information, so there is a demand for providing deeper emotional experiences.

[0858] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0859] In this invention, the server includes an input means for inputting memories from a user, a server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, a natural language processing means for analyzing the context of the memory based on the extracted phrases and keywords, a storage means for converting the analyzed memory data into a digital format and storing it in a database, a search and purchase means for the user to search for and purchase memories, and a virtual reality linkage means for providing the processed data to the user as virtual reality content. This not only enables advanced analysis of the user's memories and storage of their context, but also makes it possible to provide the memory data in a form that other users can experience realistically through virtual reality.

[0860] "User" refers to the entity that uses the system to input, search, and purchase memories.

[0861] "Memories" refer to records of events and emotions experienced by the user, and are input as audio or text data.

[0862] "Input means" refers to a device or interface that allows a user to input memories as voice or text data.

[0863] "Server means" refers to a computer system that receives and processes stored data input by a user.

[0864] "AI Module" refers to artificial intelligence software that analyzes received stored data and extracts important phrases and keywords.

[0865] "Natural language processing means" refers to technology for analyzing memory context based on extracted phrases and keywords.

[0866] "Storage means" refers to a function that converts the analyzed stored data into a digital format and stores it in a database.

[0867] "Search and purchase means" refers to an interface that allows a user to search and purchase saved stored data.

[0868] "Delivery means" refers to a mechanism for providing purchased storage data to a user.

[0869] "Virtual reality linking means" refers to a technology that provides processed stored data to users as virtual reality content.

[0870] This invention relates to a system that allows users to input personal memories, analyze and store them using natural language processing (NLP) technology, and then allows other users to search and purchase them. The purchased memory data can then be reproduced as virtual reality (VR) content.

[0871] Memory input and analysis

[0872] Memory Input:

[0873] User: To input a memory, launch a dedicated application on the device. The application offers the option of voice input or text input.

[0874] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[0875] In the case of text input, the same content is written in the text field.

[0876] Sending and receiving data:

[0877] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[0878] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[0879] Natural Language Processing:

[0880] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[0881] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[0882] Contextual analysis:

[0883] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[0884] Preserving memories

[0885] Database storage

[0886] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[0887] Server: Stores the stored data in a database after converting it into digital format.

[0888] Searching and Purchasing Memory

[0889] Memory Retrieval:

[0890] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[0891] Terminal: Sends the entered search query to the server.

[0892] Retrieving and displaying search results:

[0893] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[0894] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[0895] Purchasing and donating memory:

[0896] User: Select a memory they like and press the purchase button to purchase it.

[0897] Device: Sends a purchase request to the server.

[0898] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[0899] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[0900] Virtual reality reproduction

[0901] Virtual Reality Integration:

[0902] Server: Reconstructs the purchased memory data as virtual reality content and generates links.

[0903] User: Experience virtual reality content using smart glasses or a head-mounted display.

[0904] Specific examples and prompts for the generative AI model

[0905] Examples:

[0906] User A: Inputs a memory of a soccer tournament from high school, and talks about the changes in emotions and detailed episodes that occurred during the event.

[0907] Server: Converts voice data into text and analyzes it using NLP. Stores the analysis results in a database.

[0908] User B: Searches for "high school soccer tournament" in the virtual store and finds User A's memory.

[0909] Server: Returns search results and generates a VR link. User B enjoys the VR experience through the link.

[0910] Prompt statement:

[0911] "Tell me about a time when you participated in a high school soccer tournament. Please describe in detail any particularly memorable episodes or emotions."

[0912] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0913] Step 1:

[0914] User enters memory:

[0915] Input: The user launches a dedicated application on the device and inputs the memory by voice or text.

[0916] Processing: In the case of voice input, the application records the voice data, and in the case of text input, it receives the text data as is.

[0917] Output: Audio or text data.

[0918] Step 2:

[0919] The device converts the data:

[0920] Input: Audio data.

[0921] Processing: When a voice command is received, the device converts the voice data into text using transcription software (e.g., Google Speech-to-Text API or IBM Watson).

[0922] Output: Text data.

[0923] Step 3:

[0924] Sending data:

[0925] Input: Text data.

[0926] Processing: The terminal sends the converted text data or the directly entered text data to the server as an HTTP request.

[0927] Output: HTTP request (text data).

[0928] Step 4:

[0929] Server receives data:

[0930] Input: HTTP request (text data).

[0931] Processing: The server receives the stored data sent from the terminal.

[0932] Output: Received text data.

[0933] Step 5:

[0934] Perform natural language processing:

[0935] Input: Received text data.

[0936] Processing: The server uses AI modules (e.g., Hugging Face Transformers) to perform natural language processing (NLP) and extract key phrases and keywords.

[0937] Output: Extracted phrases and keywords.

[0938] Step 6:

[0939] Contextual analysis:

[0940] Input: Extracted phrases or keywords.

[0941] Processing: The server's AI module analyzes the memory context based on phrases and keywords, and tags and structures the details.

[0942] Output: Tagged data.

[0943] Step 7:

[0944] Save to database:

[0945] Input: Tagged data.

[0946] Processing: The server converts the tagged stored data into a digital format and stores it in a database.

[0947] Output: Memorized data stored in a database.

[0948] Step 8:

[0949] User searches memory:

[0950] Input: Search keyword.

[0951] Process: The user enters search keywords into an application on the device, and the device sends the search query to the server as an HTTP request.

[0952] Output: HTTP request (search query).

[0953] Step 9:

[0954] The server retrieves the search results:

[0955] Input: HTTP request (search query).

[0956] Processing: The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[0957] Output: HTTP response (search results).

[0958] Step 10:

[0959] Display search results:

[0960] Input: HTTP response (search result).

[0961] Processing: The terminal displays the search results received from the server to the user. For example, it displays the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament."

[0962] Output: What is displayed to the user.

[0963] Step 11:

[0964] User purchases memory:

[0965] Input: Purchase request.

[0966] Processing: The user selects a memory they like and presses the purchase button. The device sends a purchase request to the server.

[0967] Output: HTTP request (purchase request).

[0968] Step 12:

[0969] The server processes the payment:

[0970] Input: HTTP request (purchase request).

[0971] Processing: The server processes the payment and generates a download link or streaming URL for the stored data for which the purchase has been confirmed.

[0972] Output: Download link or streaming URL.

[0973] Step 13:

[0974] Providing purchase data:

[0975] Input: Download link or streaming URL.

[0976] Processing: The device will display a download link or streaming URL to the user, allowing them to access the stored data they purchased.

[0977] Output: Provide link to user.

[0978] Step 14:

[0979] Virtual reality reproduction:

[0980] Input: Purchased memory data.

[0981] Processing: Reconstructs purchased memory data into virtual reality content and generates a link.

[0982] Output: Virtual reality experience link.

[0983] Step 15:

[0984] Experience virtual reality content:

[0985] Input: Virtual reality experience link.

[0986] Processing: Users experience virtual reality content using smart glasses or a head-mounted display.

[0987] Output: Experience virtual reality content.

[0988] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0989] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memories input by a user and the emotional state of the memories. Specific embodiments will be described in detail below.

[0990] Memory input and emotion recognition

[0991] Memory input

[0992] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[0993] For example, to record "Memories of my college graduation ceremony," a user selects voice input and speaks into the microphone, "The joy I felt at my college graduation ceremony," or writes the memory in the text input field.

[0994] Emotion recognition

[0995] Device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. For example, it determines whether the user is expressing emotions such as joy, sadness, or surprise through voice tone and speed, as well as sentiment analysis of the text.

[0996] Terminal: The recognized emotion data is sent to the server along with the input data.

[0997] Sending and Receiving Data

[0998] Sending data

[0999] Terminal: The converted text data or directly entered text data and emotion data are sent to the server as an HTTP request.

[1000] Receiving data

[1001] Server: Receives emotion-recognized memory data sent from the device.

[1002] Memory analysis

[1003] Performing natural language processing

[1004] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[1005] Server: Sends data to the NLP module and starts the analysis.

[1006] Keyword extraction

[1007] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1008] Contextual Analysis

[1009] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[1010] Preserving memories

[1011] Database storage

[1012] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[1013] Server: Stores the stored data in a database after converting it into digital format.

[1014] Searching and Purchasing Memory

[1015] Memory retrieval

[1016] User: Another user searches for "joyful memories of college graduation" in an application on their device and enters search keywords and the desired emotional state.

[1017] Terminal: Sends the entered search query to the server.

[1018] Getting search results

[1019] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1020] Displaying search results

[1021] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[1022] Purchasing and donating memories

[1023] Purchasing Memory

[1024] User: Select a memory they like and press the purchase button to purchase it.

[1025] Device: Sends a purchase request to the server.

[1026] Payment Processing

[1027] Server: Receives the purchase request and processes the payment through the payment system.

[1028] Providing stored data

[1029] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[1030] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[1031] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search and purchase them. Furthermore, users who purchase them can enjoy the emotional state of their memories as well.

[1032] The processing flow will be explained below.

[1033] Memory input and emotion recognition

[1034] Step 1: Entering memory

[1035] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1036] For example, if a user wants to record "memories of college graduation," they can select voice input and speak into the microphone, "the joy I felt at college graduation," or write their memories in the text input field.

[1037] Step 2: Recognize emotions

[1038] On the device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. It uses tone and speed of speech and sentiment analysis of text to determine whether the user is expressing emotions such as joy, sadness, or surprise.

[1039] Step 3: Text Conversion (Speech Input)

[1040] Device: If voice input is selected, the recorded voice data is converted to text data using transcription software, and the converted text data is temporarily stored.

[1041] Step 4: Sending data

[1042] Terminal: The converted text data or the directly entered text data and the recognized emotion data are sent to the server as an HTTP request.

[1043] Data reception and analysis

[1044] Step 5: Receiving the data

[1045] Server: Receives emotion-recognized memory data sent from the device.

[1046] Step 6: Performing Natural Language Processing

[1047] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[1048] Server: Sends data to the NLP module and starts the analysis.

[1049] Step 7: Keyword extraction

[1050] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1051] Step 8: Analyze the sentiment data

[1052] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[1053] Preserving memories

[1054] Step 9: Prepare to save to the database

[1055] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[1056] Step 10: Save to database

[1057] Server: Stores the stored data in a database after converting it into digital format.

[1058] Searching and Purchasing Memory

[1059] Step 11: Memory Retrieval

[1060] User: When another user searches for "joyful memories of college graduation" through an application on a terminal, the user inputs search keywords and a desired emotional state.

[1061] Terminal: Sends the entered search query to the server.

[1062] Step 12: Getting search results

[1063] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1064] Step 13: Viewing search results

[1065] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[1066] Purchasing and donating memories

[1067] Step 14: Buy Memory

[1068] User: Select the memory they like and press the purchase button if they wish to purchase it.

[1069] Device: Sends a purchase request to the server.

[1070] Step 15: Payment Processing

[1071] Server: Receives the purchase request and processes the payment through the payment system.

[1072] Step 16: Confirm your purchase

[1073] Server: Once payment is complete, verify that the purchase was successful.

[1074] Step 17: Provide stored data

[1075] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[1076] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[1077] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search for and purchase them. In addition, users who purchase the memories can enjoy the emotional state of the memories.

[1078] Example 2

[1079] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1080] In existing systems, when a user inputs a memory, it is difficult to recognize the specific emotional state and analyze, store, and provide the memory data. Therefore, there is a need for a system that can accurately recognize the emotional information in memory and analyze, store, search, and provide the data based on that information.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1082] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, emotion recognition means for recognizing emotional states from voice and text data, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data to users. This allows users to input, recognize, and store memories more accurately, and enables other users to search, purchase, and share memories, including their emotional information.

[1083] "Input means" refers to the method or device that a user uses to input a memory.

[1084] "Server means" refers to a central computing device for receiving and processing stored data sent by users.

[1085] "AI Module" refers to the artificial intelligence software module used to analyze received stored data and extract key phrases and keywords.

[1086] "Emotion recognizer" refers to a system or method for recognizing emotional states from speech and text data.

[1087] "Storage means" refers to a means or device for converting the analyzed stored data into a digital format and storing it in a database.

[1088] "Search and purchase means" refers to a method or device that allows a user to search for stored data and purchase it if necessary.

[1089] "Providing means" refers to a means or system for providing the purchased storage data to the user in the form of a download link or streaming URL.

[1090] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memory input by a user and the emotional state of that memory.

[1091] The system consists of the following elements:

[1092] 1. Input Method

[1093] User: The user launches a dedicated application on the device and inputs their memories. The application offers the option of voice input or text input.

[1094] Example: To record "Memories of my college graduation ceremony," a user taps the voice input button and speaks "The joy I felt at my college graduation ceremony," or writes "The joy I felt at my college graduation ceremony" in the text input field.

[1095] 2. Server Means

[1096] Terminal: The stored data entered by the user is sent to the server in JSON format, for example, { "text": "The joy felt at the university graduation ceremony", "emotion": "Joy"}.

[1097] Server: The server receives the data sent from the device and prepares it for processing.

[1098] 3. AI Module

[1099] Server: Passes the received text data to an AI module (e.g., TensorFlow or PyTorch) for natural language processing (NLP).

[1100] Example: Using NLP to extract important phrases and keywords such as "university," "graduation," and "joy" from text data.

[1101] 4. Emotion recognition means

[1102] On the device: An emotion engine is used to analyze voice and text data, identifying emotions from the tone and rate of voice and the content of text.

[1103] Example: Recognizing emotional states such as happiness, sadness, surprise, etc. For example, identifying "happiness" based on tone and rate of speech and text.

[1104] 5. Preservation means

[1105] Server: Converts the analyzed memory data into a digital format and stores it in a database.

[1106] Example: Save data with tagging that includes category information such as "emotion," "event," "place," etc. For example, set tags like tags: ["university," "graduation ceremony," "joy"].

[1107] 6. Search and Purchase Methods

[1108] User: Another user searches a memory to find information about a specific emotion or event. Enter search keywords and an emotional state.

[1109] Example: A user types "college," "graduation," and "joy" into the app's search bar and clicks the search button.

[1110] Device: Sends a search query to the server.

[1111] 7. Means of provision

[1112] Server: Provides the found stored data to the user in the form of a download link or streaming URL.

[1113] Example: Displaying a title such as "Memories of Mr. / Ms. XX's College Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[1114] User: Selects the memory they like, purchases it, enters payment information and taps the purchase button.

[1115] Server: Receives the purchase request, processes the payment through a payment system (e.g., Stripe or PayPal), and then generates a download link or streaming URL for the purchased stored data and provides it to the user.

[1116] 8. Examples of concrete examples and prompts

[1117] Example: User A inputs the joy of his / her college graduation into a memory, and the emotion is recognized and saved as "joy." User B searches for "college," "graduation ceremony," and "joy," discovers User A's memory, and purchases it, allowing him / her to learn more about the joy.

[1118] Example prompt: "Describe a system that recognizes and stores emotional states based on memories of your college graduation. Explain in detail how the system analyzes and stores the data, using the example memory of 'summer family vacation.'"

[1119] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1120] Step 1:

[1121] Memory input

[1122] User: The user launches a dedicated application on the device and inputs their memories. The application provides the option of voice input or text input.

[1123] Specific actions: To record "Memories of my college graduation ceremony," the user taps the voice input button and speaks "The joy I felt at my college graduation ceremony" into the microphone, or writes "The joy I felt at my college graduation ceremony" in the text input field.

[1124] Input: User voice or text data

[1125] Output: Audio file or text data as stored data

[1126] Step 2:

[1127] Emotion recognition

[1128] Device: The device converts the input voice data into text using a speech recognition function, then passes the text data to an emotion recognition engine to recognize the emotional state.

[1129] What it does: It converts voice data into text using the Google Speech-to-Text API, then passes the text data to IBM Watson Emotion Analysis to identify emotional states such as "joy" or "sadness."

[1130] Input: Audio file or text data as storage data

[1131] Output: Recognized emotion data (e.g., happy, sad)

[1132] Step 3:

[1133] Sending data

[1134] Device: The device sends the recognized emotion data and text data to the server in JSON format.

[1135] Specific operation: The data is sent as an HTTP request in the format { "text": "The joy I felt at my college graduation ceremony", "emotion": "Joy"}.

[1136] Input: Recognized emotion data and text data

[1137] Output: JSON data sent to the server

[1138] Step 4:

[1139] Data reception and analysis

[1140] Server: The server receives the data sent from the device and passes the received text data to the AI ​​module for natural language processing (NLP).

[1141] What it does: It uses AI modules (e.g. TensorFlow and PyTorch) to analyze the content of text and extract important phrases and keywords.

[1142] Input: JSON data sent from the terminal

[1143] Output: Extracted phrases and keywords (e.g., "university," "graduation," "joy")

[1144] Step 5:

[1145] Contextual Analysis

[1146] AI module: Understands the context of memories based on extracted keywords and phrases, along with emotional data.

[1147] Specific operation: The memory content is tagged and structured based on extracted phrases, keywords, and emotional data.

[1148] Input: Extracted phrases and keywords, and sentiment data

[1149] Output: Structured stored data

[1150] Step 6:

[1151] Data storage

[1152] Server: Converts structured stored data into a digital format and stores it in a database.

[1153] Specific operation: Save data with tagging and category information such as "emotion," "event," "location," etc. For example, set tags as follows: tags: ["university," "graduation ceremony," "joy"]

[1154] Input: Structured stored data

[1155] Output: Memory data stored in the database

[1156] Step 7:

[1157] Memory retrieval

[1158] User: Another user uses an application on the device to search for memories, for example, searching for "college graduation joy."

[1159] Specific actions: Enter "college," "graduation," and "joy" in the search bar and click the search button.

[1160] Input: Search keyword

[1161] Output: Data sent to the server as a search query

[1162] Step 8:

[1163] Providing search results

[1164] Server: Searches the database for stored data that matches the search criteria and sends it to the terminal as an HTTP response.

[1165] Specific operation: The found stored data is sent to the device as an API response and converted into a display format.

[1166] Input: search query

[1167] Output: Corresponding stored data

[1168] Step 9:

[1169] Displaying search results

[1170] Device: Displays the received search results to the user. For example, it displays the title such as "Memories of XX's University Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[1171] What it does: It displays search results in a list format and provides a detailed view that displays more information when you click on it.

[1172] Input: Search results received from the server

[1173] Output: The search results that are displayed to the user

[1174] Step 10:

[1175] Purchasing and donating memories

[1176] User: If they want to purchase a memory they like, they press the purchase button and enter their payment information.

[1177] What happens: You tap the buy button, enter your payment information, and confirm your purchase.

[1178] Input: Purchase request and payment information

[1179] Output: Purchase request sent to the server

[1180] Step 11:

[1181] Payment processing and provision

[1182] Server: Receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal). Then, generates a download link or streaming URL for the purchased stored data and provides it to the user.

[1183] Specific operation: After payment confirmation, an access link for the stored data is generated and sent to the user.

[1184] Input: Purchase request and payment information

[1185] Output: A download link or streaming URL provided to the user

[1186] (Application example 2)

[1187] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1188] Conventional memory data management systems have limited user memory input to text or voice data, making it impossible to recognize emotions or sell memory data in virtual stores. This has limited the user experience, making it difficult to share or trade memory data effectively.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1190] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, provision means for providing purchased memory data to users, virtual store display means for selling memory data in a virtual space, emotion analysis means with emotion recognition capabilities, voice recognition means for converting voice data into text, and purchase management means for processing the purchase procedure for memory data through a payment means. This allows users to record their own memories in detail, analyze them including emotions, and sell, search, and purchase them in a virtual store.

[1191] A "user" is someone who uses this system and performs operations such as memory input, search, and purchase.

[1192] "Memories" refers to information about personal experiences and memories entered by the user.

[1193] "Input means" refers to a device or interface that allows a user to input memories, including voice and text input options.

[1194] The term "server means" refers to computer resources that receive input stored data and perform various processes such as analysis.

[1195] "AI module" refers to the artificial intelligence algorithms and software used to analyze input memory data and extract key phrases and keywords.

[1196] "Storage means" refers to a mechanism for converting the analyzed stored data into a digital format and storing it in a database.

[1197] "Search facility" refers to the interface and associated mechanisms by which a user searches the memory.

[1198] "Purchase vehicle" refers to the interface and associated mechanisms for a user to purchase selected storage.

[1199] "Providing means" refers to a mechanism for providing the purchased stored data to the user, and includes providing the data in the form of a download link or a streaming URL.

[1200] "Virtual store display means" refers to a display interface and related mechanisms for selling stored data in a virtual space.

[1201] "Emotion analysis means" refers to a mechanism for recognizing and analyzing the user's emotions from stored data.

[1202] "Speech recognition means" refers to a mechanism for converting voice data into text data.

[1203] "Purchase management means" refers to a mechanism for processing the purchase procedure for stored data through a payment means.

[1204] "Payment Method" means the electronic payment system or service used to complete a Purchase.

[1205] The present invention relates to a system that allows users to input their own memories, recognize and analyze their emotions, and sell and purchase items in a virtual store. Specific embodiments are described in detail below.

[1206] Memory input and emotion recognition

[1207] Memory input

[1208] Users install and launch a dedicated application for recording memories on their smartphone. This application offers the option of voice input or text input. For example, to record "memories of college graduation," a user can select voice input and speak into the microphone, "the joy I felt at my college graduation ceremony." Alternatively, they can write their memories in the text input field.

[1209] Emotion recognition

[1210] The application analyzes the voice or text data entered by the user in real time and uses an emotion analysis engine to recognize the user's emotional state. For voice data, it uses the Google Cloud Speech-to-Text API to convert the speech to text. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotion of text data. For example, it determines whether the user is expressing emotions such as "happiness," "sadness," or "surprise" based on the tone and speed of the voice and the emotion analysis of the text. This emotion data is sent to the server along with the input data.

[1211] Sending and Receiving Data

[1212] Sending data

[1213] The terminal transmits the converted text data or the directly input text data and the emotion data as an HTTP request to the server.

[1214] Receiving data

[1215] The server receives the emotion-recognized stored data transmitted from the terminal.

[1216] Memory analysis

[1217] Performing natural language processing

[1218] The server passes the received text data to a generative AI module, preparing it for natural language processing (NLP). The generative AI model uses, for example, the Transformers library from Hugging Face.

[1219] Keyword extraction

[1220] The AI ​​module uses NLP to analyze the text data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1221] Contextual Analysis

[1222] The AI ​​module understands the context of the memory based on extracted keywords, phrases, and emotional data, and then tags and structures the detailed content.

[1223] Preserving memories

[1224] Database storage

[1225] The server categorizes and tags the memory data based on the analysis results and emotional data. The tags include category information such as "emotion," "event," and "location." The memory data converted into digital format is stored in an SQL database (MySQL or PostgreSQL).

[1226] Searching and Purchasing Memory

[1227] Memory retrieval

[1228] A user can use the application's search function to search for memories entered by other users. By entering a desired emotional state along with search keywords, the user can find the corresponding memory data. For example, searching for "graduation ceremony" and "joy."

[1229] Getting search results

[1230] The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[1231] Displaying search results

[1232] The terminal displays the search results received from the server to the user, such as a title such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)," along with detailed information and emotional state.

[1233] Purchasing and donating memories

[1234] Purchasing Memory

[1235] When the user selects a memory they like and presses the purchase button to purchase it, the terminal sends a purchase request to the server.

[1236] Payment Processing

[1237] The server receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal).

[1238] Providing stored data

[1239] The server generates a download link or streaming URL for the purchased stored data and provides it to the user. The terminal displays the link or URL provided by the server to the user, allowing the user to access the purchased stored data.

[1240] Specific examples

[1241] Consider a scenario in which a user registers their "joy felt at their university graduation ceremony" by voice input into the system, and other users search for and purchase this item using the keywords "graduation ceremony" and "joy." This system provides a way for individual users to experience and share their memories and emotions in a virtual store.

[1242] Prompt Sentence Examples

[1243] "Tell me about the joy you felt at your college graduation."

[1244] "Please enter your emotional memory of graduation."

[1245] "Search for the keyword 'graduation' and the emotion 'joy'"

[1246] In this way, the system of the present invention inputs the user's memories and recognizes, analyzes, stores, searches, and provides emotions, thereby realizing functions that enrich the user's experience.

[1247] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1248] Step 1:

[1249] The user launches a dedicated application on their smartphone for inputting memories. They can select either voice input or text input as the input method. If voice input is selected, the user speaks their memories into a microphone. If text input is selected, the user inputs their memories using a keyboard. This input data is sent to the application as voice data or text data.

[1250] Step 2:

[1251] The device processes the input data in real time, converting voice data into text using the Google Cloud Speech-to-Text API, and then performs emotion analysis on the converted text data using IBM Watson Tone Analyzer. The analysis results in emotional data such as joy, sadness, and surprise.

[1252] Step 3:

[1253] The device then sends the converted text data and emotion data to the server as an HTTP request, which includes the user's input memory and associated emotion.

[1254] Step 4:

[1255] The server receives the memory data and emotion data sent from the terminal. This received data includes the user's memory information and emotion information.

[1256] Step 5:

[1257] The server prepares the received memory data for natural language processing (NLP) using a generative AI model, sending the data to an NLP module using a library such as Hugging Face's Transformers library, and begins analysis.

[1258] Step 6:

[1259] The AI ​​module uses NLP to analyze the memory data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1260] Step 7:

[1261] The AI ​​module understands the context of memories based on extracted keywords, phrases, and emotional data, and then tags and structures the details, assigning tags such as "emotion," "event," and "location" to the memory data.

[1262] Step 8:

[1263] The server converts the tagged analysis results into a digital format and stores them in an SQL database such as MySQL or PostgreSQL, including the tagged stored data.

[1264] Step 9:

[1265] A user makes a request to search for a memory from a terminal. The search request includes keywords, such as "graduation ceremony" and "joy," as well as a desired emotional state.

[1266] Step 10:

[1267] The server searches the database based on the received search request, retrieves the corresponding stored data as search results, and sends the search results to the terminal as an HTTP response.

[1268] Step 11:

[1269] The terminal displays the search results received from the server to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)" are displayed.

[1270] Step 12:

[1271] The user goes through the process of purchasing a memory they like. When they press the purchase button, the device sends a purchase request to the server.

[1272] Step 13:

[1273] The server receives the purchase request, processes the payment through a payment system such as Stripe or PayPal, and, if successful, generates a download link or streaming URL for the stored data.

[1274] Step 14:

[1275] The server sends the generated download link or streaming URL to the terminal, which displays it to the user, allowing the user to access the stored data they purchased.

[1276] Prompt Sentence Examples

[1277] "Tell me about the joy you felt at your college graduation."

[1278] "Please enter your emotional memory of graduation."

[1279] "Search for the keyword 'graduation' and the emotion 'joy'"

[1280] Through the above processing steps, a system is provided that allows users to record their own memories in detail, share them with other users, and trade in virtual stores.

[1281] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1282] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1283] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1284] [Third embodiment]

[1285] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1286] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1287] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1288] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1289] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1290] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1291] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1292] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1293] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1294] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1295] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1296] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1297] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments will be described below.

[1298] Memory input and analysis

[1299] Memory input

[1300] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1301] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[1302] In the case of text input, the same content is written in the text field.

[1303] Sending and Receiving Data

[1304] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[1305] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[1306] Memory analysis

[1307] Natural Language Processing

[1308] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[1309] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[1310] Contextual Analysis

[1311] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[1312] Preserving memories

[1313] Database storage

[1314] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[1315] Server: Stores the stored data in a database after converting it into digital format.

[1316] Searching and Purchasing Memory

[1317] Memory retrieval

[1318] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[1319] Terminal: Sends the entered search query to the server.

[1320] Retrieving and displaying search results

[1321] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1322] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1323] Purchasing and donating memories

[1324] User: Select a memory they like and press the purchase button to purchase it.

[1325] Device: Sends a purchase request to the server.

[1326] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[1327] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[1328] This allows users to digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy those memories.

[1329] The processing flow will be explained below.

[1330] Memory input and analysis

[1331] Step 1: Entering memory

[1332] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1333] For example, a user may select voice input and speak into a microphone to record "memories of a high school soccer tournament," or type into a text input field.

[1334] Step 2: Convert audio data to text

[1335] On your device: If you select voice input, your audio recording will be converted to text using transcription software, and the converted text will be temporarily stored in your device's memory.

[1336] Step 3: Sending data

[1337] Terminal: The converted text data or the text data entered directly is sent to the server as an HTTP request.

[1338] Step 4: Receiving the data

[1339] Server: Receives the stored data sent from the device and checks the data format (voice data or text data).

[1340] Memory analysis

[1341] Step 5: Performing Natural Language Processing

[1342] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[1343] Server: Sends data to the NLP module and starts the analysis.

[1344] Step 6: Keyword extraction

[1345] AI module: Analyzes text data using NLP to extract important phrases and keywords.

[1346] For example, important information such as "high school," "soccer tournament," and "championship" is analyzed.

[1347] Step 7: Context Analysis

[1348] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the detailed content.

[1349] Preserving memories

[1350] Step 8: Structuring the database

[1351] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[1352] Server: Prepares structured stored data for conversion into digital format.

[1353] Step 9: Save to Database

[1354] Server: Stores the stored data in a database after converting it into digital format.

[1355] Searching and Purchasing Memory

[1356] Step 10: Searching your memory

[1357] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[1358] Terminal: Sends the entered search query to the server.

[1359] Step 11: Getting search results

[1360] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1361] Step 12: Viewing search results

[1362] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1363] Purchasing and donating memories

[1364] Step 13: Buy Memory

[1365] User: Select a memory they like and press the purchase button to purchase it.

[1366] Device: Sends a purchase request to the server.

[1367] Step 14: Payment Processing

[1368] Server: Receives the purchase request and processes the payment through the payment system.

[1369] Step 15: Confirm your purchase

[1370] Server: Once payment is complete, verify that the purchase was successful.

[1371] Step 16: Providing Stored Data

[1372] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[1373] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[1374] In this way, users can digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy them.

[1375] Example 1

[1376] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1377] Until now, there has been no system that can handle the entire process of inputting, analyzing, storing, searching, and purchasing memories in a unified manner. Furthermore, converting voice data into text, and tagging and classifying analysis results are time-consuming and difficult, reducing usability. This has made it difficult for users to easily digitize their memories and share or sell them with other users.

[1378] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1379] In this invention, the server includes input means for inputting memories from users, conversion means for acquiring the input memory data and converting voice data into text data, transmission means for transmitting the acquired memory data via an HTTP request, analysis means for receiving the transmitted memory data and performing natural language processing and context analysis using an AI module, storage means for classifying, tagging, and storing the analyzed memory data in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data in the form of a download link or streaming URL. This allows users to easily digitize their memories and share or sell them with other users over the network.

[1380] A "user" is someone who uses the system to enter, search, and purchase memories.

[1381] "Input means" refers to a device or function that allows a user to input memories, and provides options for voice input and text input.

[1382] "Conversion means" refers to a device or software for converting voice data into text data, and uses voice recognition technology.

[1383] The "transmission means" refers to a function for transmitting the acquired stored data to the server, and transmits the data via an HTTP request.

[1384] "Analysis means" refers to devices and software that process received stored data using natural language processing (NLP) and contextual analysis.

[1385] "Storage means" refers to the function of converting the analyzed stored data into a digital format, appropriately classifying and tagging it, and storing it in a database.

[1386] "Search means" refers to a function that allows a user to search memories, and searches for relevant stored data from a database through a search query.

[1387] "Purchase means" refers to a function that allows a user to purchase a memory that they like, and executes payment processing.

[1388] The "means for providing" refers to a function for providing the purchased stored data to the user, and is provided in the form of a download link or a streaming URL.

[1389] "Database" refers to an electronic storage device for storing analyzed and archived stored data.

[1390] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments are described below.

[1391] Memory input

[1392] User: Launches the dedicated application and inputs a memory. The user is provided with the option of voice input or text input. For voice input, the user records the memory using the device's microphone. For example, the user speaks, "When I played in the high school soccer tournament," to obtain voice data. For text input, the user inputs the memory using the device's keyboard. For example, the user types, "When I played in the high school soccer tournament," into the text field.

[1393] Device: If the user selects voice input, the device uses speech recognition software to convert the recorded voice data into text data. Specifically, the device uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the voice data into text data. The converted text data, or text data entered directly by the user, is sent to the server as an HTTP request.

[1394] Sending and Receiving Data

[1395] Terminal: Generates an HTTP request and sends the retrieved text data to the server.

[1396] Server: Receives the HTTP request and processes the text data. If the data is audio, speech recognition is performed again on the server side.

[1397] Memory analysis

[1398] Server: The received text data is sent to the AI ​​module for natural language processing (NLP). Specifically, the text data is analyzed using advanced NLP techniques such as OpenAI's GPT-4 API. Key phrases and keywords are extracted based on the analysis results. For example, information such as "high school," "soccer tournament," and "championship" are extracted.

[1399] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[1400] Preserving memories

[1401] Server: Classifies and tags the stored data based on the analysis results. Tags include category information such as "sports," "school," and "tournament." The analyzed and classified data is saved in a database. Common database technologies such as MySQL and MongoDB are used for the database.

[1402] Searching and Purchasing Memory

[1403] User: If another user wants to use an application on their device to search for "memories of high school soccer tournaments," they enter a search keyword.

[1404] Terminal: Sends the entered search query to the server.

[1405] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1406] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1407] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[1408] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[1409] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[1410] Specific examples (prompt sentence examples)

[1411] Example voice input: "I want to record my memories of playing in a high school soccer tournament."

[1412] Example text entry: "I want to record my time playing in a high school soccer tournament."

[1413] This system allows users to digitize their memories, share them with other users, and sell them, and users who purchase them can enjoy them.

[1414] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1415] Step 1:

[1416] User: Launches the dedicated application and starts inputting memories. To input memories, the user can choose between voice input or text input. For example, the user can say "When I played in the high school soccer tournament" or input the same content in text.

[1417] Input: Audio or text data storage

[1418] Output: Continuously updated audio data or consistent text data

[1419] Specific operation: The user opens the application on the device, follows the instructions, speaks the memory aloud using the microphone, presses the voice input button to start recording, and when finished, presses the button again to stop recording.

[1420] Step 2:

[1421] Device: When voice data is acquired, the device converts the voice data into text data using speech recognition technology such as the Google Cloud Speech-to-Text API, and temporarily stores the converted text data and text data entered directly by the user.

[1422] Input: Audio data

[1423] Output: Text data

[1424] Specific operation: The device sends voice data to the speech recognition API, receives the converted text data, and displays it in a text field or stores it in a variable.

[1425] Step 3:

[1426] Terminal: The acquired text data is sent to the server as an HTTP request. The user's memory data is temporarily stored locally at this stage and then sent to the server.

[1427] Input: Text data

[1428] Output: HTTP request

[1429] Specific operation: The device application converts the text data into JSON format and sends it to the specified URL on the server using an HTTP POST request.

[1430] Step 4:

[1431] Server: Receives the HTTP request and prepares to analyze the stored data. If voice data is included, voice recognition is performed again.

[1432] Input: HTTP request

[1433] Output: Text data

[1434] Specific operation: The server side receives the HTTP request, analyzes the payload, extracts text data, and if it contains audio data, calls the speech recognition API on the server side to convert it into text.

[1435] Step 5:

[1436] Server: Sends the received text data to an AI module (e.g., OpenAI GPT-4 API) for natural language processing (NLP), which extracts key phrases and keywords.

[1437] Input: Text data

[1438] Output: Key phrases and keywords

[1439] Specific operation: The server passes the text data to the AI ​​module and sends the prompt "Analyze the following text and extract key phrases and keywords: About playing in a high school soccer tournament." The AI ​​module responds with the extracted keywords.

[1440] Step 6:

[1441] Server: Understands the context of stored data based on extracted keywords, and tags and structures detailed content.

[1442] Input: Extracted keywords

[1443] Output: Tagged text data

[1444] Specific operation: Based on the analysis results from the AI ​​module, the server applies a tagging algorithm to assign category information such as "sports," "school," or "tournament."

[1445] Step 7:

[1446] Server: The tagged and classified data is stored in a database using database technologies such as MySQL or MongoDB.

[1447] Input: Tagged and classified text data

[1448] Output: Memory data stored in the database

[1449] Specific operation: Executes an INSERT statement on the database installed on the server and saves the analysis results.

[1450] Step 8:

[1451] User: Another user enters a search keyword to search for "high school soccer tournament memories" on their device.

[1452] Input: Search keyword

[1453] Output: Search query

[1454] What happens: A user types "high school soccer tournament" into the application's search field and presses the search button.

[1455] Step 9:

[1456] Device: Sends a search query to the server via an HTTP GET request.

[1457] Input: Search keyword

[1458] Output: HTTP GET request

[1459] Specific operation: The device formats the search keywords as query parameters into an HTTP GET request and sends it to the server's search API.

[1460] Step 10:

[1461] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1462] Input: HTTP GET request

[1463] Output: Search result data

[1464] Specific operation: The server searches the database, extracts the corresponding stored data, and returns the extracted data in JSON format as an HTTP response.

[1465] Step 11:

[1466] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1467] Input: Search result data

[1468] Output: Display of search results

[1469] Specific operation: The device parses the received search results and displays them in the user interface, arranging the title, part of the text, tag information, etc. in the display area.

[1470] Step 12:

[1471] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[1472] Input: Purchase Information

[1473] Output: Purchase request

[1474] Specific operation: When the user clicks the purchase button, the device sends an HTTP POST request to the server containing the necessary information for the purchase.

[1475] Step 13:

[1476] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[1477] Input: Purchase Request

[1478] Output: Download link or streaming URL

[1479] Specific operation: The server calls a payment API (e.g., Stripe or PayPal) to process the payment. Once the purchase is confirmed, a download link or streaming URL for the corresponding stored data is generated.

[1480] Step 14:

[1481] On the device: The download link or streaming URL received from the server is displayed to the user, allowing them to access the stored data they have purchased.

[1482] Input: Download link or streaming URL

[1483] Output: Display of links and URLs

[1484] Specific operation: The device analyzes the links and URLs received from the server and displays them in the user interface. The user can click on the links to access the stored data.

[1485] This allows users to digitize their memories and share or sell them with other users, who can then enjoy the memories they have purchased.

[1486] (Application example 1)

[1487] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1488] Conventional memory data management systems analyze and store memories entered by users, allowing other users to search and purchase them, but lack the means to recreate those memories more specifically and provide them in a form that users can experience. Furthermore, the effective use of memory data is limited, and sharing of experiences is limited to visual and auditory information, so there is a demand for providing deeper emotional experiences.

[1489] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1490] In this invention, the server includes an input means for inputting memories from a user, a server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, a natural language processing means for analyzing the context of the memory based on the extracted phrases and keywords, a storage means for converting the analyzed memory data into a digital format and storing it in a database, a search and purchase means for the user to search for and purchase memories, and a virtual reality linkage means for providing the processed data to the user as virtual reality content. This not only enables advanced analysis of the user's memories and storage of their context, but also makes it possible to provide the memory data in a form that other users can experience realistically through virtual reality.

[1491] "User" refers to the entity that uses the system to input, search, and purchase memories.

[1492] "Memories" refer to records of events and emotions experienced by the user, and are input as audio or text data.

[1493] "Input means" refers to a device or interface that allows a user to input memories as voice or text data.

[1494] "Server means" refers to a computer system that receives and processes stored data input by a user.

[1495] "AI Module" refers to artificial intelligence software that analyzes received stored data and extracts important phrases and keywords.

[1496] "Natural language processing means" refers to technology for analyzing memory context based on extracted phrases and keywords.

[1497] "Storage means" refers to a function that converts the analyzed stored data into a digital format and stores it in a database.

[1498] "Search and purchase means" refers to an interface that allows a user to search and purchase saved stored data.

[1499] "Delivery means" refers to a mechanism for providing purchased storage data to a user.

[1500] "Virtual reality linking means" refers to a technology that provides processed stored data to users as virtual reality content.

[1501] This invention relates to a system that allows users to input personal memories, analyze and store them using natural language processing (NLP) technology, and then allows other users to search and purchase them. The purchased memory data can then be reproduced as virtual reality (VR) content.

[1502] Memory input and analysis

[1503] Memory Input:

[1504] User: To input a memory, launch a dedicated application on the device. The application offers the option of voice input or text input.

[1505] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[1506] In the case of text input, the same content is written in the text field.

[1507] Sending and receiving data:

[1508] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[1509] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[1510] Natural Language Processing:

[1511] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[1512] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[1513] Contextual analysis:

[1514] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[1515] Preserving memories

[1516] Database storage

[1517] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[1518] Server: Stores the stored data in a database after converting it into digital format.

[1519] Searching and Purchasing Memory

[1520] Memory Retrieval:

[1521] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[1522] Terminal: Sends the entered search query to the server.

[1523] Retrieving and displaying search results:

[1524] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1525] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1526] Purchasing and donating memory:

[1527] User: Select a memory they like and press the purchase button to purchase it.

[1528] Device: Sends a purchase request to the server.

[1529] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[1530] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[1531] Virtual reality reproduction

[1532] Virtual Reality Integration:

[1533] Server: Reconstructs the purchased memory data as virtual reality content and generates links.

[1534] User: Experience virtual reality content using smart glasses or a head-mounted display.

[1535] Specific examples and prompts for the generative AI model

[1536] Examples:

[1537] User A: Inputs a memory of a soccer tournament from high school, and talks about the changes in emotions and detailed episodes that occurred during the event.

[1538] Server: Converts voice data into text and analyzes it using NLP. Stores the analysis results in a database.

[1539] User B: Searches for "high school soccer tournament" in the virtual store and finds User A's memory.

[1540] Server: Returns search results and generates a VR link. User B enjoys the VR experience through the link.

[1541] Prompt statement:

[1542] "Tell me about a time when you participated in a high school soccer tournament. Please describe in detail any particularly memorable episodes or emotions."

[1543] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1544] Step 1:

[1545] User enters memory:

[1546] Input: The user launches a dedicated application on the device and inputs the memory by voice or text.

[1547] Processing: In the case of voice input, the application records the voice data, and in the case of text input, it receives the text data as is.

[1548] Output: Audio or text data.

[1549] Step 2:

[1550] The device converts the data:

[1551] Input: Audio data.

[1552] Processing: When a voice command is received, the device converts the voice data into text using transcription software (e.g., Google Speech-to-Text API or IBM Watson).

[1553] Output: Text data.

[1554] Step 3:

[1555] Sending data:

[1556] Input: Text data.

[1557] Processing: The terminal sends the converted text data or the directly entered text data to the server as an HTTP request.

[1558] Output: HTTP request (text data).

[1559] Step 4:

[1560] Server receives data:

[1561] Input: HTTP request (text data).

[1562] Processing: The server receives the stored data sent from the terminal.

[1563] Output: Received text data.

[1564] Step 5:

[1565] Perform natural language processing:

[1566] Input: Received text data.

[1567] Processing: The server uses AI modules (e.g., Hugging Face Transformers) to perform natural language processing (NLP) and extract key phrases and keywords.

[1568] Output: Extracted phrases and keywords.

[1569] Step 6:

[1570] Contextual analysis:

[1571] Input: Extracted phrases or keywords.

[1572] Processing: The server's AI module analyzes the memory context based on phrases and keywords, and tags and structures the details.

[1573] Output: Tagged data.

[1574] Step 7:

[1575] Save to database:

[1576] Input: Tagged data.

[1577] Processing: The server converts the tagged stored data into a digital format and stores it in a database.

[1578] Output: Memorized data stored in a database.

[1579] Step 8:

[1580] User searches memory:

[1581] Input: Search keyword.

[1582] Process: The user enters search keywords into an application on the device, and the device sends the search query to the server as an HTTP request.

[1583] Output: HTTP request (search query).

[1584] Step 9:

[1585] The server retrieves the search results:

[1586] Input: HTTP request (search query).

[1587] Processing: The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[1588] Output: HTTP response (search results).

[1589] Step 10:

[1590] Display search results:

[1591] Input: HTTP response (search result).

[1592] Processing: The terminal displays the search results received from the server to the user. For example, it displays the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament."

[1593] Output: What is displayed to the user.

[1594] Step 11:

[1595] User purchases memory:

[1596] Input: Purchase request.

[1597] Processing: The user selects a memory they like and presses the purchase button. The device sends a purchase request to the server.

[1598] Output: HTTP request (purchase request).

[1599] Step 12:

[1600] The server processes the payment:

[1601] Input: HTTP request (purchase request).

[1602] Processing: The server processes the payment and generates a download link or streaming URL for the stored data for which the purchase has been confirmed.

[1603] Output: Download link or streaming URL.

[1604] Step 13:

[1605] Providing purchase data:

[1606] Input: Download link or streaming URL.

[1607] Processing: The device will display a download link or streaming URL to the user, allowing them to access the stored data they purchased.

[1608] Output: Provide link to user.

[1609] Step 14:

[1610] Virtual reality reproduction:

[1611] Input: Purchased memory data.

[1612] Processing: Reconstructs purchased memory data into virtual reality content and generates a link.

[1613] Output: Virtual reality experience link.

[1614] Step 15:

[1615] Experience virtual reality content:

[1616] Input: Virtual reality experience link.

[1617] Processing: Users experience virtual reality content using smart glasses or a head-mounted display.

[1618] Output: Experience virtual reality content.

[1619] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1620] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memories input by a user and the emotional state of the memories. Specific embodiments will be described in detail below.

[1621] Memory input and emotion recognition

[1622] Memory input

[1623] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1624] For example, to record "Memories of my college graduation ceremony," a user selects voice input and speaks into the microphone, "The joy I felt at my college graduation ceremony," or writes the memory in the text input field.

[1625] Emotion recognition

[1626] Device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. For example, it determines whether the user is expressing emotions such as joy, sadness, or surprise through voice tone and speed, as well as sentiment analysis of the text.

[1627] Terminal: The recognized emotion data is sent to the server along with the input data.

[1628] Sending and Receiving Data

[1629] Sending data

[1630] Terminal: The converted text data or directly entered text data and emotion data are sent to the server as an HTTP request.

[1631] Receiving data

[1632] Server: Receives emotion-recognized memory data sent from the device.

[1633] Memory analysis

[1634] Performing natural language processing

[1635] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[1636] Server: Sends data to the NLP module and starts the analysis.

[1637] Keyword extraction

[1638] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1639] Contextual Analysis

[1640] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[1641] Preserving memories

[1642] Database storage

[1643] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[1644] Server: Stores the stored data in a database after converting it into digital format.

[1645] Searching and Purchasing Memory

[1646] Memory retrieval

[1647] User: Another user searches for "joyful memories of college graduation" in an application on their device and enters search keywords and the desired emotional state.

[1648] Terminal: Sends the entered search query to the server.

[1649] Getting search results

[1650] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1651] Displaying search results

[1652] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[1653] Purchasing and donating memories

[1654] Purchasing Memory

[1655] User: Select a memory they like and press the purchase button to purchase it.

[1656] Device: Sends a purchase request to the server.

[1657] Payment Processing

[1658] Server: Receives the purchase request and processes the payment through the payment system.

[1659] Providing stored data

[1660] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[1661] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[1662] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search and purchase them. Furthermore, users who purchase them can enjoy the emotional state of their memories as well.

[1663] The processing flow will be explained below.

[1664] Memory input and emotion recognition

[1665] Step 1: Entering memory

[1666] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1667] For example, if a user wants to record "memories of college graduation," they can select voice input and speak into the microphone, "the joy I felt at college graduation," or write their memories in the text input field.

[1668] Step 2: Recognize emotions

[1669] On the device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. It uses tone and speed of speech and sentiment analysis of text to determine whether the user is expressing emotions such as joy, sadness, or surprise.

[1670] Step 3: Text Conversion (Speech Input)

[1671] Device: If voice input is selected, the recorded voice data is converted to text data using transcription software, and the converted text data is temporarily stored.

[1672] Step 4: Sending data

[1673] Terminal: The converted text data or the directly entered text data and the recognized emotion data are sent to the server as an HTTP request.

[1674] Data reception and analysis

[1675] Step 5: Receiving the data

[1676] Server: Receives emotion-recognized memory data sent from the device.

[1677] Step 6: Performing Natural Language Processing

[1678] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[1679] Server: Sends data to the NLP module and starts the analysis.

[1680] Step 7: Keyword extraction

[1681] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1682] Step 8: Analyze the sentiment data

[1683] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[1684] Preserving memories

[1685] Step 9: Prepare to save to the database

[1686] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[1687] Step 10: Save to database

[1688] Server: Stores the stored data in a database after converting it into digital format.

[1689] Searching and Purchasing Memory

[1690] Step 11: Memory Retrieval

[1691] User: When another user searches for "joyful memories of college graduation" through an application on a terminal, the user inputs search keywords and a desired emotional state.

[1692] Terminal: Sends the entered search query to the server.

[1693] Step 12: Getting search results

[1694] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1695] Step 13: Viewing search results

[1696] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[1697] Purchasing and donating memories

[1698] Step 14: Buy Memory

[1699] User: Select the memory they like and press the purchase button if they wish to purchase it.

[1700] Device: Sends a purchase request to the server.

[1701] Step 15: Payment Processing

[1702] Server: Receives the purchase request and processes the payment through the payment system.

[1703] Step 16: Confirm your purchase

[1704] Server: Once payment is complete, verify that the purchase was successful.

[1705] Step 17: Provide stored data

[1706] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[1707] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[1708] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search for and purchase them. In addition, users who purchase the memories can enjoy the emotional state of the memories.

[1709] Example 2

[1710] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1711] In existing systems, when a user inputs a memory, it is difficult to recognize the specific emotional state and analyze, store, and provide the memory data. Therefore, there is a need for a system that can accurately recognize the emotional information in memory and analyze, store, search, and provide the data based on that information.

[1712] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1713] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, emotion recognition means for recognizing emotional states from voice and text data, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data to users. This allows users to input, recognize, and store memories more accurately, and enables other users to search, purchase, and share memories, including their emotional information.

[1714] "Input means" refers to the method or device that a user uses to input a memory.

[1715] "Server means" refers to a central computing device for receiving and processing stored data sent by users.

[1716] "AI Module" refers to the artificial intelligence software module used to analyze received stored data and extract key phrases and keywords.

[1717] "Emotion recognizer" refers to a system or method for recognizing emotional states from speech and text data.

[1718] "Storage means" refers to a means or device for converting the analyzed stored data into a digital format and storing it in a database.

[1719] "Search and purchase means" refers to a method or device that allows a user to search for stored data and purchase it if necessary.

[1720] "Providing means" refers to a means or system for providing the purchased storage data to the user in the form of a download link or streaming URL.

[1721] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memory input by a user and the emotional state of that memory.

[1722] The system consists of the following elements:

[1723] 1. Input Method

[1724] User: The user launches a dedicated application on the device and inputs their memories. The application offers the option of voice input or text input.

[1725] Example: To record "Memories of my college graduation ceremony," a user taps the voice input button and speaks "The joy I felt at my college graduation ceremony," or writes "The joy I felt at my college graduation ceremony" in the text input field.

[1726] 2. Server Means

[1727] Terminal: The stored data entered by the user is sent to the server in JSON format, for example, { "text": "The joy felt at the university graduation ceremony", "emotion": "Joy"}.

[1728] Server: The server receives the data sent from the device and prepares it for processing.

[1729] 3. AI Module

[1730] Server: Passes the received text data to an AI module (e.g., TensorFlow or PyTorch) for natural language processing (NLP).

[1731] Example: Using NLP to extract important phrases and keywords such as "university," "graduation," and "joy" from text data.

[1732] 4. Emotion recognition means

[1733] On the device: An emotion engine is used to analyze voice and text data, identifying emotions from the tone and rate of voice and the content of text.

[1734] Example: Recognizing emotional states such as happiness, sadness, surprise, etc. For example, identifying "happiness" based on tone and rate of speech and text.

[1735] 5. Preservation means

[1736] Server: Converts the analyzed memory data into a digital format and stores it in a database.

[1737] Example: Save data with tagging that includes category information such as "emotion," "event," "place," etc. For example, set tags like tags: ["university," "graduation ceremony," "joy"].

[1738] 6. Search and Purchase Methods

[1739] User: Another user searches a memory to find information about a specific emotion or event. Enter search keywords and an emotional state.

[1740] Example: A user types "college," "graduation," and "joy" into the app's search bar and clicks the search button.

[1741] Device: Sends a search query to the server.

[1742] 7. Means of provision

[1743] Server: Provides the found stored data to the user in the form of a download link or streaming URL.

[1744] Example: Displaying a title such as "Memories of Mr. / Ms. XX's College Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[1745] User: Selects the memory they like, purchases it, enters payment information and taps the purchase button.

[1746] Server: Receives the purchase request, processes the payment through a payment system (e.g., Stripe or PayPal), and then generates a download link or streaming URL for the purchased stored data and provides it to the user.

[1747] 8. Examples of concrete examples and prompts

[1748] Example: User A inputs the joy of his / her college graduation into a memory, and the emotion is recognized and saved as "joy." User B searches for "college," "graduation ceremony," and "joy," discovers User A's memory, and purchases it, allowing him / her to learn more about the joy.

[1749] Example prompt: "Describe a system that recognizes and stores emotional states based on memories of your college graduation. Explain in detail how the system analyzes and stores the data, using the example memory of 'summer family vacation.'"

[1750] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1751] Step 1:

[1752] Memory input

[1753] User: The user launches a dedicated application on the device and inputs their memories. The application provides the option of voice input or text input.

[1754] Specific actions: To record "Memories of my college graduation ceremony," the user taps the voice input button and speaks "The joy I felt at my college graduation ceremony" into the microphone, or writes "The joy I felt at my college graduation ceremony" in the text input field.

[1755] Input: User voice or text data

[1756] Output: Audio file or text data as stored data

[1757] Step 2:

[1758] Emotion recognition

[1759] Device: The device converts the input voice data into text using a speech recognition function, then passes the text data to an emotion recognition engine to recognize the emotional state.

[1760] What it does: It converts voice data into text using the Google Speech-to-Text API, then passes the text data to IBM Watson Emotion Analysis to identify emotional states such as "joy" or "sadness."

[1761] Input: Audio file or text data as storage data

[1762] Output: Recognized emotion data (e.g., happy, sad)

[1763] Step 3:

[1764] Sending data

[1765] Device: The device sends the recognized emotion data and text data to the server in JSON format.

[1766] Specific operation: The data is sent as an HTTP request in the format { "text": "The joy I felt at my college graduation ceremony", "emotion": "Joy"}.

[1767] Input: Recognized emotion data and text data

[1768] Output: JSON data sent to the server

[1769] Step 4:

[1770] Data reception and analysis

[1771] Server: The server receives the data sent from the device and passes the received text data to the AI ​​module for natural language processing (NLP).

[1772] What it does: It uses AI modules (e.g. TensorFlow and PyTorch) to analyze the content of text and extract important phrases and keywords.

[1773] Input: JSON data sent from the terminal

[1774] Output: Extracted phrases and keywords (e.g., "university," "graduation," "joy")

[1775] Step 5:

[1776] Contextual Analysis

[1777] AI module: Understands the context of memories based on extracted keywords and phrases, along with emotional data.

[1778] Specific operation: The memory content is tagged and structured based on extracted phrases, keywords, and emotional data.

[1779] Input: Extracted phrases and keywords, and sentiment data

[1780] Output: Structured stored data

[1781] Step 6:

[1782] Data storage

[1783] Server: Converts structured stored data into a digital format and stores it in a database.

[1784] Specific operation: Save data with tagging and category information such as "emotion," "event," "location," etc. For example, set tags as follows: tags: ["university," "graduation ceremony," "joy"]

[1785] Input: Structured stored data

[1786] Output: Memory data stored in the database

[1787] Step 7:

[1788] Memory retrieval

[1789] User: Another user uses an application on the device to search for memories, for example, searching for "college graduation joy."

[1790] Specific actions: Enter "college," "graduation," and "joy" in the search bar and click the search button.

[1791] Input: Search keyword

[1792] Output: Data sent to the server as a search query

[1793] Step 8:

[1794] Providing search results

[1795] Server: Searches the database for stored data that matches the search criteria and sends it to the terminal as an HTTP response.

[1796] Specific operation: The found stored data is sent to the device as an API response and converted into a display format.

[1797] Input: search query

[1798] Output: Corresponding stored data

[1799] Step 9:

[1800] Displaying search results

[1801] Device: Displays the received search results to the user. For example, it displays the title such as "Memories of XX's University Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[1802] What it does: It displays search results in a list format and provides a detailed view that displays more information when you click on it.

[1803] Input: Search results received from the server

[1804] Output: The search results that are displayed to the user

[1805] Step 10:

[1806] Purchasing and donating memories

[1807] User: If they want to purchase a memory they like, they press the purchase button and enter their payment information.

[1808] What happens: You tap the buy button, enter your payment information, and confirm your purchase.

[1809] Input: Purchase request and payment information

[1810] Output: Purchase request sent to the server

[1811] Step 11:

[1812] Payment processing and provision

[1813] Server: Receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal). Then, generates a download link or streaming URL for the purchased stored data and provides it to the user.

[1814] Specific operation: After payment confirmation, an access link for the stored data is generated and sent to the user.

[1815] Input: Purchase request and payment information

[1816] Output: A download link or streaming URL provided to the user

[1817] (Application example 2)

[1818] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1819] Conventional memory data management systems have limited user memory input to text or voice data, making it impossible to recognize emotions or sell memory data in virtual stores. This has limited the user experience, making it difficult to share or trade memory data effectively.

[1820] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1821] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, provision means for providing purchased memory data to users, virtual store display means for selling memory data in a virtual space, emotion analysis means with emotion recognition capabilities, voice recognition means for converting voice data into text, and purchase management means for processing the purchase procedure for memory data through a payment means. This allows users to record their own memories in detail, analyze them including emotions, and sell, search, and purchase them in a virtual store.

[1822] A "user" is someone who uses this system and performs operations such as memory input, search, and purchase.

[1823] "Memories" refers to information about personal experiences and memories entered by the user.

[1824] "Input means" refers to a device or interface that allows a user to input memories, including voice and text input options.

[1825] The term "server means" refers to computer resources that receive input stored data and perform various processes such as analysis.

[1826] "AI module" refers to the artificial intelligence algorithms and software used to analyze input memory data and extract key phrases and keywords.

[1827] "Storage means" refers to a mechanism for converting the analyzed stored data into a digital format and storing it in a database.

[1828] "Search facility" refers to the interface and associated mechanisms by which a user searches the memory.

[1829] "Purchase vehicle" refers to the interface and associated mechanisms for a user to purchase selected storage.

[1830] "Providing means" refers to a mechanism for providing the purchased stored data to the user, and includes providing the data in the form of a download link or a streaming URL.

[1831] "Virtual store display means" refers to a display interface and related mechanisms for selling stored data in a virtual space.

[1832] "Emotion analysis means" refers to a mechanism for recognizing and analyzing the user's emotions from stored data.

[1833] "Speech recognition means" refers to a mechanism for converting voice data into text data.

[1834] "Purchase management means" refers to a mechanism for processing the purchase procedure for stored data through a payment means.

[1835] "Payment Method" means the electronic payment system or service used to complete a Purchase.

[1836] The present invention relates to a system that allows users to input their own memories, recognize and analyze their emotions, and sell and purchase items in a virtual store. Specific embodiments are described in detail below.

[1837] Memory input and emotion recognition

[1838] Memory input

[1839] Users install and launch a dedicated application for recording memories on their smartphone. This application offers the option of voice input or text input. For example, to record "memories of college graduation," a user can select voice input and speak into the microphone, "the joy I felt at my college graduation ceremony." Alternatively, they can write their memories in the text input field.

[1840] Emotion recognition

[1841] The application analyzes the voice or text data entered by the user in real time and uses an emotion analysis engine to recognize the user's emotional state. For voice data, it uses the Google Cloud Speech-to-Text API to convert the speech to text. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotion of text data. For example, it determines whether the user is expressing emotions such as "happiness," "sadness," or "surprise" based on the tone and speed of the voice and the emotion analysis of the text. This emotion data is sent to the server along with the input data.

[1842] Sending and Receiving Data

[1843] Sending data

[1844] The terminal transmits the converted text data or the directly input text data and the emotion data as an HTTP request to the server.

[1845] Receiving data

[1846] The server receives the emotion-recognized stored data transmitted from the terminal.

[1847] Memory analysis

[1848] Performing natural language processing

[1849] The server passes the received text data to a generative AI module, preparing it for natural language processing (NLP). The generative AI model uses, for example, the Transformers library from Hugging Face.

[1850] Keyword extraction

[1851] The AI ​​module uses NLP to analyze the text data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1852] Contextual Analysis

[1853] The AI ​​module understands the context of the memory based on extracted keywords, phrases, and emotional data, and then tags and structures the detailed content.

[1854] Preserving memories

[1855] Database storage

[1856] The server categorizes and tags the memory data based on the analysis results and emotional data. The tags include category information such as "emotion," "event," and "location." The memory data converted into digital format is stored in an SQL database (MySQL or PostgreSQL).

[1857] Searching and Purchasing Memory

[1858] Memory retrieval

[1859] A user can use the application's search function to search for memories entered by other users. By entering a desired emotional state along with search keywords, the user can find the corresponding memory data. For example, searching for "graduation ceremony" and "joy."

[1860] Getting search results

[1861] The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[1862] Displaying search results

[1863] The terminal displays the search results received from the server to the user, such as a title such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)," along with detailed information and emotional state.

[1864] Purchasing and donating memories

[1865] Purchasing Memory

[1866] When the user selects a memory they like and presses the purchase button to purchase it, the terminal sends a purchase request to the server.

[1867] Payment Processing

[1868] The server receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal).

[1869] Providing stored data

[1870] The server generates a download link or streaming URL for the purchased stored data and provides it to the user. The terminal displays the link or URL provided by the server to the user, allowing the user to access the purchased stored data.

[1871] Specific examples

[1872] Consider a scenario in which a user registers their "joy felt at their university graduation ceremony" by voice input into the system, and other users search for and purchase this item using the keywords "graduation ceremony" and "joy." This system provides a way for individual users to experience and share their memories and emotions in a virtual store.

[1873] Prompt Sentence Examples

[1874] "Tell me about the joy you felt at your college graduation."

[1875] "Please enter your emotional memory of graduation."

[1876] "Search for the keyword 'graduation' and the emotion 'joy'"

[1877] In this way, the system of the present invention inputs the user's memories and recognizes, analyzes, stores, searches, and provides emotions, thereby realizing functions that enrich the user's experience.

[1878] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1879] Step 1:

[1880] The user launches a dedicated application on their smartphone for inputting memories. They can select either voice input or text input as the input method. If voice input is selected, the user speaks their memories into a microphone. If text input is selected, the user inputs their memories using a keyboard. This input data is sent to the application as voice data or text data.

[1881] Step 2:

[1882] The device processes the input data in real time, converting voice data into text using the Google Cloud Speech-to-Text API, and then performs emotion analysis on the converted text data using IBM Watson Tone Analyzer. The analysis results in emotional data such as joy, sadness, and surprise.

[1883] Step 3:

[1884] The device then sends the converted text data and emotion data to the server as an HTTP request, which includes the user's input memory and associated emotion.

[1885] Step 4:

[1886] The server receives the memory data and emotion data sent from the terminal. This received data includes the user's memory information and emotion information.

[1887] Step 5:

[1888] The server prepares the received memory data for natural language processing (NLP) using a generative AI model, sending the data to an NLP module using a library such as Hugging Face's Transformers library, and begins analysis.

[1889] Step 6:

[1890] The AI ​​module uses NLP to analyze the memory data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[1891] Step 7:

[1892] The AI ​​module understands the context of memories based on extracted keywords, phrases, and emotional data, and then tags and structures the details, assigning tags such as "emotion," "event," and "location" to the memory data.

[1893] Step 8:

[1894] The server converts the tagged analysis results into a digital format and stores them in an SQL database such as MySQL or PostgreSQL, including the tagged stored data.

[1895] Step 9:

[1896] A user makes a request to search for a memory from a terminal. The search request includes keywords, such as "graduation ceremony" and "joy," as well as a desired emotional state.

[1897] Step 10:

[1898] The server searches the database based on the received search request, retrieves the corresponding stored data as search results, and sends the search results to the terminal as an HTTP response.

[1899] Step 11:

[1900] The terminal displays the search results received from the server to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)" are displayed.

[1901] Step 12:

[1902] The user goes through the process of purchasing a memory they like. When they press the purchase button, the device sends a purchase request to the server.

[1903] Step 13:

[1904] The server receives the purchase request, processes the payment through a payment system such as Stripe or PayPal, and, if successful, generates a download link or streaming URL for the stored data.

[1905] Step 14:

[1906] The server sends the generated download link or streaming URL to the terminal, which displays it to the user, allowing the user to access the stored data they purchased.

[1907] Prompt Sentence Examples

[1908] "Tell me about the joy you felt at your college graduation."

[1909] "Please enter your emotional memory of graduation."

[1910] "Search for the keyword 'graduation' and the emotion 'joy'"

[1911] Through the above processing steps, a system is provided that allows users to record their own memories in detail, share them with other users, and trade in virtual stores.

[1912] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1913] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1914] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1915] [Fourth embodiment]

[1916] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1917] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1918] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1919] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1920] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1921] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1922] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1923] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1924] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1925] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1926] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1927] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1928] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1929] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments will be described below.

[1930] Memory input and analysis

[1931] Memory input

[1932] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1933] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[1934] In the case of text input, the same content is written in the text field.

[1935] Sending and Receiving Data

[1936] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[1937] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[1938] Memory analysis

[1939] Natural Language Processing

[1940] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[1941] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[1942] Contextual Analysis

[1943] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[1944] Preserving memories

[1945] Database storage

[1946] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[1947] Server: Stores the stored data in a database after converting it into digital format.

[1948] Searching and Purchasing Memory

[1949] Memory retrieval

[1950] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[1951] Terminal: Sends the entered search query to the server.

[1952] Retrieving and displaying search results

[1953] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1954] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1955] Purchasing and donating memories

[1956] User: Select a memory they like and press the purchase button to purchase it.

[1957] Device: Sends a purchase request to the server.

[1958] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[1959] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[1960] This allows users to digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy those memories.

[1961] The processing flow will be explained below.

[1962] Memory input and analysis

[1963] Step 1: Entering memory

[1964] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[1965] For example, a user may select voice input and speak into a microphone to record "memories of a high school soccer tournament," or type into a text input field.

[1966] Step 2: Convert audio data to text

[1967] On your device: If you select voice input, your audio recording will be converted to text using transcription software, and the converted text will be temporarily stored in your device's memory.

[1968] Step 3: Sending data

[1969] Terminal: The converted text data or the text data entered directly is sent to the server as an HTTP request.

[1970] Step 4: Receiving the data

[1971] Server: Receives the stored data sent from the device and checks the data format (voice data or text data).

[1972] Memory analysis

[1973] Step 5: Performing Natural Language Processing

[1974] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[1975] Server: Sends data to the NLP module and starts the analysis.

[1976] Step 6: Keyword extraction

[1977] AI module: Analyzes text data using NLP to extract important phrases and keywords.

[1978] For example, important information such as "high school," "soccer tournament," and "championship" is analyzed.

[1979] Step 7: Context Analysis

[1980] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the detailed content.

[1981] Preserving memories

[1982] Step 8: Structuring the database

[1983] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[1984] Server: Prepares structured stored data for conversion into digital format.

[1985] Step 9: Save to Database

[1986] Server: Stores the stored data in a database after converting it into digital format.

[1987] Searching and Purchasing Memory

[1988] Step 10: Searching your memory

[1989] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[1990] Terminal: Sends the entered search query to the server.

[1991] Step 11: Getting search results

[1992] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[1993] Step 12: Viewing search results

[1994] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[1995] Purchasing and donating memories

[1996] Step 13: Buy Memory

[1997] User: Select a memory they like and press the purchase button to purchase it.

[1998] Device: Sends a purchase request to the server.

[1999] Step 14: Payment Processing

[2000] Server: Receives the purchase request and processes the payment through the payment system.

[2001] Step 15: Confirm your purchase

[2002] Server: Once payment is complete, verify that the purchase was successful.

[2003] Step 16: Providing Stored Data

[2004] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[2005] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[2006] In this way, users can digitize their memories, share them with other users, and sell them. Furthermore, users who purchase them can enjoy them.

[2007] Example 1

[2008] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2009] Until now, there has been no system that can handle the entire process of inputting, analyzing, storing, searching, and purchasing memories in a unified manner. Furthermore, converting voice data into text, and tagging and classifying analysis results are time-consuming and difficult, reducing usability. This has made it difficult for users to easily digitize their memories and share or sell them with other users.

[2010] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2011] In this invention, the server includes input means for inputting memories from users, conversion means for acquiring the input memory data and converting voice data into text data, transmission means for transmitting the acquired memory data via an HTTP request, analysis means for receiving the transmitted memory data and performing natural language processing and context analysis using an AI module, storage means for classifying, tagging, and storing the analyzed memory data in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data in the form of a download link or streaming URL. This allows users to easily digitize their memories and share or sell them with other users over the network.

[2012] A "user" is someone who uses the system to enter, search, and purchase memories.

[2013] "Input means" refers to a device or function that allows a user to input memories, and provides options for voice input and text input.

[2014] "Conversion means" refers to a device or software for converting voice data into text data, and uses voice recognition technology.

[2015] The "transmission means" refers to a function for transmitting the acquired stored data to the server, and transmits the data via an HTTP request.

[2016] "Analysis means" refers to devices and software that process received stored data using natural language processing (NLP) and contextual analysis.

[2017] "Storage means" refers to the function of converting the analyzed stored data into a digital format, appropriately classifying and tagging it, and storing it in a database.

[2018] "Search means" refers to a function that allows a user to search memories, and searches for relevant stored data from a database through a search query.

[2019] "Purchase means" refers to a function that allows a user to purchase a memory that they like, and executes payment processing.

[2020] The "means for providing" refers to a function for providing the purchased stored data to the user, and is provided in the form of a download link or a streaming URL.

[2021] "Database" refers to an electronic storage device for storing analyzed and archived stored data.

[2022] The present invention relates to a system that allows users to input memories, analyze and store them, and then other users can search and purchase them. Specific embodiments are described below.

[2023] Memory input

[2024] User: Launches the dedicated application and inputs a memory. The user is provided with the option of voice input or text input. For voice input, the user records the memory using the device's microphone. For example, the user speaks, "When I played in the high school soccer tournament," to obtain voice data. For text input, the user inputs the memory using the device's keyboard. For example, the user types, "When I played in the high school soccer tournament," into the text field.

[2025] Device: If the user selects voice input, the device uses speech recognition software to convert the recorded voice data into text data. Specifically, the device uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the voice data into text data. The converted text data, or text data entered directly by the user, is sent to the server as an HTTP request.

[2026] Sending and Receiving Data

[2027] Terminal: Generates an HTTP request and sends the retrieved text data to the server.

[2028] Server: Receives the HTTP request and processes the text data. If the data is audio, speech recognition is performed again on the server side.

[2029] Memory analysis

[2030] Server: The received text data is sent to the AI ​​module for natural language processing (NLP). Specifically, the text data is analyzed using advanced NLP techniques such as OpenAI's GPT-4 API. Key phrases and keywords are extracted based on the analysis results. For example, information such as "high school," "soccer tournament," and "championship" are extracted.

[2031] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[2032] Preserving memories

[2033] Server: Classifies and tags the stored data based on the analysis results. Tags include category information such as "sports," "school," and "tournament." The analyzed and classified data is saved in a database. Common database technologies such as MySQL and MongoDB are used for the database.

[2034] Searching and Purchasing Memory

[2035] User: If another user wants to use an application on their device to search for "memories of high school soccer tournaments," they enter a search keyword.

[2036] Terminal: Sends the entered search query to the server.

[2037] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[2038] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[2039] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[2040] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[2041] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[2042] Specific examples (prompt sentence examples)

[2043] Example voice input: "I want to record my memories of playing in a high school soccer tournament."

[2044] Example text entry: "I want to record my time playing in a high school soccer tournament."

[2045] This system allows users to digitize their memories, share them with other users, and sell them, and users who purchase them can enjoy them.

[2046] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2047] Step 1:

[2048] User: Launches the dedicated application and starts inputting memories. To input memories, the user can choose between voice input or text input. For example, the user can say "When I played in the high school soccer tournament" or input the same content in text.

[2049] Input: Audio or text data storage

[2050] Output: Continuously updated audio data or consistent text data

[2051] Specific operation: The user opens the application on the device, follows the instructions, speaks the memory aloud using the microphone, presses the voice input button to start recording, and when finished, presses the button again to stop recording.

[2052] Step 2:

[2053] Device: When voice data is acquired, the device converts the voice data into text data using speech recognition technology such as the Google Cloud Speech-to-Text API, and temporarily stores the converted text data and text data entered directly by the user.

[2054] Input: Audio data

[2055] Output: Text data

[2056] Specific operation: The device sends voice data to the speech recognition API, receives the converted text data, and displays it in a text field or stores it in a variable.

[2057] Step 3:

[2058] Terminal: The acquired text data is sent to the server as an HTTP request. The user's memory data is temporarily stored locally at this stage and then sent to the server.

[2059] Input: Text data

[2060] Output: HTTP request

[2061] Specific operation: The device application converts the text data into JSON format and sends it to the specified URL on the server using an HTTP POST request.

[2062] Step 4:

[2063] Server: Receives the HTTP request and prepares to analyze the stored data. If voice data is included, voice recognition is performed again.

[2064] Input: HTTP request

[2065] Output: Text data

[2066] Specific operation: The server side receives the HTTP request, analyzes the payload, extracts text data, and if it contains audio data, calls the speech recognition API on the server side to convert it into text.

[2067] Step 5:

[2068] Server: Sends the received text data to an AI module (e.g., OpenAI GPT-4 API) for natural language processing (NLP), which extracts key phrases and keywords.

[2069] Input: Text data

[2070] Output: Key phrases and keywords

[2071] Specific operation: The server passes the text data to the AI ​​module and sends the prompt "Analyze the following text and extract key phrases and keywords: About playing in a high school soccer tournament." The AI ​​module responds with the extracted keywords.

[2072] Step 6:

[2073] Server: Understands the context of stored data based on extracted keywords, and tags and structures detailed content.

[2074] Input: Extracted keywords

[2075] Output: Tagged text data

[2076] Specific operation: Based on the analysis results from the AI ​​module, the server applies a tagging algorithm to assign category information such as "sports," "school," or "tournament."

[2077] Step 7:

[2078] Server: The tagged and classified data is stored in a database using database technologies such as MySQL or MongoDB.

[2079] Input: Tagged and classified text data

[2080] Output: Memory data stored in the database

[2081] Specific operation: Executes an INSERT statement on the database installed on the server and saves the analysis results.

[2082] Step 8:

[2083] User: Another user enters a search keyword to search for "high school soccer tournament memories" on their device.

[2084] Input: Search keyword

[2085] Output: Search query

[2086] What happens: A user types "high school soccer tournament" into the application's search field and presses the search button.

[2087] Step 9:

[2088] Device: Sends a search query to the server via an HTTP GET request.

[2089] Input: Search keyword

[2090] Output: HTTP GET request

[2091] Specific operation: The device formats the search keywords as query parameters into an HTTP GET request and sends it to the server's search API.

[2092] Step 10:

[2093] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[2094] Input: HTTP GET request

[2095] Output: Search result data

[2096] Specific operation: The server searches the database, extracts the corresponding stored data, and returns the extracted data in JSON format as an HTTP response.

[2097] Step 11:

[2098] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[2099] Input: Search result data

[2100] Output: Display of search results

[2101] Specific operation: The device parses the received search results and displays them in the user interface, arranging the title, part of the text, tag information, etc. in the display area.

[2102] Step 12:

[2103] User: Selects a memory they like and presses the purchase button to purchase it. A purchase request is sent to the server.

[2104] Input: Purchase Information

[2105] Output: Purchase request

[2106] Specific operation: When the user clicks the purchase button, the device sends an HTTP POST request to the server containing the necessary information for the purchase.

[2107] Step 13:

[2108] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[2109] Input: Purchase Request

[2110] Output: Download link or streaming URL

[2111] Specific operation: The server calls a payment API (e.g., Stripe or PayPal) to process the payment. Once the purchase is confirmed, a download link or streaming URL for the corresponding stored data is generated.

[2112] Step 14:

[2113] On the device: The download link or streaming URL received from the server is displayed to the user, allowing them to access the stored data they have purchased.

[2114] Input: Download link or streaming URL

[2115] Output: Display of links and URLs

[2116] Specific operation: The device analyzes the links and URLs received from the server and displays them in the user interface. The user can click on the links to access the stored data.

[2117] This allows users to digitize their memories and share or sell them with other users, who can then enjoy the memories they have purchased.

[2118] (Application example 1)

[2119] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2120] Conventional memory data management systems analyze and store memories entered by users, allowing other users to search and purchase them, but lack the means to recreate those memories more specifically and provide them in a form that users can experience. Furthermore, the effective use of memory data is limited, and sharing of experiences is limited to visual and auditory information, so there is a demand for providing deeper emotional experiences.

[2121] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2122] In this invention, the server includes an input means for inputting memories from a user, a server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, a natural language processing means for analyzing the context of the memory based on the extracted phrases and keywords, a storage means for converting the analyzed memory data into a digital format and storing it in a database, a search and purchase means for the user to search for and purchase memories, and a virtual reality linkage means for providing the processed data to the user as virtual reality content. This not only enables advanced analysis of the user's memories and storage of their context, but also makes it possible to provide the memory data in a form that other users can experience realistically through virtual reality.

[2123] "User" refers to the entity that uses the system to input, search, and purchase memories.

[2124] "Memories" refer to records of events and emotions experienced by the user, and are input as audio or text data.

[2125] "Input means" refers to a device or interface that allows a user to input memories as voice or text data.

[2126] "Server means" refers to a computer system that receives and processes stored data input by a user.

[2127] "AI Module" refers to artificial intelligence software that analyzes received stored data and extracts important phrases and keywords.

[2128] "Natural language processing means" refers to technology for analyzing memory context based on extracted phrases and keywords.

[2129] "Storage means" refers to a function that converts the analyzed stored data into a digital format and stores it in a database.

[2130] "Search and purchase means" refers to an interface that allows a user to search and purchase saved stored data.

[2131] "Delivery means" refers to a mechanism for providing purchased storage data to a user.

[2132] "Virtual reality linking means" refers to a technology that provides processed stored data to users as virtual reality content.

[2133] This invention relates to a system that allows users to input personal memories, analyze and store them using natural language processing (NLP) technology, and then allows other users to search and purchase them. The purchased memory data can then be reproduced as virtual reality (VR) content.

[2134] Memory input and analysis

[2135] Memory Input:

[2136] User: To input a memory, launch a dedicated application on the device. The application offers the option of voice input or text input.

[2137] For example, if a user wants to record memories of a high school soccer tournament, he or she launches the application and, in the case of voice input, says, "The time when I participated in the high school soccer tournament."

[2138] In the case of text input, the same content is written in the text field.

[2139] Sending and receiving data:

[2140] Terminal: When a user inputs voice data, the voice data is converted into text data using transcription software. The converted text data or the text data entered directly is sent to the server as an HTTP request.

[2141] Server: Receives the stored data sent from the device. If the received data is voice data, the process of converting voice to text begins.

[2142] Natural Language Processing:

[2143] Server: Sends the received text data to the AI ​​module for natural language processing (NLP).

[2144] AI module: Analyzes text data using NLP to extract key phrases and keywords, such as "high school," "soccer tournament," and "championship."

[2145] Contextual analysis:

[2146] AI module: Based on extracted keywords and phrases, it understands the context of memories and tags and structures the details, making them easier to search and organize.

[2147] Preserving memories

[2148] Database storage

[2149] Server: Based on the analysis results, the stored data is classified and tagged. Tags include category information such as "sports," "school," and "tournament."

[2150] Server: Stores the stored data in a database after converting it into digital format.

[2151] Searching and Purchasing Memory

[2152] Memory Retrieval:

[2153] User: Another user searches for "memories of high school soccer tournaments" in an application on their device and enters search keywords.

[2154] Terminal: Sends the entered search query to the server.

[2155] Retrieving and displaying search results:

[2156] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[2157] Terminal: The search results received from the server are displayed to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament" are displayed.

[2158] Purchasing and donating memory:

[2159] User: Select a memory they like and press the purchase button to purchase it.

[2160] Device: Sends a purchase request to the server.

[2161] Server: Processes the payment and generates a download link or streaming URL for the stored data once the purchase is confirmed.

[2162] On your device: You can provide the user with a download link or streaming URL to access their purchased stored data.

[2163] Virtual reality reproduction

[2164] Virtual Reality Integration:

[2165] Server: Reconstructs the purchased memory data as virtual reality content and generates links.

[2166] User: Experience virtual reality content using smart glasses or a head-mounted display.

[2167] Specific examples and prompts for the generative AI model

[2168] Examples:

[2169] User A: Inputs a memory of a soccer tournament from high school, and talks about the changes in emotions and detailed episodes that occurred during the event.

[2170] Server: Converts voice data into text and analyzes it using NLP. Stores the analysis results in a database.

[2171] User B: Searches for "high school soccer tournament" in the virtual store and finds User A's memory.

[2172] Server: Returns search results and generates a VR link. User B enjoys the VR experience through the link.

[2173] Prompt statement:

[2174] "Tell me about a time when you participated in a high school soccer tournament. Please describe in detail any particularly memorable episodes or emotions."

[2175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2176] Step 1:

[2177] User enters memory:

[2178] Input: The user launches a dedicated application on the device and inputs the memory by voice or text.

[2179] Processing: In the case of voice input, the application records the voice data, and in the case of text input, it receives the text data as is.

[2180] Output: Audio or text data.

[2181] Step 2:

[2182] The device converts the data:

[2183] Input: Audio data.

[2184] Processing: When a voice command is received, the device converts the voice data into text using transcription software (e.g., Google Speech-to-Text API or IBM Watson).

[2185] Output: Text data.

[2186] Step 3:

[2187] Sending data:

[2188] Input: Text data.

[2189] Processing: The terminal sends the converted text data or the directly entered text data to the server as an HTTP request.

[2190] Output: HTTP request (text data).

[2191] Step 4:

[2192] Server receives data:

[2193] Input: HTTP request (text data).

[2194] Processing: The server receives the stored data sent from the terminal.

[2195] Output: Received text data.

[2196] Step 5:

[2197] Perform natural language processing:

[2198] Input: Received text data.

[2199] Processing: The server uses AI modules (e.g., Hugging Face Transformers) to perform natural language processing (NLP) and extract key phrases and keywords.

[2200] Output: Extracted phrases and keywords.

[2201] Step 6:

[2202] Contextual analysis:

[2203] Input: Extracted phrases or keywords.

[2204] Processing: The server's AI module analyzes the memory context based on phrases and keywords, and tags and structures the details.

[2205] Output: Tagged data.

[2206] Step 7:

[2207] Save to database:

[2208] Input: Tagged data.

[2209] Processing: The server converts the tagged stored data into a digital format and stores it in a database.

[2210] Output: Memorized data stored in a database.

[2211] Step 8:

[2212] User searches memory:

[2213] Input: Search keyword.

[2214] Process: The user enters search keywords into an application on the device, and the device sends the search query to the server as an HTTP request.

[2215] Output: HTTP request (search query).

[2216] Step 9:

[2217] The server retrieves the search results:

[2218] Input: HTTP request (search query).

[2219] Processing: The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[2220] Output: HTTP response (search results).

[2221] Step 10:

[2222] Display search results:

[2223] Input: HTTP response (search result).

[2224] Processing: The terminal displays the search results received from the server to the user. For example, it displays the title and detailed information such as "Memories of Mr. / Ms. XX's High School Soccer Tournament."

[2225] Output: What is displayed to the user.

[2226] Step 11:

[2227] User purchases memory:

[2228] Input: Purchase request.

[2229] Processing: The user selects a memory they like and presses the purchase button. The device sends a purchase request to the server.

[2230] Output: HTTP request (purchase request).

[2231] Step 12:

[2232] The server processes the payment:

[2233] Input: HTTP request (purchase request).

[2234] Processing: The server processes the payment and generates a download link or streaming URL for the stored data for which the purchase has been confirmed.

[2235] Output: Download link or streaming URL.

[2236] Step 13:

[2237] Providing purchase data:

[2238] Input: Download link or streaming URL.

[2239] Processing: The device will display a download link or streaming URL to the user, allowing them to access the stored data they purchased.

[2240] Output: Provide link to user.

[2241] Step 14:

[2242] Virtual reality reproduction:

[2243] Input: Purchased memory data.

[2244] Processing: Reconstructs purchased memory data into virtual reality content and generates a link.

[2245] Output: Virtual reality experience link.

[2246] Step 15:

[2247] Experience virtual reality content:

[2248] Input: Virtual reality experience link.

[2249] Processing: Users experience virtual reality content using smart glasses or a head-mounted display.

[2250] Output: Experience virtual reality content.

[2251] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2252] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memories input by a user and the emotional state of the memories. Specific embodiments will be described in detail below.

[2253] Memory input and emotion recognition

[2254] Memory input

[2255] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[2256] For example, to record "Memories of my college graduation ceremony," a user selects voice input and speaks into the microphone, "The joy I felt at my college graduation ceremony," or writes the memory in the text input field.

[2257] Emotion recognition

[2258] Device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. For example, it determines whether the user is expressing emotions such as joy, sadness, or surprise through voice tone and speed, as well as sentiment analysis of the text.

[2259] Terminal: The recognized emotion data is sent to the server along with the input data.

[2260] Sending and Receiving Data

[2261] Sending data

[2262] Terminal: The converted text data or directly entered text data and emotion data are sent to the server as an HTTP request.

[2263] Receiving data

[2264] Server: Receives emotion-recognized memory data sent from the device.

[2265] Memory analysis

[2266] Performing natural language processing

[2267] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[2268] Server: Sends data to the NLP module and starts the analysis.

[2269] Keyword extraction

[2270] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[2271] Contextual Analysis

[2272] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[2273] Preserving memories

[2274] Database storage

[2275] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[2276] Server: Stores the stored data in a database after converting it into digital format.

[2277] Searching and Purchasing Memory

[2278] Memory retrieval

[2279] User: Another user searches for "joyful memories of college graduation" in an application on their device and enters search keywords and the desired emotional state.

[2280] Terminal: Sends the entered search query to the server.

[2281] Getting search results

[2282] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[2283] Displaying search results

[2284] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[2285] Purchasing and donating memories

[2286] Purchasing Memory

[2287] User: Select a memory they like and press the purchase button to purchase it.

[2288] Device: Sends a purchase request to the server.

[2289] Payment Processing

[2290] Server: Receives the purchase request and processes the payment through the payment system.

[2291] Providing stored data

[2292] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[2293] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[2294] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search and purchase them. Furthermore, users who purchase them can enjoy the emotional state of their memories as well.

[2295] The processing flow will be explained below.

[2296] Memory input and emotion recognition

[2297] Step 1: Entering memory

[2298] User: To input a memory, the user launches a dedicated application on the device. The application offers the option of voice input or text input.

[2299] For example, if a user wants to record "memories of college graduation," they can select voice input and speak into the microphone, "the joy I felt at college graduation," or write their memories in the text input field.

[2300] Step 2: Recognize emotions

[2301] On the device: Analyzes input voice or text data in real time and uses an emotion engine to recognize the user's emotional state. Determine whether the user is expressing emotions such as joy, sadness, or surprise based on the tone and speed of the voice and sentiment analysis of the text.

[2302] Step 3: Text Conversion (Speech Input)

[2303] Device: If voice input is selected, the recorded voice data is converted to text data using transcription software, and the converted text data is temporarily stored.

[2304] Step 4: Sending data

[2305] Terminal: The converted text data or the directly entered text data and the recognized emotion data are sent to the server as an HTTP request.

[2306] Data reception and analysis

[2307] Step 5: Receiving the data

[2308] Server: Receives emotion-recognized memory data sent from the device.

[2309] Step 6: Performing Natural Language Processing

[2310] Server: Passes the received text data to the AI ​​module and prepares it for natural language processing (NLP).

[2311] Server: Sends data to the NLP module and starts the analysis.

[2312] Step 7: Keyword extraction

[2313] AI module: Analyzes text data using NLP to extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[2314] Step 8: Analyze the sentiment data

[2315] AI module: Based on extracted keywords, phrases, and sentiment data, it understands the context of memories and tags and structures the detailed content.

[2316] Preserving memories

[2317] Step 9: Prepare to save to the database

[2318] Server: Based on the analysis results and emotion data, the memory data is classified and tagged. Tags include category information such as "emotion," "event," and "location."

[2319] Step 10: Save to database

[2320] Server: Stores the stored data in a database after converting it into digital format.

[2321] Searching and Purchasing Memory

[2322] Step 11: Memory Retrieval

[2323] User: When another user searches for "joyful memories of college graduation" through an application on a terminal, the user inputs search keywords and a desired emotional state.

[2324] Terminal: Sends the entered search query to the server.

[2325] Step 12: Getting search results

[2326] Server: Searches for the relevant stored data from the database and sends it to the terminal as an HTTP response.

[2327] Step 13: Viewing search results

[2328] Terminal: The search results received from the server are displayed to the user. For example, the title, detailed information, and emotional state of the user are displayed, such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)."

[2329] Purchasing and donating memories

[2330] Step 14: Buy Memory

[2331] User: Select the memory they like and press the purchase button if they wish to purchase it.

[2332] Device: Sends a purchase request to the server.

[2333] Step 15: Payment Processing

[2334] Server: Receives the purchase request and processes the payment through the payment system.

[2335] Step 16: Confirm your purchase

[2336] Server: Once payment is complete, verify that the purchase was successful.

[2337] Step 17: Provide stored data

[2338] Server: Generates and provides the download link or streaming URL for the purchased stored data to the user.

[2339] Terminal: Displays the link or URL provided by the server to the user, allowing them to access the purchased stored data.

[2340] In this way, a system is realized in which users can input their own memories, recognize and store their emotional state, and other users can search for and purchase them. In addition, users who purchase the memories can enjoy the emotional state of the memories.

[2341] Example 2

[2342] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2343] In existing systems, when a user inputs a memory, it is difficult to recognize the specific emotional state and analyze, store, and provide the memory data. Therefore, there is a need for a system that can accurately recognize the emotional information in memory and analyze, store, search, and provide the data based on that information.

[2344] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2345] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, emotion recognition means for recognizing emotional states from voice and text data, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, and provision means for providing purchased memory data to users. This allows users to input, recognize, and store memories more accurately, and enables other users to search, purchase, and share memories, including their emotional information.

[2346] "Input means" refers to the method or device that a user uses to input a memory.

[2347] "Server means" refers to a central computing device for receiving and processing stored data sent by users.

[2348] "AI Module" refers to the artificial intelligence software module used to analyze received stored data and extract key phrases and keywords.

[2349] "Emotion recognizer" refers to a system or method for recognizing emotional states from speech and text data.

[2350] "Storage means" refers to a means or device for converting the analyzed stored data into a digital format and storing it in a database.

[2351] "Search and purchase means" refers to a method or device that allows a user to search for stored data and purchase it if necessary.

[2352] "Providing means" refers to a means or system for providing the purchased storage data to the user in the form of a download link or streaming URL.

[2353] The present invention relates to a system for recognizing, analyzing, storing, retrieving, and providing memory input by a user and the emotional state of that memory.

[2354] The system consists of the following elements:

[2355] 1. Input Method

[2356] User: The user launches a dedicated application on the device and inputs their memories. The application offers the option of voice input or text input.

[2357] Example: To record "Memories of my college graduation ceremony," a user taps the voice input button and speaks "The joy I felt at my college graduation ceremony," or writes "The joy I felt at my college graduation ceremony" in the text input field.

[2358] 2. Server Means

[2359] Terminal: The stored data entered by the user is sent to the server in JSON format, for example, { "text": "The joy felt at the university graduation ceremony", "emotion": "Joy"}.

[2360] Server: The server receives the data sent from the device and prepares it for processing.

[2361] 3. AI Module

[2362] Server: Passes the received text data to an AI module (e.g., TensorFlow or PyTorch) for natural language processing (NLP).

[2363] Example: Using NLP to extract important phrases and keywords such as "university," "graduation," and "joy" from text data.

[2364] 4. Emotion recognition means

[2365] On the device: An emotion engine is used to analyze voice and text data, identifying emotions from the tone and rate of voice and the content of text.

[2366] Example: Recognizing emotional states such as happiness, sadness, surprise, etc. For example, identifying "happiness" based on tone and rate of speech and text.

[2367] 5. Preservation means

[2368] Server: Converts the analyzed memory data into a digital format and stores it in a database.

[2369] Example: Save data with tagging that includes category information such as "emotion," "event," "place," etc. For example, set tags like tags: ["university," "graduation ceremony," "joy"].

[2370] 6. Search and Purchase Methods

[2371] User: Another user searches a memory to find information about a specific emotion or event. Enter search keywords and an emotional state.

[2372] Example: A user types "college," "graduation," and "joy" into the app's search bar and clicks the search button.

[2373] Device: Sends a search query to the server.

[2374] 7. Means of provision

[2375] Server: Provides the found stored data to the user in the form of a download link or streaming URL.

[2376] Example: Displaying a title such as "Memories of Mr. / Ms. XX's College Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[2377] User: Selects the memory they like, purchases it, enters payment information and taps the purchase button.

[2378] Server: Receives the purchase request, processes the payment through a payment system (e.g., Stripe or PayPal), and then generates a download link or streaming URL for the purchased stored data and provides it to the user.

[2379] 8. Examples of concrete examples and prompts

[2380] Example: User A inputs the joy of his / her college graduation into a memory, and the emotion is recognized and saved as "joy." User B searches for "college," "graduation ceremony," and "joy," discovers User A's memory, and purchases it, allowing him / her to learn more about the joy.

[2381] Example prompt: "Describe a system that recognizes and stores emotional states based on memories of your college graduation. Explain in detail how the system analyzes and stores the data, using the example memory of 'summer family vacation.'"

[2382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2383] Step 1:

[2384] Memory input

[2385] User: The user launches a dedicated application on the device and inputs their memories. The application provides the option of voice input or text input.

[2386] Specific actions: To record "Memories of my college graduation ceremony," the user taps the voice input button and speaks "The joy I felt at my college graduation ceremony" into the microphone, or writes "The joy I felt at my college graduation ceremony" in the text input field.

[2387] Input: User voice or text data

[2388] Output: Audio file or text data as stored data

[2389] Step 2:

[2390] Emotion recognition

[2391] Device: The device converts the input voice data into text using a speech recognition function, then passes the text data to an emotion recognition engine to recognize the emotional state.

[2392] What it does: It converts voice data into text using the Google Speech-to-Text API, then passes the text data to IBM Watson Emotion Analysis to identify emotional states such as "joy" or "sadness."

[2393] Input: Audio file or text data as storage data

[2394] Output: Recognized emotion data (e.g., happy, sad)

[2395] Step 3:

[2396] Sending data

[2397] Device: The device sends the recognized emotion data and text data to the server in JSON format.

[2398] Specific operation: The data is sent as an HTTP request in the format { "text": "The joy I felt at my college graduation ceremony", "emotion": "Joy"}.

[2399] Input: Recognized emotion data and text data

[2400] Output: JSON data sent to the server

[2401] Step 4:

[2402] Data reception and analysis

[2403] Server: The server receives the data sent from the device and passes the received text data to the AI ​​module for natural language processing (NLP).

[2404] What it does: It uses AI modules (e.g. TensorFlow and PyTorch) to analyze the content of text and extract important phrases and keywords.

[2405] Input: JSON data sent from the terminal

[2406] Output: Extracted phrases and keywords (e.g., "university," "graduation," "joy")

[2407] Step 5:

[2408] Contextual Analysis

[2409] AI module: Understands the context of memories based on extracted keywords and phrases, along with emotional data.

[2410] Specific operation: The memory content is tagged and structured based on extracted phrases, keywords, and emotional data.

[2411] Input: Extracted phrases and keywords, and sentiment data

[2412] Output: Structured stored data

[2413] Step 6:

[2414] Data storage

[2415] Server: Converts structured stored data into a digital format and stores it in a database.

[2416] Specific operation: Save data with tagging and category information such as "emotion," "event," "location," etc. For example, set tags as follows: tags: ["university," "graduation ceremony," "joy"]

[2417] Input: Structured stored data

[2418] Output: Memory data stored in the database

[2419] Step 7:

[2420] Memory retrieval

[2421] User: Another user uses an application on the device to search for memories, for example, searching for "college graduation joy."

[2422] Specific actions: Enter "college," "graduation," and "joy" in the search bar and click the search button.

[2423] Input: Search keyword

[2424] Output: Data sent to the server as a search query

[2425] Step 8:

[2426] Providing search results

[2427] Server: Searches the database for stored data that matches the search criteria and sends it to the terminal as an HTTP response.

[2428] Specific operation: The found stored data is sent to the device as an API response and converted into a display format.

[2429] Input: search query

[2430] Output: Corresponding stored data

[2431] Step 9:

[2432] Displaying search results

[2433] Device: Displays the received search results to the user. For example, it displays the title such as "Memories of XX's University Graduation Ceremony (Joy)," as well as detailed information and emotional state.

[2434] What it does: It displays search results in a list format and provides a detailed view that displays more information when you click on it.

[2435] Input: Search results received from the server

[2436] Output: The search results that are displayed to the user

[2437] Step 10:

[2438] Purchasing and donating memories

[2439] User: If they want to purchase a memory they like, they press the purchase button and enter their payment information.

[2440] What happens: You tap the buy button, enter your payment information, and confirm your purchase.

[2441] Input: Purchase request and payment information

[2442] Output: Purchase request sent to the server

[2443] Step 11:

[2444] Payment processing and provision

[2445] Server: Receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal). Then, generates a download link or streaming URL for the purchased stored data and provides it to the user.

[2446] Specific operation: After payment confirmation, an access link for the stored data is generated and sent to the user.

[2447] Input: Purchase request and payment information

[2448] Output: A download link or streaming URL provided to the user

[2449] (Application example 2)

[2450] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2451] Conventional memory data management systems have limited user memory input to text or voice data, making it impossible to recognize emotions or sell memory data in virtual stores. This has limited the user experience, making it difficult to share or trade memory data effectively.

[2452] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2453] In this invention, the server includes input means for inputting memories from users, server means for receiving the input memory data, an AI module for analyzing the received memory data and extracting important phrases and keywords, storage means for converting the analyzed memory data into a digital format and storing it in a database, search and purchase means for users to search for and purchase memories, provision means for providing purchased memory data to users, virtual store display means for selling memory data in a virtual space, emotion analysis means with emotion recognition capabilities, voice recognition means for converting voice data into text, and purchase management means for processing the purchase procedure for memory data through a payment means. This allows users to record their own memories in detail, analyze them including emotions, and sell, search, and purchase them in a virtual store.

[2454] A "user" is someone who uses this system and performs operations such as memory input, search, and purchase.

[2455] "Memories" refers to information about personal experiences and memories entered by the user.

[2456] "Input means" refers to a device or interface that allows a user to input memories, including voice and text input options.

[2457] The term "server means" refers to computer resources that receive input stored data and perform various processes such as analysis.

[2458] "AI module" refers to the artificial intelligence algorithms and software used to analyze input memory data and extract key phrases and keywords.

[2459] "Storage means" refers to a mechanism for converting the analyzed stored data into a digital format and storing it in a database.

[2460] "Search facility" refers to the interface and associated mechanisms by which a user searches the memory.

[2461] "Purchase vehicle" refers to the interface and associated mechanisms for a user to purchase selected storage.

[2462] "Providing means" refers to a mechanism for providing the purchased stored data to the user, and includes providing the data in the form of a download link or a streaming URL.

[2463] "Virtual store display means" refers to a display interface and related mechanisms for selling stored data in a virtual space.

[2464] "Emotion analysis means" refers to a mechanism for recognizing and analyzing the user's emotions from stored data.

[2465] "Speech recognition means" refers to a mechanism for converting voice data into text data.

[2466] "Purchase management means" refers to a mechanism for processing the purchase procedure for stored data through a payment means.

[2467] "Payment Method" means the electronic payment system or service used to complete a Purchase.

[2468] The present invention relates to a system that allows users to input their own memories, recognize and analyze their emotions, and sell and purchase items in a virtual store. Specific embodiments are described in detail below.

[2469] Memory input and emotion recognition

[2470] Memory input

[2471] Users install and launch a dedicated application for recording memories on their smartphone. This application offers the option of voice input or text input. For example, to record "memories of college graduation," a user can select voice input and speak into the microphone, "the joy I felt at my college graduation ceremony." Alternatively, they can write their memories in the text input field.

[2472] Emotion recognition

[2473] The application analyzes the voice or text data entered by the user in real time and uses an emotion analysis engine to recognize the user's emotional state. For voice data, it uses the Google Cloud Speech-to-Text API to convert the speech to text. Furthermore, it uses IBM Watson Tone Analyzer to analyze the emotion of text data. For example, it determines whether the user is expressing emotions such as "happiness," "sadness," or "surprise" based on the tone and speed of the voice and the emotion analysis of the text. This emotion data is sent to the server along with the input data.

[2474] Sending and Receiving Data

[2475] Sending data

[2476] The terminal transmits the converted text data or the directly input text data and the emotion data as an HTTP request to the server.

[2477] Receiving data

[2478] The server receives the emotion-recognized stored data transmitted from the terminal.

[2479] Memory analysis

[2480] Performing natural language processing

[2481] The server passes the received text data to a generative AI module, preparing it for natural language processing (NLP). The generative AI model uses, for example, the Transformers library from Hugging Face.

[2482] Keyword extraction

[2483] The AI ​​module uses NLP to analyze the text data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[2484] Contextual Analysis

[2485] The AI ​​module understands the context of the memory based on extracted keywords, phrases, and emotional data, and then tags and structures the detailed content.

[2486] Preserving memories

[2487] Database storage

[2488] The server categorizes and tags the memory data based on the analysis results and emotional data. The tags include category information such as "emotion," "event," and "location." The memory data converted into digital format is stored in an SQL database (MySQL or PostgreSQL).

[2489] Searching and Purchasing Memory

[2490] Memory retrieval

[2491] A user can use the application's search function to search for memories entered by other users. By entering a desired emotional state along with search keywords, the user can find the corresponding memory data. For example, searching for "graduation ceremony" and "joy."

[2492] Getting search results

[2493] The server searches the database for the corresponding stored data and sends it to the terminal as an HTTP response.

[2494] Displaying search results

[2495] The terminal displays the search results received from the server to the user, such as a title such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)," along with detailed information and emotional state.

[2496] Purchasing and donating memories

[2497] Purchasing Memory

[2498] When the user selects a memory they like and presses the purchase button to purchase it, the terminal sends a purchase request to the server.

[2499] Payment Processing

[2500] The server receives the purchase request and processes the payment through a payment system (e.g., Stripe or PayPal).

[2501] Providing stored data

[2502] The server generates a download link or streaming URL for the purchased stored data and provides it to the user. The terminal displays the link or URL provided by the server to the user, allowing the user to access the purchased stored data.

[2503] Specific examples

[2504] Consider a scenario in which a user registers their "joy felt at their university graduation ceremony" by voice input into the system, and other users search for and purchase this item using the keywords "graduation ceremony" and "joy." This system provides a way for individual users to experience and share their memories and emotions in a virtual store.

[2505] Prompt Sentence Examples

[2506] "Tell me about the joy you felt at your college graduation."

[2507] "Please enter your emotional memory of graduation."

[2508] "Search for the keyword 'graduation' and the emotion 'joy'"

[2509] In this way, the system of the present invention inputs the user's memories and recognizes, analyzes, stores, searches, and provides emotions, thereby realizing functions that enrich the user's experience.

[2510] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2511] Step 1:

[2512] The user launches a dedicated application on their smartphone for inputting memories. They can select either voice input or text input as the input method. If voice input is selected, the user speaks their memories into a microphone. If text input is selected, the user inputs their memories using a keyboard. This input data is sent to the application as voice data or text data.

[2513] Step 2:

[2514] The device processes the input data in real time, converting voice data into text using the Google Cloud Speech-to-Text API, and then performs emotion analysis on the converted text data using IBM Watson Tone Analyzer. The analysis results in emotional data such as joy, sadness, and surprise.

[2515] Step 3:

[2516] The device then sends the converted text data and emotion data to the server as an HTTP request, which includes the user's input memory and associated emotion.

[2517] Step 4:

[2518] The server receives the memory data and emotion data sent from the terminal. This received data includes the user's memory information and emotion information.

[2519] Step 5:

[2520] The server prepares the received memory data for natural language processing (NLP) using a generative AI model, sending the data to an NLP module using a library such as Hugging Face's Transformers library, and begins analysis.

[2521] Step 6:

[2522] The AI ​​module uses NLP to analyze the memory data and extract important phrases and keywords, such as "university," "graduation ceremony," and "joy."

[2523] Step 7:

[2524] The AI ​​module understands the context of memories based on extracted keywords, phrases, and emotional data, and then tags and structures the details, assigning tags such as "emotion," "event," and "location" to the memory data.

[2525] Step 8:

[2526] The server converts the tagged analysis results into a digital format and stores them in an SQL database such as MySQL or PostgreSQL, including the tagged stored data.

[2527] Step 9:

[2528] A user makes a request to search for a memory from a terminal. The search request includes keywords, such as "graduation ceremony" and "joy," as well as a desired emotional state.

[2529] Step 10:

[2530] The server searches the database based on the received search request, retrieves the corresponding stored data as search results, and sends the search results to the terminal as an HTTP response.

[2531] Step 11:

[2532] The terminal displays the search results received from the server to the user. For example, the title and detailed information such as "Memories of Mr. / Ms. XX's University Graduation Ceremony (Joy)" are displayed.

[2533] Step 12:

[2534] The user goes through the process of purchasing a memory they like. When they press the purchase button, the device sends a purchase request to the server.

[2535] Step 13:

[2536] The server receives the purchase request, processes the payment through a payment system such as Stripe or PayPal, and, if successful, generates a download link or streaming URL for the stored data.

[2537] Step 14:

[2538] The server sends the generated download link or streaming URL to the terminal, which displays it to the user, allowing the user to access the stored data they purchased.

[2539] Prompt Sentence Examples

[2540] "Tell me about the joy you felt at your college graduation."

[2541] "Please enter your emotional memory of graduation."

[2542] "Search for the keyword 'graduation' and the emotion 'joy'"

[2543] Through the above processing steps, a system is provided that allows users to record their own memories in detail, share them with other users, and trade in virtual stores.

[2544] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the...

Claims

1. input means for inputting memories from a user; a server means for receiving input stored data; An AI module that analyzes the received memory data and extracts important phrases and keywords; a storage means for converting the analyzed stored data into a digital format and storing the digital data in a database; search and purchase means for users to search and purchase memories; and providing means for providing the purchased stored data to the user.

2. 2. The system of claim 1, wherein said input means provides a choice between voice input and text input.

3. 2. The system according to claim 1, wherein the providing means provides the purchased stored data in the form of a download link or a streaming URL.

Images