system

Abstract

Provide a system. A method for recording audio data provided by a user; means for transmitting the recorded voice data to a server; A means for converting the received voice data into text by the server; A means for analyzing the converted text data and extracting keywords and context; a means for storing the extracted data in a per-user database; means for generating a digital copy of the user using the stored data; means for responding to interaction requests using the generated digital copy; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today's elderly and terminally ill patients, as well as their families and friends, face many limitations in the means they have to pass on their precious memories and experiences to future generations. It is particularly difficult for elderly people with declining memory or those who have not had the opportunity to record their experiences to preserve them in a tangible form. Preserving memories of deceased loved ones is important as emotional support, but current technology does not adequately meet this need. There is a need for a system that can solve these issues, digitize users' memories and personalities, permanently store and share them, and utilize them as a new means of communication. [Means for solving the problem]

[0005] The present invention provides a system that records voice data provided by a user and transmits the data to a server. The server converts the received voice data into text and analyzes the converted text data to extract keywords and context. The extracted data is stored in a database for each user, and then machine learning is used to create a digital copy of the user. This digital copy is continuously trained based on new interactions. When responding to interaction requests, the system generates appropriate responses based on specific context and keywords, allowing family and friends to connect with the user's memories and personality through the interaction. This allows for a new way to connect with the deceased and can be used as a means to pass on memories and experiences to future generations.

[0006] A "user" is an individual who uses the system to provide voice data and digitize their memories and experiences.

[0007] "Voice data" is digital data that is an electronic recording of what a user says.

[0008] A "terminal" is a device used by a user (e.g., a smartphone or tablet) that records audio data and transmits it to a server.

[0009] A "server" is a central processing unit that receives voice data, analyzes it, stores it in a database, and generates digital copies and interactive responses.

[0010] "Converting to text" refers to the process of converting voice data into character string data, and is often done using voice recognition technology.

[0011] "Natural language processing" is a technology that analyzes text data to extract keywords and context.

[0012] A "database" is an electronic recording device that structures and stores each user's memories and experiences.

[0013] "Machine learning" is an artificial intelligence technology that learns patterns from large amounts of data and makes future predictions and decisions.

[0014] A "digital copy" is a digital personality model built based on a user's memories and experiences.

[0015] An "interaction request" is a request sent by a family member or friend to interact with the digital copy.

[0016] A "response" is a reply message sent by a digital copy in response to a dialogue request. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention digitizes memories and experiences based on voice data provided by a user, and then conducts a dialogue using the generated digital copy. Specific embodiments are described below.

[0039] Collection and transmission of voice data

[0040] Users start a dedicated application and provide audio data about their daily lives and past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[0041] Speech-to-text transcription and analysis

[0042] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0043] Saving to a database

[0044] The analyzed text data and keywords are stored in a database on the server. This database is structured for each user and organizes data based on different categories and timelines, allowing for a detailed record of the user's memories and experiences.

[0045] Digital copy generation and learning

[0046] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0047] Interactive functionality

[0048] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, pulls out relevant information from the digital copy, and generates a specific response. This response is then provided to the family via the device. This allows for detailed conversations about the deceased's memories and experiences, maintaining an emotional connection in a new way.

[0049] Specific examples

[0050] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0051] 2. The device records this audio data and sends it to the server.

[0052] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0053] 4. The server stores these data in a database and updates the user's digital copy.

[0054] 5. When a family member makes a request such as "Tell me about your summer vacation memories," the server uses the digital copy to generate an appropriate response, which the device then relays to the family member.

[0055] In this way, this system digitizes the user's memories and personality, not only passing on their memories to future generations but also enabling a new form of communication with family and friends.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] The user launches a dedicated application and begins speaking audio data about daily life and past events.

[0059] Step 2:

[0060] The device records the user's voice and stores the recording in a buffer.

[0061] Step 3:

[0062] The device sends the completed recording to the server's API endpoint using the HTTPS protocol.

[0063] Step 4:

[0064] The server passes the received voice data to a speech recognition engine and converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[0065] Step 5:

[0066] The server then passes the converted text data to a natural language processing (NLP) engine for keyword and context analysis. In this step, keywords such as "childhood," "mother," and "ocean" are extracted.

[0067] Step 6:

[0068] The server stores the analyzed keywords and text data in a database for each user, organizing the data based on date and category.

[0069] Step 7:

[0070] The server uses the stored data to train machine learning models, which then generate a digital copy of the user that reflects how they speak and think.

[0071] Step 8:

[0072] The device receives conversation requests from family and friends, such as "Tell me about your summer vacation memories."

[0073] Step 9:

[0074] The server analyzes the interaction request it receives and queries the digital copy of the user to extract relevant memory and experience data.

[0075] Step 10:

[0076] The server generates an appropriate response from the digital copy, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0077] Step 11:

[0078] Your device will generate a response that will be displayed to family and friends, or played aloud.

[0079] Step 12:

[0080] The server updates the machine learning model again based on the new interaction data collected, improving the accuracy of the user's digital copy.

[0081] This series of processes digitizes the user's memories and personality, enabling new forms of communication with family and friends.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Current systems for digitizing users' memories and experiences and maintaining emotional connections with family and friends in new ways face several challenges. First, accuracy is required in the conversion of voice data to text and subsequent analysis, but there is a lack of technology to improve this accuracy. Second, if the machine learning technology used to generate the digital copy does not perform sufficiently, the reproduction of the user's memories and interactions will be insufficient. Finally, the digital copy's ability to continuously learn has not been established, which can lead to a decline in the quality of the interactions.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes means for including a speech recognition module that converts speech data into text, means for including a natural language processing engine that analyzes the text data and extracts keywords and context, means for storing the extracted data in a database for each user, and means for using machine learning technology to generate a digital copy of the user using the stored data. This enables highly accurate text conversion and analysis of speech data, and by generating and continuously training the digital copy of the user, the quality of the dialogue can be improved and the user's memories and experiences can be more faithfully digitized.

[0087] "User" refers to a person who uses the system to provide voice data.

[0088] "Terminal" refers to a device that has the function of recording a user's voice data and transmitting it to a server.

[0089] A "server" refers to a device or system that receives voice data sent from a terminal, converts it into text data, analyzes it, and stores it in a database.

[0090] A "voice recognition module" refers to software or hardware for converting voice data into text data.

[0091] A "natural language processing engine" refers to technology for analyzing text data and extracting important keywords and context.

[0092] "Database" refers to a system that stores and manages analyzed text data and keywords for each user.

[0093] "Machine learning technology" refers to algorithms and models that use data to create a digital copy of a user and allow it to continuously learn.

[0094] A "digital copy" refers to a model created to mimic a user's speech and thought patterns.

[0095] "Interaction requests" refer to questions or requests for conversation that a user's family or friends make to a digital copy.

[0096] "Response" refers to a specific reply that a digital copy makes in response to an interaction request.

[0097] The system of the present invention aims to digitize memories and experiences based on voice data provided by users, and then engage in dialogue using the generated digital copies. The system of the present invention includes the following main components:

[0098] Collection and transmission of voice data

[0099] Users start a dedicated application and provide audio data about their daily lives or past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[0100] Speech-to-text transcription and analysis

[0101] The server converts the received voice data into text data using a speech recognition module (e.g., Google® Speech-to-Text). The converted text data is then analyzed using a natural language processing (NLP) engine (e.g., OpenAI® GPT-3®) to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0102] Saving to a database

[0103] The server stores the analyzed text data and keywords in a database. This database is structured for each user and organizes data based on different categories and timelines. For example, there are categories such as "childhood" and "summer memories."

[0104] Digital copy generation and learning

[0105] The server creates a digital copy of the user based on the data stored in the database. This digital copy uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns. The digital copy continuously learns based on new interaction data to form a more faithful personality model. For example, it learns about the specific words and phrases the user speaks.

[0106] Interactive functionality

[0107] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about Mom's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. This response is then provided to the family member via the device.

[0108] Specific examples

[0109] For example, a user might say, "The flowers I saw in the park yesterday were beautiful," using a dedicated application. The device records this and sends it to the server. The server then converts the received voice data into text using Google Speech-to-Text, generating the text, "The flowers I saw in the park yesterday were beautiful." The server then analyzes the text data using OpenAI GPT-3, extracting keywords such as "yesterday," "park," "flowers," and "beautiful." The server then stores this data in a database and categorizes it into categories such as "yesterday's events" and "memories of the park."

[0110] The device accepts a request from the family such as "What was beautiful yesterday?" The server analyzes this and generates a response from past data, such as "The flowers I saw in the park yesterday were beautiful," and responds to the family via the device.

[0111] Prompt Sentence Examples

[0112] "Analyze the following audio data and extract key keywords and context to generate a digital copy of the user. Example: 'When I was a child, I used to spend summer vacations at my grandparents' house.'"

[0113] In this way, the system digitizes the user's memories and personality, passing on their memories to future generations and enabling new forms of communication with family and friends.

[0114] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0115] Step 1:

[0116] The user starts the dedicated application and inputs voice information. For example, the user might say, "The flowers I saw in the park yesterday were beautiful."

[0117] Input: User's voice data

[0118] Output: Audio data recorded on the device

[0119] Specific behavior: The user presses the record button on the application, and recording stops when the user finishes speaking.

[0120] Step 2:

[0121] The terminal records the user's voice data and transmits the voice data to the server in real time or at regular intervals.

[0122] Input: Recorded audio data

[0123] Output: Audio data sent to the server

[0124] Specific operation: The device uploads audio data to a server via the Internet.

[0125] Step 3:

[0126] The server converts the received voice data into text data using a voice recognition module (e.g., Google Speech-to-Text).

[0127] Input: Audio data sent to the server

[0128] Output: Text data

[0129] Specific operation: The speech recognition module is started on the server side, the audio file is analyzed, and text is generated.

[0130] Step 4:

[0131] The server passes the converted text data to a natural language processing (NLP) engine (e.g., OpenAI GPT-3) for analysis to extract keywords and context.

[0132] Input: Text data

[0133] Output: Extracted keywords and context information

[0134] How it works: The NLP engine analyzes the text data and extracts important keywords such as "yesterday," "park," "flowers," and "beautiful."

[0135] Step 5:

[0136] The server stores the analyzed text data and keywords in a database, which is structured for each user and organized according to different categories and timelines.

[0137] Input: Extracted keywords and context information

[0138] Output: Text data for each user stored in the database

[0139] Specific operation: Text data and keywords are stored by category using database software (e.g., MySQL (registered trademark)).

[0140] Step 6:

[0141] The server creates a digital copy of the user based on the data stored in the database, which uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns.

[0142] Input: User data stored in the database

[0143] Output: Generated digital copy

[0144] What it does: A machine learning model uses your conversational data as training data to generate a digital copy.

[0145] Step 7:

[0146] The server continuously updates and trains the generated digital copy, allowing the model to continually learn based on new interaction data.

[0147] Input: New interaction data

[0148] Output: Updated digital copy

[0149] What happens: New interaction data is added to the database and the machine learning model is retrained.

[0150] Step 8:

[0151] The device accepts conversation requests from family and friends. For example, a family member might request, "I'd like to hear about my mother's summer vacation memories."

[0152] Input: Interaction request

[0153] Output: Request sent to server

[0154] Specific operation: The device receives a request in voice or text format and sends it to the server.

[0155] Step 9:

[0156] The server analyzes the interaction request and extracts relevant information from the digital copy to generate a specific response.

[0157] Input: Interaction request and digital copy

[0158] Output: The generated response

[0159] What it does: The server analyzes the request and uses the digital copy to generate an appropriate response.

[0160] Step 10:

[0161] The device then provides the generated response to family and friends, for example, "The flowers I saw in the park yesterday were beautiful."

[0162] Input: The generated response

[0163] Output: Responses to family and friends

[0164] What happens: The device plays or displays a response in audio or text format.

[0165] Through these steps, the system digitizes the user's memories and personality, enabling new forms of emotional connection with family and friends.

[0166] (Application example 1)

[0167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] While there have been systems that convert users' voice data into text and generate digital copies, few systems are capable of generating and delivering personalized content based on the user's memories and experiences, making it difficult to provide an experience optimized for each individual user.In addition, there are limited means of passing on content based on the memories and experiences of the deceased to future generations, creating a need for a new form of emotional connection.

[0169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0170] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for storing the extracted data in a database for each user, means for generating a digital copy of the user using the stored data, means for responding to dialogue requests using the generated digital copy, and means for generating and delivering personalized content based on the user's memories and experiences. This makes it possible to generate and deliver unique content based on the user's individual memories and experiences, thereby realizing a new form of emotional connection.

[0171] 1. "User-Provided Voice Data"

[0172] "User-provided voice data" refers to voice information recorded by a user using a dedicated device or application.

[0173] 2. "Method of recording"

[0174] A "recording means" is a device or software that has the function of acquiring and digitizing a user's voice data.

[0175] 3. "Method of sending to the server"

[0176] "Means for transmitting to the server" refers to a device or software that has the function of transferring recorded audio data to a central server via a network.

[0177] 4. "Method of converting to text"

[0178] A "text conversion means" is a device or software that uses speech recognition technology to convert voice data into text information.

[0179] 5. "Methods for analyzing and extracting keywords and context"

[0180] "Means for analyzing and extracting keywords and context" refers to a device or software that has the function of analyzing text data using natural language processing technology and identifying important words and context.

[0181] 6. "Method of storing data in a per-user database"

[0182] The "means for storing in a database for each user" is a database system that manages and stores analyzed text data and keywords for each individual user.

[0183] 7. "Means for generating a digital copy of the user"

[0184] "Means for generating a digital copy of a user" refers to a device or software that has the ability to create a digital avatar that mimics the user's voice and thought patterns based on stored data.

[0185] 8. "Means of responding to interaction requests"

[0186] A "means for responding to dialogue requests" is a device or software that has the function of generating appropriate responses using a digital copy in response to inquiries from the user or their family and friends.

[0187] 9. "A means of generating and delivering personalized content"

[0188] A "means for generating and delivering personalized content" is a device or software that has the function of creating customized videos, articles, or other content based on a user's memories and experiences and delivering it to a user's terminal.

[0189] The present invention provides a system for generating and distributing personalized content based on user voice data. This system is configured by combining multiple pieces of hardware and software.

[0190] 1. Collection of audio data

[0191] Users start a dedicated application and use their smartphone to provide audio data about their daily lives and past events. This audio data is digitized using the smartphone's recording function and sent to a server in real time or at regular intervals.

[0192] 2. Speech-to-text conversion and analysis

[0193] The server first passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. Next, this text data is analyzed using a natural language processing engine (e.g., spaCy, NLTK) to extract important keywords and context. For example, if a user says, "I traveled to Europe during my university days. Paris was particularly memorable," keywords such as "university days," "Europe," "travel," and "Paris" are extracted from the text data.

[0194] 3. Saving to the database

[0195] The analyzed text data and keywords are stored in a database (e.g., MongoDB) on the server. This database is structured for each user and organizes data based on different categories and timelines, providing a detailed record of the user's memories and experiences.

[0196] 4. Creating and Learning from Digital Copies

[0197] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0198] 5. Personalized content generation and delivery

[0199] The server generates personalized content (e.g., videos or articles) based on the user's memory and experience. It uses a generative AI model to generate the desired content by inputting a prompt. For example, the following prompt can be used:

[0200] "Generate a personalized video based on the user's memories of a trip to Europe during their college days."

[0201] The generated content is delivered to the user's smartphone application, allowing the user to view it.

[0202] Specific examples

[0203] 1. A user uses the app and says, "I traveled to Europe during my college years. Paris was particularly memorable."

[0204] 2. The smartphone transmits this voice data to the server in real time.

[0205] 3. The server converts the voice data into text and extracts the keywords "college days," "Europe," "travel," and "Paris."

[0206] 4. The server stores these data in a database and updates the user's digital copy.

[0207] 5. The server generates a personalized video using the prompt text and delivers it to the smartphone application.

[0208] This system enables users to generate and distribute unique content based on their memories and experiences, enabling new forms of emotional connection.

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] The user launches a dedicated application and talks about their daily life and past events.

[0212] Input: User's voice

[0213] Output: Recorded audio data

[0214] Specific operation: Records audio using the smartphone's microphone and saves it as digital data.

[0215] Step 2:

[0216] The device sends the recorded audio data to the server.

[0217] Input: Recorded audio data

[0218] Output: Audio data sent to the server

[0219] Specific operation: Uploads audio data to the server using the smartphone's communication function (Wi-Fi or mobile data).

[0220] Step 3:

[0221] The server passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[0222] Input: Audio data

[0223] Output: Text data

[0224] Specific operation: The server calls the speech recognition API and transcribes the audio data.

[0225] Step 4:

[0226] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy, NLTK) to extract keywords and context.

[0227] Input: Text data

[0228] Output: Extracted keywords and context

[0229] What it does: A natural language processing engine analyzes text data to identify important words, phrases, and context.

[0230] Step 5:

[0231] The server stores the extracted data in a per-user database (e.g. MongoDB).

[0232] Input: Keywords and context

[0233] Output: Data stored in the database

[0234] Specific operation: The server saves the analysis results in a database managed for each user.

[0235] Step 6:

[0236] The server uses the stored data to create a digital copy of the user, continually training them as needed.

[0237] Input: Data stored in a database

[0238] Output: Updated digital copy

[0239] How it works: The server uses machine learning models to generate a digital copy of the user that mimics their language and thought patterns, periodically feeding them new data to train them.

[0240] Step 7:

[0241] The server generates personalized content (e.g., videos, articles) based on the user's memories and experiences and delivers it to the device.

[0242] Input: Content-generated prompt based on the digital copy

[0243] Output: Generated content

[0244] Specific operation: Personalized content is created using a generative AI model, and the content is sent to the device to provide to the user. An example of a prompt for the generative AI model is, "Generate a personalized video based on the user's memories of traveling in Europe during their university days."

[0245] This processing step allows for the creation and delivery of unique content based on the user's memories and experiences, enabling a new form of emotional connection.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] The present invention combines an emotion engine with a system that digitizes memories and personalities based on voice data provided by a user and then allows the user to converse with the generated digital copy. Specific embodiments are described below.

[0248] Collection and transmission of voice data

[0249] The user starts a dedicated application and provides voice data about daily life and past events. This voice data is recorded by the device and sent to the server in real time or at regular intervals.

[0250] Speech-to-text transcription and analysis

[0251] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0252] Emotion recognition and storage in a database

[0253] The server passes the converted voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the emotion from the tone and context of the voice and extracts that emotion as data. For example, if the user is speaking happily, the emotion data is extracted as "joy."

[0254] The recognized emotion data is stored in a database on the server along with the analyzed keywords. The data is organized by date and category, providing a detailed record of the user's memories, experiences, and emotions.

[0255] Digital copy generation and learning

[0256] The server uses the data stored in the database to create a digital copy of the user. This digital copy is generated using machine learning techniques based on voice, text, and emotional data. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0257] Interactive functionality

[0258] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, extracts relevant information from the digital copy, and generates a specific response. The digital copy also takes emotional data into account when generating the response, so it might respond in a happy tone, for example, "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0259] Specific examples

[0260] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0261] 2. The device records this audio data and sends it to the server.

[0262] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0263] 4. The server uses an emotion engine to recognize the emotion the user is expressing as "joy" when speaking.

[0264] 5. The server stores these data in a database and updates the user's digital copy.

[0265] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[0266] In this way, by digitizing the user's memories and personality and taking emotional data into account, this system provides a new form of communication with family and friends that is more realistic and emotionally empathetic.

[0267] The processing flow will be explained below.

[0268] Step 1:

[0269] The user launches the dedicated application and begins to talk about anecdotes based on personal memories and emotions.

[0270] Step 2:

[0271] The device records the user's voice and stores the recording in a buffer.

[0272] Step 3:

[0273] The device sends the recorded audio data to the server's API endpoint using the HTTPS protocol.

[0274] Step 4:

[0275] The server passes the received voice data to a speech recognition module, which converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[0276] Step 5:

[0277] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the keywords and context, extracting keywords such as "childhood," "mother," and "ocean."

[0278] Step 6:

[0279] The server passes the analyzed text data and keywords to an emotion engine to recognize the user's emotion. For example, if the user speaks in a happy tone, the emotion "joy" is recognized.

[0280] Step 7:

[0281] The server combines the analyzed text data with emotional data and stores it in a database for each user, allowing memories, experiences, and emotions to be stored in an integrated manner.

[0282] Step 8:

[0283] The server uses the accumulated data to train machine learning models, which then generate a digital copy of the user that can reproduce the way they speak, think, and express emotions.

[0284] Step 9:

[0285] The device receives conversation requests from family and friends, such as "Tell me about your mother's summer vacation memories."

[0286] Step 10:

[0287] The server analyzes the received interaction request and queries the digital copy to extract relevant memory, experience, and emotion data.

[0288] Step 11:

[0289] The server generates an appropriate response from the digital copy, taking into account the emotional data and generating a response with an appropriate emotional expression. For example, if the emotional data indicates "joy," the server generates a response in a happy tone, such as "I enjoyed going to the beach with my mother during summer vacation and collecting seashells."

[0290] Step 12:

[0291] Your device will generate a response that will be displayed to family and friends, or played aloud.

[0292] Step 13:

[0293] The server updates the machine learning model again based on the new dialogue data and emotion data collected, improving the accuracy and emotional expression of the user's digital copy.

[0294] Through this series of processes, the user's memories and personality are digitized, and by taking emotional data into consideration, more realistic and emotional communication is achieved.

[0295] Example 2

[0296] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0297] Conventional dialogue systems using voice data often simply convert a user's speech into text and generate pre-defined, standard responses. However, these systems have difficulty responding to the user's emotions and individual contexts, making it difficult to improve empathy and intimacy in dialogues with the user. Furthermore, dialogue learning and continuous improvement are limited, making it insufficient to fully reproduce the user's memories and experiences.

[0298] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0299] In this invention, the server includes means for recording voice data provided by a user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data to extract keywords and context, means for recognizing emotions based on the analyzed text data and voice data, means for storing the recognized emotion data, keywords, and context in a database for each user, means for generating a digital copy of the user using the stored data, and means for responding to dialogue requests using the generated digital copy. This enables responses based on emotions and context, improves empathy and intimacy in dialogue with the user, and also enables improved dialogue accuracy through continuous learning.

[0300] "User" refers to a person who uses the system to provide voice data and engage in interaction.

[0301] "Audio Data" refers to information recorded as audio signals provided by a User using the System.

[0302] "Recording instrument" means any equipment or software used to collect and store a user's voice data in digital form.

[0303] "Means for transmitting" refers to the communication means for transferring the recorded audio data to the server.

[0304] "Server" refers to a central processing unit that processes received voice data and performs text conversion, data analysis, emotion recognition, database storage, digital copy generation, etc.

[0305] "Means for converting to text" refers to speech recognition technology for converting voice data into text data.

[0306] "Means of analyzing and extracting keywords and context" refers to natural language processing technology for extracting important words and linguistic background information from text data.

[0307] "Means for recognizing emotions" refers to technology for analyzing and identifying a user's emotions from voice and text data.

[0308] "Means for storing in a database" refers to a system for structuring and storing information such as analyzed text data, emotional data, keywords, and context.

[0309] "Means for generating a digital copy" refers to technology that generates a digital object that reproduces a user's memories and personality based on information stored in a database.

[0310] "Means for responding to dialogue requests" refers to a system that generates and provides appropriate responses based on dialogue requests from the user's family and friends.

[0311] "Means for continuous learning and developing a faithful personality model" refers to machine learning techniques that allow the generated digital copy to evolve based on new interaction data, creating a more realistic personality model.

[0312] "Means for generating appropriate responses based on emotional data and specific contexts and keywords" refers to the function of creating responses appropriate to the situation during a conversation, depending on the user's emotional state and indicative words.

[0313] The present invention combines an emotion engine with a system that digitizes memories and personality based on voice data provided by a user and then allows the user to interact with the generated digital copy. Specific embodiments are described below.

[0314] Collection and transmission of voice data

[0315] The user launches a dedicated application and provides voice data about daily life and past events. The device records this voice data and transmits it to the server in real time or at regular intervals.

[0316] Speech-to-text transcription and analysis

[0317] The server passes the received voice data to a voice recognition module (e.g., a cloud voice recognition API) that converts the voice into text data. This involves analyzing the voice waveform and mapping it to text. The converted text data is then passed to a natural language processing engine (e.g., a generative AI model) that extracts important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," the keywords "childhood," "mother," and "sea" are extracted.

[0318] Emotion recognition and storage in a database

[0319] The server passes the converted text data and voice data to an emotion engine (e.g., an emotion analysis system) to recognize the user's emotion. For example, if the user is speaking with a happy expression, the emotion data "joy" is extracted. The recognized emotion data, text data, keywords, and context are stored in the server's database. When the data is stored, it is organized by adding metadata such as date and category.

[0320] Digital copy generation and learning

[0321] The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user based on the voice, text, and emotion data stored in the database. The digital copy then continuously learns from new interaction data to form a more faithful personality model.

[0322] Interactive functionality

[0323] The server accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about my mother's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. The response takes into account emotional data and is generated in a happy tone, such as, "I remember going to the beach with my mother during summer vacation and collecting seashells." The device then relays the generated response to the family member.

[0324] Specific examples

[0325] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0326] 2. The device records this audio data and sends it to the server.

[0327] 3. The server converts the voice data into text data using a cloud speech recognition API and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0328] 4. The server uses an emotion analysis system to recognize the emotion the user is expressing as "joy" when speaking.

[0329] 5. The server stores these data in a database and updates the user's digital copy.

[0330] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[0331] (Example of a prompt to input to a generative AI model)

[0332] "This system digitizes memories and personality based on voice data provided by the user, and then combines it with an emotion engine to create a digital copy that can then converse with the created digital copy. Specifically, the user provides voice data about their daily life and past events, which is then converted into text data and analyzed. The emotion engine is used to recognize emotions and store them in a database. Finally, a digital copy is created based on this data, and in response to conversation requests from family and friends, it provides responses containing appropriate emotions."

[0333] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0334] Step 1:

[0335] The user starts the dedicated application and records audio data.

[0336] Input: The voice the user speaks to the application.

[0337] Processing: The device's recording function collects the audio as digital data and uses noise-canceling technology to improve the sound quality.

[0338] Output: High quality audio data file.

[0339] Step 2:

[0340] The device sends the recorded audio data to the server.

[0341] Input: High quality audio data files.

[0342] Processing: The device encrypts the voice data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0343] Output: Audio data uploaded to the server.

[0344] Step 3:

[0345] The server converts the received voice data into text.

[0346] Input: Audio data uploaded to the server.

[0347] Processing: The server uses a cloud speech recognition API (e.g., Google Cloud Speech-to-Text API) to analyze the audio data and map the audio waveform to text.

[0348] Output: Text data.

[0349] Step 4:

[0350] The server analyzes the converted text data and extracts keywords and context.

[0351] Input: Text data.

[0352] Processing: The server uses a natural language processing engine (e.g., a generative AI model) to analyze the text data and extract important keywords and context.

[0353] Output: Key words, phrases, and contextual information.

[0354] Step 5:

[0355] The server recognizes emotions based on text and voice data.

[0356] Input: Keywords, contextual information, and audio data.

[0357] Processing: The server uses an emotion analysis engine (e.g., an emotion analysis system) to analyze the user's emotion from the text and tone of voice and extract emotion data.

[0358] Output: Emotion tag (e.g., "joy", "sad").

[0359] Step 6:

[0360] The server stores the parsed information in a database.

[0361] Input: sentiment tags, key words and phrases, text data.

[0362] Processing: The server organizes and stores this data in a database along with metadata.

[0363] Output: Structured data in a database.

[0364] Step 7:

[0365] The server generates the digital copy.

[0366] Input: Structured data in a database.

[0367] Processing: The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user that mimics their memories and personality. This digital copy continuously learns based on the data it receives.

[0368] Output: A digital copy of the user.

[0369] Step 8:

[0370] The terminal receives the interaction request.

[0371] Input: Interaction requests from the user's family and friends.

[0372] Processing: The terminal receives the request and forwards it to the server.

[0373] Output: Interaction request data to the server.

[0374] Step 9:

[0375] The server generates a response based on the interaction request.

[0376] Input: Interaction request data, digital copy.

[0377] Processing: The server uses the digital copy to extract relevant information and generate a response that takes into account the emotional data, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0378] Output: A specific response containing emotion.

[0379] Step 10:

[0380] The device communicates the generated response to the family.

[0381] Input: A specific response containing emotion.

[0382] Processing: The device outputs the generated response in voice or text format and conveys it to the family.

[0383] Output: Response message to family members.

[0384] (Application example 2)

[0385] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] Existing security systems face the problem of being unable to grasp the behavior and emotions of suspicious individuals and intruders in real time and respond appropriately. Furthermore, typical security monitoring systems often lack the ability to learn individual behavioral patterns or the ability to analyze emotional data to respond flexibly. As a result, they are unable to respond quickly and appropriately, which can lead to a decline in the quality of security.

[0387] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0388] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for saving the extracted data in a database for each user, means for generating a digital copy of the user using the saved data, and means for responding to dialogue requests using the generated digital copy, as well as means for recognizing emotions from the voice data and text data and generating security information based thereon, and means for collecting and analyzing voice data of security targets in real time, detecting abnormal emotions or behavior, and notifying security guards. This enables real-time analysis and notification based on the voice data and emotion data, enabling rapid and appropriate responses.

[0389] A "user-provided means for recording voice data" is a device or function that records and stores user speech as digital voice data.

[0390] The "means for transmitting recorded voice data to a server" refers to a device or function for uploading recorded voice data to a server via a network.

[0391] The "means for converting voice data received by the server into text" refers to a device or function that converts voice data into text data using voice recognition technology.

[0392] The "means for analyzing the converted text data and extracting keywords and context" refers to a device or function that uses natural language processing technology to extract important words and context from the text data.

[0393] "Means for storing extracted data in a database for each user" refers to a device or function that organizes and stores analyzed data in a database based on individual user profiles.

[0394] A "means for generating a digital copy of a user using stored data" is a device or function that uses stored data to create a digital model that mimics the user's memories and personality.

[0395] A "means for responding to interaction requests using the generated digital copy" is a device or function that utilizes the generated digital copy to provide appropriate responses to questions from the user or other individual.

[0396] "Means for recognizing emotions from voice data and text data and generating security information based on that" refers to a device or function that determines emotions from analyzed voice and text and generates security-related information based on that.

[0397] "Means for collecting voice data from a security object, analyzing it in real time, detecting abnormal emotions or behavior, and notifying security guards" refers to a device or function that analyzes voice data captured at a security object in real time, detects abnormal emotions or behavior, and immediately reports the results to security guards.

[0398] A "means by which the digital copy of the user continues to learn and train based on new interactions" is a device or function that learns from new interactions with the user and continually improves itself.

[0399] "Means for learning the behavioral patterns of security targets and detecting unusual behavior" refers to a device or function that learns the normal behavioral patterns of security targets and detects abnormal behavior at an early stage.

[0400] This invention is a security system that digitizes memories and personalities based on user voice data and provides appropriate security information in real time. The system operates in the following steps.

[0401] Audio data collection

[0402] Users speak through security cameras and microphones installed in specific locations, which allows voice data to be collected in real time. Voice data may also be collected from smartphones and robots that can accept voice input.

[0403] Sending audio data

[0404] The recorded voice data is sent to a server via a network. The data arrives at the server in real time and processing begins immediately.

[0405] Converting audio data to text

[0406] The server converts the received voice data into text using a speech recognition module, which uses the Python programming language and the SpeechRecognition library, and also uses Google APIs for the text conversion.

[0407] Text data analysis and emotion recognition

[0408] The server analyzes the converted text data and extracts important keywords and context. This analysis uses natural language processing technologies such as TextBlob. At the same time, emotions are recognized from the speech and text. This emotion recognition is performed by an emotion engine, which can recognize positive, negative, and neutral emotions.

[0409] Preserving data and generating digital copies

[0410] The analyzed text and emotion data is stored in a database on a server based on date and category. This data is used to create a digital copy of the user, which is continually trained using new data.

[0411] Achieving security response

[0412] The collected voice data and analysis results are used to generate security information. For example, if abnormal emotions (e.g., anger or fear) are detected in a location under guard, a real-time notification is sent to security personnel. This notification allows for a rapid response. In addition, the system learns the behavioral patterns of the subject, and if any unusual behavior is detected, security personnel are similarly notified.

[0413] Specific examples

[0414] Specific examples are shown below.

[0415] 1. A user says, "I'm very tired today" while sitting in their living room.

[0416] 2. The voice data is collected by the smart home system and sent to the server.

[0417] 3. The server converts the voice data into text and extracts the keyword "tired."

[0418] 4. The emotion engine recognizes the emotion "tired" and this data is stored in the database.

[0419] 5. Update the user's digital copy based on the stored data.

[0420] 6. That night, a suspicious voice is detected in the living room and the emotion engine recognizes it as "anger."

[0421] 7. The server notifies the security guard in real time that "anger has been detected in the living room."

[0422] Prompt Sentence Examples

[0423] "Describe a security system that analyzes voice data to detect suspicious behavior and emotions. The system collects voice data, converts it into text, recognizes emotions, and notifies security personnel in real time. The system has a built-in emotion engine, allowing for rapid response."

[0424] This allows security systems to utilize advanced emotion recognition to take quick and accurate countermeasures.

[0425] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0426] Step 1:

[0427] The user provides the audio data

[0428] A user speaks using a device that can accept voice input (e.g., a smartphone or a robot). The speech is natural language such as "I'm very tired today." This voice data is collected by a microphone built into the device.

[0429] Input: User's voice

[0430] Output: Digital audio data

[0431] Step 2:

[0432] The device sends the recorded audio data to the server.

[0433] The device transmits the collected digital audio data to a server in real time or at regular intervals, using data communication over the Internet.

[0434] Input: Digital audio data

[0435] Output: Audio data sent to the server

[0436] Step 3:

[0437] The server converts the audio data into text

[0438] The server converts the received voice data into text data using a speech recognition module (using Google API). This process uses speech recognition technology.

[0439] Input: Audio data received by the server

[0440] Output: Text data

[0441] Step 4:

[0442] The server analyzes the converted text to extract keywords and context.

[0443] The server analyzes the text data using natural language processing technology (such as TextBlob) to extract important keywords and contexts. For example, it extracts the keyword "tired" from "I'm very tired today."

[0444] Input: Text data

[0445] Output: Keywords and context data

[0446] Step 5:

[0447] The server recognizes emotions from the voice and text data.

[0448] The server uses an emotion engine to recognize emotions from the tone of voice and text content. The emotion engine analyzes the data to determine "positive," "negative," or "neutral" emotions.

[0449] Input: Audio and text data

[0450] Output: Emotion data

[0451] Step 6:

[0452] The server stores the analyzed data and emotion data in a database.

[0453] The server stores the analyzed keywords and emotion data in a database for each user, organized by date and category.

[0454] Input: Keywords and sentiment data

[0455] Output: Records stored in the database

[0456] Step 7:

[0457] The server uses the stored data to create a digital copy of the user.

[0458] The server creates a digital copy of the user based on the data stored in the database, and the digital copy uses machine learning techniques to mimic the user's memories and personality.

[0459] Input: Saved data

[0460] Output: Digital copy of the user

[0461] Step 8:

[0462] The server responds to the interaction request with the generated digital copy.

[0463] Upon receiving an interaction request from a user or other individual, the server uses the generated digital copy to respond to the request, for example, "I'm very tired today," and generates a response such as "You need to take a break."

[0464] Input: Interaction request

[0465] Output: Dialogue response

[0466] Step 9:

[0467] The server collects and analyzes audio data from the security target in real time, detecting any abnormal emotions or behavior and notifying security personnel.

[0468] The server analyzes the voice data collected from the security location in real time and detects abnormal emotions (e.g., anger) or behavior. If an abnormality is detected, the server immediately notifies security personnel.

[0469] Input: Audio data of the security object

[0470] Output:Notify security guard

[0471] This enables the security system to respond quickly and accurately, improving the quality of security.

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0473] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0475] [Second embodiment]

[0476] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0477] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0478] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0480] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0482] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0483] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0484] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0487] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0488] The system of the present invention digitizes memories and experiences based on voice data provided by a user, and then conducts a dialogue using the generated digital copy. Specific embodiments are described below.

[0489] Collection and transmission of voice data

[0490] Users start a dedicated application and provide audio data about their daily lives and past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[0491] Speech-to-text transcription and analysis

[0492] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0493] Saving to a database

[0494] The analyzed text data and keywords are stored in a database on the server. This database is structured for each user and organizes data based on different categories and timelines, allowing for a detailed record of the user's memories and experiences.

[0495] Digital copy generation and learning

[0496] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0497] Interactive functionality

[0498] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, pulls out relevant information from the digital copy, and generates a specific response. This response is then provided to the family via the device. This allows for detailed conversations about the deceased's memories and experiences, maintaining an emotional connection in a new way.

[0499] Specific examples

[0500] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0501] 2. The device records this audio data and sends it to the server.

[0502] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0503] 4. The server stores these data in a database and updates the user's digital copy.

[0504] 5. When a family member makes a request such as "Tell me about your summer vacation memories," the server uses the digital copy to generate an appropriate response, which the device then relays to the family member.

[0505] In this way, this system digitizes the user's memories and personality, not only passing on their memories to future generations but also enabling a new form of communication with family and friends.

[0506] The processing flow will be explained below.

[0507] Step 1:

[0508] The user launches a dedicated application and begins speaking audio data about daily life and past events.

[0509] Step 2:

[0510] The device records the user's voice and stores the recording in a buffer.

[0511] Step 3:

[0512] The device sends the completed recording to the server's API endpoint using the HTTPS protocol.

[0513] Step 4:

[0514] The server passes the received voice data to a speech recognition engine and converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[0515] Step 5:

[0516] The server then passes the converted text data to a natural language processing (NLP) engine for keyword and context analysis. In this step, keywords such as "childhood," "mother," and "ocean" are extracted.

[0517] Step 6:

[0518] The server stores the analyzed keywords and text data in a database for each user, organizing the data based on date and category.

[0519] Step 7:

[0520] The server uses the stored data to train machine learning models, which then generate a digital copy of the user that reflects how they speak and think.

[0521] Step 8:

[0522] The device receives conversation requests from family and friends, such as "Tell me about your summer vacation memories."

[0523] Step 9:

[0524] The server analyzes the interaction request it receives and queries the digital copy of the user to extract relevant memory and experience data.

[0525] Step 10:

[0526] The server generates an appropriate response from the digital copy, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0527] Step 11:

[0528] Your device will generate a response that will be displayed to family and friends, or played aloud.

[0529] Step 12:

[0530] The server updates the machine learning model again based on the new interaction data collected, improving the accuracy of the user's digital copy.

[0531] This series of processes digitizes the user's memories and personality, enabling new forms of communication with family and friends.

[0532] Example 1

[0533] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0534] Current systems for digitizing users' memories and experiences and maintaining emotional connections with family and friends in new ways face several challenges. First, accuracy is required in the conversion of voice data to text and subsequent analysis, but there is a lack of technology to improve this accuracy. Second, if the machine learning technology used to generate the digital copy does not perform sufficiently, the reproduction of the user's memories and interactions will be insufficient. Finally, the digital copy's ability to continuously learn has not been established, which can lead to a decline in the quality of the interactions.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0536] In this invention, the server includes means for including a speech recognition module that converts speech data into text, means for including a natural language processing engine that analyzes the text data and extracts keywords and context, means for storing the extracted data in a database for each user, and means for using machine learning technology to generate a digital copy of the user using the stored data. This enables highly accurate text conversion and analysis of speech data, and by generating and continuously training the digital copy of the user, the quality of the dialogue can be improved and the user's memories and experiences can be more faithfully digitized.

[0537] "User" refers to a person who uses the system to provide voice data.

[0538] "Terminal" refers to a device that has the function of recording a user's voice data and transmitting it to a server.

[0539] A "server" refers to a device or system that receives voice data sent from a terminal, converts it into text data, analyzes it, and stores it in a database.

[0540] A "voice recognition module" refers to software or hardware for converting voice data into text data.

[0541] A "natural language processing engine" refers to technology for analyzing text data and extracting important keywords and context.

[0542] "Database" refers to a system that stores and manages analyzed text data and keywords for each user.

[0543] "Machine learning technology" refers to algorithms and models that use data to create a digital copy of a user and allow it to continuously learn.

[0544] A "digital copy" refers to a model created to mimic a user's speech and thought patterns.

[0545] "Interaction requests" refer to questions or requests for conversation that a user's family or friends make to a digital copy.

[0546] "Response" refers to a specific reply that a digital copy makes in response to an interaction request.

[0547] The system of the present invention aims to digitize memories and experiences based on voice data provided by users, and then engage in dialogue using the generated digital copies. The system of the present invention includes the following main components:

[0548] Collection and transmission of voice data

[0549] Users start a dedicated application and provide audio data about their daily lives or past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[0550] Speech-to-text transcription and analysis

[0551] The server converts the received voice data into text data using a speech recognition module (e.g., Google Speech-to-Text). The converted text data is then analyzed using a natural language processing (NLP) engine (e.g., OpenAI GPT-3) to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0552] Saving to a database

[0553] The server stores the analyzed text data and keywords in a database. This database is structured for each user and organizes data based on different categories and timelines. For example, there are categories such as "childhood" and "summer memories."

[0554] Digital copy generation and learning

[0555] The server creates a digital copy of the user based on the data stored in the database. This digital copy uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns. The digital copy continuously learns based on new interaction data to form a more faithful personality model. For example, it learns about the specific words and phrases the user speaks.

[0556] Interactive functionality

[0557] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about Mom's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. This response is then provided to the family member via the device.

[0558] Specific examples

[0559] For example, a user might say, "The flowers I saw in the park yesterday were beautiful," using a dedicated application. The device records this and sends it to the server. The server then converts the received voice data into text using Google Speech-to-Text, generating the text, "The flowers I saw in the park yesterday were beautiful." The server then analyzes the text data using OpenAI GPT-3, extracting keywords such as "yesterday," "park," "flowers," and "beautiful." The server then stores this data in a database and categorizes it into categories such as "yesterday's events" and "memories of the park."

[0560] The device accepts a request from the family such as "What was beautiful yesterday?" The server analyzes this and generates a response from past data, such as "The flowers I saw in the park yesterday were beautiful," and responds to the family via the device.

[0561] Prompt Sentence Examples

[0562] "Analyze the following audio data and extract key keywords and context to generate a digital copy of the user. Example: 'When I was a child, I used to spend summer vacations at my grandparents' house.'"

[0563] In this way, the system digitizes the user's memories and personality, passing on their memories to future generations and enabling new forms of communication with family and friends.

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1:

[0566] The user starts the dedicated application and inputs voice information. For example, the user might say, "The flowers I saw in the park yesterday were beautiful."

[0567] Input: User's voice data

[0568] Output: Audio data recorded on the device

[0569] Specific behavior: The user presses the record button on the application, and recording stops when the user finishes speaking.

[0570] Step 2:

[0571] The terminal records the user's voice data and transmits the voice data to the server in real time or at regular intervals.

[0572] Input: Recorded audio data

[0573] Output: Audio data sent to the server

[0574] Specific operation: The device uploads audio data to a server via the Internet.

[0575] Step 3:

[0576] The server converts the received voice data into text data using a voice recognition module (e.g., Google Speech-to-Text).

[0577] Input: Audio data sent to the server

[0578] Output: Text data

[0579] Specific operation: The speech recognition module is started on the server side, the audio file is analyzed, and text is generated.

[0580] Step 4:

[0581] The server passes the converted text data to a natural language processing (NLP) engine (e.g., OpenAI GPT-3) for analysis to extract keywords and context.

[0582] Input: Text data

[0583] Output: Extracted keywords and context information

[0584] How it works: The NLP engine analyzes the text data and extracts important keywords such as "yesterday," "park," "flowers," and "beautiful."

[0585] Step 5:

[0586] The server stores the analyzed text data and keywords in a database, which is structured for each user and organized according to different categories and timelines.

[0587] Input: Extracted keywords and context information

[0588] Output: Text data for each user stored in the database

[0589] Specific operation: Text data and keywords are stored by category using database software (e.g., MySQL).

[0590] Step 6:

[0591] The server creates a digital copy of the user based on the data stored in the database, which uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns.

[0592] Input: User data stored in the database

[0593] Output: Generated digital copy

[0594] What it does: A machine learning model uses your conversational data as training data to generate a digital copy.

[0595] Step 7:

[0596] The server continuously updates and trains the generated digital copy, allowing the model to continually learn based on new interaction data.

[0597] Input: New interaction data

[0598] Output: Updated digital copy

[0599] What happens: New interaction data is added to the database and the machine learning model is retrained.

[0600] Step 8:

[0601] The device accepts conversation requests from family and friends. For example, a family member might request, "I'd like to hear about my mother's summer vacation memories."

[0602] Input: Interaction request

[0603] Output: Request sent to server

[0604] Specific operation: The device receives a request in voice or text format and sends it to the server.

[0605] Step 9:

[0606] The server analyzes the interaction request and extracts relevant information from the digital copy to generate a specific response.

[0607] Input: Interaction request and digital copy

[0608] Output: The generated response

[0609] What it does: The server analyzes the request and uses the digital copy to generate an appropriate response.

[0610] Step 10:

[0611] The device then provides the generated response to family and friends, for example, "The flowers I saw in the park yesterday were beautiful."

[0612] Input: The generated response

[0613] Output: Responses to family and friends

[0614] What happens: The device plays or displays a response in audio or text format.

[0615] Through these steps, the system digitizes the user's memories and personality, enabling new forms of emotional connection with family and friends.

[0616] (Application example 1)

[0617] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] While there have been systems that convert users' voice data into text and generate digital copies, few systems are capable of generating and delivering personalized content based on the user's memories and experiences, making it difficult to provide an experience optimized for each individual user.In addition, there are limited means of passing on content based on the memories and experiences of the deceased to future generations, creating a need for a new form of emotional connection.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0620] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for storing the extracted data in a database for each user, means for generating a digital copy of the user using the stored data, means for responding to dialogue requests using the generated digital copy, and means for generating and delivering personalized content based on the user's memories and experiences. This makes it possible to generate and deliver unique content based on the user's individual memories and experiences, thereby realizing a new form of emotional connection.

[0621] 1. "User-Provided Voice Data"

[0622] "User-provided voice data" refers to voice information recorded by a user using a dedicated device or application.

[0623] 2. "Method of recording"

[0624] A "recording means" is a device or software that has the function of acquiring and digitizing a user's voice data.

[0625] 3. "Method of sending to the server"

[0626] "Means for transmitting to the server" refers to a device or software that has the function of transferring recorded audio data to a central server via a network.

[0627] 4. "Method of converting to text"

[0628] A "text conversion means" is a device or software that uses speech recognition technology to convert voice data into text information.

[0629] 5. "Methods for analyzing and extracting keywords and context"

[0630] "Means for analyzing and extracting keywords and context" refers to a device or software that has the function of analyzing text data using natural language processing technology and identifying important words and context.

[0631] 6. "Method of storing data in a per-user database"

[0632] The "means for storing in a database for each user" is a database system that manages and stores analyzed text data and keywords for each individual user.

[0633] 7. "Means for generating a digital copy of the user"

[0634] "Means for generating a digital copy of a user" refers to a device or software that has the ability to create a digital avatar that mimics the user's voice and thought patterns based on stored data.

[0635] 8. "Means of responding to interaction requests"

[0636] A "means for responding to dialogue requests" is a device or software that has the function of generating appropriate responses using a digital copy in response to inquiries from the user or their family and friends.

[0637] 9. "A means of generating and delivering personalized content"

[0638] A "means for generating and delivering personalized content" is a device or software that has the function of creating customized videos, articles, or other content based on a user's memories and experiences and delivering it to a user's terminal.

[0639] The present invention provides a system for generating and distributing personalized content based on user voice data. This system is configured by combining multiple pieces of hardware and software.

[0640] 1. Collection of audio data

[0641] Users start a dedicated application and use their smartphone to provide audio data about their daily lives and past events. This audio data is digitized using the smartphone's recording function and sent to a server in real time or at regular intervals.

[0642] 2. Speech-to-text conversion and analysis

[0643] The server first passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. Next, this text data is analyzed using a natural language processing engine (e.g., spaCy, NLTK) to extract important keywords and context. For example, if a user says, "I traveled to Europe during my university days. Paris was particularly memorable," keywords such as "university days," "Europe," "travel," and "Paris" are extracted from the text data.

[0644] 3. Saving to the database

[0645] The analyzed text data and keywords are stored in a database (e.g., MongoDB) on the server. This database is structured for each user and organizes data based on different categories and timelines, providing a detailed record of the user's memories and experiences.

[0646] 4. Creating and Learning from Digital Copies

[0647] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0648] 5. Personalized content generation and delivery

[0649] The server generates personalized content (e.g., videos or articles) based on the user's memory and experience. It uses a generative AI model to generate the desired content by inputting a prompt. For example, the following prompt can be used:

[0650] "Generate a personalized video based on the user's memories of a trip to Europe during their college days."

[0651] The generated content is delivered to the user's smartphone application, allowing the user to view it.

[0652] Specific examples

[0653] 1. A user uses the app and says, "I traveled to Europe during my college years. Paris was particularly memorable."

[0654] 2. The smartphone transmits this voice data to the server in real time.

[0655] 3. The server converts the voice data into text and extracts the keywords "college days," "Europe," "travel," and "Paris."

[0656] 4. The server stores these data in a database and updates the user's digital copy.

[0657] 5. The server generates a personalized video using the prompt text and delivers it to the smartphone application.

[0658] This system enables users to generate and distribute unique content based on their memories and experiences, enabling new forms of emotional connection.

[0659] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0660] Step 1:

[0661] The user launches a dedicated application and talks about their daily life and past events.

[0662] Input: User's voice

[0663] Output: Recorded audio data

[0664] Specific operation: Records audio using the smartphone's microphone and saves it as digital data.

[0665] Step 2:

[0666] The device sends the recorded audio data to the server.

[0667] Input: Recorded audio data

[0668] Output: Audio data sent to the server

[0669] Specific operation: Uploads audio data to the server using the smartphone's communication function (Wi-Fi or mobile data).

[0670] Step 3:

[0671] The server passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[0672] Input: Audio data

[0673] Output: Text data

[0674] Specific operation: The server calls the speech recognition API and transcribes the audio data.

[0675] Step 4:

[0676] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy, NLTK) to extract keywords and context.

[0677] Input: Text data

[0678] Output: Extracted keywords and context

[0679] What it does: A natural language processing engine analyzes text data to identify important words, phrases, and context.

[0680] Step 5:

[0681] The server stores the extracted data in a per-user database (e.g. MongoDB).

[0682] Input: Keywords and context

[0683] Output: Data stored in the database

[0684] Specific operation: The server saves the analysis results in a database managed for each user.

[0685] Step 6:

[0686] The server uses the stored data to create a digital copy of the user, continually training them as needed.

[0687] Input: Data stored in a database

[0688] Output: Updated digital copy

[0689] How it works: The server uses machine learning models to generate a digital copy of the user that mimics their language and thought patterns, periodically feeding them new data to train them.

[0690] Step 7:

[0691] The server generates personalized content (e.g., videos, articles) based on the user's memories and experiences and delivers it to the device.

[0692] Input: Content-generated prompt based on the digital copy

[0693] Output: Generated content

[0694] Specific operation: Personalized content is created using a generative AI model, and the content is sent to the device to provide to the user. An example of a prompt for the generative AI model is, "Generate a personalized video based on the user's memories of traveling in Europe during their university days."

[0695] This processing step allows for the creation and delivery of unique content based on the user's memories and experiences, enabling a new form of emotional connection.

[0696] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0697] The present invention combines an emotion engine with a system that digitizes memories and personalities based on voice data provided by a user and then allows the user to converse with the generated digital copy. Specific embodiments are described below.

[0698] Collection and transmission of voice data

[0699] The user starts a dedicated application and provides voice data about daily life and past events. This voice data is recorded by the device and sent to the server in real time or at regular intervals.

[0700] Speech-to-text transcription and analysis

[0701] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0702] Emotion recognition and storage in a database

[0703] The server passes the converted voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the emotion from the tone and context of the voice and extracts that emotion as data. For example, if the user is speaking happily, the emotion data is extracted as "joy."

[0704] The recognized emotion data is stored in a database on the server along with the analyzed keywords. The data is organized by date and category, providing a detailed record of the user's memories, experiences, and emotions.

[0705] Digital copy generation and learning

[0706] The server uses the data stored in the database to create a digital copy of the user. This digital copy is generated using machine learning techniques based on voice, text, and emotional data. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0707] Interactive functionality

[0708] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, extracts relevant information from the digital copy, and generates a specific response. The digital copy also takes emotional data into account when generating the response, so it might respond in a happy tone, for example, "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0709] Specific examples

[0710] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0711] 2. The device records this audio data and sends it to the server.

[0712] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0713] 4. The server uses an emotion engine to recognize the emotion the user is expressing as "joy" when speaking.

[0714] 5. The server stores these data in a database and updates the user's digital copy.

[0715] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[0716] In this way, by digitizing the user's memories and personality and taking emotional data into account, this system provides a new form of communication with family and friends that is more realistic and emotionally empathetic.

[0717] The processing flow will be explained below.

[0718] Step 1:

[0719] The user launches the dedicated application and begins to talk about anecdotes based on personal memories and emotions.

[0720] Step 2:

[0721] The device records the user's voice and stores the recording in a buffer.

[0722] Step 3:

[0723] The device sends the recorded audio data to the server's API endpoint using the HTTPS protocol.

[0724] Step 4:

[0725] The server passes the received voice data to a speech recognition module, which converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[0726] Step 5:

[0727] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the keywords and context, extracting keywords such as "childhood," "mother," and "ocean."

[0728] Step 6:

[0729] The server passes the analyzed text data and keywords to an emotion engine to recognize the user's emotion. For example, if the user speaks in a happy tone, the emotion "joy" is recognized.

[0730] Step 7:

[0731] The server combines the analyzed text data with emotional data and stores it in a database for each user, allowing memories, experiences, and emotions to be stored in an integrated manner.

[0732] Step 8:

[0733] The server uses the accumulated data to train machine learning models, which then generate a digital copy of the user that can reproduce the way they speak, think, and express emotions.

[0734] Step 9:

[0735] The device receives conversation requests from family and friends, such as "Tell me about your mother's summer vacation memories."

[0736] Step 10:

[0737] The server analyzes the received interaction request and queries the digital copy to extract relevant memory, experience, and emotion data.

[0738] Step 11:

[0739] The server generates an appropriate response from the digital copy, taking into account the emotional data and generating a response with an appropriate emotional expression. For example, if the emotional data indicates "joy," the server generates a response in a happy tone, such as "I enjoyed going to the beach with my mother during summer vacation and collecting seashells."

[0740] Step 12:

[0741] Your device will generate a response that will be displayed to family and friends, or played aloud.

[0742] Step 13:

[0743] The server updates the machine learning model again based on the new dialogue data and emotion data collected, improving the accuracy and emotional expression of the user's digital copy.

[0744] Through this series of processes, the user's memories and personality are digitized, and by taking emotional data into consideration, more realistic and emotional communication is achieved.

[0745] Example 2

[0746] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] Conventional dialogue systems using voice data often simply convert a user's speech into text and generate pre-defined, standard responses. However, these systems have difficulty responding to the user's emotions and individual contexts, making it difficult to improve empathy and intimacy in dialogues with the user. Furthermore, dialogue learning and continuous improvement are limited, making it insufficient to fully reproduce the user's memories and experiences.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0749] In this invention, the server includes means for recording voice data provided by a user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data to extract keywords and context, means for recognizing emotions based on the analyzed text data and voice data, means for storing the recognized emotion data, keywords, and context in a database for each user, means for generating a digital copy of the user using the stored data, and means for responding to dialogue requests using the generated digital copy. This enables responses based on emotions and context, improves empathy and intimacy in dialogue with the user, and also enables improved dialogue accuracy through continuous learning.

[0750] "User" refers to a person who uses the system to provide voice data and engage in interaction.

[0751] "Audio Data" refers to information recorded as audio signals provided by a User using the System.

[0752] "Recording instrument" means any equipment or software used to collect and store a user's voice data in digital form.

[0753] "Means for transmitting" refers to the communication means for transferring the recorded audio data to the server.

[0754] "Server" refers to a central processing unit that processes received voice data and performs text conversion, data analysis, emotion recognition, database storage, digital copy generation, etc.

[0755] "Means for converting to text" refers to speech recognition technology for converting voice data into text data.

[0756] "Means of analyzing and extracting keywords and context" refers to natural language processing technology for extracting important words and linguistic background information from text data.

[0757] "Means for recognizing emotions" refers to technology for analyzing and identifying a user's emotions from voice and text data.

[0758] "Means for storing in a database" refers to a system for structuring and storing information such as analyzed text data, emotional data, keywords, and context.

[0759] "Means for generating a digital copy" refers to technology that generates a digital object that reproduces a user's memories and personality based on information stored in a database.

[0760] "Means for responding to dialogue requests" refers to a system that generates and provides appropriate responses based on dialogue requests from the user's family and friends.

[0761] "Means for continuous learning and developing a faithful personality model" refers to machine learning techniques that allow the generated digital copy to evolve based on new interaction data, creating a more realistic personality model.

[0762] "Means for generating appropriate responses based on emotional data and specific contexts and keywords" refers to the function of creating responses appropriate to the situation during a conversation, depending on the user's emotional state and indicative words.

[0763] The present invention combines an emotion engine with a system that digitizes memories and personality based on voice data provided by a user and then allows the user to interact with the generated digital copy. Specific embodiments are described below.

[0764] Collection and transmission of voice data

[0765] The user launches a dedicated application and provides voice data about daily life and past events. The device records this voice data and transmits it to the server in real time or at regular intervals.

[0766] Speech-to-text transcription and analysis

[0767] The server passes the received voice data to a voice recognition module (e.g., a cloud voice recognition API) that converts the voice into text data. This involves analyzing the voice waveform and mapping it to text. The converted text data is then passed to a natural language processing engine (e.g., a generative AI model) that extracts important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," the keywords "childhood," "mother," and "sea" are extracted.

[0768] Emotion recognition and storage in a database

[0769] The server passes the converted text data and voice data to an emotion engine (e.g., an emotion analysis system) to recognize the user's emotion. For example, if the user is speaking with a happy expression, the emotion data "joy" is extracted. The recognized emotion data, text data, keywords, and context are stored in the server's database. When the data is stored, it is organized by adding metadata such as date and category.

[0770] Digital copy generation and learning

[0771] The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user based on the voice, text, and emotion data stored in the database. The digital copy then continuously learns from new interaction data to form a more faithful personality model.

[0772] Interactive functionality

[0773] The server accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about my mother's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. The response takes into account emotional data and is generated in a happy tone, such as, "I remember going to the beach with my mother during summer vacation and collecting seashells." The device then relays the generated response to the family member.

[0774] Specific examples

[0775] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0776] 2. The device records this audio data and sends it to the server.

[0777] 3. The server converts the voice data into text data using a cloud speech recognition API and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0778] 4. The server uses an emotion analysis system to recognize the emotion the user is expressing as "joy" when speaking.

[0779] 5. The server stores these data in a database and updates the user's digital copy.

[0780] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[0781] (Example of a prompt to input to a generative AI model)

[0782] "This system digitizes memories and personality based on voice data provided by the user, and then combines it with an emotion engine to create a digital copy that can then converse with the created digital copy. Specifically, the user provides voice data about their daily life and past events, which is then converted into text data and analyzed. The emotion engine is used to recognize emotions and store them in a database. Finally, a digital copy is created based on this data, and in response to conversation requests from family and friends, it provides responses containing appropriate emotions."

[0783] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0784] Step 1:

[0785] The user starts the dedicated application and records audio data.

[0786] Input: The voice the user speaks to the application.

[0787] Processing: The device's recording function collects the audio as digital data and uses noise-canceling technology to improve the sound quality.

[0788] Output: High quality audio data file.

[0789] Step 2:

[0790] The device sends the recorded audio data to the server.

[0791] Input: High quality audio data files.

[0792] Processing: The device encrypts the voice data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0793] Output: Audio data uploaded to the server.

[0794] Step 3:

[0795] The server converts the received voice data into text.

[0796] Input: Audio data uploaded to the server.

[0797] Processing: The server uses a cloud speech recognition API (e.g., Google Cloud Speech-to-Text API) to analyze the audio data and map the audio waveform to text.

[0798] Output: Text data.

[0799] Step 4:

[0800] The server analyzes the converted text data and extracts keywords and context.

[0801] Input: Text data.

[0802] Processing: The server uses a natural language processing engine (e.g., a generative AI model) to analyze the text data and extract important keywords and context.

[0803] Output: Key words, phrases, and contextual information.

[0804] Step 5:

[0805] The server recognizes emotions based on text and voice data.

[0806] Input: Keywords, contextual information, and audio data.

[0807] Processing: The server uses an emotion analysis engine (e.g., an emotion analysis system) to analyze the user's emotion from the text and tone of voice and extract emotion data.

[0808] Output: Emotion tag (e.g., "joy", "sad").

[0809] Step 6:

[0810] The server stores the parsed information in a database.

[0811] Input: sentiment tags, key words and phrases, text data.

[0812] Processing: The server organizes and stores this data in a database along with metadata.

[0813] Output: Structured data in a database.

[0814] Step 7:

[0815] The server generates the digital copy.

[0816] Input: Structured data in a database.

[0817] Processing: The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user that mimics their memories and personality. This digital copy continuously learns based on the data it receives.

[0818] Output: A digital copy of the user.

[0819] Step 8:

[0820] The terminal receives the interaction request.

[0821] Input: Interaction requests from the user's family and friends.

[0822] Processing: The terminal receives the request and forwards it to the server.

[0823] Output: Interaction request data to the server.

[0824] Step 9:

[0825] The server generates a response based on the interaction request.

[0826] Input: Interaction request data, digital copy.

[0827] Processing: The server uses the digital copy to extract relevant information and generate a response that takes into account the emotional data, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0828] Output: A specific response containing emotion.

[0829] Step 10:

[0830] The device communicates the generated response to the family.

[0831] Input: A specific response containing emotion.

[0832] Processing: The device outputs the generated response in voice or text format and conveys it to the family.

[0833] Output: Response message to family members.

[0834] (Application example 2)

[0835] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0836] Existing security systems face the problem of being unable to grasp the behavior and emotions of suspicious individuals and intruders in real time and respond appropriately. Furthermore, typical security monitoring systems often lack the ability to learn individual behavioral patterns or the ability to analyze emotional data to respond flexibly. As a result, they are unable to respond quickly and appropriately, which can lead to a decline in the quality of security.

[0837] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0838] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for saving the extracted data in a database for each user, means for generating a digital copy of the user using the saved data, and means for responding to dialogue requests using the generated digital copy, as well as means for recognizing emotions from the voice data and text data and generating security information based thereon, and means for collecting and analyzing voice data of security targets in real time, detecting abnormal emotions or behavior, and notifying security guards. This enables real-time analysis and notification based on the voice data and emotion data, enabling rapid and appropriate responses.

[0839] A "user-provided means for recording voice data" is a device or function that records and stores user speech as digital voice data.

[0840] The "means for transmitting recorded voice data to a server" refers to a device or function for uploading recorded voice data to a server via a network.

[0841] The "means for converting voice data received by the server into text" refers to a device or function that converts voice data into text data using voice recognition technology.

[0842] The "means for analyzing the converted text data and extracting keywords and context" refers to a device or function that uses natural language processing technology to extract important words and context from the text data.

[0843] "Means for storing extracted data in a database for each user" refers to a device or function that organizes and stores analyzed data in a database based on individual user profiles.

[0844] A "means for generating a digital copy of a user using stored data" is a device or function that uses stored data to create a digital model that mimics the user's memories and personality.

[0845] A "means for responding to interaction requests using the generated digital copy" is a device or function that utilizes the generated digital copy to provide appropriate responses to questions from the user or other individual.

[0846] "Means for recognizing emotions from voice data and text data and generating security information based on that" refers to a device or function that determines emotions from analyzed voice and text and generates security-related information based on that.

[0847] "Means for collecting voice data from a security object, analyzing it in real time, detecting abnormal emotions or behavior, and notifying security guards" refers to a device or function that analyzes voice data captured at a security object in real time, detects abnormal emotions or behavior, and immediately reports the results to security guards.

[0848] A "means by which the digital copy of the user continues to learn and train based on new interactions" is a device or function that learns from new interactions with the user and continually improves itself.

[0849] "Means for learning the behavioral patterns of security targets and detecting unusual behavior" refers to a device or function that learns the normal behavioral patterns of security targets and detects abnormal behavior at an early stage.

[0850] This invention is a security system that digitizes memories and personalities based on user voice data and provides appropriate security information in real time. The system operates in the following steps.

[0851] Audio data collection

[0852] Users speak through security cameras and microphones installed in specific locations, which allows voice data to be collected in real time. Voice data may also be collected from smartphones and robots that can accept voice input.

[0853] Sending audio data

[0854] The recorded voice data is sent to a server via a network. The data arrives at the server in real time and processing begins immediately.

[0855] Converting audio data to text

[0856] The server converts the received voice data into text using a speech recognition module, which uses the Python programming language and the SpeechRecognition library, and also uses Google APIs for the text conversion.

[0857] Text data analysis and emotion recognition

[0858] The server analyzes the converted text data and extracts important keywords and context. This analysis uses natural language processing technologies such as TextBlob. At the same time, emotions are recognized from the speech and text. This emotion recognition is performed by an emotion engine, which can recognize positive, negative, and neutral emotions.

[0859] Preserving data and generating digital copies

[0860] The analyzed text and emotion data is stored in a database on a server based on date and category. This data is used to create a digital copy of the user, which is continually trained using new data.

[0861] Achieving security response

[0862] The collected voice data and analysis results are used to generate security information. For example, if abnormal emotions (e.g., anger or fear) are detected in a location under guard, a real-time notification is sent to security personnel. This notification allows for a rapid response. In addition, the system learns the behavioral patterns of the subject, and if any unusual behavior is detected, security personnel are similarly notified.

[0863] Specific examples

[0864] Specific examples are shown below.

[0865] 1. A user says, "I'm very tired today" while sitting in their living room.

[0866] 2. The voice data is collected by the smart home system and sent to the server.

[0867] 3. The server converts the voice data into text and extracts the keyword "tired."

[0868] 4. The emotion engine recognizes the emotion "tired" and this data is stored in the database.

[0869] 5. Update the user's digital copy based on the stored data.

[0870] 6. That night, a suspicious voice is detected in the living room and the emotion engine recognizes it as "anger."

[0871] 7. The server notifies the security guard in real time that "anger has been detected in the living room."

[0872] Prompt Sentence Examples

[0873] "Describe a security system that analyzes voice data to detect suspicious behavior and emotions. The system collects voice data, converts it into text, recognizes emotions, and notifies security personnel in real time. The system has a built-in emotion engine, allowing for rapid response."

[0874] This allows security systems to utilize advanced emotion recognition to take quick and accurate countermeasures.

[0875] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0876] Step 1:

[0877] The user provides the audio data

[0878] A user speaks using a device that can accept voice input (e.g., a smartphone or a robot). The speech is natural language such as "I'm very tired today." This voice data is collected by a microphone built into the device.

[0879] Input: User's voice

[0880] Output: Digital audio data

[0881] Step 2:

[0882] The device sends the recorded audio data to the server.

[0883] The device transmits the collected digital audio data to a server in real time or at regular intervals, using data communication over the Internet.

[0884] Input: Digital audio data

[0885] Output: Audio data sent to the server

[0886] Step 3:

[0887] The server converts the audio data into text

[0888] The server converts the received voice data into text data using a speech recognition module (using Google API). This process uses speech recognition technology.

[0889] Input: Audio data received by the server

[0890] Output: Text data

[0891] Step 4:

[0892] The server analyzes the converted text to extract keywords and context.

[0893] The server analyzes the text data using natural language processing technology (such as TextBlob) to extract important keywords and contexts. For example, it extracts the keyword "tired" from "I'm very tired today."

[0894] Input: Text data

[0895] Output: Keywords and context data

[0896] Step 5:

[0897] The server recognizes emotions from the voice and text data.

[0898] The server uses an emotion engine to recognize emotions from the tone of voice and text content. The emotion engine analyzes the data to determine "positive," "negative," or "neutral" emotions.

[0899] Input: Audio and text data

[0900] Output: Emotion data

[0901] Step 6:

[0902] The server stores the analyzed data and emotion data in a database.

[0903] The server stores the analyzed keywords and emotion data in a database for each user, organized by date and category.

[0904] Input: Keywords and sentiment data

[0905] Output: Records stored in the database

[0906] Step 7:

[0907] The server uses the stored data to create a digital copy of the user.

[0908] The server creates a digital copy of the user based on the data stored in the database, and the digital copy uses machine learning techniques to mimic the user's memories and personality.

[0909] Input: Saved data

[0910] Output: Digital copy of the user

[0911] Step 8:

[0912] The server responds to the interaction request with the generated digital copy.

[0913] Upon receiving an interaction request from a user or other individual, the server uses the generated digital copy to respond to the request, for example, "I'm very tired today," and generates a response such as "You need to take a break."

[0914] Input: Interaction request

[0915] Output: Dialogue response

[0916] Step 9:

[0917] The server collects and analyzes audio data from the security target in real time, detecting any abnormal emotions or behavior and notifying security personnel.

[0918] The server analyzes the voice data collected from the security location in real time and detects abnormal emotions (e.g., anger) or behavior. If an abnormality is detected, the server immediately notifies security personnel.

[0919] Input: Audio data of the security object

[0920] Output:Notify security guard

[0921] This enables the security system to respond quickly and accurately, improving the quality of security.

[0922] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0923] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0924] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0925] [Third embodiment]

[0926] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0927] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0928] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0929] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0930] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0931] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0932] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0933] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0934] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0935] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0936] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0937] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0938] The system of the present invention digitizes memories and experiences based on voice data provided by a user, and then conducts a dialogue using the generated digital copy. Specific embodiments are described below.

[0939] Collection and transmission of voice data

[0940] Users start a dedicated application and provide audio data about their daily lives and past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[0941] Speech-to-text transcription and analysis

[0942] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[0943] Saving to a database

[0944] The analyzed text data and keywords are stored in a database on the server. This database is structured for each user and organizes data based on different categories and timelines, allowing for a detailed record of the user's memories and experiences.

[0945] Digital copy generation and learning

[0946] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[0947] Interactive functionality

[0948] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, pulls out relevant information from the digital copy, and generates a specific response. This response is then provided to the family via the device. This allows for detailed conversations about the deceased's memories and experiences, maintaining an emotional connection in a new way.

[0949] Specific examples

[0950] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[0951] 2. The device records this audio data and sends it to the server.

[0952] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[0953] 4. The server stores these data in a database and updates the user's digital copy.

[0954] 5. When a family member makes a request such as "Tell me about your summer vacation memories," the server uses the digital copy to generate an appropriate response, which the device then relays to the family member.

[0955] In this way, this system digitizes the user's memories and personality, not only passing on their memories to future generations but also enabling a new form of communication with family and friends.

[0956] The processing flow will be explained below.

[0957] Step 1:

[0958] The user launches a dedicated application and begins speaking audio data about daily life and past events.

[0959] Step 2:

[0960] The device records the user's voice and stores the recording in a buffer.

[0961] Step 3:

[0962] The device sends the completed recording to the server's API endpoint using the HTTPS protocol.

[0963] Step 4:

[0964] The server passes the received voice data to a speech recognition engine and converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[0965] Step 5:

[0966] The server then passes the converted text data to a natural language processing (NLP) engine for keyword and context analysis. In this step, keywords such as "childhood," "mother," and "ocean" are extracted.

[0967] Step 6:

[0968] The server stores the analyzed keywords and text data in a database for each user, organizing the data based on date and category.

[0969] Step 7:

[0970] The server uses the stored data to train machine learning models, which then generate a digital copy of the user that reflects how they speak and think.

[0971] Step 8:

[0972] The device receives conversation requests from family and friends, such as "Tell me about your summer vacation memories."

[0973] Step 9:

[0974] The server analyzes the interaction request it receives and queries the digital copy of the user to extract relevant memory and experience data.

[0975] Step 10:

[0976] The server generates an appropriate response from the digital copy, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[0977] Step 11:

[0978] Your device will generate a response that will be displayed to family and friends, or played aloud.

[0979] Step 12:

[0980] The server updates the machine learning model again based on the new interaction data collected, improving the accuracy of the user's digital copy.

[0981] This series of processes digitizes the user's memories and personality, enabling new forms of communication with family and friends.

[0982] Example 1

[0983] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0984] Current systems for digitizing users' memories and experiences and maintaining emotional connections with family and friends in new ways face several challenges. First, accuracy is required in the conversion of voice data to text and subsequent analysis, but there is a lack of technology to improve this accuracy. Second, if the machine learning technology used to generate the digital copy does not perform sufficiently, the reproduction of the user's memories and interactions will be insufficient. Finally, the digital copy's ability to continuously learn has not been established, which can lead to a decline in the quality of the interactions.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0986] In this invention, the server includes means for including a speech recognition module that converts speech data into text, means for including a natural language processing engine that analyzes the text data and extracts keywords and context, means for storing the extracted data in a database for each user, and means for using machine learning technology to generate a digital copy of the user using the stored data. This enables highly accurate text conversion and analysis of speech data, and by generating and continuously training the digital copy of the user, the quality of the dialogue can be improved and the user's memories and experiences can be more faithfully digitized.

[0987] "User" refers to a person who uses the system to provide voice data.

[0988] "Terminal" refers to a device that has the function of recording a user's voice data and transmitting it to a server.

[0989] A "server" refers to a device or system that receives voice data sent from a terminal, converts it into text data, analyzes it, and stores it in a database.

[0990] A "voice recognition module" refers to software or hardware for converting voice data into text data.

[0991] A "natural language processing engine" refers to technology for analyzing text data and extracting important keywords and context.

[0992] "Database" refers to a system that stores and manages analyzed text data and keywords for each user.

[0993] "Machine learning technology" refers to algorithms and models that use data to create a digital copy of a user and allow it to continuously learn.

[0994] A "digital copy" refers to a model created to mimic a user's speech and thought patterns.

[0995] "Interaction requests" refer to questions or requests for conversation that a user's family or friends make to a digital copy.

[0996] "Response" refers to a specific reply that a digital copy makes in response to an interaction request.

[0997] The system of the present invention aims to digitize memories and experiences based on voice data provided by users, and then engage in dialogue using the generated digital copies. The system of the present invention includes the following main components:

[0998] Collection and transmission of voice data

[0999] Users start a dedicated application and provide audio data about their daily lives or past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[1000] Speech-to-text transcription and analysis

[1001] The server converts the received voice data into text data using a speech recognition module (e.g., Google Speech-to-Text). The converted text data is then analyzed using a natural language processing (NLP) engine (e.g., OpenAI GPT-3) to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[1002] Saving to a database

[1003] The server stores the analyzed text data and keywords in a database. This database is structured for each user and organizes data based on different categories and timelines. For example, there are categories such as "childhood" and "summer memories."

[1004] Digital copy generation and learning

[1005] The server creates a digital copy of the user based on the data stored in the database. This digital copy uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns. The digital copy continuously learns based on new interaction data to form a more faithful personality model. For example, it learns about the specific words and phrases the user speaks.

[1006] Interactive functionality

[1007] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about Mom's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. This response is then provided to the family member via the device.

[1008] Specific examples

[1009] For example, a user might say, "The flowers I saw in the park yesterday were beautiful," using a dedicated application. The device records this and sends it to the server. The server then converts the received voice data into text using Google Speech-to-Text, generating the text, "The flowers I saw in the park yesterday were beautiful." The server then analyzes the text data using OpenAI GPT-3, extracting keywords such as "yesterday," "park," "flowers," and "beautiful." The server then stores this data in a database and categorizes it into categories such as "yesterday's events" and "memories of the park."

[1010] The device accepts a request from the family such as "What was beautiful yesterday?" The server analyzes this and generates a response from past data, such as "The flowers I saw in the park yesterday were beautiful," and responds to the family via the device.

[1011] Prompt Sentence Examples

[1012] "Analyze the following audio data and extract key keywords and context to generate a digital copy of the user. Example: 'When I was a child, I used to spend summer vacations at my grandparents' house.'"

[1013] In this way, the system digitizes the user's memories and personality, passing on their memories to future generations and enabling new forms of communication with family and friends.

[1014] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1015] Step 1:

[1016] The user starts the dedicated application and inputs voice information. For example, the user might say, "The flowers I saw in the park yesterday were beautiful."

[1017] Input: User's voice data

[1018] Output: Audio data recorded on the device

[1019] Specific behavior: The user presses the record button on the application, and recording stops when the user finishes speaking.

[1020] Step 2:

[1021] The terminal records the user's voice data and transmits the voice data to the server in real time or at regular intervals.

[1022] Input: Recorded audio data

[1023] Output: Audio data sent to the server

[1024] Specific operation: The device uploads audio data to a server via the Internet.

[1025] Step 3:

[1026] The server converts the received voice data into text data using a voice recognition module (e.g., Google Speech-to-Text).

[1027] Input: Audio data sent to the server

[1028] Output: Text data

[1029] Specific operation: The speech recognition module is started on the server side, the audio file is analyzed, and text is generated.

[1030] Step 4:

[1031] The server passes the converted text data to a natural language processing (NLP) engine (e.g., OpenAI GPT-3) for analysis to extract keywords and context.

[1032] Input: Text data

[1033] Output: Extracted keywords and context information

[1034] How it works: The NLP engine analyzes the text data and extracts important keywords such as "yesterday," "park," "flowers," and "beautiful."

[1035] Step 5:

[1036] The server stores the analyzed text data and keywords in a database, which is structured for each user and organized according to different categories and timelines.

[1037] Input: Extracted keywords and context information

[1038] Output: Text data for each user stored in the database

[1039] Specific operation: Text data and keywords are stored by category using database software (e.g., MySQL).

[1040] Step 6:

[1041] The server creates a digital copy of the user based on the data stored in the database, which uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns.

[1042] Input: User data stored in the database

[1043] Output: Generated digital copy

[1044] What it does: A machine learning model uses your conversational data as training data to generate a digital copy.

[1045] Step 7:

[1046] The server continuously updates and trains the generated digital copy, allowing the model to continually learn based on new interaction data.

[1047] Input: New interaction data

[1048] Output: Updated digital copy

[1049] What happens: New interaction data is added to the database and the machine learning model is retrained.

[1050] Step 8:

[1051] The device accepts conversation requests from family and friends. For example, a family member might request, "I'd like to hear about my mother's summer vacation memories."

[1052] Input: Interaction request

[1053] Output: Request sent to server

[1054] Specific operation: The device receives a request in voice or text format and sends it to the server.

[1055] Step 9:

[1056] The server analyzes the interaction request and extracts relevant information from the digital copy to generate a specific response.

[1057] Input: Interaction request and digital copy

[1058] Output: The generated response

[1059] What it does: The server analyzes the request and uses the digital copy to generate an appropriate response.

[1060] Step 10:

[1061] The device then provides the generated response to family and friends, for example, "The flowers I saw in the park yesterday were beautiful."

[1062] Input: The generated response

[1063] Output: Responses to family and friends

[1064] What happens: The device plays or displays a response in audio or text format.

[1065] Through these steps, the system digitizes the user's memories and personality, enabling new forms of emotional connection with family and friends.

[1066] (Application example 1)

[1067] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1068] While there have been systems that convert users' voice data into text and generate digital copies, few systems are capable of generating and delivering personalized content based on the user's memories and experiences, making it difficult to provide an experience optimized for each individual user.In addition, there are limited means of passing on content based on the memories and experiences of the deceased to future generations, creating a need for a new form of emotional connection.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1070] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for storing the extracted data in a database for each user, means for generating a digital copy of the user using the stored data, means for responding to dialogue requests using the generated digital copy, and means for generating and delivering personalized content based on the user's memories and experiences. This makes it possible to generate and deliver unique content based on the user's individual memories and experiences, thereby realizing a new form of emotional connection.

[1071] 1. "User-Provided Voice Data"

[1072] "User-provided voice data" refers to voice information recorded by a user using a dedicated device or application.

[1073] 2. "Method of recording"

[1074] A "recording means" is a device or software that has the function of acquiring and digitizing a user's voice data.

[1075] 3. "Method of sending to the server"

[1076] "Means for transmitting to the server" refers to a device or software that has the function of transferring recorded audio data to a central server via a network.

[1077] 4. "Method of converting to text"

[1078] A "text conversion means" is a device or software that uses speech recognition technology to convert voice data into text information.

[1079] 5. "Methods for analyzing and extracting keywords and context"

[1080] "Means for analyzing and extracting keywords and context" refers to a device or software that has the function of analyzing text data using natural language processing technology and identifying important words and context.

[1081] 6. "Method of storing data in a per-user database"

[1082] The "means for storing in a database for each user" is a database system that manages and stores analyzed text data and keywords for each individual user.

[1083] 7. "Means for generating a digital copy of the user"

[1084] "Means for generating a digital copy of a user" refers to a device or software that has the ability to create a digital avatar that mimics the user's voice and thought patterns based on stored data.

[1085] 8. "Means of responding to interaction requests"

[1086] A "means for responding to dialogue requests" is a device or software that has the function of generating appropriate responses using a digital copy in response to inquiries from the user or their family and friends.

[1087] 9. "A means of generating and delivering personalized content"

[1088] A "means for generating and delivering personalized content" is a device or software that has the function of creating customized videos, articles, or other content based on a user's memories and experiences and delivering it to a user's terminal.

[1089] The present invention provides a system for generating and distributing personalized content based on user voice data. This system is configured by combining multiple pieces of hardware and software.

[1090] 1. Collection of audio data

[1091] Users start a dedicated application and use their smartphone to provide audio data about their daily lives and past events. This audio data is digitized using the smartphone's recording function and sent to a server in real time or at regular intervals.

[1092] 2. Speech-to-text conversion and analysis

[1093] The server first passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. Next, this text data is analyzed using a natural language processing engine (e.g., spaCy, NLTK) to extract important keywords and context. For example, if a user says, "I traveled to Europe during my university days. Paris was particularly memorable," keywords such as "university days," "Europe," "travel," and "Paris" are extracted from the text data.

[1094] 3. Saving to the database

[1095] The analyzed text data and keywords are stored in a database (e.g., MongoDB) on the server. This database is structured for each user and organizes data based on different categories and timelines, providing a detailed record of the user's memories and experiences.

[1096] 4. Creating and Learning from Digital Copies

[1097] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[1098] 5. Personalized content generation and delivery

[1099] The server generates personalized content (e.g., videos or articles) based on the user's memory and experience. It uses a generative AI model to generate the desired content by inputting a prompt. For example, the following prompt can be used:

[1100] "Generate a personalized video based on the user's memories of a trip to Europe during their college days."

[1101] The generated content is delivered to the user's smartphone application, allowing the user to view it.

[1102] Specific examples

[1103] 1. A user uses the app and says, "I traveled to Europe during my college years. Paris was particularly memorable."

[1104] 2. The smartphone transmits this voice data to the server in real time.

[1105] 3. The server converts the voice data into text and extracts the keywords "college days," "Europe," "travel," and "Paris."

[1106] 4. The server stores these data in a database and updates the user's digital copy.

[1107] 5. The server generates a personalized video using the prompt text and delivers it to the smartphone application.

[1108] This system enables users to generate and distribute unique content based on their memories and experiences, enabling new forms of emotional connection.

[1109] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1110] Step 1:

[1111] The user launches a dedicated application and talks about their daily life and past events.

[1112] Input: User's voice

[1113] Output: Recorded audio data

[1114] Specific operation: Records audio using the smartphone's microphone and saves it as digital data.

[1115] Step 2:

[1116] The device sends the recorded audio data to the server.

[1117] Input: Recorded audio data

[1118] Output: Audio data sent to the server

[1119] Specific operation: Uploads audio data to the server using the smartphone's communication function (Wi-Fi or mobile data).

[1120] Step 3:

[1121] The server passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[1122] Input: Audio data

[1123] Output: Text data

[1124] Specific operation: The server calls the speech recognition API and transcribes the audio data.

[1125] Step 4:

[1126] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy, NLTK) to extract keywords and context.

[1127] Input: Text data

[1128] Output: Extracted keywords and context

[1129] What it does: A natural language processing engine analyzes text data to identify important words, phrases, and context.

[1130] Step 5:

[1131] The server stores the extracted data in a per-user database (e.g. MongoDB).

[1132] Input: Keywords and context

[1133] Output: Data stored in the database

[1134] Specific operation: The server saves the analysis results in a database managed for each user.

[1135] Step 6:

[1136] The server uses the stored data to create a digital copy of the user, continually training them as needed.

[1137] Input: Data stored in a database

[1138] Output: Updated digital copy

[1139] How it works: The server uses machine learning models to generate a digital copy of the user that mimics their language and thought patterns, periodically feeding them new data to train them.

[1140] Step 7:

[1141] The server generates personalized content (e.g., videos, articles) based on the user's memories and experiences and delivers it to the device.

[1142] Input: Content-generated prompt based on the digital copy

[1143] Output: Generated content

[1144] Specific operation: Personalized content is created using a generative AI model, and the content is sent to the device to provide to the user. An example of a prompt for the generative AI model is, "Generate a personalized video based on the user's memories of traveling in Europe during their university days."

[1145] This processing step allows for the creation and delivery of unique content based on the user's memories and experiences, enabling a new form of emotional connection.

[1146] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1147] The present invention combines an emotion engine with a system that digitizes memories and personalities based on voice data provided by a user and then allows the user to converse with the generated digital copy. Specific embodiments are described below.

[1148] Collection and transmission of voice data

[1149] The user starts a dedicated application and provides voice data about daily life and past events. This voice data is recorded by the device and sent to the server in real time or at regular intervals.

[1150] Speech-to-text transcription and analysis

[1151] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[1152] Emotion recognition and storage in a database

[1153] The server passes the converted voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the emotion from the tone and context of the voice and extracts that emotion as data. For example, if the user is speaking happily, the emotion data is extracted as "joy."

[1154] The recognized emotion data is stored in a database on the server along with the analyzed keywords. The data is organized by date and category, providing a detailed record of the user's memories, experiences, and emotions.

[1155] Digital copy generation and learning

[1156] The server uses the data stored in the database to create a digital copy of the user. This digital copy is generated using machine learning techniques based on voice, text, and emotional data. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[1157] Interactive functionality

[1158] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, extracts relevant information from the digital copy, and generates a specific response. The digital copy also takes emotional data into account when generating the response, so it might respond in a happy tone, for example, "I remember going to the beach with my mother during summer vacation and collecting seashells."

[1159] Specific examples

[1160] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[1161] 2. The device records this audio data and sends it to the server.

[1162] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[1163] 4. The server uses an emotion engine to recognize the emotion the user is expressing as "joy" when speaking.

[1164] 5. The server stores these data in a database and updates the user's digital copy.

[1165] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[1166] In this way, by digitizing the user's memories and personality and taking emotional data into account, this system provides a new form of communication with family and friends that is more realistic and emotionally empathetic.

[1167] The processing flow will be explained below.

[1168] Step 1:

[1169] The user launches the dedicated application and begins to talk about anecdotes based on personal memories and emotions.

[1170] Step 2:

[1171] The device records the user's voice and stores the recording in a buffer.

[1172] Step 3:

[1173] The device sends the recorded audio data to the server's API endpoint using the HTTPS protocol.

[1174] Step 4:

[1175] The server passes the received voice data to a speech recognition module, which converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[1176] Step 5:

[1177] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the keywords and context, extracting keywords such as "childhood," "mother," and "ocean."

[1178] Step 6:

[1179] The server passes the analyzed text data and keywords to an emotion engine to recognize the user's emotion. For example, if the user speaks in a happy tone, the emotion "joy" is recognized.

[1180] Step 7:

[1181] The server combines the analyzed text data with emotional data and stores it in a database for each user, allowing memories, experiences, and emotions to be stored in an integrated manner.

[1182] Step 8:

[1183] The server uses the accumulated data to train machine learning models, which then generate a digital copy of the user that can reproduce the way they speak, think, and express emotions.

[1184] Step 9:

[1185] The device receives conversation requests from family and friends, such as "Tell me about your mother's summer vacation memories."

[1186] Step 10:

[1187] The server analyzes the received interaction request and queries the digital copy to extract relevant memory, experience, and emotion data.

[1188] Step 11:

[1189] The server generates an appropriate response from the digital copy, taking into account the emotional data and generating a response with an appropriate emotional expression. For example, if the emotional data indicates "joy," the server generates a response in a happy tone, such as "I enjoyed going to the beach with my mother during summer vacation and collecting seashells."

[1190] Step 12:

[1191] Your device will generate a response that will be displayed to family and friends, or played aloud.

[1192] Step 13:

[1193] The server updates the machine learning model again based on the new dialogue data and emotion data collected, improving the accuracy and emotional expression of the user's digital copy.

[1194] Through this series of processes, the user's memories and personality are digitized, and by taking emotional data into consideration, more realistic and emotional communication is achieved.

[1195] Example 2

[1196] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1197] Conventional dialogue systems using voice data often simply convert a user's speech into text and generate pre-defined, standard responses. However, these systems have difficulty responding to the user's emotions and individual contexts, making it difficult to improve empathy and intimacy in dialogues with the user. Furthermore, dialogue learning and continuous improvement are limited, making it insufficient to fully reproduce the user's memories and experiences.

[1198] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1199] In this invention, the server includes means for recording voice data provided by a user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data to extract keywords and context, means for recognizing emotions based on the analyzed text data and voice data, means for storing the recognized emotion data, keywords, and context in a database for each user, means for generating a digital copy of the user using the stored data, and means for responding to dialogue requests using the generated digital copy. This enables responses based on emotions and context, improves empathy and intimacy in dialogue with the user, and also enables improved dialogue accuracy through continuous learning.

[1200] "User" refers to a person who uses the system to provide voice data and engage in interaction.

[1201] "Audio Data" refers to information recorded as audio signals provided by a User using the System.

[1202] "Recording instrument" means any equipment or software used to collect and store a user's voice data in digital form.

[1203] "Means for transmitting" refers to the communication means for transferring the recorded audio data to the server.

[1204] "Server" refers to a central processing unit that processes received voice data and performs text conversion, data analysis, emotion recognition, database storage, digital copy generation, etc.

[1205] "Means for converting to text" refers to speech recognition technology for converting voice data into text data.

[1206] "Means of analyzing and extracting keywords and context" refers to natural language processing technology for extracting important words and linguistic background information from text data.

[1207] "Means for recognizing emotions" refers to technology for analyzing and identifying a user's emotions from voice and text data.

[1208] "Means for storing in a database" refers to a system for structuring and storing information such as analyzed text data, emotional data, keywords, and context.

[1209] "Means for generating a digital copy" refers to technology that generates a digital object that reproduces a user's memories and personality based on information stored in a database.

[1210] "Means for responding to dialogue requests" refers to a system that generates and provides appropriate responses based on dialogue requests from the user's family and friends.

[1211] "Means for continuous learning and developing a faithful personality model" refers to machine learning techniques that allow the generated digital copy to evolve based on new interaction data, creating a more realistic personality model.

[1212] "Means for generating appropriate responses based on emotional data and specific contexts and keywords" refers to the function of creating responses appropriate to the situation during a conversation, depending on the user's emotional state and indicative words.

[1213] The present invention combines an emotion engine with a system that digitizes memories and personality based on voice data provided by a user and then allows the user to interact with the generated digital copy. Specific embodiments are described below.

[1214] Collection and transmission of voice data

[1215] The user launches a dedicated application and provides voice data about daily life and past events. The device records this voice data and transmits it to the server in real time or at regular intervals.

[1216] Speech-to-text transcription and analysis

[1217] The server passes the received voice data to a voice recognition module (e.g., a cloud voice recognition API) that converts the voice into text data. This involves analyzing the voice waveform and mapping it to text. The converted text data is then passed to a natural language processing engine (e.g., a generative AI model) that extracts important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," the keywords "childhood," "mother," and "sea" are extracted.

[1218] Emotion recognition and storage in a database

[1219] The server passes the converted text data and voice data to an emotion engine (e.g., an emotion analysis system) to recognize the user's emotion. For example, if the user is speaking with a happy expression, the emotion data "joy" is extracted. The recognized emotion data, text data, keywords, and context are stored in the server's database. When the data is stored, it is organized by adding metadata such as date and category.

[1220] Digital copy generation and learning

[1221] The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user based on the voice, text, and emotion data stored in the database. The digital copy then continuously learns from new interaction data to form a more faithful personality model.

[1222] Interactive functionality

[1223] The server accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about my mother's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. The response takes into account emotional data and is generated in a happy tone, such as, "I remember going to the beach with my mother during summer vacation and collecting seashells." The device then relays the generated response to the family member.

[1224] Specific examples

[1225] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[1226] 2. The device records this audio data and sends it to the server.

[1227] 3. The server converts the voice data into text data using a cloud speech recognition API and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[1228] 4. The server uses an emotion analysis system to recognize the emotion the user is expressing as "joy" when speaking.

[1229] 5. The server stores these data in a database and updates the user's digital copy.

[1230] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[1231] (Example of a prompt to input to a generative AI model)

[1232] "This system digitizes memories and personality based on voice data provided by the user, and then combines it with an emotion engine to create a digital copy that can then converse with the created digital copy. Specifically, the user provides voice data about their daily life and past events, which is then converted into text data and analyzed. The emotion engine is used to recognize emotions and store them in a database. Finally, a digital copy is created based on this data, and in response to conversation requests from family and friends, it provides responses containing appropriate emotions."

[1233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1234] Step 1:

[1235] The user starts the dedicated application and records audio data.

[1236] Input: The voice the user speaks to the application.

[1237] Processing: The device's recording function collects the audio as digital data and uses noise-canceling technology to improve the sound quality.

[1238] Output: High quality audio data file.

[1239] Step 2:

[1240] The device sends the recorded audio data to the server.

[1241] Input: High quality audio data files.

[1242] Processing: The device encrypts the voice data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1243] Output: Audio data uploaded to the server.

[1244] Step 3:

[1245] The server converts the received voice data into text.

[1246] Input: Audio data uploaded to the server.

[1247] Processing: The server uses a cloud speech recognition API (e.g., Google Cloud Speech-to-Text API) to analyze the audio data and map the audio waveform to text.

[1248] Output: Text data.

[1249] Step 4:

[1250] The server analyzes the converted text data and extracts keywords and context.

[1251] Input: Text data.

[1252] Processing: The server uses a natural language processing engine (e.g., a generative AI model) to analyze the text data and extract important keywords and context.

[1253] Output: Key words, phrases, and contextual information.

[1254] Step 5:

[1255] The server recognizes emotions based on text and voice data.

[1256] Input: Keywords, contextual information, and audio data.

[1257] Processing: The server uses an emotion analysis engine (e.g., an emotion analysis system) to analyze the user's emotion from the text and tone of voice and extract emotion data.

[1258] Output: Emotion tag (e.g., "joy", "sad").

[1259] Step 6:

[1260] The server stores the parsed information in a database.

[1261] Input: sentiment tags, key words and phrases, text data.

[1262] Processing: The server organizes and stores this data in a database along with metadata.

[1263] Output: Structured data in a database.

[1264] Step 7:

[1265] The server generates the digital copy.

[1266] Input: Structured data in a database.

[1267] Processing: The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user that mimics their memories and personality. This digital copy continuously learns based on the data it receives.

[1268] Output: A digital copy of the user.

[1269] Step 8:

[1270] The terminal receives the interaction request.

[1271] Input: Interaction requests from the user's family and friends.

[1272] Processing: The terminal receives the request and forwards it to the server.

[1273] Output: Interaction request data to the server.

[1274] Step 9:

[1275] The server generates a response based on the interaction request.

[1276] Input: Interaction request data, digital copy.

[1277] Processing: The server uses the digital copy to extract relevant information and generate a response that takes into account the emotional data, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[1278] Output: A specific response containing emotion.

[1279] Step 10:

[1280] The device communicates the generated response to the family.

[1281] Input: A specific response containing emotion.

[1282] Processing: The device outputs the generated response in voice or text format and conveys it to the family.

[1283] Output: Response message to family members.

[1284] (Application example 2)

[1285] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1286] Existing security systems face the problem of being unable to grasp the behavior and emotions of suspicious individuals and intruders in real time and respond appropriately. Furthermore, typical security monitoring systems often lack the ability to learn individual behavioral patterns or the ability to analyze emotional data to respond flexibly. As a result, they are unable to respond quickly and appropriately, which can lead to a decline in the quality of security.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1288] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for saving the extracted data in a database for each user, means for generating a digital copy of the user using the saved data, and means for responding to dialogue requests using the generated digital copy, as well as means for recognizing emotions from the voice data and text data and generating security information based thereon, and means for collecting and analyzing voice data of security targets in real time, detecting abnormal emotions or behavior, and notifying security guards. This enables real-time analysis and notification based on the voice data and emotion data, enabling rapid and appropriate responses.

[1289] A "user-provided means for recording voice data" is a device or function that records and stores user speech as digital voice data.

[1290] The "means for transmitting recorded voice data to a server" refers to a device or function for uploading recorded voice data to a server via a network.

[1291] The "means for converting voice data received by the server into text" refers to a device or function that converts voice data into text data using voice recognition technology.

[1292] The "means for analyzing the converted text data and extracting keywords and context" refers to a device or function that uses natural language processing technology to extract important words and context from the text data.

[1293] "Means for storing extracted data in a database for each user" refers to a device or function that organizes and stores analyzed data in a database based on individual user profiles.

[1294] A "means for generating a digital copy of a user using stored data" is a device or function that uses stored data to create a digital model that mimics the user's memories and personality.

[1295] A "means for responding to interaction requests using the generated digital copy" is a device or function that utilizes the generated digital copy to provide appropriate responses to questions from the user or other individual.

[1296] "Means for recognizing emotions from voice data and text data and generating security information based on that" refers to a device or function that determines emotions from analyzed voice and text and generates security-related information based on that.

[1297] "Means for collecting voice data from a security object, analyzing it in real time, detecting abnormal emotions or behavior, and notifying security guards" refers to a device or function that analyzes voice data captured at a security object in real time, detects abnormal emotions or behavior, and immediately reports the results to security guards.

[1298] A "means by which the digital copy of the user continues to learn and train based on new interactions" is a device or function that learns from new interactions with the user and continually improves itself.

[1299] "Means for learning the behavioral patterns of security targets and detecting unusual behavior" refers to a device or function that learns the normal behavioral patterns of security targets and detects abnormal behavior at an early stage.

[1300] This invention is a security system that digitizes memories and personalities based on user voice data and provides appropriate security information in real time. The system operates in the following steps.

[1301] Audio data collection

[1302] Users speak through security cameras and microphones installed in specific locations, which allows voice data to be collected in real time. Voice data may also be collected from smartphones and robots that can accept voice input.

[1303] Sending audio data

[1304] The recorded voice data is sent to a server via a network. The data arrives at the server in real time and processing begins immediately.

[1305] Converting audio data to text

[1306] The server converts the received voice data into text using a speech recognition module, which uses the Python programming language and the SpeechRecognition library, and also uses Google APIs for the text conversion.

[1307] Text data analysis and emotion recognition

[1308] The server analyzes the converted text data and extracts important keywords and context. This analysis uses natural language processing technologies such as TextBlob. At the same time, emotions are recognized from the speech and text. This emotion recognition is performed by an emotion engine, which can recognize positive, negative, and neutral emotions.

[1309] Preserving data and generating digital copies

[1310] The analyzed text and emotion data is stored in a database on a server based on date and category. This data is used to create a digital copy of the user, which is continually trained using new data.

[1311] Achieving security response

[1312] The collected voice data and analysis results are used to generate security information. For example, if abnormal emotions (e.g., anger or fear) are detected in a location under guard, a real-time notification is sent to security personnel. This notification allows for a rapid response. In addition, the system learns the behavioral patterns of the subject, and if any unusual behavior is detected, security personnel are similarly notified.

[1313] Specific examples

[1314] Specific examples are shown below.

[1315] 1. A user says, "I'm very tired today" while sitting in their living room.

[1316] 2. The voice data is collected by the smart home system and sent to the server.

[1317] 3. The server converts the voice data into text and extracts the keyword "tired."

[1318] 4. The emotion engine recognizes the emotion "tired" and this data is stored in the database.

[1319] 5. Update the user's digital copy based on the stored data.

[1320] 6. That night, a suspicious voice is detected in the living room and the emotion engine recognizes it as "anger."

[1321] 7. The server notifies the security guard in real time that "anger has been detected in the living room."

[1322] Prompt Sentence Examples

[1323] "Describe a security system that analyzes voice data to detect suspicious behavior and emotions. The system collects voice data, converts it into text, recognizes emotions, and notifies security personnel in real time. The system has a built-in emotion engine, allowing for rapid response."

[1324] This allows security systems to utilize advanced emotion recognition to take quick and accurate countermeasures.

[1325] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1326] Step 1:

[1327] The user provides the audio data

[1328] A user speaks using a device that can accept voice input (e.g., a smartphone or a robot). The speech is natural language such as "I'm very tired today." This voice data is collected by a microphone built into the device.

[1329] Input: User's voice

[1330] Output: Digital audio data

[1331] Step 2:

[1332] The device sends the recorded audio data to the server.

[1333] The device transmits the collected digital audio data to a server in real time or at regular intervals, using data communication over the Internet.

[1334] Input: Digital audio data

[1335] Output: Audio data sent to the server

[1336] Step 3:

[1337] The server converts the audio data into text

[1338] The server converts the received voice data into text data using a speech recognition module (using Google API). This process uses speech recognition technology.

[1339] Input: Audio data received by the server

[1340] Output: Text data

[1341] Step 4:

[1342] The server analyzes the converted text to extract keywords and context.

[1343] The server analyzes the text data using natural language processing technology (such as TextBlob) to extract important keywords and contexts. For example, it extracts the keyword "tired" from "I'm very tired today."

[1344] Input: Text data

[1345] Output: Keywords and context data

[1346] Step 5:

[1347] The server recognizes emotions from the voice and text data.

[1348] The server uses an emotion engine to recognize emotions from the tone of voice and text content. The emotion engine analyzes the data to determine "positive," "negative," or "neutral" emotions.

[1349] Input: Audio and text data

[1350] Output: Emotion data

[1351] Step 6:

[1352] The server stores the analyzed data and emotion data in a database.

[1353] The server stores the analyzed keywords and emotion data in a database for each user, organized by date and category.

[1354] Input: Keywords and sentiment data

[1355] Output: Records stored in the database

[1356] Step 7:

[1357] The server uses the stored data to create a digital copy of the user.

[1358] The server creates a digital copy of the user based on the data stored in the database, and the digital copy uses machine learning techniques to mimic the user's memories and personality.

[1359] Input: Saved data

[1360] Output: Digital copy of the user

[1361] Step 8:

[1362] The server responds to the interaction request with the generated digital copy.

[1363] Upon receiving an interaction request from a user or other individual, the server uses the generated digital copy to respond to the request, for example, "I'm very tired today," and generates a response such as "You need to take a break."

[1364] Input: Interaction request

[1365] Output: Dialogue response

[1366] Step 9:

[1367] The server collects and analyzes audio data from the security target in real time, detecting any abnormal emotions or behavior and notifying security personnel.

[1368] The server analyzes the voice data collected from the security location in real time and detects abnormal emotions (e.g., anger) or behavior. If an abnormality is detected, the server immediately notifies security personnel.

[1369] Input: Audio data of the security object

[1370] Output:Notify security guard

[1371] This enables the security system to respond quickly and accurately, improving the quality of security.

[1372] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1373] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1374] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1375] [Fourth embodiment]

[1376] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1377] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1378] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1379] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1380] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1381] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1382] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1383] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1384] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1385] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1386] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1387] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1388] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1389] The system of the present invention digitizes memories and experiences based on voice data provided by a user, and then conducts a dialogue using the generated digital copy. Specific embodiments are described below.

[1390] Collection and transmission of voice data

[1391] Users start a dedicated application and provide audio data about their daily lives and past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[1392] Speech-to-text transcription and analysis

[1393] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[1394] Saving to a database

[1395] The analyzed text data and keywords are stored in a database on the server. This database is structured for each user and organizes data based on different categories and timelines, allowing for a detailed record of the user's memories and experiences.

[1396] Digital copy generation and learning

[1397] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[1398] Interactive functionality

[1399] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, pulls out relevant information from the digital copy, and generates a specific response. This response is then provided to the family via the device. This allows for detailed conversations about the deceased's memories and experiences, maintaining an emotional connection in a new way.

[1400] Specific examples

[1401] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[1402] 2. The device records this audio data and sends it to the server.

[1403] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[1404] 4. The server stores these data in a database and updates the user's digital copy.

[1405] 5. When a family member makes a request such as "Tell me about your summer vacation memories," the server uses the digital copy to generate an appropriate response, which the device then relays to the family member.

[1406] In this way, this system digitizes the user's memories and personality, not only passing on their memories to future generations but also enabling a new form of communication with family and friends.

[1407] The processing flow will be explained below.

[1408] Step 1:

[1409] The user launches a dedicated application and begins speaking audio data about daily life and past events.

[1410] Step 2:

[1411] The device records the user's voice and stores the recording in a buffer.

[1412] Step 3:

[1413] The device sends the completed recording to the server's API endpoint using the HTTPS protocol.

[1414] Step 4:

[1415] The server passes the received voice data to a speech recognition engine and converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[1416] Step 5:

[1417] The server then passes the converted text data to a natural language processing (NLP) engine for keyword and context analysis. In this step, keywords such as "childhood," "mother," and "ocean" are extracted.

[1418] Step 6:

[1419] The server stores the analyzed keywords and text data in a database for each user, organizing the data based on date and category.

[1420] Step 7:

[1421] The server uses the stored data to train machine learning models, which then generate a digital copy of the user that reflects how they speak and think.

[1422] Step 8:

[1423] The device receives conversation requests from family and friends, such as "Tell me about your summer vacation memories."

[1424] Step 9:

[1425] The server analyzes the interaction request it receives and queries the digital copy of the user to extract relevant memory and experience data.

[1426] Step 10:

[1427] The server generates an appropriate response from the digital copy, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[1428] Step 11:

[1429] Your device will generate a response that will be displayed to family and friends, or played aloud.

[1430] Step 12:

[1431] The server updates the machine learning model again based on the new interaction data collected, improving the accuracy of the user's digital copy.

[1432] This series of processes digitizes the user's memories and personality, enabling new forms of communication with family and friends.

[1433] Example 1

[1434] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1435] Current systems for digitizing users' memories and experiences and maintaining emotional connections with family and friends in new ways face several challenges. First, accuracy is required in the conversion of voice data to text and subsequent analysis, but there is a lack of technology to improve this accuracy. Second, if the machine learning technology used to generate the digital copy does not perform sufficiently, the reproduction of the user's memories and interactions will be insufficient. Finally, the digital copy's ability to continuously learn has not been established, which can lead to a decline in the quality of the interactions.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1437] In this invention, the server includes means for including a speech recognition module that converts speech data into text, means for including a natural language processing engine that analyzes the text data and extracts keywords and context, means for storing the extracted data in a database for each user, and means for using machine learning technology to generate a digital copy of the user using the stored data. This enables highly accurate text conversion and analysis of speech data, and by generating and continuously training the digital copy of the user, the quality of the dialogue can be improved and the user's memories and experiences can be more faithfully digitized.

[1438] "User" refers to a person who uses the system to provide voice data.

[1439] "Terminal" refers to a device that has the function of recording a user's voice data and transmitting it to a server.

[1440] A "server" refers to a device or system that receives voice data sent from a terminal, converts it into text data, analyzes it, and stores it in a database.

[1441] A "voice recognition module" refers to software or hardware for converting voice data into text data.

[1442] A "natural language processing engine" refers to technology for analyzing text data and extracting important keywords and context.

[1443] "Database" refers to a system that stores and manages analyzed text data and keywords for each user.

[1444] "Machine learning technology" refers to algorithms and models that use data to create a digital copy of a user and allow it to continuously learn.

[1445] A "digital copy" refers to a model created to mimic a user's speech and thought patterns.

[1446] "Interaction requests" refer to questions or requests for conversation that a user's family or friends make to a digital copy.

[1447] "Response" refers to a specific reply that a digital copy makes in response to an interaction request.

[1448] The system of the present invention aims to digitize memories and experiences based on voice data provided by users, and then engage in dialogue using the generated digital copies. The system of the present invention includes the following main components:

[1449] Collection and transmission of voice data

[1450] Users start a dedicated application and provide audio data about their daily lives or past events. This audio data is recorded by the device and sent to a server in real time or at regular intervals.

[1451] Speech-to-text transcription and analysis

[1452] The server converts the received voice data into text data using a speech recognition module (e.g., Google Speech-to-Text). The converted text data is then analyzed using a natural language processing (NLP) engine (e.g., OpenAI GPT-3) to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[1453] Saving to a database

[1454] The server stores the analyzed text data and keywords in a database. This database is structured for each user and organizes data based on different categories and timelines. For example, there are categories such as "childhood" and "summer memories."

[1455] Digital copy generation and learning

[1456] The server creates a digital copy of the user based on the data stored in the database. This digital copy uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns. The digital copy continuously learns based on new interaction data to form a more faithful personality model. For example, it learns about the specific words and phrases the user speaks.

[1457] Interactive functionality

[1458] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about Mom's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. This response is then provided to the family member via the device.

[1459] Specific examples

[1460] For example, a user might say, "The flowers I saw in the park yesterday were beautiful," using a dedicated application. The device records this and sends it to the server. The server then converts the received voice data into text using Google Speech-to-Text, generating the text, "The flowers I saw in the park yesterday were beautiful." The server then analyzes the text data using OpenAI GPT-3, extracting keywords such as "yesterday," "park," "flowers," and "beautiful." The server then stores this data in a database and categorizes it into categories such as "yesterday's events" and "memories of the park."

[1461] The device accepts a request from the family such as "What was beautiful yesterday?" The server analyzes this and generates a response from past data, such as "The flowers I saw in the park yesterday were beautiful," and responds to the family via the device.

[1462] Prompt Sentence Examples

[1463] "Analyze the following audio data and extract key keywords and context to generate a digital copy of the user. Example: 'When I was a child, I used to spend summer vacations at my grandparents' house.'"

[1464] In this way, the system digitizes the user's memories and personality, passing on their memories to future generations and enabling new forms of communication with family and friends.

[1465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1466] Step 1:

[1467] The user starts the dedicated application and inputs voice information. For example, the user might say, "The flowers I saw in the park yesterday were beautiful."

[1468] Input: User's voice data

[1469] Output: Audio data recorded on the device

[1470] Specific behavior: The user presses the record button on the application, and recording stops when the user finishes speaking.

[1471] Step 2:

[1472] The terminal records the user's voice data and transmits the voice data to the server in real time or at regular intervals.

[1473] Input: Recorded audio data

[1474] Output: Audio data sent to the server

[1475] Specific operation: The device uploads audio data to a server via the Internet.

[1476] Step 3:

[1477] The server converts the received voice data into text data using a voice recognition module (e.g., Google Speech-to-Text).

[1478] Input: Audio data sent to the server

[1479] Output: Text data

[1480] Specific operation: The speech recognition module is started on the server side, the audio file is analyzed, and text is generated.

[1481] Step 4:

[1482] The server passes the converted text data to a natural language processing (NLP) engine (e.g., OpenAI GPT-3) for analysis to extract keywords and context.

[1483] Input: Text data

[1484] Output: Extracted keywords and context information

[1485] How it works: The NLP engine analyzes the text data and extracts important keywords such as "yesterday," "park," "flowers," and "beautiful."

[1486] Step 5:

[1487] The server stores the analyzed text data and keywords in a database, which is structured for each user and organized according to different categories and timelines.

[1488] Input: Extracted keywords and context information

[1489] Output: Text data for each user stored in the database

[1490] Specific operation: Text data and keywords are stored by category using database software (e.g., MySQL).

[1491] Step 6:

[1492] The server creates a digital copy of the user based on the data stored in the database, which uses machine learning techniques (e.g., OpenAI GPT-3) to mimic the user's speech and thought patterns.

[1493] Input: User data stored in the database

[1494] Output: Generated digital copy

[1495] What it does: A machine learning model uses your conversational data as training data to generate a digital copy.

[1496] Step 7:

[1497] The server continuously updates and trains the generated digital copy, allowing the model to continually learn based on new interaction data.

[1498] Input: New interaction data

[1499] Output: Updated digital copy

[1500] What happens: New interaction data is added to the database and the machine learning model is retrained.

[1501] Step 8:

[1502] The device accepts conversation requests from family and friends. For example, a family member might request, "I'd like to hear about my mother's summer vacation memories."

[1503] Input: Interaction request

[1504] Output: Request sent to server

[1505] Specific operation: The device receives a request in voice or text format and sends it to the server.

[1506] Step 9:

[1507] The server analyzes the interaction request and extracts relevant information from the digital copy to generate a specific response.

[1508] Input: Interaction request and digital copy

[1509] Output: The generated response

[1510] What it does: The server analyzes the request and uses the digital copy to generate an appropriate response.

[1511] Step 10:

[1512] The device then provides the generated response to family and friends, for example, "The flowers I saw in the park yesterday were beautiful."

[1513] Input: The generated response

[1514] Output: Responses to family and friends

[1515] What happens: The device plays or displays a response in audio or text format.

[1516] Through these steps, the system digitizes the user's memories and personality, enabling new forms of emotional connection with family and friends.

[1517] (Application example 1)

[1518] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] While there have been systems that convert users' voice data into text and generate digital copies, few systems are capable of generating and delivering personalized content based on the user's memories and experiences, making it difficult to provide an experience optimized for each individual user.In addition, there are limited means of passing on content based on the memories and experiences of the deceased to future generations, creating a need for a new form of emotional connection.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1521] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for storing the extracted data in a database for each user, means for generating a digital copy of the user using the stored data, means for responding to dialogue requests using the generated digital copy, and means for generating and delivering personalized content based on the user's memories and experiences. This makes it possible to generate and deliver unique content based on the user's individual memories and experiences, thereby realizing a new form of emotional connection.

[1522] 1. "User-Provided Voice Data"

[1523] "User-provided voice data" refers to voice information recorded by a user using a dedicated device or application.

[1524] 2. "Method of recording"

[1525] A "recording means" is a device or software that has the function of acquiring and digitizing a user's voice data.

[1526] 3. "Method of sending to the server"

[1527] "Means for transmitting to the server" refers to a device or software that has the function of transferring recorded audio data to a central server via a network.

[1528] 4. "Method of converting to text"

[1529] A "text conversion means" is a device or software that uses speech recognition technology to convert voice data into text information.

[1530] 5. "Methods for analyzing and extracting keywords and context"

[1531] "Means for analyzing and extracting keywords and context" refers to a device or software that has the function of analyzing text data using natural language processing technology and identifying important words and context.

[1532] 6. "Method of storing data in a per-user database"

[1533] The "means for storing in a database for each user" is a database system that manages and stores analyzed text data and keywords for each individual user.

[1534] 7. "Means for generating a digital copy of the user"

[1535] "Means for generating a digital copy of a user" refers to a device or software that has the ability to create a digital avatar that mimics the user's voice and thought patterns based on stored data.

[1536] 8. "Means of responding to interaction requests"

[1537] A "means for responding to dialogue requests" is a device or software that has the function of generating appropriate responses using a digital copy in response to inquiries from the user or their family and friends.

[1538] 9. "A means of generating and delivering personalized content"

[1539] A "means for generating and delivering personalized content" is a device or software that has the function of creating customized videos, articles, or other content based on a user's memories and experiences and delivering it to a user's terminal.

[1540] The present invention provides a system for generating and distributing personalized content based on user voice data. This system is configured by combining multiple pieces of hardware and software.

[1541] 1. Collection of audio data

[1542] Users start a dedicated application and use their smartphone to provide audio data about their daily lives and past events. This audio data is digitized using the smartphone's recording function and sent to a server in real time or at regular intervals.

[1543] 2. Speech-to-text conversion and analysis

[1544] The server first passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. Next, this text data is analyzed using a natural language processing engine (e.g., spaCy, NLTK) to extract important keywords and context. For example, if a user says, "I traveled to Europe during my university days. Paris was particularly memorable," keywords such as "university days," "Europe," "travel," and "Paris" are extracted from the text data.

[1545] 3. Saving to the database

[1546] The analyzed text data and keywords are stored in a database (e.g., MongoDB) on the server. This database is structured for each user and organizes data based on different categories and timelines, providing a detailed record of the user's memories and experiences.

[1547] 4. Creating and Learning from Digital Copies

[1548] The server uses the data stored in the database to create a digital copy of the user. This digital copy uses machine learning techniques to mimic the user's speech and thought patterns. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[1549] 5. Personalized content generation and delivery

[1550] The server generates personalized content (e.g., videos or articles) based on the user's memory and experience. It uses a generative AI model to generate the desired content by inputting a prompt. For example, the following prompt can be used:

[1551] "Generate a personalized video based on the user's memories of a trip to Europe during their college days."

[1552] The generated content is delivered to the user's smartphone application, allowing the user to view it.

[1553] Specific examples

[1554] 1. A user uses the app and says, "I traveled to Europe during my college years. Paris was particularly memorable."

[1555] 2. The smartphone transmits this voice data to the server in real time.

[1556] 3. The server converts the voice data into text and extracts the keywords "college days," "Europe," "travel," and "Paris."

[1557] 4. The server stores these data in a database and updates the user's digital copy.

[1558] 5. The server generates a personalized video using the prompt text and delivers it to the smartphone application.

[1559] This system enables users to generate and distribute unique content based on their memories and experiences, enabling new forms of emotional connection.

[1560] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1561] Step 1:

[1562] The user launches a dedicated application and talks about their daily life and past events.

[1563] Input: User's voice

[1564] Output: Recorded audio data

[1565] Specific operation: Records audio using the smartphone's microphone and saves it as digital data.

[1566] Step 2:

[1567] The device sends the recorded audio data to the server.

[1568] Input: Recorded audio data

[1569] Output: Audio data sent to the server

[1570] Specific operation: Uploads audio data to the server using the smartphone's communication function (Wi-Fi or mobile data).

[1571] Step 3:

[1572] The server passes the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[1573] Input: Audio data

[1574] Output: Text data

[1575] Specific operation: The server calls the speech recognition API and transcribes the audio data.

[1576] Step 4:

[1577] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy, NLTK) to extract keywords and context.

[1578] Input: Text data

[1579] Output: Extracted keywords and context

[1580] What it does: A natural language processing engine analyzes text data to identify important words, phrases, and context.

[1581] Step 5:

[1582] The server stores the extracted data in a per-user database (e.g. MongoDB).

[1583] Input: Keywords and context

[1584] Output: Data stored in the database

[1585] Specific operation: The server saves the analysis results in a database managed for each user.

[1586] Step 6:

[1587] The server uses the stored data to create a digital copy of the user, continually training them as needed.

[1588] Input: Data stored in a database

[1589] Output: Updated digital copy

[1590] How it works: The server uses machine learning models to generate a digital copy of the user that mimics their language and thought patterns, periodically feeding them new data to train them.

[1591] Step 7:

[1592] The server generates personalized content (e.g., videos, articles) based on the user's memories and experiences and delivers it to the device.

[1593] Input: Content-generated prompt based on the digital copy

[1594] Output: Generated content

[1595] Specific operation: Personalized content is created using a generative AI model, and the content is sent to the device to provide to the user. An example of a prompt for the generative AI model is, "Generate a personalized video based on the user's memories of traveling in Europe during their university days."

[1596] This processing step allows for the creation and delivery of unique content based on the user's memories and experiences, enabling a new form of emotional connection.

[1597] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1598] The present invention combines an emotion engine with a system that digitizes memories and personalities based on voice data provided by a user and then allows the user to converse with the generated digital copy. Specific embodiments are described below.

[1599] Collection and transmission of voice data

[1600] The user starts a dedicated application and provides voice data about daily life and past events. This voice data is recorded by the device and sent to the server in real time or at regular intervals.

[1601] Speech-to-text transcription and analysis

[1602] The server passes the received voice data to a speech recognition module, which converts it into text data. This text data is then analyzed by a natural language processing (NLP) engine to extract important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," keywords such as "childhood," "mother," and "sea" are extracted from the text data.

[1603] Emotion recognition and storage in a database

[1604] The server passes the converted voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the emotion from the tone and context of the voice and extracts that emotion as data. For example, if the user is speaking happily, the emotion data is extracted as "joy."

[1605] The recognized emotion data is stored in a database on the server along with the analyzed keywords. The data is organized by date and category, providing a detailed record of the user's memories, experiences, and emotions.

[1606] Digital copy generation and learning

[1607] The server uses the data stored in the database to create a digital copy of the user. This digital copy is generated using machine learning techniques based on voice, text, and emotional data. The created digital copy continuously learns based on new interaction data, forming a more faithful personality model.

[1608] Interactive functionality

[1609] The device accepts conversation requests from the user's family and friends. For example, if a family member requests, "I'd like to hear about your mother's summer vacation memories," the server analyzes the request, extracts relevant information from the digital copy, and generates a specific response. The digital copy also takes emotional data into account when generating the response, so it might respond in a happy tone, for example, "I remember going to the beach with my mother during summer vacation and collecting seashells."

[1610] Specific examples

[1611] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[1612] 2. The device records this audio data and sends it to the server.

[1613] 3. The server converts the voice data into text data and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[1614] 4. The server uses an emotion engine to recognize the emotion the user is expressing as "joy" when speaking.

[1615] 5. The server stores these data in a database and updates the user's digital copy.

[1616] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[1617] In this way, by digitizing the user's memories and personality and taking emotional data into account, this system provides a new form of communication with family and friends that is more realistic and emotionally empathetic.

[1618] The processing flow will be explained below.

[1619] Step 1:

[1620] The user launches the dedicated application and begins to talk about anecdotes based on personal memories and emotions.

[1621] Step 2:

[1622] The device records the user's voice and stores the recording in a buffer.

[1623] Step 3:

[1624] The device sends the recorded audio data to the server's API endpoint using the HTTPS protocol.

[1625] Step 4:

[1626] The server passes the received voice data to a speech recognition module, which converts it into text data. The converted result is "When I was a child, I went to the beach with my mother."

[1627] Step 5:

[1628] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the keywords and context, extracting keywords such as "childhood," "mother," and "ocean."

[1629] Step 6:

[1630] The server passes the analyzed text data and keywords to an emotion engine to recognize the user's emotion. For example, if the user speaks in a happy tone, the emotion "joy" is recognized.

[1631] Step 7:

[1632] The server combines the analyzed text data with emotional data and stores it in a database for each user, allowing memories, experiences, and emotions to be stored in an integrated manner.

[1633] Step 8:

[1634] The server uses the accumulated data to train machine learning models, which then generate a digital copy of the user that can reproduce the way they speak, think, and express emotions.

[1635] Step 9:

[1636] The device receives conversation requests from family and friends, such as "Tell me about your mother's summer vacation memories."

[1637] Step 10:

[1638] The server analyzes the received interaction request and queries the digital copy to extract relevant memory, experience, and emotion data.

[1639] Step 11:

[1640] The server generates an appropriate response from the digital copy, taking into account the emotional data and generating a response with an appropriate emotional expression. For example, if the emotional data indicates "joy," the server generates a response in a happy tone, such as "I enjoyed going to the beach with my mother during summer vacation and collecting seashells."

[1641] Step 12:

[1642] Your device will generate a response that will be displayed to family and friends, or played aloud.

[1643] Step 13:

[1644] The server updates the machine learning model again based on the new dialogue data and emotion data collected, improving the accuracy and emotional expression of the user's digital copy.

[1645] Through this series of processes, the user's memories and personality are digitized, and by taking emotional data into consideration, more realistic and emotional communication is achieved.

[1646] Example 2

[1647] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1648] Conventional dialogue systems using voice data often simply convert a user's speech into text and generate pre-defined, standard responses. However, these systems have difficulty responding to the user's emotions and individual contexts, making it difficult to improve empathy and intimacy in dialogues with the user. Furthermore, dialogue learning and continuous improvement are limited, making it insufficient to fully reproduce the user's memories and experiences.

[1649] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1650] In this invention, the server includes means for recording voice data provided by a user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data to extract keywords and context, means for recognizing emotions based on the analyzed text data and voice data, means for storing the recognized emotion data, keywords, and context in a database for each user, means for generating a digital copy of the user using the stored data, and means for responding to dialogue requests using the generated digital copy. This enables responses based on emotions and context, improves empathy and intimacy in dialogue with the user, and also enables improved dialogue accuracy through continuous learning.

[1651] "User" refers to a person who uses the system to provide voice data and engage in interaction.

[1652] "Audio Data" refers to information recorded as audio signals provided by a User using the System.

[1653] "Recording instrument" means any equipment or software used to collect and store a user's voice data in digital form.

[1654] "Means for transmitting" refers to the communication means for transferring the recorded audio data to the server.

[1655] "Server" refers to a central processing unit that processes received voice data and performs text conversion, data analysis, emotion recognition, database storage, digital copy generation, etc.

[1656] "Means for converting to text" refers to speech recognition technology for converting voice data into text data.

[1657] "Means of analyzing and extracting keywords and context" refers to natural language processing technology for extracting important words and linguistic background information from text data.

[1658] "Means for recognizing emotions" refers to technology for analyzing and identifying a user's emotions from voice and text data.

[1659] "Means for storing in a database" refers to a system for structuring and storing information such as analyzed text data, emotional data, keywords, and context.

[1660] "Means for generating a digital copy" refers to technology that generates a digital object that reproduces a user's memories and personality based on information stored in a database.

[1661] "Means for responding to dialogue requests" refers to a system that generates and provides appropriate responses based on dialogue requests from the user's family and friends.

[1662] "Means for continuous learning and developing a faithful personality model" refers to machine learning techniques that allow the generated digital copy to evolve based on new interaction data, creating a more realistic personality model.

[1663] "Means for generating appropriate responses based on emotional data and specific contexts and keywords" refers to the function of creating responses appropriate to the situation during a conversation, depending on the user's emotional state and indicative words.

[1664] The present invention combines an emotion engine with a system that digitizes memories and personality based on voice data provided by a user and then allows the user to interact with the generated digital copy. Specific embodiments are described below.

[1665] Collection and transmission of voice data

[1666] The user launches a dedicated application and provides voice data about daily life and past events. The device records this voice data and transmits it to the server in real time or at regular intervals.

[1667] Speech-to-text transcription and analysis

[1668] The server passes the received voice data to a voice recognition module (e.g., a cloud voice recognition API) that converts the voice into text data. This involves analyzing the voice waveform and mapping it to text. The converted text data is then passed to a natural language processing engine (e.g., a generative AI model) that extracts important keywords and context. For example, if a user says, "When I was a child, I went to the beach with my mother," the keywords "childhood," "mother," and "sea" are extracted.

[1669] Emotion recognition and storage in a database

[1670] The server passes the converted text data and voice data to an emotion engine (e.g., an emotion analysis system) to recognize the user's emotion. For example, if the user is speaking with a happy expression, the emotion data "joy" is extracted. The recognized emotion data, text data, keywords, and context are stored in the server's database. When the data is stored, it is organized by adding metadata such as date and category.

[1671] Digital copy generation and learning

[1672] The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user based on the voice, text, and emotion data stored in the database. The digital copy then continuously learns from new interaction data to form a more faithful personality model.

[1673] Interactive functionality

[1674] The server accepts conversation requests from the user's family and friends. For example, if a family member requests, "I want to hear about my mother's summer vacation memories," the server analyzes the request, pulls relevant information from the digital copy, and generates a specific response. The response takes into account emotional data and is generated in a happy tone, such as, "I remember going to the beach with my mother during summer vacation and collecting seashells." The device then relays the generated response to the family member.

[1675] Specific examples

[1676] 1. The user uses a dedicated application to say, "When I was a child, I used to spend summer vacations at my grandparents' house."

[1677] 2. The device records this audio data and sends it to the server.

[1678] 3. The server converts the voice data into text data using a cloud speech recognition API and extracts the keywords "childhood," "summer vacation," and "grandparents' house."

[1679] 4. The server uses an emotion analysis system to recognize the emotion the user is expressing as "joy" when speaking.

[1680] 5. The server stores these data in a database and updates the user's digital copy.

[1681] 6. When a family member makes a request such as, "Tell me about your summer vacation memories," the server uses the digital copy to generate a response containing the appropriate emotion: "During summer vacation, I often went to the beach with my mother and had a lot of fun collecting seashells," and the device relays this response to the family member.

[1682] (Example of a prompt to input to a generative AI model)

[1683] "This system digitizes memories and personality based on voice data provided by the user, and then combines it with an emotion engine to create a digital copy that can then converse with the created digital copy. Specifically, the user provides voice data about their daily life and past events, which is then converted into text data and analyzed. The emotion engine is used to recognize emotions and store them in a database. Finally, a digital copy is created based on this data, and in response to conversation requests from family and friends, it provides responses containing appropriate emotions."

[1684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1685] Step 1:

[1686] The user starts the dedicated application and records audio data.

[1687] Input: The voice the user speaks to the application.

[1688] Processing: The device's recording function collects the audio as digital data and uses noise-canceling technology to improve the sound quality.

[1689] Output: High quality audio data file.

[1690] Step 2:

[1691] The device sends the recorded audio data to the server.

[1692] Input: High quality audio data files.

[1693] Processing: The device encrypts the voice data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1694] Output: Audio data uploaded to the server.

[1695] Step 3:

[1696] The server converts the received voice data into text.

[1697] Input: Audio data uploaded to the server.

[1698] Processing: The server uses a cloud speech recognition API (e.g., Google Cloud Speech-to-Text API) to analyze the audio data and map the audio waveform to text.

[1699] Output: Text data.

[1700] Step 4:

[1701] The server analyzes the converted text data and extracts keywords and context.

[1702] Input: Text data.

[1703] Processing: The server uses a natural language processing engine (e.g., a generative AI model) to analyze the text data and extract important keywords and context.

[1704] Output: Key words, phrases, and contextual information.

[1705] Step 5:

[1706] The server recognizes emotions based on text and voice data.

[1707] Input: Keywords, contextual information, and audio data.

[1708] Processing: The server uses an emotion analysis engine (e.g., an emotion analysis system) to analyze the user's emotion from the text and tone of voice and extract emotion data.

[1709] Output: Emotion tag (e.g., "joy", "sad").

[1710] Step 6:

[1711] The server stores the parsed information in a database.

[1712] Input: sentiment tags, key words and phrases, text data.

[1713] Processing: The server organizes and stores this data in a database along with metadata.

[1714] Output: Structured data in a database.

[1715] Step 7:

[1716] The server generates the digital copy.

[1717] Input: Structured data in a database.

[1718] Processing: The server uses machine learning techniques (e.g., neural network models) to generate a digital copy of the user that mimics their memories and personality. This digital copy continuously learns based on the data it receives.

[1719] Output: A digital copy of the user.

[1720] Step 8:

[1721] The terminal receives the interaction request.

[1722] Input: Interaction requests from the user's family and friends.

[1723] Processing: The terminal receives the request and forwards it to the server.

[1724] Output: Interaction request data to the server.

[1725] Step 9:

[1726] The server generates a response based on the interaction request.

[1727] Input: Interaction request data, digital copy.

[1728] Processing: The server uses the digital copy to extract relevant information and generate a response that takes into account the emotional data, such as "I remember going to the beach with my mother during summer vacation and collecting seashells."

[1729] Output: A specific response containing emotion.

[1730] Step 10:

[1731] The device communicates the generated response to the family.

[1732] Input: A specific response containing emotion.

[1733] Processing: The device outputs the generated response in voice or text format and conveys it to the family.

[1734] Output: Response message to family members.

[1735] (Application example 2)

[1736] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1737] Existing security systems face the problem of being unable to grasp the behavior and emotions of suspicious individuals and intruders in real time and respond appropriately. Furthermore, typical security monitoring systems often lack the ability to learn individual behavioral patterns or the ability to analyze emotional data to respond flexibly. As a result, they are unable to respond quickly and appropriately, which can lead to a decline in the quality of security.

[1738] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1739] In this invention, the server includes means for recording voice data provided by the user, means for transmitting the recorded voice data to the server, means for converting the voice data received by the server into text, means for analyzing the converted text data and extracting keywords and context, means for saving the extracted data in a database for each user, means for generating a digital copy of the user using the saved data, and means for responding to dialogue requests using the generated digital copy, as well as means for recognizing emotions from the voice data and text data and generating security information based thereon, and means for collecting and analyzing voice data of security targets in real time, detecting abnormal emotions or behavior, and notifying security guards. This enables real-time analysis and notification based on the voice data and emotion data, enabling rapid and appropriate responses.

[1740] A "user-provided means for recording voice data" is a device or function that records and stores user speech as digital voice data.

[1741] The "means for transmitting recorded voice data to a server" refers to a device or function for uploading recorded voice data to a server via a network.

[1742] The "means for converting voice data received by the server into text" refers to a device or function that converts voice data into text data using voice recognition technology.

[1743] The "means for analyzing the converted text data and extracting keywords and context" refers to a device or function that uses natural language processing technology to extract important words and context from the text data.

[1744] "Means for storing extracted data in a database for each user" refers to a device or function that organizes and stores analyzed data in a database based on individual user profiles.

[1745] A "means for generating a digital copy of a user using stored data" is a device or function that uses stored data to create a digital model that mimics the user's memories and personality.

[1746] A "means for responding to interaction requests using the generated digital copy" is a device or function that utilizes the generated digital copy to provide appropriate responses to questions from the user or other individual.

[1747] "Means for recognizing emotions from voice data and text data and generating security information based on that" refers to a device or function that determines emotions from analyzed voice and text and generates security-related information based on that.

[1748] "Means for collecting voice data from a security object, analyzing it in real time, detecting abnormal emotions or behavior, and notifying security guards" refers to a device or function that analyzes voice data captured at a security object in real time, detects abnormal emotions or behavior, and immediately reports the results to security guards.

[1749] A "means by which the digital copy of the user continues to learn and train based on new interactions" is a device or function that learns from new interactions with the user and continually improves itself.

[1750] "Means for learning the behavioral patterns of security targets and detecting unusual behavior" refers to a device or function that learns the normal behavioral patterns of security targets and detects abnormal behavior at an early stage.

[1751] This invention is a security system that digitizes memories and personalities based on user voice data and provides appropriate security information in real time. The system operates in the following steps.

[1752] Audio data collection

[1753] Users speak through security cameras and microphones installed in specific locations, which allows voice data to be collected in real time. Voice data may also be collected from smartphones and robots that can accept voice input.

[1754] Sending audio data

[1755] The recorded voice data is sent to a server via a network. The data arrives at the server in real time and processing begins immediately.

[1756] Converting audio data to text

[1757] The server converts the received voice data into text using a speech recognition module, which uses the Python programming language and the SpeechRecognition library, and also uses Google APIs for the text conversion.

[1758] Text data analysis and emotion recognition

[1759] The server analyzes the converted text data and extracts important keywords and context. This analysis uses natural language processing technologies such as TextBlob. At the same time, emotions are recognized from the speech and text. This emotion recognition is performed by an emotion engine, which can recognize positive, negative, and neutral emotions.

[1760] Preserving data and generating digital copies

[1761] The analyzed text and emotion data is stored in a database on a server based on date and category. This data is used to create a digital copy of the user, which is continually trained using new data.

[1762] Achieving security response

[1763] The collected voice data and analysis results are used to generate security information. For example, if abnormal emotions (e.g., anger or fear) are detected in a location under guard, a real-time notification is sent to security personnel. This notification allows for a rapid response. In addition, the system learns the behavioral patterns of the subject, and if any unusual behavior is detected, security personnel are similarly notified.

[1764] Specific examples

[1765] Specific examples are shown below.

[1766] 1. A user says, "I'm very tired today" while sitting in their living room.

[1767] 2. The voice data is collected by the smart home system and sent to the server.

[1768] 3. The server converts the voice data into text and extracts the keyword "tired."

[1769] 4. The emotion engine recognizes the emotion "tired" and this data is stored in the database.

[1770] 5. Update the user's digital copy based on the stored data.

[1771] 6. That night, a suspicious voice is detected in the living room and the emotion engine recognizes it as "anger."

[1772] 7. The server notifies the security guard in real time that "anger has been detected in the living room."

[1773] Prompt Sentence Examples

[1774] "Describe a security system that analyzes voice data to detect suspicious behavior and emotions. The system collects voice data, converts it into text, recognizes emotions, and notifies security personnel in real time. The system has a built-in emotion engine, allowing for rapid response."

[1775] This allows security systems to utilize advanced emotion recognition to take quick and accurate countermeasures.

[1776] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1777] Step 1:

[1778] The user provides the audio data

[1779] A user speaks using a device that can accept voice input (e.g., a smartphone or a robot). The speech is natural language such as "I'm very tired today." This voice data is collected by a microphone built into the device.

[1780] Input: User's voice

[1781] Output: Digital audio data

[1782] Step 2:

[1783] The device sends the recorded audio data to the server.

[1784] The device transmits the collected digital audio data to a server in real time or at regular intervals, using data communication over the Internet.

[1785] Input: Digital audio data

[1786] Output: Audio data sent to the server

[1787] Step 3:

[1788] The server converts the audio data into text

[1789] The server converts the received voice data into text data using a speech recognition module (using Google API). This process uses speech recognition technology.

[1790] Input: Audio data received by the server

[1791] Output: Text data

[1792] Step 4:

[1793] The server analyzes the converted text to extract keywords and context.

[1794] The server analyzes the text data using natural language processing technology (such as TextBlob) to extract important keywords and contexts. For example, it extracts the keyword "tired" from "I'm very tired today."

[1795] Input: Text data

[1796] Output: Keywords and context data

[1797] Step 5:

[1798] The server recognizes emotions from the voice and text data.

[1799] The server uses an emotion engine to recognize emotions from the tone of voice and text content. The emotion engine analyzes the data to determine "positive," "negative," or "neutral" emotions.

[1800] Input: Audio and text data

[1801] Output: Emotion data

[1802] Step 6:

[1803] The server stores the analyzed data and emotion data in a database.

[1804] The server stores the analyzed keywords and emotion data in a database for each user, organized by date and category.

[1805] Input: Keywords and sentiment data

[1806] Output: Records stored in the database

[1807] Step 7:

[1808] The server uses the stored data to create a digital copy of the user.

[1809] The server creates a digital copy of the user based on the data stored in the database, and the digital copy uses machine learning techniques to mimic the user's memories and personality.

[1810] Input: Saved data

[1811] Output: Digital copy of the user

[1812] Step 8:

[1813] The server responds to the interaction request with the generated digital copy.

[1814] Upon receiving an interaction request from a user or other individual, the server uses the generated digital copy to respond to the request, for example, "I'm very tired today," and generates a response such as "You need to take a break."

[1815] Input: Interaction request

[1816] Output: Dialogue response

[1817] Step 9:

[1818] The server collects and analyzes audio data from the security target in real time, detecting any abnormal emotions or behavior and notifying security personnel.

[1819] The server analyzes the voice data collected from the security location in real time and detects abnormal emotions (e.g., anger) or behavior. If an abnormality is detected, the server immediately notifies security personnel.

[1820] Input: Audio data of the security object

[1821] Output:Notify security guard

[1822] This enables the security system to respond quickly and accurately, improving the quality of security.

[1823] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1824] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1825] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1826] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1827] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1828] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1829] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1830] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1831] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1832] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1833] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1834] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1835] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1836] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1837] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1838] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1839] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1840] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1841] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1842] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1843] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1844] The following is further disclosed regarding the above embodiment.

[1845] (Claim 1)

[1846] means for recording user-provided voice data;

[1847] means for transmitting the recorded voice data to a server;

[1848] A means for converting the received voice data into text by the server;

[1849] A means for analyzing the converted text data and extracting keywords and context;

[1850] a means for storing the extracted data in a per-user database;

[1851] means for generating a digital copy of the user using the stored data;

[1852] means for responding to interaction requests using the generated digital copy;

[1853] A system including:

[1854] (Claim 2)

[1855] 10. The system of claim 1, including means for the digital copy of the user to continue to learn and train based on new interactions.

[1856] (Claim 3)

[1857] 10. The system of claim 1, wherein the digital copy of the user, when responding to an interaction request, includes means for generating an appropriate response based on specific context or keywords.

[1858] "Example 1"

[1859] (Claim 1)

[1860] means for recording user-provided voice data;

[1861] means for transmitting the recorded voice data to a server;

[1862] a server having a speech recognition module for converting received speech data into text;

[1863] a means for providing a natural language processing engine that analyzes the converted text data and extracts keywords and context;

[1864] a means for storing the extracted data in a per-user database;

[1865] a means for using machine learning techniques to generate a digital copy of the user using the stored data;

[1866] means for responding to interaction requests using the generated digital copy;

[1867] A system including:

[1868] (Claim 2)

[1869] 10. The system of claim 1, further comprising means for the digital copy of the user to continually learn and train based on new interactions.

[1870] (Claim 3)

[1871] 10. The system of claim 1, wherein the digital copy of the user, when responding to an interaction request, includes means for generating an appropriate response based on specific context or keywords.

[1872] "Application Example 1"

[1873] (Claim 1)

[1874] means for recording user-provided voice data;

[1875] means for transmitting the recorded voice data to a server;

[1876] A means for converting the received voice data into text by the server;

[1877] A means for analyzing the converted text data and extracting keywords and context;

[1878] a means for storing the extracted data in a per-user database;

[1879] means for generating a digital copy of the user using the stored data;

[1880] means for responding to interaction requests using the generated digital copy;

[1881] means for generating and delivering personalized content based on a user's memories and experiences;

[1882] A system including:

[1883] (Claim 2)

[1884] 10. The system of claim 1, including means for the digital copy of the user to continue to learn and train based on new interactions.

[1885] (Claim 3)

[1886] 10. The system of claim 1, wherein the digital copy of the user, when responding to an interaction request, includes means for generating an appropriate response based on specific context or keywords.

[1887] "Example 2: Combining Emotion Engines"

[1888] (Claim 1)

[1889] means for recording user-provided voice data;

[1890] means for transmitting the recorded voice data to a server;

[1891] A means for converting the received voice data into text by the server;

[1892] A means for analyzing the converted text data and extracting keywords and context;

[1893] A means for recognizing emotions based on the analyzed text data and voice data;

[1894] means for storing the recognized emotion data, keywords and context in a per-user database;

[1895] means for generating a digital copy of the user using the stored data;

[1896] means for responding to interaction requests using the generated digital copy;

[1897] A system including:

[1898] (Claim 2)

[1899] 10. The system of claim 1, wherein the digital copy of the user includes means for continuously learning based on new interaction data to form a faithful personality model.

[1900] (Claim 3)

[1901] 10. The system of claim 1, wherein the digital copy of the user, when responding to an interaction request, includes means for generating an appropriate response based on the recognized emotion data and specific context and keywords.

[1902] "Application example 2 when combining emotion engines"

[1903] (Claim 1)

[1904] means for recording user-provided voice data;

[1905] means for transmitting the recorded voice data to a server;

[1906] A means for converting the received voice data into text by the server;

[1907] A means for analyzing the converted text data and extracting keywords and context;

[1908] a means for storing the extracted data in a per-user database;

[1909] means for generating a digital copy of the user using the stored data;

[1910] In addition to means for responding to interaction requests using the generated digital copy,

[1911] means for recognizing emotions from voice data and text data and generating security information based thereon;

[1912] A system that includes a means of collecting audio data from security targets, analyzing it in real time, detecting abnormal emotions or behavior, and notifying security personnel.

[1913] (Claim 2)

[1914] In addition to the means by which a digital copy of the user continues to learn and train based on new interactions.

[1915] 2. The system according to claim 1, further comprising means for learning the behavioral patterns of the security target and detecting any unusual behavior.

[1916] (Claim 3)

[1917] In addition to the means by which a digital copy of the user generates appropriate responses based on specific context and keywords when responding to interaction requests.

[1918] 2. The system according to claim 1, further comprising means for generating security information based on emotion data and notifying security personnel in real time. [Explanation of symbols]

[1919] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for recording user-provided voice data; means for transmitting the recorded voice data to a server; A means for converting the received voice data into text by the server; A means for analyzing the converted text data and extracting keywords and context; a means for storing the extracted data in a per-user database; means for generating a digital copy of the user using the stored data; means for responding to interaction requests using the generated digital copy; A system including:

2. 10. The system of claim 1, further comprising means for the digital copy of the user to continue to learn and train based on new interactions.

3. 10. The system of claim 1, wherein the digital copy of the user, when responding to an interaction request, includes means for generating appropriate responses based on specific context and keywords.

Images