System

Abstract

A system is provided.SOLUTION: The system includes a means for transmitting a message in a specific event by using an avatar generated on the basis of the data of the deceased, a means for inputting event information and storing the information, and a means for generating the avatar on the basis of the event date and displaying it.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, opportunities to share memories with deceased family and friends have decreased due to busy lives and physical distance between family and friends. Furthermore, without direct expressions of will or blessings from the deceased on special events or anniversaries, the presence of the deceased is diminished, and ties with loved ones tend to be severed. There is a need for a way to solve these problems and continue to make the presence of the deceased felt. [Means for solving the problem]

[0005] The present invention provides a system for transmitting messages at specific events using avatars generated based on data on deceased people. Specifically, the problem is solved by the following means.

[0006] 1. A means of entering and saving event information.

[0007] 2. A means to generate and display an avatar based on the event date and time.

[0008] 3. A means for avatars to send messages to attendees of memorial services during the event.

[0009] 4. A means of notifying users to light incense on a specific day.

[0010] 5. A means for the avatar to generate and display a thank you message in response to the notification.

[0011] 6. A means of sending congratulatory messages to the user's grandchildren or great-grandchildren on specific anniversaries and awarding points at the same time.

[0012] 7. A means to spawn an avatar based on that event and send that message.

[0013] This allows you to maintain the presence of the deceased while directly conveying your thoughts and blessings to them on special days and events.

[0014] "Data of the deceased" refers to the deceased's voice, images, text messages, and other personal information from their lifetime, and is the basis for the generating AI.

[0015] "Generative AI" is a program that uses artificial intelligence technology to generate the voice, image, and message of the deceased, creating an avatar of the deceased.

[0016] An "avatar" is a virtual image generated by generative AI based on the data of the deceased, and serves as a medium for displaying messages from the deceased.

[0017] "Event" refers to special occasions or anniversaries related to the deceased, such as memorial services, Obon, New Year's, birthdays, entrance ceremonies, and graduation ceremonies.

[0018] A "message" is a communication such as a message of thanks or congratulations sent by the avatar of the deceased person at a specific event.

[0019] A "user" is a relative, friend, acquaintance, etc. related to the deceased, who receives the deceased's avatar and messages through the system.

[0020] "Terminal" refers to a device used by a user to access the system, including a smartphone, tablet, or PC.

[0021] The "server" is the central computer of the system that stores data on the deceased and runs the generative AI, processing and managing various data.

[0022] "Points" are rewards provided to users by the system and are given to users in the form of digital currency or electronic money. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] This invention is a system that uses an avatar generated based on the data of the deceased to send a message from the deceased at a specific event. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, and a server are used.

[0045] Overall system configuration

[0046] The system mainly consists of the following components:

[0047] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[0048] 2. Server: The central computer that manages the database and the AI ​​generation. It stores event information, generates avatars, generates messages, and awards points.

[0049] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[0050] 4. Database: Storage for storing data of deceased people, user information, and event information.

[0051] Program processing overview

[0052] The program processing in this system is explained as follows.

[0053] Event support function for memorial services, etc.

[0054] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and when the date and time of the memorial service approaches, it requests the AI ​​to generate an avatar of the deceased. On the day of the memorial service, the user's device displays this avatar and a message, conveying the deceased's words of gratitude to attendees.

[0055] Functions for Obon and New Year holidays

[0056] The user device notifies the server that they will light incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user device. The user device then displays the message and avatar, conveying the deceased's gratitude.

[0057] Blessing function at the time of growth

[0058] The server manages the anniversaries of the growth of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device and simultaneously awards points. The user's device displays the congratulatory message and a notification that points have been awarded.

[0059] Specific examples

[0060] Example 1: Responses at memorial services

[0061] 1. The user device enters the date and time of the memorial service and sends it to the server.

[0062] 2. The server stores the information in a database.

[0063] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[0064] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[0065] 5. The user device plays the message "Thank you for coming to the memorial service today."

[0066] Example 2: How to handle visits during Obon and New Year's

[0067] 1. The user device notifies the server when it is time to light the incense.

[0068] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[0069] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[0070] 4. The server sends the message and avatar to the user's device.

[0071] 5. The user device displays the message "Thank you for visiting during Obon."

[0072] Example 3: Grandchild's birthday celebration

[0073] 1. The server checks the date of birth from the database.

[0074] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[0075] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[0076] 4. The user's device displays the message "Happy Birthday" and an avatar.

[0077] 5. The server awards points to the user and notifies them on the device.

[0078] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[0079] The processing flow will be explained below.

[0080] Event support function for memorial services, etc.

[0081] Step 1:

[0082] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[0083] Step 2:

[0084] The terminal transmits the entered date and time information of the memorial service to the server.

[0085] Step 3:

[0086] The server stores the received date and time information in a database.

[0087] Step 4:

[0088] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[0089] Step 5:

[0090] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[0091] Step 6:

[0092] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[0093] Step 7:

[0094] The server sends the generated avatar and message data to the device.

[0095] Step 8:

[0096] The device displays the generated avatar and message and plays it to the attendees.

[0097] Functions for Obon and New Year holidays

[0098] Step 1:

[0099] The user launches the app on their device and enters the information when lighting the incense.

[0100] Step 2:

[0101] The terminal transmits the input information to the server.

[0102] Step 3:

[0103] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[0104] Step 4:

[0105] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[0106] Step 5:

[0107] The server sends the generated message and avatar to the device.

[0108] Step 6:

[0109] The device displays the generated thank you message and avatar to notify the user.

[0110] Blessing function at the time of growth

[0111] Step 1:

[0112] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[0113] Step 2:

[0114] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[0115] Step 3:

[0116] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[0117] Step 4:

[0118] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[0119] Step 5:

[0120] The device displays the generated congratulatory message and avatar to notify the user.

[0121] Step 6:

[0122] At the same time, the server performs a point awarding process and awards points to the user.

[0123] Step 7:

[0124] The terminal notifies the user of the point allocation notice so that the user can check it.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] The present invention aims to deepen the bond with the deceased by sending messages from the deceased at specific events using avatars generated based on the deceased's data. However, current systems lack a mechanism for dynamically generating messages and avatars for specific dates or events, and may not meet user expectations. Furthermore, there is a need for an efficient system that can accommodate multiple events and anniversaries and send messages from the deceased in a timely manner.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for sending a prompt to a generative AI model based on the event date and time to generate an avatar and a message, and a means for sending and displaying the generated avatar and message to a user terminal, thereby enabling messages and avatars from the deceased to be generated and sent in a timely manner and provided to users.

[0130] "Data of the deceased" refers to personal information and recorded data relating to the deceased, including the deceased's voice, images, messages, etc.

[0131] An "avatar" is a digital character or image created based on the data of a deceased person.

[0132] An "event" is an event or ceremony that takes place on a specific day or time, and includes memorial services, anniversaries, Obon, New Year's, etc.

[0133] A "message" is a verbal message in the form of text, audio, video, etc., generated based on the data of the deceased.

[0134] A "generative AI model" is a model that uses artificial intelligence to generate digital content such as text, images, and audio. In this invention, it generates an avatar and message based on data about the deceased.

[0135] A "prompt" is an instruction that is input to a generative AI model to generate specific content.

[0136] "User device" refers to a digital device such as a smartphone or computer used by a user, and is used to input event information and receive and display avatars and messages.

[0137] The "server" is a central control device that manages the database, controls the generative AI model, and requests and sends messages and avatars.

[0138] "Event information" refers to information such as the date, time, and content related to a specific event, which is input by the user and sent to the server.

[0139] "Means for storing" refers to the function of storing input event information and data on the deceased in storage such as a database.

[0140] "Display means" refers to a function for visually displaying the generated avatar and message on the screen of the user terminal.

[0141] This invention is a system that uses an avatar generated based on the data of the deceased to send messages from the deceased at specific events. To implement this system, hardware and software such as a user terminal, a server, a generative AI model, and a database are used.

[0142] Hardware and software used

[0143] 1. User Device

[0144] A device that allows users to enter event information and receive messages. Examples include smartphones and PCs.

[0145] 2. Server

[0146] A central control unit that manages the entire system and links with the database and generative AI model.

[0147] 3. Generative AI Models

[0148] An artificial intelligence model that generates an avatar and message based on data about the deceased. A specific example would be OpenAI GPT-3.

[0149] 4. Database

[0150] A storage system that stores data on deceased people, user information, and event information. Examples include MySQL and PostgreSQL.

[0151] System configuration and processing overview

[0152] Event support function for memorial services, etc.

[0153] The user enters the date and time of the memorial service through the user's device and sends it to the server. The server stores this information in a database and, the day before the memorial service, sends a prompt to the generative AI model to generate an avatar of the deceased. An example of a prompt is: "Please generate a message of thanks for the memorial service in the voice of the deceased." On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays it.

[0154] Functions for Obon and New Year holidays

[0155] The user uses their device to notify the server that they will light incense during Obon or New Year's visits. The server then sends a prompt to the generative AI model, which generates a message of gratitude and an avatar. An example of a prompt is: "Please generate a message of gratitude during Obon in the voice of the deceased." The generated avatar and message are then sent from the server to the user's device, where they are displayed and played back on the device.

[0156] Blessing function at the time of growth

[0157] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, it sends a prompt to the generative AI model, which generates a congratulatory message and avatar. An example of a prompt is: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased." On the anniversary day, the server sends the generated avatar and message to the user's device, and also notifies the user of the points awarded. The user's device displays and plays the message, and notifies the user of the points.

[0158] Specific examples

[0159] Example 1: Responses at memorial services

[0160] 1. The user opens the smartphone app and enters the date of the memorial service.

[0161] 2. The device sends the information to the server, which stores it in a database.

[0162] 3. The server sends a prompt to the generative AI model the day before the memorial service, generating an avatar and message.

[0163] 4. On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays them.

[0164] Example 2: How to handle visits during Obon and New Year's

[0165] 1. Notify the user when lighting incense.

[0166] 2. The server sends a prompt to the generative AI model, which generates a thank you message and an avatar.

[0167] 3. The generative AI model sends the avatar and message back to the server, which then sends it to the user's device.

[0168] 4. The user device displays and plays the message and avatar.

[0169] Example 3: Grandchild's birthday celebration

[0170] 1. The server checks the date of birth from the database.

[0171] 2. On the day before the anniversary, send a prompt to the generative AI model to generate a congratulatory message and avatar.

[0172] 3. On the anniversary day, the server sends the generated avatar and a message to the user's device.

[0173] 4. The user's device displays and plays the message and avatar, and also notifies them of points.

[0174] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[0175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0176] Processing steps for event response functions such as memorial services

[0177] Step 1: Enter your event information

[0178] The user inputs the date and time of the memorial service into the user terminal and presses the "send" button.

[0179] Input: Date and time of the memorial service.

[0180] Output: Data sent to the server.

[0181] Specific operation: The user enters the date of the memorial service (October 31, 2023) and the time (10:00) on the application screen and clicks the send button.

[0182] Step 2: Save your data

[0183] The server stores the date and time of the received memorial service in a database.

[0184] Input: Date and time data for the memorial service.

[0185] Output: Event information stored in a database.

[0186] Specific operation: The date "October 31, 2023" and time "10:00" received by the server are inserted into the event information table in the database.

[0187] Step 3: Request avatar creation

[0188] The day before the memorial service, the server requests the generative AI model to generate an avatar of the deceased person, using a prompt text.

[0189] Input: prompt statement.

[0190] Output: A request to the generative AI model to generate a new model.

[0191] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of gratitude for the memorial service in the voice of the deceased."

[0192] Step 4: Receiving avatars and messages

[0193] The generative AI model generates an avatar and message based on the request and sends it back to the server.

[0194] Input: prompt statement.

[0195] Output: The generated avatar and message.

[0196] Specific operation: The generative AI model generates an avatar that says, "Thank you for coming to the memorial service today," along with a message, and sends it back to the server.

[0197] Step 5: Message and Avatar Notifications

[0198] On the day of the memorial service, the server sends the generated avatar and a message to the user's terminal.

[0199] Input: The generated avatar and message.

[0200] Output: Data sent to the user's terminal.

[0201] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for coming to the memorial service today."

[0202] Step 6: Display your avatar and message

[0203] The user terminal displays and plays the avatar and message received from the server.

[0204] Input: Avatar and message sent from the server.

[0205] Output: User notification and display.

[0206] Specific operation: The user's device displays an avatar on the application and plays a message saying, "Thank you for coming to the memorial service today."

[0207] Processing steps for Obon and New Year's holiday support functions

[0208] Step 1: Incense Notification

[0209] A user uses a user terminal to notify that incense will be lit when visiting during Obon or New Year's.

[0210] Input: Notification of incense being lit.

[0211] Output: Notification data to the server.

[0212] Specific operation: The user clicks the "Light Incense" button in the application, notifying the server.

[0213] Step 2: Request avatar creation

[0214] The server sends a message of thanks and requests the AI ​​model to generate an avatar. A prompt is used.

[0215] Input: prompt statement.

[0216] Output: A request to the generative AI model to generate a new model.

[0217] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of thanks for Obon in the voice of the deceased."

[0218] Step 3: Receiving avatars and messages

[0219] The generative AI model generates a thank you message and avatar and sends them back to the server.

[0220] Input: prompt statement.

[0221] Output: The generated avatar and message.

[0222] Specific operation: The generative AI model generates an avatar that says, "Thank you for visiting during Obon," along with a message, and sends it back to the server.

[0223] Step 4: Message and Avatar Notifications

[0224] The server sends the generated message and avatar to the user terminal.

[0225] Input: The generated avatar and message.

[0226] Output: Data sent to the user's terminal.

[0227] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for visiting during Obon."

[0228] Step 5: Displaying your message and avatar

[0229] The user terminal displays it and plays back the message "Thank you for visiting during Obon."

[0230] Input: Avatar and message sent from the server.

[0231] Output: User notification and display.

[0232] Specific operation: The user's device displays an avatar on the application and plays the message "Thank you for visiting during Obon."

[0233] Processing steps for the blessing function during growth

[0234] Step 1: Manage your anniversaries

[0235] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database.

[0236] Input: Anniversary data in the database.

[0237] Output: Anniversary check result.

[0238] What it does: Every day the server checks the database to see if an anniversary is approaching.

[0239] Step 2: Request avatar creation

[0240] When an anniversary approaches, the server asks the generative AI model to generate a congratulatory message and an avatar, using a prompt text.

[0241] Input: prompt statement.

[0242] Output: A request to the generative AI model to generate a new model.

[0243] Specific operation: The server sends a prompt to the generation AI model: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased."

[0244] Step 3: Receiving avatars and messages

[0245] The generative AI model generates a congratulatory message and avatar and sends them back to the server.

[0246] Input: prompt statement.

[0247] Output: The generated avatar and message.

[0248] Specific operation: The generative AI model generates an avatar that says "Happy Birthday" and the message, and sends it back to the server.

[0249] Step 4: Message and Avatar Notifications

[0250] On the anniversary day, the server sends the generated message and avatar to the user's terminal.

[0251] Input: The generated avatar and message.

[0252] Output: Data sent to the user's terminal.

[0253] Specific operation: The server sends the generated avatar and a message saying "Happy Birthday" to the user's device.

[0254] Step 5: Points awarded

[0255] The server awards points to the user and sends a notification to the user's terminal.

[0256] Input: User information and point data.

[0257] Output: Points awarded notification.

[0258] Specific operation: The server will add 100 points to the user's account and send a notification to the user's device.

[0259] Step 6: Displaying your message and avatar

[0260] The user terminal displays the generated avatar and message, and plays back the message "Happy Birthday."

[0261] Input: Avatar and message sent from the server.

[0262] Output: User notification and display.

[0263] Specific operation: The user's device displays an avatar on the application and plays the message "Happy Birthday."

[0264] (Application example 1)

[0265] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0266] Previous systems that allowed users to relive memories with the deceased were limited to specific events and dates, limiting the opportunities for users to feel a connection with the deceased in their daily lives. Furthermore, they were inadequate in supporting visits other than memorial services and specific days, or memorable places and items, and there was no system that allowed users to feel a deeper connection with the deceased.

[0267] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0268] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for generating an avatar based on the event date and time and displaying it, a means for the user to use a device that recognizes a specific place or product and transmit that information, and a means for using a device that displays the avatar and message generated based on data of the deceased, allowing the user to relive memories of the deceased in their daily lives and receive messages related to specific places or products.

[0269] A "deceased person" is someone who has already passed away and is recreated within the system based on data from their life.

[0270] "Data" refers to information about the deceased person during their lifetime, including text data, audio data, image data, and the like.

[0271] An "avatar" is a digital character generated based on the data of a deceased person, and its role is to display a message or image of the deceased to the user.

[0272] An "event" is an event or activity that takes place at a specific date, time, or place, and includes memorial services, anniversaries, and the like.

[0273] A "message" is a message or thank you that the deceased person's avatar conveys to the user.

[0274] "Means" are the tools and methods a system uses to achieve its goals.

[0275] A "user" is an individual who uses the system and operates it to relive memories of a deceased loved one.

[0276] "Device" is a general term for the hardware required for users to use the system, including smart glasses and smartphones.

[0277] A "server" is a computer that manages and processes data at the center of the system, and is where the generative AI and database run.

[0278] "Generative AI" is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[0279] A "database" is a storage system for storing data on deceased people, user information, event information, etc.

[0280] The system for implementing this invention comprises the following components: a user terminal, a server, a generating AI, a database, smart glasses, and, if necessary, a recognition device (such as a camera).

[0281] Overall system flow

[0282] 1. User Device:

[0283] Users use smart glasses or a smartphone to scan a specific location or product.

[0284] Used to enter event information into the system.

[0285] 2. Server:

[0286] Responsible for managing and processing data.

[0287] Event information is stored in a database, and avatars and messages are generated through generative AI.

[0288] 3. Generation AI:

[0289] It is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[0290] OpenAI's GPT-4 is used as a concrete example.

[0291] 4. Database:

[0292] A storage system for storing data on deceased individuals, user information, and event information. A database management system such as MySQL is used.

[0293] 5. Smart Glasses:

[0294] A device used by a user to recognize a specific place or product, such as Google Glass.

[0295] The deceased person's avatar and message are displayed in the user's field of view.

[0296] Specific processing of the program

[0297] 1. Data Collection:

[0298] The user puts on the smart glasses and scans memorable places and products.

[0299] 2. Data transmission:

[0300] The scanned data is sent from the user device to the server.

[0301] 3. Data Retention:

[0302] The server stores the received data in a database.

[0303] 4. Avatar and message generation:

[0304] A generation AI on the server generates avatars and messages based on the data.

[0305] GPT-4 and other AIs are used as generative AI.

[0306] 5. Displaying the results:

[0307] The generated avatar and message are sent from the server to the smart glasses via the user terminal.

[0308] Smart glasses display avatars and messages in the user's field of vision.

[0309] Specific examples

[0310] For example, when a user visits a specific location in a restaurant that holds special memories for the deceased, they can scan the location with the smart glasses. The server receives the data and asks the AI ​​to generate an avatar of the deceased and create a message related to the location. A message from the deceased, such as "I had a wonderful time eating here," is displayed on the screen.

[0311] Prompt Sentence Examples

[0312] "A user visits a specific seat in a restaurant. Generate a thank you message based on memories of the deceased person being at that location."

[0313] In this way, users can receive messages from the deceased related to specific places or items, deepening their connection with the deceased as part of their daily lives.

[0314] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0315] Step 1:

[0316] Users scan memorable places and products

[0317] A user puts on the smart glasses and scans a specific location or item.

[0318] Input: The user visually recognizes a specific location or product through the camera on the smart glasses.

[0319] Output: Scanned data (image data, QR code, etc.) is generated and sent to the next step.

[0320] Step 2:

[0321] Sending scan data to the server

[0322] The user terminal transmits the scan data to the server.

[0323] Input: The scan data generated in step 1.

[0324] Output: The scanned data is sent to the server.

[0325] Step 3:

[0326] The server stores the scan data in a database

[0327] The server stores the received scan data in a database.

[0328] Input: Scan data sent to the server.

[0329] Output: The scan data is saved to the database and the result of the save success is returned.

[0330] Step 4:

[0331] The server asks the AI ​​to generate an avatar and a message.

[0332] The server asks the generation AI to generate an avatar and message based on the saved scan data.

[0333] Input: Scan data stored in the database.

[0334] Output: The prompt is sent to the generation AI, and a request is made to generate an avatar and message.

[0335] Step 5:

[0336] Generative AI generates avatars and messages

[0337] The generation AI generates an avatar and message based on the prompt text.

[0338] A specific example is the prompt: "The user visited a specific seat in a restaurant. Generate a thank-you message based on memories of the deceased person being at that location."

[0339] Input: The prompt sent to the generation AI.

[0340] Output: The generated avatar and message.

[0341] Step 6:

[0342] Send the generated avatar and message to the server

[0343] The server receives the generated avatar and message.

[0344] Input: Avatar and message returned by the generation AI.

[0345] Output: The avatar and message are stored on the server and ready to be sent to the user's device.

[0346] Step 7:

[0347] The server sends the avatar and message to the user's device.

[0348] The server sends the avatar and the message to the user's smart glasses.

[0349] Input: Avatar and message stored on the server.

[0350] Output: The avatar and message are transferred to the user's device and displayed on the user's device.

[0351] Step 8:

[0352] Users receive avatars and messages through smart glasses

[0353] The smart glasses display an avatar and a message in the user's field of vision.

[0354] Input: Avatar and message sent from the server.

[0355] Output: An avatar is displayed on the user's smart glasses and a message is read out loud.

[0356] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0357] This invention is a system that uses an avatar generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[0358] Overall system configuration

[0359] The system mainly consists of the following components:

[0360] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[0361] 2. Server: The central computer that manages the database, generation AI, and emotion engine. It stores event information, generates avatars, generates messages, recognizes emotions, and awards points.

[0362] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[0363] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[0364] 5. Database: Storage for storing data of deceased persons, user information, and event information.

[0365] Program processing overview

[0366] The program processing in this system is explained as follows.

[0367] Event support function for memorial services, etc.

[0368] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and as the date and time of the memorial service approaches, it requests the generation AI to generate an avatar of the deceased. On the day of the memorial service, the emotion engine recognizes the user's emotions in real time, and the avatar and message displayed will match those emotions. For example, if the user has a sad expression, the avatar will display an encouraging message such as, "Thank you for coming at this difficult time."

[0369] Functions for Obon and New Year holidays

[0370] The user's device notifies the server that they will be offering incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user's device. The emotion engine analyzes the user's emotions, and the avatar personalizes the thank-you message. For example, if the user has a happy expression, it will display a message such as, "Thank you for coming to visit us. We're happy to see your smile."

[0371] Blessing function at the time of growth

[0372] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device, and at the same time, it personalizes the message using an emotion engine. For example, if the user is emotional, it displays an emotional message such as "Happy birthday. I'm so happy to have been able to watch you grow up." The server also performs point allocation processing and awards points to the user.

[0373] Specific examples

[0374] Example 1: Responses at memorial services

[0375] 1. The user device enters the date and time of the memorial service and sends it to the server.

[0376] 2. The server stores the information in a database.

[0377] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[0378] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[0379] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[0380] 6. The user's device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging message such as, "Thank you for coming during this difficult time."

[0381] Example 2: How to handle visits during Obon and New Year's

[0382] 1. The user device notifies the server when it is time to light the incense.

[0383] 2. The server sends a thank you message and asks the AI ​​to generate an avatar.

[0384] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[0385] 4. The server sends the message and avatar to the user's device.

[0386] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[0387] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message changes to "Thank you for visiting. I'm happy to see your smile."

[0388] Example 3: Grandchild's birthday celebration

[0389] 1. The server checks the date of birth from the database.

[0390] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[0391] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[0392] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[0393] 5. The user's device displays the message "Happy Birthday" and an avatar. If the user is touched, the message changes to "Happy Birthday. I'm so happy to watch you grow up."

[0394] 6. The server awards points to the user and notifies them on the device.

[0395] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[0396] The processing flow will be explained below.

[0397] Event support function for memorial services, etc.

[0398] Step 1:

[0399] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[0400] Step 2:

[0401] The terminal transmits the entered date and time information of the memorial service to the server.

[0402] Step 3:

[0403] The server stores the received date and time information in a database.

[0404] Step 4:

[0405] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[0406] Step 5:

[0407] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[0408] Step 6:

[0409] The server sends the generated avatar and message data to the device.

[0410] Step 7:

[0411] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[0412] Step 8:

[0413] The device will start displaying your avatar and message.

[0414] Step 9:

[0415] The emotion engine recognizes the user's emotions in real time.

[0416] Step 10:

[0417] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[0418] Step 11:

[0419] The device displays a personalized message and plays it to attendees.

[0420] Functions for Obon and New Year holidays

[0421] Step 1:

[0422] The user launches the app on their device and enters the information when lighting the incense.

[0423] Step 2:

[0424] The terminal transmits the input information to the server.

[0425] Step 3:

[0426] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[0427] Step 4:

[0428] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[0429] Step 5:

[0430] The server sends the generated message and avatar to the device.

[0431] Step 6:

[0432] The device will begin displaying a generated thank you message and avatar.

[0433] Step 7:

[0434] The emotion engine analyzes the user's emotions.

[0435] Step 8:

[0436] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[0437] Step 9:

[0438] The device will notify the user by displaying a personalized thank you message and avatar.

[0439] Blessing function at the time of growth

[0440] Step 1:

[0441] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[0442] Step 2:

[0443] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[0444] Step 3:

[0445] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[0446] Step 4:

[0447] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[0448] Step 5:

[0449] The terminal will begin displaying a congratulatory message and the generated avatar.

[0450] Step 6:

[0451] The emotion engine analyzes the user's emotions.

[0452] Step 7:

[0453] The emotion engine personalizes messages based on the results of emotion analysis and sends them to the device.

[0454] Step 8:

[0455] The device will notify the user by displaying a personalized congratulatory message and avatar.

[0456] Step 9:

[0457] The server performs a point giving process and gives points to the user.

[0458] Step 10:

[0459] The terminal notifies the user of the point allocation notice so that the user can check it.

[0460] Example 2

[0461] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0462] Conventional memorial events lacked a means for users to feel a sense of dialogue with the deceased, making it difficult to deepen an emotional connection. Furthermore, there was no way to provide messages tailored to the user's emotions at specific events or occasions. This prevented users from feeling closer to the deceased, making it difficult for them to empathize or feel comforted. Furthermore, there was a lack of mechanisms for appropriately managing the timing of event notifications and the display of messages from the deceased.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0464] In this invention, the server includes a means for sending a message at a specific event using a model generated based on data of the deceased, a means for inputting event information and saving the information, a means for generating a model based on the event date and time and displaying it, a means for recognizing the user's emotions in real time and personalizing the message based on the recognition results, and a means for transmitting and displaying the generated message and model to a user terminal. This allows the user to feel a sense of dialogue with the deceased and deepen their emotional connection. Furthermore, by providing a personalized message tailored to the user's emotions, the user can gain empathy and comfort. Furthermore, the timing of event notifications and message display can be appropriately managed.

[0465] A "model" is a virtual character or message generation program generated based on data about the deceased.

[0466] "Events" are events or ceremonies that are held on specific dates or times, such as memorial services, Obon, and New Year's.

[0467] A "message" is information such as text, audio, or video sent from the deceased to the user.

[0468] A "user terminal" is a device used by a user, such as a smartphone or a personal computer.

[0469] The "server" is a central computer that manages and operates the database, generative AI, and emotion engine.

[0470] The "input means" is an interface through which the user inputs event information and transmits it to the server.

[0471] "Storage means" is a function that stores input event information in a database.

[0472] "Display means" is a function that displays the generated model and messages on the user terminal.

[0473] "Emotion recognition means" is a technology for analyzing a user's emotions in real time and obtaining the results.

[0474] "Personalization means" is a function that individually optimizes messages based on acquired emotional information.

[0475] A "notification means" is a communication means for reporting specific actions or events from a user terminal to a server.

[0476] The "point giving means" is a function for giving points to a user and notifying the user.

[0477] The present invention is a system that uses a model generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, a model, an event information input means, a model display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[0478] Overall system configuration

[0479] The system mainly consists of the following components:

[0480] 1. User terminal: A device such as a smartphone or PC that users use to enter event information and receive messages.

[0481] 2. Server: The central computer that manages the database, generative AI, and emotion engine. It stores event information, generates models, generates messages, recognizes emotions, and awards points.

[0482] 3. Generative AI: An artificial intelligence program that generates models and messages based on data about the deceased.

[0483] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[0484] 5. Database: Storage for storing data of the deceased, user information, and event information.

[0485] Program processing overview

[0486] The program processing in this system is explained as follows.

[0487] Functions for handling events such as memorial services

[0488] The user enters the date and time of the memorial service into their device, which then sends the information to the server. The server stores the received information in a database. The day before the memorial service, the server requests the generative AI to generate a model of the deceased. On the day of the memorial service, when the user opens the app, the server sends the model and a message to the user's device and displays them. The emotion engine recognizes the user's emotions in real time and personalizes the message accordingly. For example, if the user has a sad expression, the model will display an encouraging message such as, "Thank you for coming at this difficult time."

[0489] Functions for Obon and New Year holidays

[0490] When a user lights incense during Obon or New Year's, the information is sent from the user's device to the server. Based on this information, the server requests the generation AI to generate a message of gratitude and a model. The generated data is sent from the server to the user's device. The emotion engine analyzes the user's emotions and personalizes the message. For example, if the user has a happy expression, a message such as "Thank you for coming to visit us. We're happy to see your smile" is displayed.

[0491] Blessing function at the time of growth

[0492] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, the server requests the generation AI to generate a congratulatory message and model. On the anniversary day, the server sends the generated message and model to the user's device, and the emotion engine analyzes the user's emotions and personalizes the message. For example, if the user is emotional, a message such as "Happy birthday. I'm so happy to have watched you grow up" is displayed. The server also performs the process of awarding points and notifies the user of the points.

[0493] Specific examples

[0494] Functions for handling events such as memorial services

[0495] Examples:

[0496] 1. The user opens the app on their smartphone and enters the date of the memorial service (December 25, 2023) and the time (10:00).

[0497] 2. The user device sends the data "Memorial date: December 25, 2023, time: 10:00" to the server.

[0498] 3. The server stores the received information in a database.

[0499] 4. On December 24, 2023, the server asks the generation AI to "generate a model for a memorial service."

[0500] 5. When the user opens the app on the day of the memorial service, the server sends the model and a message.

[0501] 6. The emotion engine analyzes the user's sad expression and generates a personalized message.

[0502] Prompt Sentence Examples

[0503] Here are some example prompts to input to a generative AI model:

[0504] "The memorial date is approaching. Please generate a model of the deceased and an appropriate message. Include an encouraging message if the user has a sad expression."

[0505] This allows users to feel like they are having a conversation with the deceased, deepening their emotional connection. Furthermore, by providing personalized messages tailored to the user's emotions, it is possible to gain empathy and comfort. Furthermore, it is possible to appropriately manage the timing of event notifications and message display.

[0506] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0507] Processing steps for events such as memorial services

[0508] Step 1:

[0509] The user enters the date and time of the memorial service.

[0510] Input: Date and time of memorial service (e.g. December 25, 2023, 10:00).

[0511] Output: The input information is saved on the user's terminal.

[0512] Specific operation: The user opens a dedicated app on their smartphone and enters the date and time of the memorial service in the form that appears.

[0513] Step 2:

[0514] The user terminal transmits information about the memorial service to the server.

[0515] Input: Date and time of memorial service.

[0516] Output: Memorial service information is sent to the server.

[0517] Specific operation: The date and time of the memorial service entered by the user is sent to the server as an HTTP request.

[0518] Step 3:

[0519] The server stores the memorial information in a database.

[0520] Input: Date and time of memorial service.

[0521] Output: Memorial information stored in a database.

[0522] Specific operation: The server saves the date and time of the memorial service received as a new record in a database table.

[0523] Step 4:

[0524] The day before the memorial service, the server requests the generation AI to generate a model.

[0525] Input: Trigger based on date and time before and after the memorial service, memorial service information.

[0526] Output: The generated model and messages.

[0527] Specific operation: The server uses a trigger set on the day before the memorial service to send a prompt message to the generation AI saying, "Please generate a model for the memorial service."

[0528] Step 5:

[0529] On the day of the memorial service, the user opens the application.

[0530] Input: User action.

[0531] Output: The application sends a request to the server.

[0532] Specific action: The user launches the app on their smartphone.

[0533] Step 6:

[0534] The server sends the avatar and message to the user's terminal.

[0535] Input: The generated avatar and message.

[0536] Output: The avatar and message displayed on the user's device.

[0537] Specific operation: The server sends the avatar and message received from the generated AI to the user's device in JSON format.

[0538] Step 7:

[0539] The emotion engine recognizes the user's emotions.

[0540] Input: User's facial expression data.

[0541] Output: Recognized emotion data (e.g. sadness, joy).

[0542] Specific operation: Using the camera on the user's device, the emotion engine analyzes the user's facial expressions in real time.

[0543] Step 8:

[0544] The server personalizes the message and displays it on the user's terminal.

[0545] Input: Recognized emotion data, generated message.

[0546] Output: A personalized message.

[0547] Specific operation: The server modifies the message based on the emotion data and sends it to the user's device. If the user has a sad expression, an encouraging message such as "Thank you for coming during this difficult time" will be displayed.

[0548] (Application example 2)

[0549] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0550] Conventional dialogue systems for the deceased only send messages using an avatar generated based on the deceased's data, but are unable to provide personalized messages based on the user's emotions. Furthermore, it is difficult to provide an emotionally rich experience for specific events. Therefore, there is a need for a system that allows users to empathize and feel deeply moved.

[0551] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0552] In this invention, the server includes means for sending a message at a specific event using an avatar generated based on data of the deceased, means for inputting and saving event information, means for generating an avatar based on the event date and time and displaying it, and means for recognizing a user's emotions in real time and personalizing a message based on the emotions, thereby making it possible to recognize a user's emotions and provide a personalized message.

[0553] "Data of the deceased" refers to information about a deceased person, including digital data such as images, audio, video, and text.

[0554] An "avatar" is a digital image created based on the data of a deceased person, recreating the deceased in a virtual space.

[0555] "Specific events" refers to special occasions or anniversaries related to the deceased, including memorial services, birthdays, Obon, New Year's, and other regular events.

[0556] The "means for sending a message" is a method by which the avatar can provide a text or voice message to the user depending on the date and situation of the event.

[0557] "Event information" is detailed data about a particular event, such as the date, time, and location.

[0558] "Storage means" refers to the methods and technologies used to store event information and other related data in a database or the like.

[0559] "Display means" refers to a technique for visually displaying the generated avatar or message on the user's terminal.

[0560] "Means for recognizing a user's emotions in real time and personalizing messages based on those emotions" refers to a method that uses emotion recognition technology to determine a user's current emotional state and appropriately change the message content accordingly.

[0561] A "personalized message" is a text or voice message that is individually tailored to the user's emotions and situation.

[0562] This invention is a system that uses an avatar generated based on the data of a deceased person to send a message at a specific event. By combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. Details of the embodiment are described below.

[0563] 1. System Configuration

[0564] The system mainly consists of the following components:

[0565] User device: A device such as a smartphone or computer that users use to enter event information and receive messages.

[0566] Server: A central computer that manages the database, generative AI, and emotion engine. It stores event information, generates avatars, generates messages, and recognizes emotions.

[0567] Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[0568] Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[0569] Database: Storage for storing data of deceased persons, user information, and event information.

[0570] 2. Program Processing Overview

[0571] The user enters event information on the user device and sends it to the server, which stores it in a database. As the event date approaches, the server requests the generation AI to generate an avatar of the deceased person. On the day of the event, the emotion engine recognizes the user's emotions in real time, and the avatar and message are displayed according to those emotions.

[0572] 3. Hardware and Software Used

[0573] Hardware:

[0574] Smartphone

[0575] server

[0576] software:

[0577] EmotionEngine: Software for recognizing user emotions.

[0578] Avatar Generator (generative AI): Software that generates avatars based on data of deceased people.

[0579] Message Generator (Generative AI): Software that generates messages based on data about the deceased and event information.

[0580] Database: Database management software for storing user and event information.

[0581] 4. Specific Examples

[0582] ■ Responses to memorial services:

[0583] 1. The user terminal enters the date and time of the memorial service and sends it to the server.

[0584] 2. The server stores the information in a database.

[0585] 3. The day before the memorial service, the server requests the generation AI to generate an avatar.

[0586] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[0587] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[0588] 6. The user device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging one such as, "Thank you for coming during this difficult time."

[0589] ■ Visits during Obon and New Year's:

[0590] 1. The user device notifies the server when it is time to light the incense.

[0591] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[0592] 3. The generation AI generates a thank you message and an avatar and sends them back to the server.

[0593] 4. The server sends the message and avatar to the user's device.

[0594] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[0595] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message is changed to "Thank you for visiting. I'm happy to see your smile."

[0596] ■ Grandchild's birthday celebration:

[0597] 1. The server checks the date of birth from the database.

[0598] 2. On the day before the anniversary, ask the generation AI to generate a congratulatory message and an avatar.

[0599] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[0600] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[0601] 5. The user device displays the message "Happy Birthday" and the avatar. If the user is touched, the message changes to "Happy Birthday. I'm so glad to watch you grow up."

[0602] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[0603] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0604] Step 1:

[0605] The user enters event information (e.g., the date and time of the memorial service) into the terminal and sends it to the server.

[0606] Input: Event information (date, time)

[0607] Data processing: User terminal packages event information

[0608] Output: Request sent to server

[0609] Step 2:

[0610] The server stores the received event information in a database.

[0611] Input: Received event information

[0612] Data processing: Converting event information into the appropriate format

[0613] Output: Save results to database

[0614] Step 3:

[0615] As the event date approaches, the server asks the generation AI to generate an avatar of the deceased person.

[0616] Input: Event information, deceased person's data

[0617] Data processing: Generative AI analyzes the data of the deceased and generates an avatar

[0618] Output: The generated avatar

[0619] Step 4:

[0620] On the day of the event, the server sends an avatar and a message to the user's terminal.

[0621] Input: Generated avatar, standard message

[0622] Data processing: Packaging avatars and messages

[0623] Output: Send request to user terminal

[0624] Step 5:

[0625] The user's device displays the received avatar and message.

[0626] Input: avatar, message

[0627] Data processing: Convert received data into a display format

[0628] Output: Display avatar and message on user's device

[0629] Step 6:

[0630] The emotion engine recognizes the user's emotions in real time.

[0631] Input: User's facial expression data (camera input)

[0632] Data calculation: Emotion engine analyzes facial expression data

[0633] Output: Recognized emotion data

[0634] Step 7:

[0635] The server personalizes the message based on the recognized emotion data.

[0636] Input: Standard message, recognized emotion data

[0637] Data processing: Message generation based on emotional data

[0638] Output: Personalized message

[0639] Step 8:

[0640] The user terminal displays the personalized message.

[0641] Input: Personalized message

[0642] Data processing: Convert received messages into a display format

[0643] Output: Display a personalized message on the user's device

[0644] Through the above steps, a personalized message according to emotions is provided based on the event information entered by the user.

[0645] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0646] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0647] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0648] [Second embodiment]

[0649] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0650] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0651] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0652] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0653] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0654] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0655] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0656] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0657] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0658] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0659] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0660] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0661] This invention is a system that uses an avatar generated based on the data of the deceased to send a message from the deceased at a specific event. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, and a server are used.

[0662] Overall system configuration

[0663] The system mainly consists of the following components:

[0664] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[0665] 2. Server: The central computer that manages the database and the AI ​​generation. It stores event information, generates avatars, generates messages, and awards points.

[0666] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[0667] 4. Database: Storage for storing data of deceased people, user information, and event information.

[0668] Program processing overview

[0669] The program processing in this system is explained as follows.

[0670] Event support function for memorial services, etc.

[0671] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and when the date and time of the memorial service approaches, it requests the AI ​​to generate an avatar of the deceased. On the day of the memorial service, the user's device displays this avatar and a message, conveying the deceased's words of gratitude to attendees.

[0672] Functions for Obon and New Year holidays

[0673] The user device notifies the server that they will light incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user device. The user device then displays the message and avatar, conveying the deceased's gratitude.

[0674] Blessing function at the time of growth

[0675] The server manages the anniversaries of the growth of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device and simultaneously awards points. The user's device displays the congratulatory message and a notification that points have been awarded.

[0676] Specific examples

[0677] Example 1: Responses at memorial services

[0678] 1. The user device enters the date and time of the memorial service and sends it to the server.

[0679] 2. The server stores the information in a database.

[0680] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[0681] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[0682] 5. The user device plays the message "Thank you for coming to the memorial service today."

[0683] Example 2: How to handle visits during Obon and New Year's

[0684] 1. The user device notifies the server when it is time to light the incense.

[0685] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[0686] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[0687] 4. The server sends the message and avatar to the user's device.

[0688] 5. The user device displays the message "Thank you for visiting during Obon."

[0689] Example 3: Grandchild's birthday celebration

[0690] 1. The server checks the date of birth from the database.

[0691] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[0692] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[0693] 4. The user's device displays the message "Happy Birthday" and an avatar.

[0694] 5. The server awards points to the user and notifies them on the device.

[0695] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[0696] The processing flow will be explained below.

[0697] Event support function for memorial services, etc.

[0698] Step 1:

[0699] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[0700] Step 2:

[0701] The terminal transmits the entered date and time information of the memorial service to the server.

[0702] Step 3:

[0703] The server stores the received date and time information in a database.

[0704] Step 4:

[0705] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[0706] Step 5:

[0707] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[0708] Step 6:

[0709] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[0710] Step 7:

[0711] The server sends the generated avatar and message data to the device.

[0712] Step 8:

[0713] The device displays the generated avatar and message and plays it to the attendees.

[0714] Functions for Obon and New Year holidays

[0715] Step 1:

[0716] The user launches the app on their device and enters the information when lighting the incense.

[0717] Step 2:

[0718] The terminal transmits the input information to the server.

[0719] Step 3:

[0720] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[0721] Step 4:

[0722] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[0723] Step 5:

[0724] The server sends the generated message and avatar to the device.

[0725] Step 6:

[0726] The device displays the generated thank you message and avatar to notify the user.

[0727] Blessing function at the time of growth

[0728] Step 1:

[0729] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[0730] Step 2:

[0731] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[0732] Step 3:

[0733] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[0734] Step 4:

[0735] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[0736] Step 5:

[0737] The device displays the generated congratulatory message and avatar to notify the user.

[0738] Step 6:

[0739] At the same time, the server performs a point awarding process and awards points to the user.

[0740] Step 7:

[0741] The terminal notifies the user of the point allocation notice so that the user can check it.

[0742] Example 1

[0743] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] The present invention aims to deepen the bond with the deceased by sending messages from the deceased at specific events using avatars generated based on the deceased's data. However, current systems lack a mechanism for dynamically generating messages and avatars for specific dates or events, and may not meet user expectations. Furthermore, there is a need for an efficient system that can accommodate multiple events and anniversaries and send messages from the deceased in a timely manner.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0746] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for sending a prompt to a generative AI model based on the event date and time to generate an avatar and a message, and a means for sending and displaying the generated avatar and message to a user terminal, thereby enabling messages and avatars from the deceased to be generated and sent in a timely manner and provided to users.

[0747] "Data of the deceased" refers to personal information and recorded data relating to the deceased, including the deceased's voice, images, messages, etc.

[0748] An "avatar" is a digital character or image created based on the data of a deceased person.

[0749] An "event" is an event or ceremony that takes place on a specific day or time, and includes memorial services, anniversaries, Obon, New Year's, etc.

[0750] A "message" is a verbal message in the form of text, audio, video, etc., generated based on the data of the deceased.

[0751] A "generative AI model" is a model that uses artificial intelligence to generate digital content such as text, images, and audio. In this invention, it generates an avatar and message based on data about the deceased.

[0752] A "prompt" is an instruction that is input to a generative AI model to generate specific content.

[0753] "User device" refers to a digital device such as a smartphone or computer used by a user, and is used to input event information and receive and display avatars and messages.

[0754] The "server" is a central control device that manages the database, controls the generative AI model, and requests and sends messages and avatars.

[0755] "Event information" refers to information such as the date, time, and content related to a specific event, which is input by the user and sent to the server.

[0756] "Means for storing" refers to the function of storing input event information and data on the deceased in storage such as a database.

[0757] "Display means" refers to a function for visually displaying the generated avatar and message on the screen of the user terminal.

[0758] This invention is a system that uses an avatar generated based on the data of the deceased to send messages from the deceased at specific events. To implement this system, hardware and software such as a user terminal, a server, a generative AI model, and a database are used.

[0759] Hardware and software used

[0760] 1. User Device

[0761] A device that allows users to enter event information and receive messages. Examples include smartphones and PCs.

[0762] 2. Server

[0763] A central control unit that manages the entire system and links with the database and generative AI model.

[0764] 3. Generative AI Models

[0765] An artificial intelligence model that generates an avatar and message based on data about the deceased. A specific example would be OpenAI GPT-3.

[0766] 4. Database

[0767] A storage system that stores data on deceased people, user information, and event information. Examples include MySQL and PostgreSQL.

[0768] System configuration and processing overview

[0769] Event support function for memorial services, etc.

[0770] The user enters the date and time of the memorial service through the user's device and sends it to the server. The server stores this information in a database and, the day before the memorial service, sends a prompt to the generative AI model to generate an avatar of the deceased. An example of a prompt is: "Please generate a message of thanks for the memorial service in the voice of the deceased." On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays it.

[0771] Functions for Obon and New Year holidays

[0772] The user uses their device to notify the server that they will light incense during Obon or New Year's visits. The server then sends a prompt to the generative AI model, which generates a message of gratitude and an avatar. An example of a prompt is: "Please generate a message of gratitude during Obon in the voice of the deceased." The generated avatar and message are then sent from the server to the user's device, where they are displayed and played back on the device.

[0773] Blessing function at the time of growth

[0774] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, it sends a prompt to the generative AI model, which generates a congratulatory message and avatar. An example of a prompt is: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased." On the anniversary day, the server sends the generated avatar and message to the user's device, and also notifies the user of the points awarded. The user's device displays and plays the message, and notifies the user of the points.

[0775] Specific examples

[0776] Example 1: Responses at memorial services

[0777] 1. The user opens the smartphone app and enters the date of the memorial service.

[0778] 2. The device sends the information to the server, which stores it in a database.

[0779] 3. The server sends a prompt to the generative AI model the day before the memorial service, generating an avatar and message.

[0780] 4. On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays them.

[0781] Example 2: How to handle visits during Obon and New Year's

[0782] 1. Notify the user when lighting incense.

[0783] 2. The server sends a prompt to the generative AI model, which generates a thank you message and an avatar.

[0784] 3. The generative AI model sends the avatar and message back to the server, which then sends it to the user's device.

[0785] 4. The user device displays and plays the message and avatar.

[0786] Example 3: Grandchild's birthday celebration

[0787] 1. The server checks the date of birth from the database.

[0788] 2. On the day before the anniversary, send a prompt to the generative AI model to generate a congratulatory message and avatar.

[0789] 3. On the anniversary day, the server sends the generated avatar and a message to the user's device.

[0790] 4. The user's device displays and plays the message and avatar, and also notifies them of points.

[0791] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[0792] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0793] Processing steps for event response functions such as memorial services

[0794] Step 1: Enter your event information

[0795] The user inputs the date and time of the memorial service into the user terminal and presses the "send" button.

[0796] Input: Date and time of the memorial service.

[0797] Output: Data sent to the server.

[0798] Specific operation: The user enters the date of the memorial service (October 31, 2023) and the time (10:00) on the application screen and clicks the send button.

[0799] Step 2: Save your data

[0800] The server stores the date and time of the received memorial service in a database.

[0801] Input: Date and time data for the memorial service.

[0802] Output: Event information stored in a database.

[0803] Specific operation: The date "October 31, 2023" and time "10:00" received by the server are inserted into the event information table in the database.

[0804] Step 3: Request avatar creation

[0805] The day before the memorial service, the server requests the generative AI model to generate an avatar of the deceased person, using a prompt text.

[0806] Input: prompt statement.

[0807] Output: A request to the generative AI model to generate a new model.

[0808] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of gratitude for the memorial service in the voice of the deceased."

[0809] Step 4: Receiving avatars and messages

[0810] The generative AI model generates an avatar and message based on the request and sends it back to the server.

[0811] Input: prompt statement.

[0812] Output: The generated avatar and message.

[0813] Specific operation: The generative AI model generates an avatar that says, "Thank you for coming to the memorial service today," along with a message, and sends it back to the server.

[0814] Step 5: Message and Avatar Notifications

[0815] On the day of the memorial service, the server sends the generated avatar and a message to the user's terminal.

[0816] Input: The generated avatar and message.

[0817] Output: Data sent to the user's terminal.

[0818] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for coming to the memorial service today."

[0819] Step 6: Display your avatar and message

[0820] The user terminal displays and plays the avatar and message received from the server.

[0821] Input: Avatar and message sent from the server.

[0822] Output: User notification and display.

[0823] Specific operation: The user's device displays an avatar on the application and plays a message saying, "Thank you for coming to the memorial service today."

[0824] Processing steps for Obon and New Year's holiday support functions

[0825] Step 1: Incense Notification

[0826] A user uses a user terminal to notify that incense will be lit when visiting during Obon or New Year's.

[0827] Input: Notification of incense being lit.

[0828] Output: Notification data to the server.

[0829] Specific operation: The user clicks the "Light Incense" button in the application, notifying the server.

[0830] Step 2: Request avatar creation

[0831] The server sends a message of thanks and requests the AI ​​model to generate an avatar. A prompt is used.

[0832] Input: prompt statement.

[0833] Output: A request to the generative AI model to generate a new model.

[0834] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of thanks for Obon in the voice of the deceased."

[0835] Step 3: Receiving avatars and messages

[0836] The generative AI model generates a thank you message and avatar and sends them back to the server.

[0837] Input: prompt statement.

[0838] Output: The generated avatar and message.

[0839] Specific operation: The generative AI model generates an avatar that says, "Thank you for visiting during Obon," along with a message, and sends it back to the server.

[0840] Step 4: Message and Avatar Notifications

[0841] The server sends the generated message and avatar to the user terminal.

[0842] Input: The generated avatar and message.

[0843] Output: Data sent to the user's terminal.

[0844] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for visiting during Obon."

[0845] Step 5: Displaying your message and avatar

[0846] The user terminal displays it and plays back the message "Thank you for visiting during Obon."

[0847] Input: Avatar and message sent from the server.

[0848] Output: User notification and display.

[0849] Specific operation: The user's device displays an avatar on the application and plays the message "Thank you for visiting during Obon."

[0850] Processing steps for the blessing function during growth

[0851] Step 1: Manage your anniversaries

[0852] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database.

[0853] Input: Anniversary data in the database.

[0854] Output: Anniversary check result.

[0855] What it does: Every day the server checks the database to see if an anniversary is approaching.

[0856] Step 2: Request avatar creation

[0857] When an anniversary approaches, the server asks the generative AI model to generate a congratulatory message and an avatar, using a prompt text.

[0858] Input: prompt statement.

[0859] Output: A request to the generative AI model to generate a new model.

[0860] Specific operation: The server sends a prompt to the generation AI model: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased."

[0861] Step 3: Receiving avatars and messages

[0862] The generative AI model generates a congratulatory message and avatar and sends them back to the server.

[0863] Input: prompt statement.

[0864] Output: The generated avatar and message.

[0865] Specific operation: The generative AI model generates an avatar that says "Happy Birthday" and the message, and sends it back to the server.

[0866] Step 4: Message and Avatar Notifications

[0867] On the anniversary day, the server sends the generated message and avatar to the user's terminal.

[0868] Input: The generated avatar and message.

[0869] Output: Data sent to the user's terminal.

[0870] Specific operation: The server sends the generated avatar and a message saying "Happy Birthday" to the user's device.

[0871] Step 5: Points awarded

[0872] The server awards points to the user and sends a notification to the user's terminal.

[0873] Input: User information and point data.

[0874] Output: Points awarded notification.

[0875] Specific operation: The server will add 100 points to the user's account and send a notification to the user's device.

[0876] Step 6: Displaying your message and avatar

[0877] The user terminal displays the generated avatar and message, and plays back the message "Happy Birthday."

[0878] Input: Avatar and message sent from the server.

[0879] Output: User notification and display.

[0880] Specific operation: The user's device displays an avatar on the application and plays the message "Happy Birthday."

[0881] (Application example 1)

[0882] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0883] Previous systems that allowed users to relive memories with the deceased were limited to specific events and dates, limiting the opportunities for users to feel a connection with the deceased in their daily lives. Furthermore, they were inadequate in supporting visits other than memorial services and specific days, or memorable places and items, and there was no system that allowed users to feel a deeper connection with the deceased.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0885] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for generating an avatar based on the event date and time and displaying it, a means for the user to use a device that recognizes a specific place or product and transmit that information, and a means for using a device that displays the avatar and message generated based on data of the deceased, allowing the user to relive memories of the deceased in their daily lives and receive messages related to specific places or products.

[0886] A "deceased person" is someone who has already passed away and is recreated within the system based on data from their life.

[0887] "Data" refers to information about the deceased person during their lifetime, including text data, audio data, image data, and the like.

[0888] An "avatar" is a digital character generated based on the data of a deceased person, and its role is to display a message or image of the deceased to the user.

[0889] An "event" is an event or activity that takes place at a specific date, time, or place, and includes memorial services, anniversaries, and the like.

[0890] A "message" is a message or thank you that the deceased person's avatar conveys to the user.

[0891] "Means" are the tools and methods a system uses to achieve its goals.

[0892] A "user" is an individual who uses the system and operates it to relive memories of a deceased loved one.

[0893] "Device" is a general term for the hardware required for users to use the system, including smart glasses and smartphones.

[0894] A "server" is a computer that manages and processes data at the center of the system, and is where the generative AI and database run.

[0895] "Generative AI" is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[0896] A "database" is a storage system for storing data on deceased people, user information, event information, etc.

[0897] The system for implementing this invention comprises the following components: a user terminal, a server, a generating AI, a database, smart glasses, and, if necessary, a recognition device (such as a camera).

[0898] Overall system flow

[0899] 1. User Device:

[0900] Users use smart glasses or a smartphone to scan a specific location or product.

[0901] Used to enter event information into the system.

[0902] 2. Server:

[0903] Responsible for managing and processing data.

[0904] Event information is stored in a database, and avatars and messages are generated through generative AI.

[0905] 3. Generation AI:

[0906] It is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[0907] OpenAI's GPT-4 is used as a concrete example.

[0908] 4. Database:

[0909] A storage system for storing data on deceased individuals, user information, and event information. A database management system such as MySQL is used.

[0910] 5. Smart Glasses:

[0911] A device used by a user to recognize a specific place or product, such as Google Glass.

[0912] The deceased person's avatar and message are displayed in the user's field of view.

[0913] Specific processing of the program

[0914] 1. Data Collection:

[0915] The user puts on the smart glasses and scans memorable places and products.

[0916] 2. Data transmission:

[0917] The scanned data is sent from the user device to the server.

[0918] 3. Data Retention:

[0919] The server stores the received data in a database.

[0920] 4. Avatar and message generation:

[0921] A generation AI on the server generates avatars and messages based on the data.

[0922] GPT-4 and other AIs are used as generative AI.

[0923] 5. Displaying the results:

[0924] The generated avatar and message are sent from the server to the smart glasses via the user terminal.

[0925] Smart glasses display avatars and messages in the user's field of vision.

[0926] Specific examples

[0927] For example, when a user visits a specific location in a restaurant that holds special memories for the deceased, they can scan the location with the smart glasses. The server receives the data and asks the AI ​​to generate an avatar of the deceased and create a message related to the location. A message from the deceased, such as "I had a wonderful time eating here," is displayed on the screen.

[0928] Prompt Sentence Examples

[0929] "A user visits a specific seat in a restaurant. Generate a thank you message based on memories of the deceased person being at that location."

[0930] In this way, users can receive messages from the deceased related to specific places or items, deepening their connection with the deceased as part of their daily lives.

[0931] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0932] Step 1:

[0933] Users scan memorable places and products

[0934] A user puts on the smart glasses and scans a specific location or item.

[0935] Input: The user visually recognizes a specific location or product through the camera on the smart glasses.

[0936] Output: Scanned data (image data, QR code, etc.) is generated and sent to the next step.

[0937] Step 2:

[0938] Sending scan data to the server

[0939] The user terminal transmits the scan data to the server.

[0940] Input: The scan data generated in step 1.

[0941] Output: The scanned data is sent to the server.

[0942] Step 3:

[0943] The server stores the scan data in a database

[0944] The server stores the received scan data in a database.

[0945] Input: Scan data sent to the server.

[0946] Output: The scan data is saved to the database and the result of the save success is returned.

[0947] Step 4:

[0948] The server asks the AI ​​to generate an avatar and a message.

[0949] The server asks the generation AI to generate an avatar and message based on the saved scan data.

[0950] Input: Scan data stored in the database.

[0951] Output: The prompt is sent to the generation AI, and a request is made to generate an avatar and message.

[0952] Step 5:

[0953] Generative AI generates avatars and messages

[0954] The generation AI generates an avatar and message based on the prompt text.

[0955] A specific example is the prompt: "The user visited a specific seat in a restaurant. Generate a thank-you message based on memories of the deceased person being at that location."

[0956] Input: The prompt sent to the generation AI.

[0957] Output: The generated avatar and message.

[0958] Step 6:

[0959] Send the generated avatar and message to the server

[0960] The server receives the generated avatar and message.

[0961] Input: Avatar and message returned by the generation AI.

[0962] Output: The avatar and message are stored on the server and ready to be sent to the user's device.

[0963] Step 7:

[0964] The server sends the avatar and message to the user's device.

[0965] The server sends the avatar and the message to the user's smart glasses.

[0966] Input: Avatar and message stored on the server.

[0967] Output: The avatar and message are transferred to the user's device and displayed on the user's device.

[0968] Step 8:

[0969] Users receive avatars and messages through smart glasses

[0970] The smart glasses display an avatar and a message in the user's field of vision.

[0971] Input: Avatar and message sent from the server.

[0972] Output: An avatar is displayed on the user's smart glasses and a message is read out loud.

[0973] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0974] This invention is a system that uses an avatar generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[0975] Overall system configuration

[0976] The system mainly consists of the following components:

[0977] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[0978] 2. Server: The central computer that manages the database, generation AI, and emotion engine. It stores event information, generates avatars, generates messages, recognizes emotions, and awards points.

[0979] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[0980] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[0981] 5. Database: Storage for storing data of deceased persons, user information, and event information.

[0982] Program processing overview

[0983] The program processing in this system is explained as follows.

[0984] Event support function for memorial services, etc.

[0985] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and as the date and time of the memorial service approaches, it requests the generation AI to generate an avatar of the deceased. On the day of the memorial service, the emotion engine recognizes the user's emotions in real time, and the avatar and message displayed will match those emotions. For example, if the user has a sad expression, the avatar will display an encouraging message such as, "Thank you for coming at this difficult time."

[0986] Functions for Obon and New Year holidays

[0987] The user's device notifies the server that they will be offering incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user's device. The emotion engine analyzes the user's emotions, and the avatar personalizes the thank-you message. For example, if the user has a happy expression, it will display a message such as, "Thank you for coming to visit us. We're happy to see your smile."

[0988] Blessing function at the time of growth

[0989] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device, and at the same time, it personalizes the message using an emotion engine. For example, if the user is emotional, it displays an emotional message such as "Happy birthday. I'm so happy to have been able to watch you grow up." The server also performs point allocation processing and awards points to the user.

[0990] Specific examples

[0991] Example 1: Responses at memorial services

[0992] 1. The user device enters the date and time of the memorial service and sends it to the server.

[0993] 2. The server stores the information in a database.

[0994] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[0995] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[0996] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[0997] 6. The user's device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging message such as, "Thank you for coming during this difficult time."

[0998] Example 2: How to handle visits during Obon and New Year's

[0999] 1. The user device notifies the server when it is time to light the incense.

[1000] 2. The server sends a thank you message and asks the AI ​​to generate an avatar.

[1001] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[1002] 4. The server sends the message and avatar to the user's device.

[1003] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[1004] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message changes to "Thank you for visiting. I'm happy to see your smile."

[1005] Example 3: Grandchild's birthday celebration

[1006] 1. The server checks the date of birth from the database.

[1007] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[1008] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[1009] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[1010] 5. The user's device displays the message "Happy Birthday" and an avatar. If the user is touched, the message changes to "Happy Birthday. I'm so happy to watch you grow up."

[1011] 6. The server awards points to the user and notifies them on the device.

[1012] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[1013] The processing flow will be explained below.

[1014] Event support function for memorial services, etc.

[1015] Step 1:

[1016] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[1017] Step 2:

[1018] The terminal transmits the entered date and time information of the memorial service to the server.

[1019] Step 3:

[1020] The server stores the received date and time information in a database.

[1021] Step 4:

[1022] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[1023] Step 5:

[1024] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[1025] Step 6:

[1026] The server sends the generated avatar and message data to the device.

[1027] Step 7:

[1028] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[1029] Step 8:

[1030] The device will start displaying your avatar and message.

[1031] Step 9:

[1032] The emotion engine recognizes the user's emotions in real time.

[1033] Step 10:

[1034] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[1035] Step 11:

[1036] The device displays a personalized message and plays it to attendees.

[1037] Functions for Obon and New Year holidays

[1038] Step 1:

[1039] The user launches the app on their device and enters the information when lighting the incense.

[1040] Step 2:

[1041] The terminal transmits the input information to the server.

[1042] Step 3:

[1043] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[1044] Step 4:

[1045] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[1046] Step 5:

[1047] The server sends the generated message and avatar to the device.

[1048] Step 6:

[1049] The device will begin displaying a generated thank you message and avatar.

[1050] Step 7:

[1051] The emotion engine analyzes the user's emotions.

[1052] Step 8:

[1053] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[1054] Step 9:

[1055] The device will notify the user by displaying a personalized thank you message and avatar.

[1056] Blessing function at the time of growth

[1057] Step 1:

[1058] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[1059] Step 2:

[1060] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[1061] Step 3:

[1062] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[1063] Step 4:

[1064] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[1065] Step 5:

[1066] The terminal will begin displaying a congratulatory message and the generated avatar.

[1067] Step 6:

[1068] The emotion engine analyzes the user's emotions.

[1069] Step 7:

[1070] The emotion engine personalizes messages based on the results of emotion analysis and sends them to the device.

[1071] Step 8:

[1072] The device will notify the user by displaying a personalized congratulatory message and avatar.

[1073] Step 9:

[1074] The server performs a point giving process and gives points to the user.

[1075] Step 10:

[1076] The terminal notifies the user of the point allocation notice so that the user can check it.

[1077] Example 2

[1078] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1079] Conventional memorial events lacked a means for users to feel a sense of dialogue with the deceased, making it difficult to deepen an emotional connection. Furthermore, there was no way to provide messages tailored to the user's emotions at specific events or occasions. This prevented users from feeling closer to the deceased, making it difficult for them to empathize or feel comforted. Furthermore, there was a lack of mechanisms for appropriately managing the timing of event notifications and the display of messages from the deceased.

[1080] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1081] In this invention, the server includes a means for sending a message at a specific event using a model generated based on data of the deceased, a means for inputting event information and saving the information, a means for generating a model based on the event date and time and displaying it, a means for recognizing the user's emotions in real time and personalizing the message based on the recognition results, and a means for transmitting and displaying the generated message and model to a user terminal. This allows the user to feel a sense of dialogue with the deceased and deepen their emotional connection. Furthermore, by providing a personalized message tailored to the user's emotions, the user can gain empathy and comfort. Furthermore, the timing of event notifications and message display can be appropriately managed.

[1082] A "model" is a virtual character or message generation program generated based on data about the deceased.

[1083] "Events" are events or ceremonies that are held on specific dates or times, such as memorial services, Obon, and New Year's.

[1084] A "message" is information such as text, audio, or video sent from the deceased to the user.

[1085] A "user terminal" is a device used by a user, such as a smartphone or a personal computer.

[1086] The "server" is a central computer that manages and operates the database, generative AI, and emotion engine.

[1087] The "input means" is an interface through which the user inputs event information and transmits it to the server.

[1088] "Storage means" is a function that stores input event information in a database.

[1089] "Display means" is a function that displays the generated model and messages on the user terminal.

[1090] "Emotion recognition means" is a technology for analyzing a user's emotions in real time and obtaining the results.

[1091] "Personalization means" is a function that individually optimizes messages based on acquired emotional information.

[1092] A "notification means" is a communication means for reporting specific actions or events from a user terminal to a server.

[1093] The "point giving means" is a function for giving points to a user and notifying the user.

[1094] The present invention is a system that uses a model generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, a model, an event information input means, a model display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[1095] Overall system configuration

[1096] The system mainly consists of the following components:

[1097] 1. User terminal: A device such as a smartphone or PC that users use to enter event information and receive messages.

[1098] 2. Server: The central computer that manages the database, generative AI, and emotion engine. It stores event information, generates models, generates messages, recognizes emotions, and awards points.

[1099] 3. Generative AI: An artificial intelligence program that generates models and messages based on data about the deceased.

[1100] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[1101] 5. Database: Storage for storing data of the deceased, user information, and event information.

[1102] Program processing overview

[1103] The program processing in this system is explained as follows.

[1104] Functions for handling events such as memorial services

[1105] The user enters the date and time of the memorial service into their device, which then sends the information to the server. The server stores the received information in a database. The day before the memorial service, the server requests the generative AI to generate a model of the deceased. On the day of the memorial service, when the user opens the app, the server sends the model and a message to the user's device and displays them. The emotion engine recognizes the user's emotions in real time and personalizes the message accordingly. For example, if the user has a sad expression, the model will display an encouraging message such as, "Thank you for coming at this difficult time."

[1106] Functions for Obon and New Year holidays

[1107] When a user lights incense during Obon or New Year's, the information is sent from the user's device to the server. Based on this information, the server requests the generation AI to generate a message of gratitude and a model. The generated data is sent from the server to the user's device. The emotion engine analyzes the user's emotions and personalizes the message. For example, if the user has a happy expression, a message such as "Thank you for coming to visit us. We're happy to see your smile" is displayed.

[1108] Blessing function at the time of growth

[1109] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, the server requests the generation AI to generate a congratulatory message and model. On the anniversary day, the server sends the generated message and model to the user's device, and the emotion engine analyzes the user's emotions and personalizes the message. For example, if the user is emotional, a message such as "Happy birthday. I'm so happy to have watched you grow up" is displayed. The server also performs the process of awarding points and notifies the user of the points.

[1110] Specific examples

[1111] Functions for handling events such as memorial services

[1112] Examples:

[1113] 1. The user opens the app on their smartphone and enters the date of the memorial service (December 25, 2023) and the time (10:00).

[1114] 2. The user device sends the data "Memorial date: December 25, 2023, time: 10:00" to the server.

[1115] 3. The server stores the received information in a database.

[1116] 4. On December 24, 2023, the server asks the generation AI to "generate a model for a memorial service."

[1117] 5. When the user opens the app on the day of the memorial service, the server sends the model and a message.

[1118] 6. The emotion engine analyzes the user's sad expression and generates a personalized message.

[1119] Prompt Sentence Examples

[1120] Here are some example prompts to input to a generative AI model:

[1121] "The memorial date is approaching. Please generate a model of the deceased and an appropriate message. Include an encouraging message if the user has a sad expression."

[1122] This allows users to feel like they are having a conversation with the deceased, deepening their emotional connection. Furthermore, by providing personalized messages tailored to the user's emotions, it is possible to gain empathy and comfort. Furthermore, it is possible to appropriately manage the timing of event notifications and message display.

[1123] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1124] Processing steps for events such as memorial services

[1125] Step 1:

[1126] The user enters the date and time of the memorial service.

[1127] Input: Date and time of memorial service (e.g. December 25, 2023, 10:00).

[1128] Output: The input information is saved on the user's terminal.

[1129] Specific operation: The user opens a dedicated app on their smartphone and enters the date and time of the memorial service in the form that appears.

[1130] Step 2:

[1131] The user terminal transmits information about the memorial service to the server.

[1132] Input: Date and time of memorial service.

[1133] Output: Memorial service information is sent to the server.

[1134] Specific operation: The date and time of the memorial service entered by the user is sent to the server as an HTTP request.

[1135] Step 3:

[1136] The server stores the memorial information in a database.

[1137] Input: Date and time of memorial service.

[1138] Output: Memorial information stored in a database.

[1139] Specific operation: The server saves the date and time of the memorial service received as a new record in a database table.

[1140] Step 4:

[1141] The day before the memorial service, the server requests the generation AI to generate a model.

[1142] Input: Trigger based on date and time before and after the memorial service, memorial service information.

[1143] Output: The generated model and messages.

[1144] Specific operation: The server uses a trigger set on the day before the memorial service to send a prompt message to the generation AI saying, "Please generate a model for the memorial service."

[1145] Step 5:

[1146] On the day of the memorial service, the user opens the application.

[1147] Input: User action.

[1148] Output: The application sends a request to the server.

[1149] Specific action: The user launches the app on their smartphone.

[1150] Step 6:

[1151] The server sends the avatar and message to the user's terminal.

[1152] Input: The generated avatar and message.

[1153] Output: The avatar and message displayed on the user's device.

[1154] Specific operation: The server sends the avatar and message received from the generated AI to the user's device in JSON format.

[1155] Step 7:

[1156] The emotion engine recognizes the user's emotions.

[1157] Input: User's facial expression data.

[1158] Output: Recognized emotion data (e.g. sadness, joy).

[1159] Specific operation: Using the camera on the user's device, the emotion engine analyzes the user's facial expressions in real time.

[1160] Step 8:

[1161] The server personalizes the message and displays it on the user's terminal.

[1162] Input: Recognized emotion data, generated message.

[1163] Output: A personalized message.

[1164] Specific operation: The server modifies the message based on the emotion data and sends it to the user's device. If the user has a sad expression, an encouraging message such as "Thank you for coming during this difficult time" will be displayed.

[1165] (Application example 2)

[1166] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1167] Conventional dialogue systems for the deceased only send messages using an avatar generated based on the deceased's data, but are unable to provide personalized messages based on the user's emotions. Furthermore, it is difficult to provide an emotionally rich experience for specific events. Therefore, there is a need for a system that allows users to empathize and feel deeply moved.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1169] In this invention, the server includes means for sending a message at a specific event using an avatar generated based on data of the deceased, means for inputting and saving event information, means for generating an avatar based on the event date and time and displaying it, and means for recognizing a user's emotions in real time and personalizing a message based on the emotions, thereby making it possible to recognize a user's emotions and provide a personalized message.

[1170] "Data of the deceased" refers to information about a deceased person, including digital data such as images, audio, video, and text.

[1171] An "avatar" is a digital image created based on the data of a deceased person, recreating the deceased in a virtual space.

[1172] "Specific events" refers to special occasions or anniversaries related to the deceased, including memorial services, birthdays, Obon, New Year's, and other regular events.

[1173] The "means for sending a message" is a method by which the avatar can provide a text or voice message to the user depending on the date and situation of the event.

[1174] "Event information" is detailed data about a particular event, such as the date, time, and location.

[1175] "Storage means" refers to the methods and technologies used to store event information and other related data in a database or the like.

[1176] "Display means" refers to a technique for visually displaying the generated avatar or message on the user's terminal.

[1177] "Means for recognizing a user's emotions in real time and personalizing messages based on those emotions" refers to a method that uses emotion recognition technology to determine a user's current emotional state and appropriately change the message content accordingly.

[1178] A "personalized message" is a text or voice message that is individually tailored to the user's emotions and situation.

[1179] This invention is a system that uses an avatar generated based on the data of a deceased person to send a message at a specific event. By combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. Details of the embodiment are described below.

[1180] 1. System Configuration

[1181] The system mainly consists of the following components:

[1182] User device: A device such as a smartphone or computer that users use to enter event information and receive messages.

[1183] Server: A central computer that manages the database, generative AI, and emotion engine. It stores event information, generates avatars, generates messages, and recognizes emotions.

[1184] Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[1185] Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[1186] Database: Storage for storing data of deceased persons, user information, and event information.

[1187] 2. Program Processing Overview

[1188] The user enters event information on the user device and sends it to the server, which stores it in a database. As the event date approaches, the server requests the generation AI to generate an avatar of the deceased person. On the day of the event, the emotion engine recognizes the user's emotions in real time, and the avatar and message are displayed according to those emotions.

[1189] 3. Hardware and Software Used

[1190] Hardware:

[1191] Smartphone

[1192] server

[1193] software:

[1194] EmotionEngine: Software for recognizing user emotions.

[1195] Avatar Generator (generative AI): Software that generates avatars based on data of deceased people.

[1196] Message Generator (Generative AI): Software that generates messages based on data about the deceased and event information.

[1197] Database: Database management software for storing user and event information.

[1198] 4. Specific Examples

[1199] ■ Responses to memorial services:

[1200] 1. The user terminal enters the date and time of the memorial service and sends it to the server.

[1201] 2. The server stores the information in a database.

[1202] 3. The day before the memorial service, the server requests the generation AI to generate an avatar.

[1203] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[1204] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[1205] 6. The user device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging one such as, "Thank you for coming during this difficult time."

[1206] ■ Visits during Obon and New Year's:

[1207] 1. The user device notifies the server when it is time to light the incense.

[1208] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[1209] 3. The generation AI generates a thank you message and an avatar and sends them back to the server.

[1210] 4. The server sends the message and avatar to the user's device.

[1211] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[1212] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message is changed to "Thank you for visiting. I'm happy to see your smile."

[1213] ■ Grandchild's birthday celebration:

[1214] 1. The server checks the date of birth from the database.

[1215] 2. On the day before the anniversary, ask the generation AI to generate a congratulatory message and an avatar.

[1216] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[1217] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[1218] 5. The user device displays the message "Happy Birthday" and the avatar. If the user is touched, the message changes to "Happy Birthday. I'm so glad to watch you grow up."

[1219] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[1220] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1221] Step 1:

[1222] The user enters event information (e.g., the date and time of the memorial service) into the terminal and sends it to the server.

[1223] Input: Event information (date, time)

[1224] Data processing: User terminal packages event information

[1225] Output: Request sent to server

[1226] Step 2:

[1227] The server stores the received event information in a database.

[1228] Input: Received event information

[1229] Data processing: Converting event information into the appropriate format

[1230] Output: Save results to database

[1231] Step 3:

[1232] As the event date approaches, the server asks the generation AI to generate an avatar of the deceased person.

[1233] Input: Event information, deceased person's data

[1234] Data processing: Generative AI analyzes the data of the deceased and generates an avatar

[1235] Output: The generated avatar

[1236] Step 4:

[1237] On the day of the event, the server sends an avatar and a message to the user's terminal.

[1238] Input: Generated avatar, standard message

[1239] Data processing: Packaging avatars and messages

[1240] Output: Send request to user terminal

[1241] Step 5:

[1242] The user's device displays the received avatar and message.

[1243] Input: avatar, message

[1244] Data processing: Convert received data into a display format

[1245] Output: Display avatar and message on user's device

[1246] Step 6:

[1247] The emotion engine recognizes the user's emotions in real time.

[1248] Input: User's facial expression data (camera input)

[1249] Data calculation: Emotion engine analyzes facial expression data

[1250] Output: Recognized emotion data

[1251] Step 7:

[1252] The server personalizes the message based on the recognized emotion data.

[1253] Input: Standard message, recognized emotion data

[1254] Data processing: Message generation based on emotional data

[1255] Output: Personalized message

[1256] Step 8:

[1257] The user terminal displays the personalized message.

[1258] Input: Personalized message

[1259] Data processing: Convert received messages into a display format

[1260] Output: Display a personalized message on the user's device

[1261] Through the above steps, a personalized message according to emotions is provided based on the event information entered by the user.

[1262] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1263] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1264] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1265] [Third embodiment]

[1266] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1267] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1268] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1269] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1270] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1271] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1272] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1273] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1274] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1275] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1276] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1277] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1278] This invention is a system that uses an avatar generated based on the data of the deceased to send a message from the deceased at a specific event. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, and a server are used.

[1279] Overall system configuration

[1280] The system mainly consists of the following components:

[1281] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[1282] 2. Server: The central computer that manages the database and the AI ​​generation. It stores event information, generates avatars, generates messages, and awards points.

[1283] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[1284] 4. Database: Storage for storing data of deceased people, user information, and event information.

[1285] Program processing overview

[1286] The program processing in this system is explained as follows.

[1287] Event support function for memorial services, etc.

[1288] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and when the date and time of the memorial service approaches, it requests the AI ​​to generate an avatar of the deceased. On the day of the memorial service, the user's device displays this avatar and a message, conveying the deceased's words of gratitude to attendees.

[1289] Functions for Obon and New Year holidays

[1290] The user device notifies the server that they will light incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user device. The user device then displays the message and avatar, conveying the deceased's gratitude.

[1291] Blessing function at the time of growth

[1292] The server manages the anniversaries of the growth of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device and simultaneously awards points. The user's device displays the congratulatory message and a notification that points have been awarded.

[1293] Specific examples

[1294] Example 1: Responses at memorial services

[1295] 1. The user device enters the date and time of the memorial service and sends it to the server.

[1296] 2. The server stores the information in a database.

[1297] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[1298] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[1299] 5. The user device plays the message "Thank you for coming to the memorial service today."

[1300] Example 2: How to handle visits during Obon and New Year's

[1301] 1. The user device notifies the server when it is time to light the incense.

[1302] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[1303] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[1304] 4. The server sends the message and avatar to the user's device.

[1305] 5. The user device displays the message "Thank you for visiting during Obon."

[1306] Example 3: Grandchild's birthday celebration

[1307] 1. The server checks the date of birth from the database.

[1308] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[1309] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[1310] 4. The user's device displays the message "Happy Birthday" and an avatar.

[1311] 5. The server awards points to the user and notifies them on the device.

[1312] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[1313] The processing flow will be explained below.

[1314] Event support function for memorial services, etc.

[1315] Step 1:

[1316] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[1317] Step 2:

[1318] The terminal transmits the entered date and time information of the memorial service to the server.

[1319] Step 3:

[1320] The server stores the received date and time information in a database.

[1321] Step 4:

[1322] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[1323] Step 5:

[1324] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[1325] Step 6:

[1326] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[1327] Step 7:

[1328] The server sends the generated avatar and message data to the device.

[1329] Step 8:

[1330] The device displays the generated avatar and message and plays it to the attendees.

[1331] Functions for Obon and New Year holidays

[1332] Step 1:

[1333] The user launches the app on their device and enters the information when lighting the incense.

[1334] Step 2:

[1335] The terminal transmits the input information to the server.

[1336] Step 3:

[1337] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[1338] Step 4:

[1339] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[1340] Step 5:

[1341] The server sends the generated message and avatar to the device.

[1342] Step 6:

[1343] The device displays the generated thank you message and avatar to notify the user.

[1344] Blessing function at the time of growth

[1345] Step 1:

[1346] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[1347] Step 2:

[1348] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[1349] Step 3:

[1350] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[1351] Step 4:

[1352] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[1353] Step 5:

[1354] The device displays the generated congratulatory message and avatar to notify the user.

[1355] Step 6:

[1356] At the same time, the server performs a point awarding process and awards points to the user.

[1357] Step 7:

[1358] The terminal notifies the user of the point allocation notice so that the user can check it.

[1359] Example 1

[1360] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1361] The present invention aims to deepen the bond with the deceased by sending messages from the deceased at specific events using avatars generated based on the deceased's data. However, current systems lack a mechanism for dynamically generating messages and avatars for specific dates or events, and may not meet user expectations. Furthermore, there is a need for an efficient system that can accommodate multiple events and anniversaries and send messages from the deceased in a timely manner.

[1362] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1363] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for sending a prompt to a generative AI model based on the event date and time to generate an avatar and a message, and a means for sending and displaying the generated avatar and message to a user terminal, thereby enabling messages and avatars from the deceased to be generated and sent in a timely manner and provided to users.

[1364] "Data of the deceased" refers to personal information and recorded data relating to the deceased, including the deceased's voice, images, messages, etc.

[1365] An "avatar" is a digital character or image created based on the data of a deceased person.

[1366] An "event" is an event or ceremony that takes place on a specific day or time, and includes memorial services, anniversaries, Obon, New Year's, etc.

[1367] A "message" is a verbal message in the form of text, audio, video, etc., generated based on the data of the deceased.

[1368] A "generative AI model" is a model that uses artificial intelligence to generate digital content such as text, images, and audio. In this invention, it generates an avatar and message based on data about the deceased.

[1369] A "prompt" is an instruction that is input to a generative AI model to generate specific content.

[1370] "User device" refers to a digital device such as a smartphone or computer used by a user, and is used to input event information and receive and display avatars and messages.

[1371] The "server" is a central control device that manages the database, controls the generative AI model, and requests and sends messages and avatars.

[1372] "Event information" refers to information such as the date, time, and content related to a specific event, which is input by the user and sent to the server.

[1373] "Means for storing" refers to the function of storing input event information and data on the deceased in storage such as a database.

[1374] "Display means" refers to a function for visually displaying the generated avatar and message on the screen of the user terminal.

[1375] This invention is a system that uses an avatar generated based on the data of the deceased to send messages from the deceased at specific events. To implement this system, hardware and software such as a user terminal, a server, a generative AI model, and a database are used.

[1376] Hardware and software used

[1377] 1. User Device

[1378] A device that allows users to enter event information and receive messages. Examples include smartphones and PCs.

[1379] 2. Server

[1380] A central control unit that manages the entire system and links with the database and generative AI model.

[1381] 3. Generative AI Models

[1382] An artificial intelligence model that generates an avatar and message based on data about the deceased. A specific example would be OpenAI GPT-3.

[1383] 4. Database

[1384] A storage system that stores data on deceased people, user information, and event information. Examples include MySQL and PostgreSQL.

[1385] System configuration and processing overview

[1386] Event support function for memorial services, etc.

[1387] The user enters the date and time of the memorial service through the user's device and sends it to the server. The server stores this information in a database and, the day before the memorial service, sends a prompt to the generative AI model to generate an avatar of the deceased. An example of a prompt is: "Please generate a message of thanks for the memorial service in the voice of the deceased." On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays it.

[1388] Functions for Obon and New Year holidays

[1389] The user uses their device to notify the server that they will light incense during Obon or New Year's visits. The server then sends a prompt to the generative AI model, which generates a message of gratitude and an avatar. An example of a prompt is: "Please generate a message of gratitude during Obon in the voice of the deceased." The generated avatar and message are then sent from the server to the user's device, where they are displayed and played back on the device.

[1390] Blessing function at the time of growth

[1391] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, it sends a prompt to the generative AI model, which generates a congratulatory message and avatar. An example of a prompt is: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased." On the anniversary day, the server sends the generated avatar and message to the user's device, and also notifies the user of the points awarded. The user's device displays and plays the message, and notifies the user of the points.

[1392] Specific examples

[1393] Example 1: Responses at memorial services

[1394] 1. The user opens the smartphone app and enters the date of the memorial service.

[1395] 2. The device sends the information to the server, which stores it in a database.

[1396] 3. The server sends a prompt to the generative AI model the day before the memorial service, generating an avatar and message.

[1397] 4. On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays them.

[1398] Example 2: How to handle visits during Obon and New Year's

[1399] 1. Notify the user when lighting incense.

[1400] 2. The server sends a prompt to the generative AI model, which generates a thank you message and an avatar.

[1401] 3. The generative AI model sends the avatar and message back to the server, which then sends it to the user's device.

[1402] 4. The user device displays and plays the message and avatar.

[1403] Example 3: Grandchild's birthday celebration

[1404] 1. The server checks the date of birth from the database.

[1405] 2. On the day before the anniversary, send a prompt to the generative AI model to generate a congratulatory message and avatar.

[1406] 3. On the anniversary day, the server sends the generated avatar and a message to the user's device.

[1407] 4. The user's device displays and plays the message and avatar, and also notifies them of points.

[1408] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[1409] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1410] Processing steps for event response functions such as memorial services

[1411] Step 1: Enter your event information

[1412] The user inputs the date and time of the memorial service into the user terminal and presses the "send" button.

[1413] Input: Date and time of the memorial service.

[1414] Output: Data sent to the server.

[1415] Specific operation: The user enters the date of the memorial service (October 31, 2023) and the time (10:00) on the application screen and clicks the send button.

[1416] Step 2: Save your data

[1417] The server stores the date and time of the received memorial service in a database.

[1418] Input: Date and time data for the memorial service.

[1419] Output: Event information stored in a database.

[1420] Specific operation: The date "October 31, 2023" and time "10:00" received by the server are inserted into the event information table in the database.

[1421] Step 3: Request avatar creation

[1422] The day before the memorial service, the server requests the generative AI model to generate an avatar of the deceased person, using a prompt text.

[1423] Input: prompt statement.

[1424] Output: A request to the generative AI model to generate a new model.

[1425] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of gratitude for the memorial service in the voice of the deceased."

[1426] Step 4: Receiving avatars and messages

[1427] The generative AI model generates an avatar and message based on the request and sends it back to the server.

[1428] Input: prompt statement.

[1429] Output: The generated avatar and message.

[1430] Specific operation: The generative AI model generates an avatar that says, "Thank you for coming to the memorial service today," along with a message, and sends it back to the server.

[1431] Step 5: Message and Avatar Notifications

[1432] On the day of the memorial service, the server sends the generated avatar and a message to the user's terminal.

[1433] Input: The generated avatar and message.

[1434] Output: Data sent to the user's terminal.

[1435] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for coming to the memorial service today."

[1436] Step 6: Display your avatar and message

[1437] The user terminal displays and plays the avatar and message received from the server.

[1438] Input: Avatar and message sent from the server.

[1439] Output: User notification and display.

[1440] Specific operation: The user's device displays an avatar on the application and plays a message saying, "Thank you for coming to the memorial service today."

[1441] Processing steps for Obon and New Year's holiday support functions

[1442] Step 1: Incense Notification

[1443] A user uses a user terminal to notify that incense will be lit when visiting during Obon or New Year's.

[1444] Input: Notification of incense being lit.

[1445] Output: Notification data to the server.

[1446] Specific operation: The user clicks the "Light Incense" button in the application, notifying the server.

[1447] Step 2: Request avatar creation

[1448] The server sends a message of thanks and requests the AI ​​model to generate an avatar. A prompt is used.

[1449] Input: prompt statement.

[1450] Output: A request to the generative AI model to generate a new model.

[1451] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of thanks for Obon in the voice of the deceased."

[1452] Step 3: Receiving avatars and messages

[1453] The generative AI model generates a thank you message and avatar and sends them back to the server.

[1454] Input: prompt statement.

[1455] Output: The generated avatar and message.

[1456] Specific operation: The generative AI model generates an avatar that says, "Thank you for visiting during Obon," along with a message, and sends it back to the server.

[1457] Step 4: Message and Avatar Notifications

[1458] The server sends the generated message and avatar to the user terminal.

[1459] Input: The generated avatar and message.

[1460] Output: Data sent to the user's terminal.

[1461] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for visiting during Obon."

[1462] Step 5: Displaying your message and avatar

[1463] The user terminal displays it and plays back the message "Thank you for visiting during Obon."

[1464] Input: Avatar and message sent from the server.

[1465] Output: User notification and display.

[1466] Specific operation: The user's device displays an avatar on the application and plays the message "Thank you for visiting during Obon."

[1467] Processing steps for the blessing function during growth

[1468] Step 1: Manage your anniversaries

[1469] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database.

[1470] Input: Anniversary data in the database.

[1471] Output: Anniversary check result.

[1472] What it does: Every day the server checks the database to see if an anniversary is approaching.

[1473] Step 2: Request avatar creation

[1474] When an anniversary approaches, the server asks the generative AI model to generate a congratulatory message and an avatar, using a prompt text.

[1475] Input: prompt statement.

[1476] Output: A request to the generative AI model to generate a new model.

[1477] Specific operation: The server sends a prompt to the generation AI model: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased."

[1478] Step 3: Receiving avatars and messages

[1479] The generative AI model generates a congratulatory message and avatar and sends them back to the server.

[1480] Input: prompt statement.

[1481] Output: The generated avatar and message.

[1482] Specific operation: The generative AI model generates an avatar that says "Happy Birthday" and the message, and sends it back to the server.

[1483] Step 4: Message and Avatar Notifications

[1484] On the anniversary day, the server sends the generated message and avatar to the user's terminal.

[1485] Input: The generated avatar and message.

[1486] Output: Data sent to the user's terminal.

[1487] Specific operation: The server sends the generated avatar and a message saying "Happy Birthday" to the user's device.

[1488] Step 5: Points awarded

[1489] The server awards points to the user and sends a notification to the user's terminal.

[1490] Input: User information and point data.

[1491] Output: Points awarded notification.

[1492] Specific operation: The server will add 100 points to the user's account and send a notification to the user's device.

[1493] Step 6: Displaying your message and avatar

[1494] The user terminal displays the generated avatar and message, and plays back the message "Happy Birthday."

[1495] Input: Avatar and message sent from the server.

[1496] Output: User notification and display.

[1497] Specific operation: The user's device displays an avatar on the application and plays the message "Happy Birthday."

[1498] (Application example 1)

[1499] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1500] Previous systems that allowed users to relive memories with the deceased were limited to specific events and dates, limiting the opportunities for users to feel a connection with the deceased in their daily lives. Furthermore, they were inadequate in supporting visits other than memorial services and specific days, or memorable places and items, and there was no system that allowed users to feel a deeper connection with the deceased.

[1501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1502] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for generating an avatar based on the event date and time and displaying it, a means for the user to use a device that recognizes a specific place or product and transmit that information, and a means for using a device that displays the avatar and message generated based on data of the deceased, allowing the user to relive memories of the deceased in their daily lives and receive messages related to specific places or products.

[1503] A "deceased person" is someone who has already passed away and is recreated within the system based on data from their life.

[1504] "Data" refers to information about the deceased person during their lifetime, including text data, audio data, image data, and the like.

[1505] An "avatar" is a digital character generated based on the data of a deceased person, and its role is to display a message or image of the deceased to the user.

[1506] An "event" is an event or activity that takes place at a specific date, time, or place, and includes memorial services, anniversaries, and the like.

[1507] A "message" is a message or thank you that the deceased person's avatar conveys to the user.

[1508] "Means" are the tools and methods a system uses to achieve its goals.

[1509] A "user" is an individual who uses the system and operates it to relive memories of a deceased loved one.

[1510] "Device" is a general term for the hardware required for users to use the system, including smart glasses and smartphones.

[1511] A "server" is a computer that manages and processes data at the center of the system, and is where the generative AI and database run.

[1512] "Generative AI" is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[1513] A "database" is a storage system for storing data on deceased people, user information, event information, etc.

[1514] The system for implementing this invention comprises the following components: a user terminal, a server, a generating AI, a database, smart glasses, and, if necessary, a recognition device (such as a camera).

[1515] Overall system flow

[1516] 1. User Device:

[1517] Users use smart glasses or a smartphone to scan a specific location or product.

[1518] Used to enter event information into the system.

[1519] 2. Server:

[1520] Responsible for managing and processing data.

[1521] Event information is stored in a database, and avatars and messages are generated through generative AI.

[1522] 3. Generation AI:

[1523] It is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[1524] OpenAI's GPT-4 is used as a concrete example.

[1525] 4. Database:

[1526] A storage system for storing data on deceased individuals, user information, and event information. A database management system such as MySQL is used.

[1527] 5. Smart Glasses:

[1528] A device used by a user to recognize a specific place or product, such as Google Glass.

[1529] The deceased person's avatar and message are displayed in the user's field of view.

[1530] Specific processing of the program

[1531] 1. Data Collection:

[1532] The user puts on the smart glasses and scans memorable places and products.

[1533] 2. Data transmission:

[1534] The scanned data is sent from the user device to the server.

[1535] 3. Data Retention:

[1536] The server stores the received data in a database.

[1537] 4. Avatar and message generation:

[1538] A generation AI on the server generates avatars and messages based on the data.

[1539] GPT-4 and other AIs are used as generative AI.

[1540] 5. Displaying the results:

[1541] The generated avatar and message are sent from the server to the smart glasses via the user terminal.

[1542] Smart glasses display avatars and messages in the user's field of vision.

[1543] Specific examples

[1544] For example, when a user visits a specific location in a restaurant that holds special memories for the deceased, they can scan the location with the smart glasses. The server receives the data and asks the AI ​​to generate an avatar of the deceased and create a message related to the location. A message from the deceased, such as "I had a wonderful time eating here," is displayed on the screen.

[1545] Prompt Sentence Examples

[1546] "A user visits a specific seat in a restaurant. Generate a thank you message based on memories of the deceased person being at that location."

[1547] In this way, users can receive messages from the deceased related to specific places or items, deepening their connection with the deceased as part of their daily lives.

[1548] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1549] Step 1:

[1550] Users scan memorable places and products

[1551] A user puts on the smart glasses and scans a specific location or item.

[1552] Input: The user visually recognizes a specific location or product through the camera on the smart glasses.

[1553] Output: Scanned data (image data, QR code, etc.) is generated and sent to the next step.

[1554] Step 2:

[1555] Sending scan data to the server

[1556] The user terminal transmits the scan data to the server.

[1557] Input: The scan data generated in step 1.

[1558] Output: The scanned data is sent to the server.

[1559] Step 3:

[1560] The server stores the scan data in a database

[1561] The server stores the received scan data in a database.

[1562] Input: Scan data sent to the server.

[1563] Output: The scan data is saved to the database and the result of the save success is returned.

[1564] Step 4:

[1565] The server asks the AI ​​to generate an avatar and a message.

[1566] The server asks the generation AI to generate an avatar and message based on the saved scan data.

[1567] Input: Scan data stored in the database.

[1568] Output: The prompt is sent to the generation AI, and a request is made to generate an avatar and message.

[1569] Step 5:

[1570] Generative AI generates avatars and messages

[1571] The generation AI generates an avatar and message based on the prompt text.

[1572] A specific example is the prompt: "The user visited a specific seat in a restaurant. Generate a thank-you message based on memories of the deceased person being at that location."

[1573] Input: The prompt sent to the generation AI.

[1574] Output: The generated avatar and message.

[1575] Step 6:

[1576] Send the generated avatar and message to the server

[1577] The server receives the generated avatar and message.

[1578] Input: Avatar and message returned by the generation AI.

[1579] Output: The avatar and message are stored on the server and ready to be sent to the user's device.

[1580] Step 7:

[1581] The server sends the avatar and message to the user's device.

[1582] The server sends the avatar and the message to the user's smart glasses.

[1583] Input: Avatar and message stored on the server.

[1584] Output: The avatar and message are transferred to the user's device and displayed on the user's device.

[1585] Step 8:

[1586] Users receive avatars and messages through smart glasses

[1587] The smart glasses display an avatar and a message in the user's field of vision.

[1588] Input: Avatar and message sent from the server.

[1589] Output: An avatar is displayed on the user's smart glasses and a message is read out loud.

[1590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1591] This invention is a system that uses an avatar generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[1592] Overall system configuration

[1593] The system mainly consists of the following components:

[1594] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[1595] 2. Server: The central computer that manages the database, generation AI, and emotion engine. It stores event information, generates avatars, generates messages, recognizes emotions, and awards points.

[1596] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[1597] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[1598] 5. Database: Storage for storing data of deceased persons, user information, and event information.

[1599] Program processing overview

[1600] The program processing in this system is explained as follows.

[1601] Event support function for memorial services, etc.

[1602] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and as the date and time of the memorial service approaches, it requests the generation AI to generate an avatar of the deceased. On the day of the memorial service, the emotion engine recognizes the user's emotions in real time, and the avatar and message displayed will match those emotions. For example, if the user has a sad expression, the avatar will display an encouraging message such as, "Thank you for coming at this difficult time."

[1603] Functions for Obon and New Year holidays

[1604] The user's device notifies the server that they will be offering incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user's device. The emotion engine analyzes the user's emotions, and the avatar personalizes the thank-you message. For example, if the user has a happy expression, it will display a message such as, "Thank you for coming to visit us. We're happy to see your smile."

[1605] Blessing function at the time of growth

[1606] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device, and at the same time, it personalizes the message using an emotion engine. For example, if the user is emotional, it displays an emotional message such as "Happy birthday. I'm so happy to have been able to watch you grow up." The server also performs point allocation processing and awards points to the user.

[1607] Specific examples

[1608] Example 1: Responses at memorial services

[1609] 1. The user device enters the date and time of the memorial service and sends it to the server.

[1610] 2. The server stores the information in a database.

[1611] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[1612] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[1613] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[1614] 6. The user's device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging message such as, "Thank you for coming during this difficult time."

[1615] Example 2: How to handle visits during Obon and New Year's

[1616] 1. The user device notifies the server when it is time to light the incense.

[1617] 2. The server sends a thank you message and asks the AI ​​to generate an avatar.

[1618] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[1619] 4. The server sends the message and avatar to the user's device.

[1620] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[1621] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message changes to "Thank you for visiting. I'm happy to see your smile."

[1622] Example 3: Grandchild's birthday celebration

[1623] 1. The server checks the date of birth from the database.

[1624] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[1625] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[1626] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[1627] 5. The user's device displays the message "Happy Birthday" and an avatar. If the user is touched, the message changes to "Happy Birthday. I'm so happy to watch you grow up."

[1628] 6. The server awards points to the user and notifies them on the device.

[1629] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[1630] The processing flow will be explained below.

[1631] Event support function for memorial services, etc.

[1632] Step 1:

[1633] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[1634] Step 2:

[1635] The terminal transmits the entered date and time information of the memorial service to the server.

[1636] Step 3:

[1637] The server stores the received date and time information in a database.

[1638] Step 4:

[1639] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[1640] Step 5:

[1641] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[1642] Step 6:

[1643] The server sends the generated avatar and message data to the device.

[1644] Step 7:

[1645] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[1646] Step 8:

[1647] The device will start displaying your avatar and message.

[1648] Step 9:

[1649] The emotion engine recognizes the user's emotions in real time.

[1650] Step 10:

[1651] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[1652] Step 11:

[1653] The device displays a personalized message and plays it to attendees.

[1654] Functions for Obon and New Year holidays

[1655] Step 1:

[1656] The user launches the app on their device and enters the information when lighting the incense.

[1657] Step 2:

[1658] The terminal transmits the input information to the server.

[1659] Step 3:

[1660] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[1661] Step 4:

[1662] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[1663] Step 5:

[1664] The server sends the generated message and avatar to the device.

[1665] Step 6:

[1666] The device will begin displaying a generated thank you message and avatar.

[1667] Step 7:

[1668] The emotion engine analyzes the user's emotions.

[1669] Step 8:

[1670] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[1671] Step 9:

[1672] The device will notify the user by displaying a personalized thank you message and avatar.

[1673] Blessing function at the time of growth

[1674] Step 1:

[1675] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[1676] Step 2:

[1677] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[1678] Step 3:

[1679] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[1680] Step 4:

[1681] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[1682] Step 5:

[1683] The terminal will begin displaying a congratulatory message and the generated avatar.

[1684] Step 6:

[1685] The emotion engine analyzes the user's emotions.

[1686] Step 7:

[1687] The emotion engine personalizes messages based on the results of emotion analysis and sends them to the device.

[1688] Step 8:

[1689] The device will notify the user by displaying a personalized congratulatory message and avatar.

[1690] Step 9:

[1691] The server performs a point giving process and gives points to the user.

[1692] Step 10:

[1693] The terminal notifies the user of the point allocation notice so that the user can check it.

[1694] Example 2

[1695] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1696] Conventional memorial events lacked a means for users to feel a sense of dialogue with the deceased, making it difficult to deepen an emotional connection. Furthermore, there was no way to provide messages tailored to the user's emotions at specific events or occasions. This prevented users from feeling closer to the deceased, making it difficult for them to empathize or feel comforted. Furthermore, there was a lack of mechanisms for appropriately managing the timing of event notifications and the display of messages from the deceased.

[1697] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1698] In this invention, the server includes a means for sending a message at a specific event using a model generated based on data of the deceased, a means for inputting event information and saving the information, a means for generating a model based on the event date and time and displaying it, a means for recognizing the user's emotions in real time and personalizing the message based on the recognition results, and a means for transmitting and displaying the generated message and model to a user terminal. This allows the user to feel a sense of dialogue with the deceased and deepen their emotional connection. Furthermore, by providing a personalized message tailored to the user's emotions, the user can gain empathy and comfort. Furthermore, the timing of event notifications and message display can be appropriately managed.

[1699] A "model" is a virtual character or message generation program generated based on data about the deceased.

[1700] "Events" are events or ceremonies that are held on specific dates or times, such as memorial services, Obon, and New Year's.

[1701] A "message" is information such as text, audio, or video sent from the deceased to the user.

[1702] A "user terminal" is a device used by a user, such as a smartphone or a personal computer.

[1703] The "server" is a central computer that manages and operates the database, generative AI, and emotion engine.

[1704] The "input means" is an interface through which the user inputs event information and transmits it to the server.

[1705] "Storage means" is a function that stores input event information in a database.

[1706] "Display means" is a function that displays the generated model and messages on the user terminal.

[1707] "Emotion recognition means" is a technology for analyzing a user's emotions in real time and obtaining the results.

[1708] "Personalization means" is a function that individually optimizes messages based on acquired emotional information.

[1709] A "notification means" is a communication means for reporting specific actions or events from a user terminal to a server.

[1710] The "point giving means" is a function for giving points to a user and notifying the user.

[1711] The present invention is a system that uses a model generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, a model, an event information input means, a model display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[1712] Overall system configuration

[1713] The system mainly consists of the following components:

[1714] 1. User terminal: A device such as a smartphone or PC that users use to enter event information and receive messages.

[1715] 2. Server: The central computer that manages the database, generative AI, and emotion engine. It stores event information, generates models, generates messages, recognizes emotions, and awards points.

[1716] 3. Generative AI: An artificial intelligence program that generates models and messages based on data about the deceased.

[1717] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[1718] 5. Database: Storage for storing data of the deceased, user information, and event information.

[1719] Program processing overview

[1720] The program processing in this system is explained as follows.

[1721] Functions for handling events such as memorial services

[1722] The user enters the date and time of the memorial service into their device, which then sends the information to the server. The server stores the received information in a database. The day before the memorial service, the server requests the generative AI to generate a model of the deceased. On the day of the memorial service, when the user opens the app, the server sends the model and a message to the user's device and displays them. The emotion engine recognizes the user's emotions in real time and personalizes the message accordingly. For example, if the user has a sad expression, the model will display an encouraging message such as, "Thank you for coming at this difficult time."

[1723] Functions for Obon and New Year holidays

[1724] When a user lights incense during Obon or New Year's, the information is sent from the user's device to the server. Based on this information, the server requests the generation AI to generate a message of gratitude and a model. The generated data is sent from the server to the user's device. The emotion engine analyzes the user's emotions and personalizes the message. For example, if the user has a happy expression, a message such as "Thank you for coming to visit us. We're happy to see your smile" is displayed.

[1725] Blessing function at the time of growth

[1726] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, the server requests the generation AI to generate a congratulatory message and model. On the anniversary day, the server sends the generated message and model to the user's device, and the emotion engine analyzes the user's emotions and personalizes the message. For example, if the user is emotional, a message such as "Happy birthday. I'm so happy to have watched you grow up" is displayed. The server also performs the process of awarding points and notifies the user of the points.

[1727] Specific examples

[1728] Functions for handling events such as memorial services

[1729] Examples:

[1730] 1. The user opens the app on their smartphone and enters the date of the memorial service (December 25, 2023) and the time (10:00).

[1731] 2. The user device sends the data "Memorial date: December 25, 2023, time: 10:00" to the server.

[1732] 3. The server stores the received information in a database.

[1733] 4. On December 24, 2023, the server asks the generation AI to "generate a model for a memorial service."

[1734] 5. When the user opens the app on the day of the memorial service, the server sends the model and a message.

[1735] 6. The emotion engine analyzes the user's sad expression and generates a personalized message.

[1736] Prompt Sentence Examples

[1737] Here are some example prompts to input to a generative AI model:

[1738] "The memorial date is approaching. Please generate a model of the deceased and an appropriate message. Include an encouraging message if the user has a sad expression."

[1739] This allows users to feel like they are having a conversation with the deceased, deepening their emotional connection. Furthermore, by providing personalized messages tailored to the user's emotions, it is possible to gain empathy and comfort. Furthermore, it is possible to appropriately manage the timing of event notifications and message display.

[1740] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1741] Processing steps for events such as memorial services

[1742] Step 1:

[1743] The user enters the date and time of the memorial service.

[1744] Input: Date and time of memorial service (e.g. December 25, 2023, 10:00).

[1745] Output: The input information is saved on the user's terminal.

[1746] Specific operation: The user opens a dedicated app on their smartphone and enters the date and time of the memorial service in the form that appears.

[1747] Step 2:

[1748] The user terminal transmits information about the memorial service to the server.

[1749] Input: Date and time of memorial service.

[1750] Output: Memorial service information is sent to the server.

[1751] Specific operation: The date and time of the memorial service entered by the user is sent to the server as an HTTP request.

[1752] Step 3:

[1753] The server stores the memorial information in a database.

[1754] Input: Date and time of memorial service.

[1755] Output: Memorial information stored in a database.

[1756] Specific operation: The server saves the date and time of the memorial service received as a new record in a database table.

[1757] Step 4:

[1758] The day before the memorial service, the server requests the generation AI to generate a model.

[1759] Input: Trigger based on date and time before and after the memorial service, memorial service information.

[1760] Output: The generated model and messages.

[1761] Specific operation: The server uses a trigger set on the day before the memorial service to send a prompt message to the generation AI saying, "Please generate a model for the memorial service."

[1762] Step 5:

[1763] On the day of the memorial service, the user opens the application.

[1764] Input: User action.

[1765] Output: The application sends a request to the server.

[1766] Specific action: The user launches the app on their smartphone.

[1767] Step 6:

[1768] The server sends the avatar and message to the user's terminal.

[1769] Input: The generated avatar and message.

[1770] Output: The avatar and message displayed on the user's device.

[1771] Specific operation: The server sends the avatar and message received from the generated AI to the user's device in JSON format.

[1772] Step 7:

[1773] The emotion engine recognizes the user's emotions.

[1774] Input: User's facial expression data.

[1775] Output: Recognized emotion data (e.g. sadness, joy).

[1776] Specific operation: Using the camera on the user's device, the emotion engine analyzes the user's facial expressions in real time.

[1777] Step 8:

[1778] The server personalizes the message and displays it on the user's terminal.

[1779] Input: Recognized emotion data, generated message.

[1780] Output: A personalized message.

[1781] Specific operation: The server modifies the message based on the emotion data and sends it to the user's device. If the user has a sad expression, an encouraging message such as "Thank you for coming during this difficult time" will be displayed.

[1782] (Application example 2)

[1783] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1784] Conventional dialogue systems for the deceased only send messages using an avatar generated based on the deceased's data, but are unable to provide personalized messages based on the user's emotions. Furthermore, it is difficult to provide an emotionally rich experience for specific events. Therefore, there is a need for a system that allows users to empathize and feel deeply moved.

[1785] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1786] In this invention, the server includes means for sending a message at a specific event using an avatar generated based on data of the deceased, means for inputting and saving event information, means for generating an avatar based on the event date and time and displaying it, and means for recognizing a user's emotions in real time and personalizing a message based on the emotions, thereby making it possible to recognize a user's emotions and provide a personalized message.

[1787] "Data of the deceased" refers to information about a deceased person, including digital data such as images, audio, video, and text.

[1788] An "avatar" is a digital image created based on the data of a deceased person, recreating the deceased in a virtual space.

[1789] "Specific events" refers to special occasions or anniversaries related to the deceased, including memorial services, birthdays, Obon, New Year's, and other regular events.

[1790] The "means for sending a message" is a method by which the avatar can provide a text or voice message to the user depending on the date and situation of the event.

[1791] "Event information" is detailed data about a particular event, such as the date, time, and location.

[1792] "Storage means" refers to the methods and technologies used to store event information and other related data in a database or the like.

[1793] "Display means" refers to a technique for visually displaying the generated avatar or message on the user's terminal.

[1794] "Means for recognizing a user's emotions in real time and personalizing messages based on those emotions" refers to a method that uses emotion recognition technology to determine a user's current emotional state and appropriately change the message content accordingly.

[1795] A "personalized message" is a text or voice message that is individually tailored to the user's emotions and situation.

[1796] This invention is a system that uses an avatar generated based on the data of a deceased person to send a message at a specific event. By combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. Details of the embodiment are described below.

[1797] 1. System Configuration

[1798] The system mainly consists of the following components:

[1799] User device: A device such as a smartphone or computer that users use to enter event information and receive messages.

[1800] Server: A central computer that manages the database, generative AI, and emotion engine. It stores event information, generates avatars, generates messages, and recognizes emotions.

[1801] Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[1802] Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[1803] Database: Storage for storing data of deceased persons, user information, and event information.

[1804] 2. Program Processing Overview

[1805] The user enters event information on the user device and sends it to the server, which stores it in a database. As the event date approaches, the server requests the generation AI to generate an avatar of the deceased person. On the day of the event, the emotion engine recognizes the user's emotions in real time, and the avatar and message are displayed according to those emotions.

[1806] 3. Hardware and Software Used

[1807] Hardware:

[1808] Smartphone

[1809] server

[1810] software:

[1811] EmotionEngine: Software for recognizing user emotions.

[1812] Avatar Generator (generative AI): Software that generates avatars based on data of deceased people.

[1813] Message Generator (Generative AI): Software that generates messages based on data about the deceased and event information.

[1814] Database: Database management software for storing user and event information.

[1815] 4. Specific Examples

[1816] ■ Responses to memorial services:

[1817] 1. The user terminal enters the date and time of the memorial service and sends it to the server.

[1818] 2. The server stores the information in a database.

[1819] 3. The day before the memorial service, the server requests the generation AI to generate an avatar.

[1820] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[1821] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[1822] 6. The user device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging one such as, "Thank you for coming during this difficult time."

[1823] ■ Visits during Obon and New Year's:

[1824] 1. The user device notifies the server when it is time to light the incense.

[1825] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[1826] 3. The generation AI generates a thank you message and an avatar and sends them back to the server.

[1827] 4. The server sends the message and avatar to the user's device.

[1828] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[1829] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message is changed to "Thank you for visiting. I'm happy to see your smile."

[1830] ■ Grandchild's birthday celebration:

[1831] 1. The server checks the date of birth from the database.

[1832] 2. On the day before the anniversary, ask the generation AI to generate a congratulatory message and an avatar.

[1833] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[1834] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[1835] 5. The user device displays the message "Happy Birthday" and the avatar. If the user is touched, the message changes to "Happy Birthday. I'm so glad to watch you grow up."

[1836] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[1837] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1838] Step 1:

[1839] The user enters event information (e.g., the date and time of the memorial service) into the terminal and sends it to the server.

[1840] Input: Event information (date, time)

[1841] Data processing: User terminal packages event information

[1842] Output: Request sent to server

[1843] Step 2:

[1844] The server stores the received event information in a database.

[1845] Input: Received event information

[1846] Data processing: Converting event information into the appropriate format

[1847] Output: Save results to database

[1848] Step 3:

[1849] As the event date approaches, the server asks the generation AI to generate an avatar of the deceased person.

[1850] Input: Event information, deceased person's data

[1851] Data processing: Generative AI analyzes the data of the deceased and generates an avatar

[1852] Output: The generated avatar

[1853] Step 4:

[1854] On the day of the event, the server sends an avatar and a message to the user's terminal.

[1855] Input: Generated avatar, standard message

[1856] Data processing: Packaging avatars and messages

[1857] Output: Send request to user terminal

[1858] Step 5:

[1859] The user's device displays the received avatar and message.

[1860] Input: avatar, message

[1861] Data processing: Convert received data into a display format

[1862] Output: Display avatar and message on user's device

[1863] Step 6:

[1864] The emotion engine recognizes the user's emotions in real time.

[1865] Input: User's facial expression data (camera input)

[1866] Data calculation: Emotion engine analyzes facial expression data

[1867] Output: Recognized emotion data

[1868] Step 7:

[1869] The server personalizes the message based on the recognized emotion data.

[1870] Input: Standard message, recognized emotion data

[1871] Data processing: Message generation based on emotional data

[1872] Output: Personalized message

[1873] Step 8:

[1874] The user terminal displays the personalized message.

[1875] Input: Personalized message

[1876] Data processing: Convert received messages into a display format

[1877] Output: Display a personalized message on the user's device

[1878] Through the above steps, a personalized message according to emotions is provided based on the event information entered by the user.

[1879] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1880] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1881] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1882] [Fourth embodiment]

[1883] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1884] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1885] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1886] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1887] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1888] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1889] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1890] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1891] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1892] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1893] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1894] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1895] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1896] This invention is a system that uses an avatar generated based on the data of the deceased to send a message from the deceased at a specific event. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, and a server are used.

[1897] Overall system configuration

[1898] The system mainly consists of the following components:

[1899] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[1900] 2. Server: The central computer that manages the database and the AI ​​generation. It stores event information, generates avatars, generates messages, and awards points.

[1901] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[1902] 4. Database: Storage for storing data of deceased people, user information, and event information.

[1903] Program processing overview

[1904] The program processing in this system is explained as follows.

[1905] Event support function for memorial services, etc.

[1906] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and when the date and time of the memorial service approaches, it requests the AI ​​to generate an avatar of the deceased. On the day of the memorial service, the user's device displays this avatar and a message, conveying the deceased's words of gratitude to attendees.

[1907] Functions for Obon and New Year holidays

[1908] The user device notifies the server that they will light incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user device. The user device then displays the message and avatar, conveying the deceased's gratitude.

[1909] Blessing function at the time of growth

[1910] The server manages the anniversaries of the growth of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device and simultaneously awards points. The user's device displays the congratulatory message and a notification that points have been awarded.

[1911] Specific examples

[1912] Example 1: Responses at memorial services

[1913] 1. The user device enters the date and time of the memorial service and sends it to the server.

[1914] 2. The server stores the information in a database.

[1915] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[1916] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[1917] 5. The user device plays the message "Thank you for coming to the memorial service today."

[1918] Example 2: How to handle visits during Obon and New Year's

[1919] 1. The user device notifies the server when it is time to light the incense.

[1920] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[1921] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[1922] 4. The server sends the message and avatar to the user's device.

[1923] 5. The user device displays the message "Thank you for visiting during Obon."

[1924] Example 3: Grandchild's birthday celebration

[1925] 1. The server checks the date of birth from the database.

[1926] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[1927] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[1928] 4. The user's device displays the message "Happy Birthday" and an avatar.

[1929] 5. The server awards points to the user and notifies them on the device.

[1930] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[1931] The processing flow will be explained below.

[1932] Event support function for memorial services, etc.

[1933] Step 1:

[1934] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[1935] Step 2:

[1936] The terminal transmits the entered date and time information of the memorial service to the server.

[1937] Step 3:

[1938] The server stores the received date and time information in a database.

[1939] Step 4:

[1940] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[1941] Step 5:

[1942] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[1943] Step 6:

[1944] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[1945] Step 7:

[1946] The server sends the generated avatar and message data to the device.

[1947] Step 8:

[1948] The device displays the generated avatar and message and plays it to the attendees.

[1949] Functions for Obon and New Year holidays

[1950] Step 1:

[1951] The user launches the app on their device and enters the information when lighting the incense.

[1952] Step 2:

[1953] The terminal transmits the input information to the server.

[1954] Step 3:

[1955] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[1956] Step 4:

[1957] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[1958] Step 5:

[1959] The server sends the generated message and avatar to the device.

[1960] Step 6:

[1961] The device displays the generated thank you message and avatar to notify the user.

[1962] Blessing function at the time of growth

[1963] Step 1:

[1964] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[1965] Step 2:

[1966] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[1967] Step 3:

[1968] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[1969] Step 4:

[1970] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[1971] Step 5:

[1972] The device displays the generated congratulatory message and avatar to notify the user.

[1973] Step 6:

[1974] At the same time, the server performs a point awarding process and awards points to the user.

[1975] Step 7:

[1976] The terminal notifies the user of the point allocation notice so that the user can check it.

[1977] Example 1

[1978] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1979] The present invention aims to deepen the bond with the deceased by sending messages from the deceased at specific events using avatars generated based on the deceased's data. However, current systems lack a mechanism for dynamically generating messages and avatars for specific dates or events, and may not meet user expectations. Furthermore, there is a need for an efficient system that can accommodate multiple events and anniversaries and send messages from the deceased in a timely manner.

[1980] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1981] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for sending a prompt to a generative AI model based on the event date and time to generate an avatar and a message, and a means for sending and displaying the generated avatar and message to a user terminal, thereby enabling messages and avatars from the deceased to be generated and sent in a timely manner and provided to users.

[1982] "Data of the deceased" refers to personal information and recorded data relating to the deceased, including the deceased's voice, images, messages, etc.

[1983] An "avatar" is a digital character or image created based on the data of a deceased person.

[1984] An "event" is an event or ceremony that takes place on a specific day or time, and includes memorial services, anniversaries, Obon, New Year's, etc.

[1985] A "message" is a verbal message in the form of text, audio, video, etc., generated based on the data of the deceased.

[1986] A "generative AI model" is a model that uses artificial intelligence to generate digital content such as text, images, and audio. In this invention, it generates an avatar and message based on data about the deceased.

[1987] A "prompt" is an instruction that is input to a generative AI model to generate specific content.

[1988] "User device" refers to a digital device such as a smartphone or computer used by a user, and is used to input event information and receive and display avatars and messages.

[1989] The "server" is a central control device that manages the database, controls the generative AI model, and requests and sends messages and avatars.

[1990] "Event information" refers to information such as the date, time, and content related to a specific event, which is input by the user and sent to the server.

[1991] "Means for storing" refers to the function of storing input event information and data on the deceased in storage such as a database.

[1992] "Display means" refers to a function for visually displaying the generated avatar and message on the screen of the user terminal.

[1993] This invention is a system that uses an avatar generated based on the data of the deceased to send messages from the deceased at specific events. To implement this system, hardware and software such as a user terminal, a server, a generative AI model, and a database are used.

[1994] Hardware and software used

[1995] 1. User Device

[1996] A device that allows users to enter event information and receive messages. Examples include smartphones and PCs.

[1997] 2. Server

[1998] A central control unit that manages the entire system and links with the database and generative AI model.

[1999] 3. Generative AI Models

[2000] An artificial intelligence model that generates an avatar and message based on data about the deceased. A specific example would be OpenAI GPT-3.

[2001] 4. Database

[2002] A storage system that stores data on deceased people, user information, and event information. Examples include MySQL and PostgreSQL.

[2003] System configuration and processing overview

[2004] Event support function for memorial services, etc.

[2005] The user enters the date and time of the memorial service through the user's device and sends it to the server. The server stores this information in a database and, the day before the memorial service, sends a prompt to the generative AI model to generate an avatar of the deceased. An example of a prompt is: "Please generate a message of thanks for the memorial service in the voice of the deceased." On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays it.

[2006] Functions for Obon and New Year holidays

[2007] The user uses their device to notify the server that they will light incense during Obon or New Year's visits. The server then sends a prompt to the generative AI model, which generates a message of gratitude and an avatar. An example of a prompt is: "Please generate a message of gratitude during Obon in the voice of the deceased." The generated avatar and message are then sent from the server to the user's device, where they are displayed and played back on the device.

[2008] Blessing function at the time of growth

[2009] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, it sends a prompt to the generative AI model, which generates a congratulatory message and avatar. An example of a prompt is: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased." On the anniversary day, the server sends the generated avatar and message to the user's device, and also notifies the user of the points awarded. The user's device displays and plays the message, and notifies the user of the points.

[2010] Specific examples

[2011] Example 1: Responses at memorial services

[2012] 1. The user opens the smartphone app and enters the date of the memorial service.

[2013] 2. The device sends the information to the server, which stores it in a database.

[2014] 3. The server sends a prompt to the generative AI model the day before the memorial service, generating an avatar and message.

[2015] 4. On the day of the memorial service, the server sends the generated avatar and message to the user's device, which displays and plays them.

[2016] Example 2: How to handle visits during Obon and New Year's

[2017] 1. Notify the user when lighting incense.

[2018] 2. The server sends a prompt to the generative AI model, which generates a thank you message and an avatar.

[2019] 3. The generative AI model sends the avatar and message back to the server, which then sends it to the user's device.

[2020] 4. The user device displays and plays the message and avatar.

[2021] Example 3: Grandchild's birthday celebration

[2022] 1. The server checks the date of birth from the database.

[2023] 2. On the day before the anniversary, send a prompt to the generative AI model to generate a congratulatory message and avatar.

[2024] 3. On the anniversary day, the server sends the generated avatar and a message to the user's device.

[2025] 4. The user's device displays and plays the message and avatar, and also notifies them of points.

[2026] In this way, the system can be used to convey the thoughts and messages of the deceased to living family and friends, deepening the bond with the deceased.

[2027] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2028] Processing steps for event response functions such as memorial services

[2029] Step 1: Enter your event information

[2030] The user inputs the date and time of the memorial service into the user terminal and presses the "send" button.

[2031] Input: Date and time of the memorial service.

[2032] Output: Data sent to the server.

[2033] Specific operation: The user enters the date of the memorial service (October 31, 2023) and the time (10:00) on the application screen and clicks the send button.

[2034] Step 2: Save your data

[2035] The server stores the date and time of the received memorial service in a database.

[2036] Input: Date and time data for the memorial service.

[2037] Output: Event information stored in a database.

[2038] Specific operation: The date "October 31, 2023" and time "10:00" received by the server are inserted into the event information table in the database.

[2039] Step 3: Request avatar creation

[2040] The day before the memorial service, the server requests the generative AI model to generate an avatar of the deceased person, using a prompt text.

[2041] Input: prompt statement.

[2042] Output: A request to the generative AI model to generate a new model.

[2043] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of gratitude for the memorial service in the voice of the deceased."

[2044] Step 4: Receiving avatars and messages

[2045] The generative AI model generates an avatar and message based on the request and sends it back to the server.

[2046] Input: prompt statement.

[2047] Output: The generated avatar and message.

[2048] Specific operation: The generative AI model generates an avatar that says, "Thank you for coming to the memorial service today," along with a message, and sends it back to the server.

[2049] Step 5: Message and Avatar Notifications

[2050] On the day of the memorial service, the server sends the generated avatar and a message to the user's terminal.

[2051] Input: The generated avatar and message.

[2052] Output: Data sent to the user's terminal.

[2053] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for coming to the memorial service today."

[2054] Step 6: Display your avatar and message

[2055] The user terminal displays and plays the avatar and message received from the server.

[2056] Input: Avatar and message sent from the server.

[2057] Output: User notification and display.

[2058] Specific operation: The user's device displays an avatar on the application and plays a message saying, "Thank you for coming to the memorial service today."

[2059] Processing steps for Obon and New Year's holiday support functions

[2060] Step 1: Incense Notification

[2061] A user uses a user terminal to notify that incense will be lit when visiting during Obon or New Year's.

[2062] Input: Notification of incense being lit.

[2063] Output: Notification data to the server.

[2064] Specific operation: The user clicks the "Light Incense" button in the application, notifying the server.

[2065] Step 2: Request avatar creation

[2066] The server sends a message of thanks and requests the AI ​​model to generate an avatar. A prompt is used.

[2067] Input: prompt statement.

[2068] Output: A request to the generative AI model to generate a new model.

[2069] Specific operation: The server sends a prompt to the generation AI model saying, "Please generate a message of thanks for Obon in the voice of the deceased."

[2070] Step 3: Receiving avatars and messages

[2071] The generative AI model generates a thank you message and avatar and sends them back to the server.

[2072] Input: prompt statement.

[2073] Output: The generated avatar and message.

[2074] Specific operation: The generative AI model generates an avatar that says, "Thank you for visiting during Obon," along with a message, and sends it back to the server.

[2075] Step 4: Message and Avatar Notifications

[2076] The server sends the generated message and avatar to the user terminal.

[2077] Input: The generated avatar and message.

[2078] Output: Data sent to the user's terminal.

[2079] Specific operation: The server sends the generated avatar to the user's device along with a message saying "Thank you for visiting during Obon."

[2080] Step 5: Displaying your message and avatar

[2081] The user terminal displays it and plays back the message "Thank you for visiting during Obon."

[2082] Input: Avatar and message sent from the server.

[2083] Output: User notification and display.

[2084] Specific operation: The user's device displays an avatar on the application and plays the message "Thank you for visiting during Obon."

[2085] Processing steps for the blessing function during growth

[2086] Step 1: Manage your anniversaries

[2087] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database.

[2088] Input: Anniversary data in the database.

[2089] Output: Anniversary check result.

[2090] What it does: Every day the server checks the database to see if an anniversary is approaching.

[2091] Step 2: Request avatar creation

[2092] When an anniversary approaches, the server asks the generative AI model to generate a congratulatory message and an avatar, using a prompt text.

[2093] Input: prompt statement.

[2094] Output: A request to the generative AI model to generate a new model.

[2095] Specific operation: The server sends a prompt to the generation AI model: "Please generate a message celebrating the grandchild's birthday in the voice of the deceased."

[2096] Step 3: Receiving avatars and messages

[2097] The generative AI model generates a congratulatory message and avatar and sends them back to the server.

[2098] Input: prompt statement.

[2099] Output: The generated avatar and message.

[2100] Specific operation: The generative AI model generates an avatar that says "Happy Birthday" and the message, and sends it back to the server.

[2101] Step 4: Message and Avatar Notifications

[2102] On the anniversary day, the server sends the generated message and avatar to the user's terminal.

[2103] Input: The generated avatar and message.

[2104] Output: Data sent to the user's terminal.

[2105] Specific operation: The server sends the generated avatar and a message saying "Happy Birthday" to the user's device.

[2106] Step 5: Points awarded

[2107] The server awards points to the user and sends a notification to the user's terminal.

[2108] Input: User information and point data.

[2109] Output: Points awarded notification.

[2110] Specific operation: The server will add 100 points to the user's account and send a notification to the user's device.

[2111] Step 6: Displaying your message and avatar

[2112] The user terminal displays the generated avatar and message, and plays back the message "Happy Birthday."

[2113] Input: Avatar and message sent from the server.

[2114] Output: User notification and display.

[2115] Specific operation: The user's device displays an avatar on the application and plays the message "Happy Birthday."

[2116] (Application example 1)

[2117] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2118] Previous systems that allowed users to relive memories with the deceased were limited to specific events and dates, limiting the opportunities for users to feel a connection with the deceased in their daily lives. Furthermore, they were inadequate in supporting visits other than memorial services and specific days, or memorable places and items, and there was no system that allowed users to feel a deeper connection with the deceased.

[2119] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2120] In this invention, the server includes a means for sending a message at a specific event using an avatar generated based on data of the deceased, a means for inputting and saving event information, a means for generating an avatar based on the event date and time and displaying it, a means for the user to use a device that recognizes a specific place or product and transmit that information, and a means for using a device that displays the avatar and message generated based on data of the deceased, allowing the user to relive memories of the deceased in their daily lives and receive messages related to specific places or products.

[2121] A "deceased person" is someone who has already passed away and is recreated within the system based on data from their life.

[2122] "Data" refers to information about the deceased person during their lifetime, including text data, audio data, image data, and the like.

[2123] An "avatar" is a digital character generated based on the data of a deceased person, and its role is to display a message or image of the deceased to the user.

[2124] An "event" is an event or activity that takes place at a specific date, time, or place, and includes memorial services, anniversaries, and the like.

[2125] A "message" is a message or thank you that the deceased person's avatar conveys to the user.

[2126] "Means" are the tools and methods a system uses to achieve its goals.

[2127] A "user" is an individual who uses the system and operates it to relive memories of a deceased loved one.

[2128] "Device" is a general term for the hardware required for users to use the system, including smart glasses and smartphones.

[2129] A "server" is a computer that manages and processes data at the center of the system, and is where the generative AI and database run.

[2130] "Generative AI" is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[2131] A "database" is a storage system for storing data on deceased people, user information, event information, etc.

[2132] The system for implementing this invention comprises the following components: a user terminal, a server, a generating AI, a database, smart glasses, and, if necessary, a recognition device (such as a camera).

[2133] Overall system flow

[2134] 1. User Device:

[2135] Users use smart glasses or a smartphone to scan a specific location or product.

[2136] Used to enter event information into the system.

[2137] 2. Server:

[2138] Responsible for managing and processing data.

[2139] Event information is stored in a database, and avatars and messages are generated through generative AI.

[2140] 3. Generation AI:

[2141] It is an artificial intelligence program that generates avatars and messages based on data about the deceased.

[2142] OpenAI's GPT-4 is used as a concrete example.

[2143] 4. Database:

[2144] A storage system for storing data on deceased individuals, user information, and event information. A database management system such as MySQL is used.

[2145] 5. Smart Glasses:

[2146] A device used by a user to recognize a specific place or product, such as Google Glass.

[2147] The deceased person's avatar and message are displayed in the user's field of view.

[2148] Specific processing of the program

[2149] 1. Data Collection:

[2150] The user puts on the smart glasses and scans memorable places and products.

[2151] 2. Data transmission:

[2152] The scanned data is sent from the user device to the server.

[2153] 3. Data Retention:

[2154] The server stores the received data in a database.

[2155] 4. Avatar and message generation:

[2156] A generation AI on the server generates avatars and messages based on the data.

[2157] GPT-4 and other AIs are used as generative AI.

[2158] 5. Displaying the results:

[2159] The generated avatar and message are sent from the server to the smart glasses via the user terminal.

[2160] Smart glasses display avatars and messages in the user's field of vision.

[2161] Specific examples

[2162] For example, when a user visits a specific location in a restaurant that holds special memories for the deceased, they can scan the location with the smart glasses. The server receives the data and asks the AI ​​to generate an avatar of the deceased and create a message related to the location. A message from the deceased, such as "I had a wonderful time eating here," is displayed on the screen.

[2163] Prompt Sentence Examples

[2164] "A user visits a specific seat in a restaurant. Generate a thank you message based on memories of the deceased person being at that location."

[2165] In this way, users can receive messages from the deceased related to specific places or items, deepening their connection with the deceased as part of their daily lives.

[2166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2167] Step 1:

[2168] Users scan memorable places and products

[2169] A user puts on the smart glasses and scans a specific location or item.

[2170] Input: The user visually recognizes a specific location or product through the camera on the smart glasses.

[2171] Output: Scanned data (image data, QR code, etc.) is generated and sent to the next step.

[2172] Step 2:

[2173] Sending scan data to the server

[2174] The user terminal transmits the scan data to the server.

[2175] Input: The scan data generated in step 1.

[2176] Output: The scanned data is sent to the server.

[2177] Step 3:

[2178] The server stores the scan data in a database

[2179] The server stores the received scan data in a database.

[2180] Input: Scan data sent to the server.

[2181] Output: The scan data is saved to the database and the result of the save success is returned.

[2182] Step 4:

[2183] The server asks the AI ​​to generate an avatar and a message.

[2184] The server asks the generation AI to generate an avatar and message based on the saved scan data.

[2185] Input: Scan data stored in the database.

[2186] Output: The prompt is sent to the generation AI, and a request is made to generate an avatar and message.

[2187] Step 5:

[2188] Generative AI generates avatars and messages

[2189] The generation AI generates an avatar and message based on the prompt text.

[2190] A specific example is the prompt: "The user visited a specific seat in a restaurant. Generate a thank-you message based on memories of the deceased person being at that location."

[2191] Input: The prompt sent to the generation AI.

[2192] Output: The generated avatar and message.

[2193] Step 6:

[2194] Send the generated avatar and message to the server

[2195] The server receives the generated avatar and message.

[2196] Input: Avatar and message returned by the generation AI.

[2197] Output: The avatar and message are stored on the server and ready to be sent to the user's device.

[2198] Step 7:

[2199] The server sends the avatar and message to the user's device.

[2200] The server sends the avatar and the message to the user's smart glasses.

[2201] Input: Avatar and message stored on the server.

[2202] Output: The avatar and message are transferred to the user's device and displayed on the user's device.

[2203] Step 8:

[2204] Users receive avatars and messages through smart glasses

[2205] The smart glasses display an avatar and a message in the user's field of vision.

[2206] Input: Avatar and message sent from the server.

[2207] Output: An avatar is displayed on the user's smart glasses and a message is read out loud.

[2208] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2209] This invention is a system that uses an avatar generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, an avatar, an event information input means, an avatar display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[2210] Overall system configuration

[2211] The system mainly consists of the following components:

[2212] 1. User device: A device such as a smartphone or PC that users use to enter event information and receive messages.

[2213] 2. Server: The central computer that manages the database, generation AI, and emotion engine. It stores event information, generates avatars, generates messages, recognizes emotions, and awards points.

[2214] 3. Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[2215] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[2216] 5. Database: Storage for storing data of deceased persons, user information, and event information.

[2217] Program processing overview

[2218] The program processing in this system is explained as follows.

[2219] Event support function for memorial services, etc.

[2220] The user enters the date and time of the memorial service on the user's device and sends it to the server. The server stores this information in a database, and as the date and time of the memorial service approaches, it requests the generation AI to generate an avatar of the deceased. On the day of the memorial service, the emotion engine recognizes the user's emotions in real time, and the avatar and message displayed will match those emotions. For example, if the user has a sad expression, the avatar will display an encouraging message such as, "Thank you for coming at this difficult time."

[2221] Functions for Obon and New Year holidays

[2222] The user's device notifies the server that they will be offering incense during Obon or New Year's visits. The server then requests the AI ​​to generate a thank-you message and avatar, and sends the generated data to the user's device. The emotion engine analyzes the user's emotions, and the avatar personalizes the thank-you message. For example, if the user has a happy expression, it will display a message such as, "Thank you for coming to visit us. We're happy to see your smile."

[2223] Blessing function at the time of growth

[2224] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren. As anniversaries approach, it requests the generation AI to generate a congratulatory message and avatar from the deceased. On the anniversary day, the server sends the generated message and avatar to the user's device, and at the same time, it personalizes the message using an emotion engine. For example, if the user is emotional, it displays an emotional message such as "Happy birthday. I'm so happy to have been able to watch you grow up." The server also performs point allocation processing and awards points to the user.

[2225] Specific examples

[2226] Example 1: Responses at memorial services

[2227] 1. The user device enters the date and time of the memorial service and sends it to the server.

[2228] 2. The server stores the information in a database.

[2229] 3. The day before the memorial service, the server asks the AI ​​to generate an avatar.

[2230] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[2231] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[2232] 6. The user's device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging message such as, "Thank you for coming during this difficult time."

[2233] Example 2: How to handle visits during Obon and New Year's

[2234] 1. The user device notifies the server when it is time to light the incense.

[2235] 2. The server sends a thank you message and asks the AI ​​to generate an avatar.

[2236] 3. The generation AI generates a thank you message and avatar and sends them back to the server.

[2237] 4. The server sends the message and avatar to the user's device.

[2238] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[2239] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message changes to "Thank you for visiting. I'm happy to see your smile."

[2240] Example 3: Grandchild's birthday celebration

[2241] 1. The server checks the date of birth from the database.

[2242] 2. On the day before the anniversary, ask the AI ​​to generate a congratulatory message and avatar.

[2243] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[2244] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[2245] 5. The user's device displays the message "Happy Birthday" and an avatar. If the user is touched, the message changes to "Happy Birthday. I'm so happy to watch you grow up."

[2246] 6. The server awards points to the user and notifies them on the device.

[2247] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[2248] The processing flow will be explained below.

[2249] Event support function for memorial services, etc.

[2250] Step 1:

[2251] The user uses the terminal, starts the application, and inputs the date and time of the memorial service.

[2252] Step 2:

[2253] The terminal transmits the entered date and time information of the memorial service to the server.

[2254] Step 3:

[2255] The server stores the received date and time information in a database.

[2256] Step 4:

[2257] The day before the memorial service, the server asks the AI ​​to generate an avatar based on the deceased's data.

[2258] Step 5:

[2259] The generating AI generates an avatar and message for the deceased person and sends the data back to the server.

[2260] Step 6:

[2261] The server sends the generated avatar and message data to the device.

[2262] Step 7:

[2263] On the day of the memorial service, the user opens the application using the terminal and connects to the server.

[2264] Step 8:

[2265] The device will start displaying your avatar and message.

[2266] Step 9:

[2267] The emotion engine recognizes the user's emotions in real time.

[2268] Step 10:

[2269] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[2270] Step 11:

[2271] The device displays a personalized message and plays it to attendees.

[2272] Functions for Obon and New Year holidays

[2273] Step 1:

[2274] The user launches the app on their device and enters the information when lighting the incense.

[2275] Step 2:

[2276] The terminal transmits the input information to the server.

[2277] Step 3:

[2278] The server receives the input information and sends a thank you message to the generating AI, requesting it to generate an avatar.

[2279] Step 4:

[2280] The generating AI generates a thank-you message and an avatar and sends the data back to the server.

[2281] Step 5:

[2282] The server sends the generated message and avatar to the device.

[2283] Step 6:

[2284] The device will begin displaying a generated thank you message and avatar.

[2285] Step 7:

[2286] The emotion engine analyzes the user's emotions.

[2287] Step 8:

[2288] The emotion engine personalizes messages based on the user's emotions and sends them to the device.

[2289] Step 9:

[2290] The device will notify the user by displaying a personalized thank you message and avatar.

[2291] Blessing function at the time of growth

[2292] Step 1:

[2293] The server retrieves information about the growth anniversaries of the user's grandchildren and great-grandchildren from the database.

[2294] Step 2:

[2295] On the day before the anniversary, the server asks the generation AI to generate a congratulatory message and an avatar.

[2296] Step 3:

[2297] The generation AI generates a congratulatory message and an avatar and sends the data back to the server.

[2298] Step 4:

[2299] On the anniversary day, the server transmits the generated message and avatar to the user terminal.

[2300] Step 5:

[2301] The terminal will begin displaying a congratulatory message and the generated avatar.

[2302] Step 6:

[2303] The emotion engine analyzes the user's emotions.

[2304] Step 7:

[2305] The emotion engine personalizes messages based on the results of emotion analysis and sends them to the device.

[2306] Step 8:

[2307] The device will notify the user by displaying a personalized congratulatory message and avatar.

[2308] Step 9:

[2309] The server performs a point giving process and gives points to the user.

[2310] Step 10:

[2311] The terminal notifies the user of the point allocation notice so that the user can check it.

[2312] Example 2

[2313] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2314] Conventional memorial events lacked a means for users to feel a sense of dialogue with the deceased, making it difficult to deepen an emotional connection. Furthermore, there was no way to provide messages tailored to the user's emotions at specific events or occasions. This prevented users from feeling closer to the deceased, making it difficult for them to empathize or feel comforted. Furthermore, there was a lack of mechanisms for appropriately managing the timing of event notifications and the display of messages from the deceased.

[2315] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2316] In this invention, the server includes a means for sending a message at a specific event using a model generated based on data of the deceased, a means for inputting event information and saving the information, a means for generating a model based on the event date and time and displaying it, a means for recognizing the user's emotions in real time and personalizing the message based on the recognition results, and a means for transmitting and displaying the generated message and model to a user terminal. This allows the user to feel a sense of dialogue with the deceased and deepen their emotional connection. Furthermore, by providing a personalized message tailored to the user's emotions, the user can gain empathy and comfort. Furthermore, the timing of event notifications and message display can be appropriately managed.

[2317] A "model" is a virtual character or message generation program generated based on data about the deceased.

[2318] "Events" are events or ceremonies that are held on specific dates or times, such as memorial services, Obon, and New Year's.

[2319] A "message" is information such as text, audio, or video sent from the deceased to the user.

[2320] A "user terminal" is a device used by a user, such as a smartphone or a personal computer.

[2321] The "server" is a central computer that manages and operates the database, generative AI, and emotion engine.

[2322] The "input means" is an interface through which the user inputs event information and transmits it to the server.

[2323] "Storage means" is a function that stores input event information in a database.

[2324] "Display means" is a function that displays the generated model and messages on the user terminal.

[2325] "Emotion recognition means" is a technology for analyzing a user's emotions in real time and obtaining the results.

[2326] "Personalization means" is a function that individually optimizes messages based on acquired emotional information.

[2327] A "notification means" is a communication means for reporting specific actions or events from a user terminal to a server.

[2328] The "point giving means" is a function for giving points to a user and notifying the user.

[2329] The present invention is a system that uses a model generated based on data of the deceased to send a message from the deceased at a specific event. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. To implement this system, data of the deceased, a generation AI, a model, an event information input means, a model display means, a message generation means, a user terminal, a server, and an emotion engine are used.

[2330] Overall system configuration

[2331] The system mainly consists of the following components:

[2332] 1. User terminal: A device such as a smartphone or PC that users use to enter event information and receive messages.

[2333] 2. Server: The central computer that manages the database, generative AI, and emotion engine. It stores event information, generates models, generates messages, recognizes emotions, and awards points.

[2334] 3. Generative AI: An artificial intelligence program that generates models and messages based on data about the deceased.

[2335] 4. Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[2336] 5. Database: Storage for storing data of the deceased, user information, and event information.

[2337] Program processing overview

[2338] The program processing in this system is explained as follows.

[2339] Functions for handling events such as memorial services

[2340] The user enters the date and time of the memorial service into their device, which then sends the information to the server. The server stores the received information in a database. The day before the memorial service, the server requests the generative AI to generate a model of the deceased. On the day of the memorial service, when the user opens the app, the server sends the model and a message to the user's device and displays them. The emotion engine recognizes the user's emotions in real time and personalizes the message accordingly. For example, if the user has a sad expression, the model will display an encouraging message such as, "Thank you for coming at this difficult time."

[2341] Functions for Obon and New Year holidays

[2342] When a user lights incense during Obon or New Year's, the information is sent from the user's device to the server. Based on this information, the server requests the generation AI to generate a message of gratitude and a model. The generated data is sent from the server to the user's device. The emotion engine analyzes the user's emotions and personalizes the message. For example, if the user has a happy expression, a message such as "Thank you for coming to visit us. We're happy to see your smile" is displayed.

[2343] Blessing function at the time of growth

[2344] The server manages the growth anniversaries of the user's grandchildren and great-grandchildren in a database. As anniversaries approach, the server requests the generation AI to generate a congratulatory message and model. On the anniversary day, the server sends the generated message and model to the user's device, and the emotion engine analyzes the user's emotions and personalizes the message. For example, if the user is emotional, a message such as "Happy birthday. I'm so happy to have watched you grow up" is displayed. The server also performs the process of awarding points and notifies the user of the points.

[2345] Specific examples

[2346] Functions for handling events such as memorial services

[2347] Examples:

[2348] 1. The user opens the app on their smartphone and enters the date of the memorial service (December 25, 2023) and the time (10:00).

[2349] 2. The user device sends the data "Memorial date: December 25, 2023, time: 10:00" to the server.

[2350] 3. The server stores the received information in a database.

[2351] 4. On December 24, 2023, the server asks the generation AI to "generate a model for a memorial service."

[2352] 5. When the user opens the app on the day of the memorial service, the server sends the model and a message.

[2353] 6. The emotion engine analyzes the user's sad expression and generates a personalized message.

[2354] Prompt Sentence Examples

[2355] Here are some example prompts to input to a generative AI model:

[2356] "The memorial date is approaching. Please generate a model of the deceased and an appropriate message. Include an encouraging message if the user has a sad expression."

[2357] This allows users to feel like they are having a conversation with the deceased, deepening their emotional connection. Furthermore, by providing personalized messages tailored to the user's emotions, it is possible to gain empathy and comfort. Furthermore, it is possible to appropriately manage the timing of event notifications and message display.

[2358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2359] Processing steps for events such as memorial services

[2360] Step 1:

[2361] The user enters the date and time of the memorial service.

[2362] Input: Date and time of memorial service (e.g. December 25, 2023, 10:00).

[2363] Output: The input information is saved on the user's terminal.

[2364] Specific operation: The user opens a dedicated app on their smartphone and enters the date and time of the memorial service in the form that appears.

[2365] Step 2:

[2366] The user terminal transmits information about the memorial service to the server.

[2367] Input: Date and time of memorial service.

[2368] Output: Memorial service information is sent to the server.

[2369] Specific operation: The date and time of the memorial service entered by the user is sent to the server as an HTTP request.

[2370] Step 3:

[2371] The server stores the memorial information in a database.

[2372] Input: Date and time of memorial service.

[2373] Output: Memorial information stored in a database.

[2374] Specific operation: The server saves the date and time of the memorial service received as a new record in a database table.

[2375] Step 4:

[2376] The day before the memorial service, the server requests the generation AI to generate a model.

[2377] Input: Trigger based on date and time before and after the memorial service, memorial service information.

[2378] Output: The generated model and messages.

[2379] Specific operation: The server uses a trigger set on the day before the memorial service to send a prompt message to the generation AI saying, "Please generate a model for the memorial service."

[2380] Step 5:

[2381] On the day of the memorial service, the user opens the application.

[2382] Input: User action.

[2383] Output: The application sends a request to the server.

[2384] Specific action: The user launches the app on their smartphone.

[2385] Step 6:

[2386] The server sends the avatar and message to the user's terminal.

[2387] Input: The generated avatar and message.

[2388] Output: The avatar and message displayed on the user's device.

[2389] Specific operation: The server sends the avatar and message received from the generated AI to the user's device in JSON format.

[2390] Step 7:

[2391] The emotion engine recognizes the user's emotions.

[2392] Input: User's facial expression data.

[2393] Output: Recognized emotion data (e.g. sadness, joy).

[2394] Specific operation: Using the camera on the user's device, the emotion engine analyzes the user's facial expressions in real time.

[2395] Step 8:

[2396] The server personalizes the message and displays it on the user's terminal.

[2397] Input: Recognized emotion data, generated message.

[2398] Output: A personalized message.

[2399] Specific operation: The server modifies the message based on the emotion data and sends it to the user's device. If the user has a sad expression, an encouraging message such as "Thank you for coming during this difficult time" will be displayed.

[2400] (Application example 2)

[2401] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2402] Conventional dialogue systems for the deceased only send messages using an avatar generated based on the deceased's data, but are unable to provide personalized messages based on the user's emotions. Furthermore, it is difficult to provide an emotionally rich experience for specific events. Therefore, there is a need for a system that allows users to empathize and feel deeply moved.

[2403] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2404] In this invention, the server includes means for sending a message at a specific event using an avatar generated based on data of the deceased, means for inputting and saving event information, means for generating an avatar based on the event date and time and displaying it, and means for recognizing a user's emotions in real time and personalizing a message based on the emotions, thereby making it possible to recognize a user's emotions and provide a personalized message.

[2405] "Data of the deceased" refers to information about a deceased person, including digital data such as images, audio, video, and text.

[2406] An "avatar" is a digital image created based on the data of a deceased person, recreating the deceased in a virtual space.

[2407] "Specific events" refers to special occasions or anniversaries related to the deceased, including memorial services, birthdays, Obon, New Year's, and other regular events.

[2408] The "means for sending a message" is a method by which the avatar can provide a text or voice message to the user depending on the date and situation of the event.

[2409] "Event information" is detailed data about a particular event, such as the date, time, and location.

[2410] "Storage means" refers to the methods and technologies used to store event information and other related data in a database or the like.

[2411] "Display means" refers to a technique for visually displaying the generated avatar or message on the user's terminal.

[2412] "Means for recognizing a user's emotions in real time and personalizing messages based on those emotions" refers to a method that uses emotion recognition technology to determine a user's current emotional state and appropriately change the message content accordingly.

[2413] A "personalized message" is a text or voice message that is individually tailored to the user's emotions and situation.

[2414] This invention is a system that uses an avatar generated based on the data of a deceased person to send a message at a specific event. By combining it with an emotion engine that recognizes the user's emotions, it is possible to provide a more personalized message. Details of the embodiment are described below.

[2415] 1. System Configuration

[2416] The system mainly consists of the following components:

[2417] User device: A device such as a smartphone or computer that users use to enter event information and receive messages.

[2418] Server: A central computer that manages the database, generative AI, and emotion engine. It stores event information, generates avatars, generates messages, and recognizes emotions.

[2419] Generative AI: An artificial intelligence program that generates avatars and messages based on data about the deceased.

[2420] Emotion Engine: A program that recognizes a user's emotions in real time and personalizes messages based on those emotions.

[2421] Database: Storage for storing data of deceased persons, user information, and event information.

[2422] 2. Program Processing Overview

[2423] The user enters event information on the user device and sends it to the server, which stores it in a database. As the event date approaches, the server requests the generation AI to generate an avatar of the deceased person. On the day of the event, the emotion engine recognizes the user's emotions in real time, and the avatar and message are displayed according to those emotions.

[2424] 3. Hardware and Software Used

[2425] Hardware:

[2426] Smartphone

[2427] server

[2428] software:

[2429] EmotionEngine: Software for recognizing user emotions.

[2430] Avatar Generator (generative AI): Software that generates avatars based on data of deceased people.

[2431] Message Generator (Generative AI): Software that generates messages based on data about the deceased and event information.

[2432] Database: Database management software for storing user and event information.

[2433] 4. Specific Examples

[2434] ■ Responses to memorial services:

[2435] 1. The user terminal enters the date and time of the memorial service and sends it to the server.

[2436] 2. The server stores the information in a database.

[2437] 3. The day before the memorial service, the server requests the generation AI to generate an avatar.

[2438] 4. On the day of the memorial service, the user opens the app, retrieves the avatar and message from the server, and displays them.

[2439] 5. The emotion engine recognizes users' emotions in real time and personalizes messages.

[2440] 6. The user device plays an avatar and a message saying, "Thank you for coming to the memorial service today." If the user has a sad expression, the message changes to an encouraging one such as, "Thank you for coming during this difficult time."

[2441] ■ Visits during Obon and New Year's:

[2442] 1. The user device notifies the server when it is time to light the incense.

[2443] 2. The server sends a thank you message to the AI ​​and asks it to generate an avatar.

[2444] 3. The generation AI generates a thank you message and an avatar and sends them back to the server.

[2445] 4. The server sends the message and avatar to the user's device.

[2446] 5. The emotion engine analyzes the user's emotions and personalizes the message.

[2447] 6. The user device displays the message "Thank you for visiting during Obon." If the user has a happy expression, the message is changed to "Thank you for visiting. I'm happy to see your smile."

[2448] ■ Grandchild's birthday celebration:

[2449] 1. The server checks the date of birth from the database.

[2450] 2. On the day before the anniversary, ask the generation AI to generate a congratulatory message and an avatar.

[2451] 3. On the anniversary day, the server sends the generated message and avatar to the user's device.

[2452] 4. The emotion engine analyzes the user's emotions and personalizes the message.

[2453] 5. The user device displays the message "Happy Birthday" and the avatar. If the user is touched, the message changes to "Happy Birthday. I'm so glad to watch you grow up."

[2454] In this way, by combining emotion engines, it is possible to provide personalized messages that match the user's emotions, generating deeper empathy and emotion.

[2455] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2456] Step 1:

[2457] The user enters event information (e.g., the date and time of the memorial service) into the terminal and sends it to the server.

[2458] Input: Event information (date, time)

[2459] Data processing: User terminal packages event information

[2460] Output: Request sent to server

[2461] Step 2:

[2462] The server stores the received event information in a database.

[2463] Input: Received event information

[2464] Data processing: Converting event information into the appropriate format

[2465] Output: Save results to database

[2466] Step 3:

[2467] As the event date approaches, the server asks the generation AI to generate an avatar of the deceased person.

[2468] Input: Event information, deceased person's data

[2469] Data processing: Generative AI analyzes the data of the deceased and generates an avatar

[2470] Output: The generated avatar

[2471] Step 4:

[2472] On the day of the event, the server sends an avatar and a message to the user's terminal.

[2473] Input: Generated avatar, standard message

[2474] Data processing: Packaging avatars and messages

[2475] Output: Send request to user terminal

[2476] Step 5:

[2477] The user's device displays the received avatar and message.

[2478] Input: avatar, message

[2479] Data processing: Convert received data into a display format

[2480] Output: Display avatar and message on user's device

[2481] Step 6:

[2482] The emotion engine recognizes the user's emotions in real time.

[2483] Input: User's facial expression data (camera input)

[2484] Data calculation: Emotion engine analyzes facial expression data

[2485] Output: Recognized emotion data

[2486] Step 7:

[2487] The server personalizes the message based on the recognized emotion data.

[2488] Input: Standard message, recognized emotion data

[2489] Data processing: Message generation based on emotional data

[2490] Output: Personalized message

[2491] Step 8:

[2492] The user terminal displays the personalized message.

[2493] Input: Personalized message

[2494] Data processing: Convert received messages into a display format

[2495] Output: Display a personalized message on the user's device

[2496] Through the above steps, a personalized message according to emotions is provided based on the event information entered by the user.

[2497] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2498] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2499] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2500] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2501] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2502] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2503] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2504] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2505] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2506] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2507] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2508] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2509] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2510] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2511] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2512] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2513] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2514] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2515] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2516] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2517] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if...

Claims

1. A means for transmitting a message at a specific event using an avatar generated based on data of the deceased person; a means for inputting event information and saving the information; A means for generating an avatar based on the event date and time and displaying the avatar; A system including:

2. 10. The system of claim 1, wherein the avatar sends messages to attendees of the memorial service during the event.

3. means for informing a user that on a specific day, incense will be lit; a means for the avatar to generate and display a thank you message in response to the notification; The system of claim 1 , comprising:

4. A means for sending a congratulatory message to a user's grandchild or great-grandchild on a specific anniversary and awarding points at that time; a means for generating an avatar based on the event and sending the message; The system of claim 1 , comprising:

5. 5. The system according to claim 4, wherein a congratulatory message is generated along with the avatar on the anniversary of the growth, and points are awarded.

Images