System

Abstract

An object of the system according to the embodiment is to generate a 3D model based on the date of the deceased and perform a dialogue.SOLUTION: A system includes an acquisition unit, a generation unit, a database, and an interaction unit. The acquisition unit acquires appearance data. The acquisition unit acquires voice data. The generating unit generates a 3D model based on the information acquired by the acquiring unit. The database stores the epipolar data. The dialogue unit performs a dialogue by referring to the episode data stored in the database.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies do not adequately acquire and manage data necessary to enable dialogue with the deceased, and there is room for improvement.

[0005] The system according to the embodiment aims to generate a 3D model based on data of the deceased person and to conduct a dialogue with them. [Means for solving the problem]

[0006] The system according to the embodiment includes an acquisition unit, a generation unit, a database, and a dialogue unit. The acquisition unit acquires appearance data. The acquisition unit acquires voice data. The generation unit generates a 3D model based on the data acquired by the acquisition unit. The database stores episode data. The dialogue unit conducts dialogue by referring to the episode data stored in the database. [Effects of the Invention]

[0007] The system according to the embodiment generates a 3D model based on data of the deceased person and can conduct dialogue. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A system according to an embodiment of the present invention generates a 3D virtual human from data recorded and acquired during a person's lifetime, allowing for conversations with the deceased in a metaverse space after death. This system includes an acquisition unit that acquires appearance data, an acquisition unit that acquires voice data, a generation unit that generates a 3D model based on the acquired data, a database that stores episode data, and a dialogue unit that engages in dialogue by referring to the episode data stored in the database. For example, the system records and acquires the appearance and voice of the deceased during their lifetime. For example, the system records detailed data related to appearance, such as the deceased's facial expressions, skeletal structure, movements, and habits. It also acquires data related to speech, such as the deceased's voice, speaking style, tone, and catchphrases. Next, the system generates a 3D virtual human based on the acquired data. The generation AI analyzes the collected data and generates a 3D model that faithfully reproduces the appearance and voice of the deceased. Furthermore, the system stores the acquired episodes in a database, allowing for conversations with the deceased by referring to past events while utilizing the conversational AI. This allows users to enjoy conversations with the deceased in a metaverse space. For example, the conversational AI can refer to and speak stories that the deceased shared while they were alive, allowing the user to reminisce about their memories with the deceased. Also, by faithfully recreating the appearance and voice of the deceased, the user can feel as if they are having a more realistic conversation with the deceased.

[0029] The system according to the embodiment includes an acquisition unit that acquires appearance data, an acquisition unit that acquires audio data, a generation unit that generates a 3D model based on the acquired data, a database that stores episode data, and a dialogue unit that engages in dialogue by referring to the episode data stored in the database. The acquisition unit acquires the appearance data. The appearance data includes, for example, but is not limited to, the facial expressions, skeletal structure, movements, and habits of the deceased. For example, the acquisition unit captures the facial expressions of the deceased with a camera and collects data on changes in facial expressions. The acquisition unit can also scan the deceased's skeleton with a 3D scanner to acquire skeletal data. The acquisition unit can also record the movements and habits of the deceased using motion capture technology. For example, the acquisition unit records the deceased's walking patterns in detail and saves the data. The acquisition unit acquires audio data, such as the deceased's voice, speaking style, tone, and catchphrases. The audio data includes, for example, but is not limited to, the deceased's tone of voice, speaking style, and catchphrases. For example, the acquisition unit records the deceased's voice with a microphone and saves the voice data. The acquisition unit can also record the deceased's speaking style and tone in detail. For example, the acquisition unit may record in detail subtle changes in the tone and speaking style of the deceased when they speak and reflect them in the audio data. The generation unit analyzes the acquired appearance data and generates a 3D model that faithfully reproduces the appearance of the deceased. For example, the generation unit creates a 3D model based on the collected data and faithfully reproduces the appearance of the deceased. The generation unit also analyzes the acquired audio data and generates audio that faithfully reproduces the deceased's voice and speaking style. For example, the generation unit analyzes the audio data and uses an algorithm to reproduce the deceased's voice and speaking style. The database stores the acquired episode data. The episode data includes, for example, events and memories that the deceased talked about before their death, but is not limited to such examples. For example, the database records in detail specific episodes that the deceased spent with family and friends and stores them in the database. The dialogue unit refers to the episode data stored in the database and conducts a dialogue based on past events. For example, the dialogue unit refers to episodes that the deceased talked about before their death and conducts a dialogue.As a result, the system according to the embodiment generates a 3D virtual human from data recorded and acquired during a person's lifetime, and allows the user to converse with the deceased in the Metaverse space after death. For example, the user can reminisce about memories with the deceased, making conversation with the deceased feel more real.

[0030] The acquisition unit can acquire appearance data such as the deceased's facial expressions, skeletal structure, movements, and habits. For example, the acquisition unit can capture the deceased's facial expressions with a camera and collect changes in facial expressions as data. The acquisition unit can also scan the deceased's skeleton with a 3D scanner to acquire skeletal data. The acquisition unit can also record the deceased's movements and habits using motion capture technology. For example, the acquisition unit can record the deceased's walking patterns in detail and save them as data. This allows data to be acquired to faithfully reproduce the deceased's appearance. Some or all of the above-mentioned processing in the acquisition unit may be performed using, for example, AI, or may be performed without AI. For example, the acquisition unit can input facial expression data of the deceased captured with a camera into a generation AI to analyze changes in facial expressions.

[0031] The acquisition unit can acquire audio data such as the deceased's voice, speaking style, tone, and catchphrases. For example, the acquisition unit can record the deceased's voice using a microphone and save it as audio data. The acquisition unit can also record the deceased's speaking style and tone in detail. For example, the acquisition unit can record in detail subtle changes in the tone and speaking style of the deceased when they speak and reflect them in the audio data. This makes it possible to acquire data for faithfully reproducing the deceased's voice. Some or all of the above-mentioned processing in the acquisition unit may be performed using, for example, AI, or may be performed without using AI. For example, the acquisition unit can input the deceased's voice data recorded using a microphone into a generation AI and analyze changes in the voice tone and speaking style.

[0032] The generation unit can analyze the acquired appearance data and generate a 3D model that faithfully reproduces the appearance of the deceased. For example, the generation unit creates a 3D model based on the collected data and faithfully reproduces the appearance of the deceased. The generation unit also analyzes the acquired appearance data and uses an algorithm to reproduce the facial expressions and movements of the deceased. For example, the generation unit reproduces in detail the facial expression of the deceased when they smile and reflects this in the 3D model. The generation unit can also reproduce in detail the movements of the deceased when they perform a specific action (e.g., waving their hands). This makes it possible to generate a 3D model that faithfully reproduces the appearance of the deceased. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the collected appearance data into the generation AI and generates a 3D model that faithfully reproduces the appearance of the deceased.

[0033] The generation unit can analyze the acquired voice data and generate voice that faithfully reproduces the voice and speaking style of the deceased. For example, the generation unit analyzes the voice data and uses an algorithm to reproduce the voice and speaking style of the deceased. The generation unit records changes in the tone of the voice and speaking style of the deceased in detail and reflects them in the voice data. For example, the generation unit reproduces subtle changes in the tone and speaking style of the deceased when they speak in detail and reflects them in the voice data. The generation unit can also reproduce the tone of voice and speaking style of the deceased when they express specific emotions. This makes it possible to generate voice that faithfully reproduces the voice and speaking style of the deceased. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the collected voice data into a generation AI and generates voice that faithfully reproduces the voice and speaking style of the deceased.

[0034] The dialogue unit can refer to episode data stored in a database and conduct a dialogue based on past events. The dialogue unit, for example, refers to episodes that the deceased talked about while alive and conducts a dialogue. The dialogue unit searches the episode data stored in the database and uses an algorithm for conducting a dialogue based on past events. For example, the dialogue unit refers to a specific episode that the deceased spent with family or friends and conducts a dialogue. The dialogue unit can also refer to events that the deceased experienced at a specific event and conduct a dialogue. This enables a more realistic conversation by conducting a dialogue based on past events. Some or all of the above-mentioned processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data stored in a database into AI and conducts a dialogue based on past events.

[0035] The acquisition unit can optimize the method of acquiring appearance data based on the living environment and activity status of the deceased. For example, the acquisition unit acquires appearance data at places where the deceased spent their daily lives and records natural movements and facial expressions. The acquisition unit can also acquire appearance data when the deceased is engaged in a specific activity and record in detail the movements and facial expressions related to that activity. Furthermore, the acquisition unit can acquire appearance data in different environments, such as indoors and outdoors, depending on the living environment of the deceased, to recreate various situations. In this way, by optimizing the method of acquiring appearance data based on the living environment and activity status of the deceased, more natural data can be acquired. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs data on the living environment of the deceased into a generation AI to optimize the method of acquiring appearance data.

[0036] When acquiring appearance data, the acquisition unit can simultaneously acquire information about the deceased's clothing and accessories. For example, the acquisition unit may record in detail the clothing and accessories that the deceased often wore and reflect this in the 3D model. The acquisition unit may also acquire the clothing and accessories that the deceased wore at a particular event and recreate that situation. Furthermore, the acquisition unit may also record in detail the color and design of the deceased's clothing and accessories and reflect this in the appearance data. In this way, by simultaneously acquiring information about the deceased's clothing and accessories, a more detailed 3D model can be generated. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit may input data about the deceased's clothing and accessories into a generation AI, which analyzes the detailed information.

[0037] When acquiring appearance data, the acquisition unit records the deceased's movement patterns in detail, thereby improving the naturalness of the movements. For example, the acquisition unit may record the deceased's walking movement patterns in detail and reflect them in the 3D model. The acquisition unit may also acquire the movement patterns of the deceased when performing specific gestures and reproduce natural movements. Furthermore, the acquisition unit may record the deceased's daily movements (e.g., sitting, standing, etc.) in detail and reflect them in the 3D model. In this way, by recording the deceased's movement patterns in detail, more natural movements can be reproduced. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit may input the deceased's movement data into a generation AI, which analyzes the movement patterns and reproduces natural movements.

[0038] When acquiring appearance data, the acquisition unit can prioritize acquiring movements related to the deceased's hobbies and special skills. For example, the acquisition unit prioritizes acquiring movements that the deceased performed as hobbies (e.g., drawing, playing a musical instrument, etc.). The acquisition unit can also record in detail movements that the deceased performed as special skills (e.g., sports movements) and reflect these in the 3D model. Furthermore, the acquisition unit can prioritize acquiring facial expressions and movements of the deceased performing their hobbies or special skills. In this way, by prioritizing the acquisition of movements related to the deceased's hobbies and special skills, a more detailed 3D model can be generated. Some or all of the above-described processing by the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs movement data related to the deceased's hobbies and special skills into a generation AI, which analyzes the detailed information.

[0039] When acquiring appearance data, the acquisition unit can acquire data corresponding to the deceased's health condition and age. For example, the acquisition unit can record in detail the movements and facial expressions of the deceased corresponding to their health condition and reflect them in the 3D model. The acquisition unit can also acquire appearance data corresponding to the deceased's age and reproduce the appearance and movements corresponding to that age. Furthermore, the acquisition unit can prioritize acquiring specific movements and facial expressions related to the deceased's health condition and age. In this way, by acquiring data corresponding to the deceased's health condition and age, a more detailed 3D model can be generated. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs data related to the deceased's health condition and age into the generation AI, which analyzes the detailed information.

[0040] The acquisition unit can record the deceased's interactions with family and friends when acquiring appearance data. For example, the acquisition unit can record in detail the facial expressions and movements of the deceased while spending time with family and friends and reflect these in the 3D model. The acquisition unit can also prioritize acquiring specific activities (e.g., eating, conversation, etc.) that the deceased engaged in with family and friends. Furthermore, the acquisition unit can record the deceased's natural movements and expressions while interacting with family and friends. This allows for the generation of a more detailed 3D model by recording the deceased's interactions with family and friends. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs interaction data of the deceased with family and friends into a generation AI, which analyzes the detailed information.

[0041] The acquisition unit can record in detail changes in the tone and speaking style of the deceased's voice to achieve more natural voice reproduction. For example, the acquisition unit can record in detail subtle changes in the tone and speaking style of the deceased when they speak and reflect them in the voice data. The acquisition unit can also acquire changes in the tone and speaking style of the deceased when they express different emotions to achieve natural voice reproduction. Furthermore, the acquisition unit can also record in detail changes in the tone and speaking style of the deceased when they speak in specific situations and reflect them in the voice data. This enables more natural voice reproduction by recording changes in the tone and speaking style of the deceased in detail. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI or without AI. For example, the acquisition unit inputs the deceased's voice data into a generation AI, analyzes changes in tone and speaking style, and achieves natural voice reproduction.

[0042] The acquisition unit can record in detail specific emotional expressions of the deceased (such as joy, sadness, and anger) when acquiring voice data. For example, the acquisition unit can record in detail the tone of voice and manner of speaking of the deceased when expressing joy and reflect this in the voice data. The acquisition unit can also record in detail the tone of voice and manner of speaking of the deceased when expressing sadness and reflect this in the voice data. Furthermore, the acquisition unit can also record in detail the tone of voice and manner of speaking of the deceased when expressing anger and reflect this in the voice data. In this way, by recording in detail the specific emotional expressions of the deceased, more natural voice reproduction is possible. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs the emotional expression data of the deceased into a generation AI, which analyzes the detailed information.

[0043] When acquiring voice data, the acquisition unit can focus on recording specific phrases and catchphrases of the deceased. For example, the acquisition unit can record in detail specific phrases that the deceased frequently used and reflect them in the voice data. The acquisition unit can also record in detail the catchphrases of the deceased and reflect them in the voice of the 3D model. Furthermore, the acquisition unit can focus on acquiring phrases and catchphrases that the deceased frequently used in specific situations. By focusing on recording the specific phrases and catchphrases of the deceased, more natural voice reproduction is possible. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs data on the specific phrases and catchphrases of the deceased into a generation AI, which analyzes the detailed information.

[0044] When acquiring audio data, the acquisition unit can acquire special audio, such as the deceased's singing voice or poetry recitation. For example, the acquisition unit can record detailed audio data of the deceased singing and reflect it in the audio of the 3D model. The acquisition unit can also acquire audio data of the deceased reciting poetry and recreate it in a special situation. Furthermore, the acquisition unit can preferentially acquire audio data of the deceased singing or reciting at a specific event. This allows for the acquisition of special audio of the deceased, thereby generating a more detailed 3D model. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the deceased's singing voice or poetry recitation data into a generation AI, which analyzes the detailed information.

[0045] When acquiring voice data, the acquisition unit can record voice changes according to the deceased's different ages and situations. For example, the acquisition unit can record in detail the deceased's voice when young and when older, and reflect this in the 3D model. The acquisition unit can also acquire changes in the deceased's tone of voice and speaking style when speaking in different situations, thereby achieving natural voice reproduction. Furthermore, the acquisition unit can also record in detail voice changes according to the deceased's age and situation, and reflect these in the voice data. This allows for the generation of a more detailed 3D model by recording voice changes according to the deceased's different ages and situations. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs data on the deceased's voice changes into a generation AI, which analyzes the detailed information.

[0046] The acquisition unit can record conversations between the deceased and family and friends when acquiring voice data. For example, the acquisition unit can record detailed voice data of the deceased talking with family and friends and reflect the voice data in the 3D model. The acquisition unit can also prioritize acquisition of specific conversations the deceased had with family and friends. Furthermore, the acquisition unit can record the natural speaking style and tone of the deceased when talking with family and friends. This allows for the generation of a more detailed 3D model by recording the deceased's conversations with family and friends. Some or all of the above-described processing in the acquisition unit can be performed using AI or without AI. For example, the acquisition unit can input conversation data between the deceased and family and friends into a generation AI and analyze the detailed information.

[0047] The generation unit can reproduce in detail specific facial expressions and movements of the deceased when generating the 3D model. For example, the generation unit can reproduce in detail the facial expression of the deceased when they smile and reflect this in the 3D model. The generation unit can also reproduce in detail the movements of the deceased when they perform a specific action (for example, waving their hands). Furthermore, the generation unit can also reproduce in detail the facial expressions and movements of the deceased when they express a specific emotion and reflect this in the 3D model. In this way, by reproducing in detail the specific facial expressions and movements of the deceased, a more natural 3D model can be generated. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs facial expression and movement data of the deceased into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0048] The generation unit can reproduce details of the deceased's clothing and accessories when generating the 3D model. For example, the generation unit can reproduce in detail the clothing and accessories that the deceased often wore and reflect them in the 3D model. The generation unit can also reproduce clothing and accessories that the deceased wore at specific events. Furthermore, the generation unit can reproduce in detail the colors and designs of the deceased's clothing and accessories and reflect them in the 3D model. This allows for the generation of a more natural-looking 3D model by reproducing the details of the deceased's clothing and accessories. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs data on the deceased's clothing and accessories into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0049] When generating the 3D model, the generation unit analyzes the movement patterns of the deceased to achieve more natural movements. For example, the generation unit analyzes the movement patterns of the deceased when walking and reflects them in the 3D model. The generation unit can also analyze the movement patterns of the deceased when performing specific gestures and reproduce natural movements. Furthermore, the generation unit can analyze the movements that the deceased performed daily (e.g., sitting, standing, etc.) and reflect them in the 3D model. In this way, by analyzing the movement patterns of the deceased, more natural movements can be reproduced. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the movement data of the deceased into the generation AI, analyzes detailed information, and reflects it in the 3D model.

[0050] When generating the 3D model, the generation unit can reproduce movements related to the deceased's hobbies and special skills. For example, the generation unit reproduces movements that the deceased performed as a hobby (e.g., drawing, playing an instrument, etc.). The generation unit can also reproduce in detail movements that the deceased performed as a special skill (e.g., sports movements) and reflect them in the 3D model. Furthermore, the generation unit can also reproduce the facial expressions and movements of the deceased while performing their hobbies or special skills. In this way, by reproducing movements related to the deceased's hobbies and special skills, a more detailed 3D model can be generated. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs movement data related to the deceased's hobbies and special skills into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0051] When generating the 3D model, the generation unit can reproduce the appearance of the deceased according to their health condition and age. For example, the generation unit can reproduce in detail the appearance and movements of the deceased according to their health condition and reflect this in the 3D model. The generation unit can also obtain appearance data according to the age of the deceased and reproduce the appearance and movements according to their age. Furthermore, the generation unit can reproduce specific movements and facial expressions related to the health condition and age of the deceased. This allows for the generation of a more detailed 3D model by reproducing the appearance according to the health condition and age of the deceased. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs data related to the health condition and age of the deceased into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0052] The generation unit can recreate the deceased's interactions with family and friends when generating the 3D model. For example, the generation unit can recreate in detail the facial expressions and movements of the deceased when spending time with family and friends and reflect them in the 3D model. The generation unit can also recreate specific activities (e.g., eating, conversation, etc.) that the deceased engaged in with family and friends. Furthermore, the generation unit can recreate the natural movements and facial expressions of the deceased when interacting with family and friends. This allows for the generation of a more detailed 3D model by recreating the deceased's interactions with family and friends. Some or all of the above-described processing in the generation unit is performed using a generation AI. For example, the generation unit inputs interaction data of the deceased with family and friends into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0053] When storing episode data, the database can record in detail episodes including specific emotional expressions of the deceased. For example, the database can record in detail episodes when the deceased expressed joy and store them in the database. The database can also record in detail episodes when the deceased expressed sadness and store them in the database. Furthermore, the database can also record in detail episodes when the deceased expressed anger and store them in the database. In this way, by recording in detail episodes including specific emotional expressions of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs emotional expression data of the deceased into a generation AI, which analyzes the detailed information and stores it as episode data.

[0054] When storing episode data, the database can focus on recording specific events and memories of the deceased. For example, the database can record in detail events experienced by the deceased at a specific event and store them in the database. The database can also record in detail specific memories the deceased had with family and friends and store them in the database. Furthermore, the database can also record in detail events and memories the deceased experienced in a specific place and store them in the database. This allows for more detailed data to be stored by focusing on recording specific events and memories of the deceased. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database inputs the deceased's event and memory data into a generation AI, analyzes the detailed information, and stores it as episode data.

[0055] When storing episode data, the database can record in detail episodes with the deceased's family and friends. For example, the database can record in detail specific episodes the deceased spent with their family and store them in the database. The database can also record in detail specific episodes the deceased spent with their friends and store them in the database. Furthermore, the database can also record in detail specific activities and events the deceased had with their family and friends and store them in the database. This allows for more detailed data to be stored by recording episodes with the deceased's family and friends in detail. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database inputs episode data with the deceased's family and friends into a generation AI, which analyzes the detailed information and stores it as episode data.

[0056] When storing episode data, the database can record episodes related to the hobbies and special skills of the deceased. For example, the database can record in detail episodes related to activities that the deceased engaged in as hobbies and store them in the database. The database can also record in detail episodes related to activities that the deceased considered to be special skills and store them in the database. Furthermore, the database can also record in detail episodes of the deceased engaging in hobbies and special skills and store them in the database. In this way, by recording episodes related to the hobbies and special skills of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs episode data related to the hobbies and special skills of the deceased into a generation AI, which analyzes the detailed information and stores it as episode data.

[0057] When storing episode data, the database can record episodes according to different ages and circumstances of the deceased. For example, the database can record in detail episodes from the deceased's youth and episodes from when the deceased was older and store them in the database. The database can also record in detail episodes experienced by the deceased in different circumstances and store them in the database. Furthermore, the database can also record in detail episodes according to the deceased's age and circumstances and store them in the database. In this way, by recording episodes according to different ages and circumstances of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs episode data according to the deceased's age and circumstances into a generation AI, which analyzes the detailed information and stores it as episode data.

[0058] When storing episode data, the database can record episodes with the deceased's family and friends. For example, the database can record in detail specific episodes the deceased spent with their family and store them in the database. The database can also record in detail specific episodes the deceased spent with their friends and store them in the database. Furthermore, the database can also record in detail specific activities and events the deceased had with their family and friends and store them in the database. This allows for more detailed data to be stored by recording episodes with the deceased's family and friends. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database can input episode data with the deceased's family and friends into a generation AI, analyze the detailed information, and store it as episode data.

[0059] During the dialogue, the dialogue unit can conduct a conversation by referring to episodes containing specific emotional expressions of the deceased. For example, the dialogue unit can conduct a dialogue by referring to episodes in which the deceased expressed joy. The dialogue unit can also conduct a dialogue by referring to episodes in which the deceased expressed sadness. Furthermore, the dialogue unit can conduct a dialogue by referring to episodes in which the deceased expressed anger. In this way, by referring to episodes containing specific emotional expressions of the deceased, more detailed dialogue is possible. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs emotional expression data of the deceased into a generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0060] During the dialogue, the dialogue unit can conduct a conversation by referring to specific events or memories of the deceased. For example, the dialogue unit can conduct a dialogue by referring to events experienced by the deceased at a specific event. The dialogue unit can also conduct a dialogue by referring to specific memories the deceased had with family and friends. Furthermore, the dialogue unit can conduct a dialogue by referring to events or memories the deceased experienced in a specific place. This enables a more detailed dialogue by referring to specific events and memories of the deceased. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs data on events and memories of the deceased into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0061] During the dialogue, the dialogue unit can refer to episodes with the deceased's family and friends to conduct the conversation. For example, the dialogue unit can refer to specific episodes the deceased spent with their family and conduct the dialogue. The dialogue unit can also refer to specific episodes the deceased spent with their friends to conduct the dialogue. Furthermore, the dialogue unit can refer to specific activities or events the deceased had with their family and friends to conduct the dialogue. This enables more detailed dialogue by referring to episodes with the deceased's family and friends. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data with the deceased's family and friends into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0062] During the dialogue, the dialogue unit can conduct the conversation by referring to episodes related to the hobbies and special skills of the deceased. For example, the dialogue unit can conduct the dialogue by referring to episodes related to activities that the deceased engaged in as hobbies. The dialogue unit can also conduct the dialogue by referring to episodes related to activities that the deceased considered to be special skills. Furthermore, the dialogue unit can conduct the dialogue by referring to episodes of the deceased engaging in hobbies and special skills. In this way, by referring to episodes related to the hobbies and special skills of the deceased, more detailed dialogue is possible. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data related to the hobbies and special skills of the deceased into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0063] During the dialogue, the dialogue unit can conduct a conversation by referring to episodes that correspond to different ages and situations of the deceased. For example, the dialogue unit may conduct a dialogue by referring to episodes from the deceased's youth and episodes from when the deceased was older. The dialogue unit may also conduct a dialogue by referring to episodes that the deceased experienced in different situations. Furthermore, the dialogue unit may conduct a dialogue by referring to episodes that correspond to the deceased's ages and situations. This enables a more detailed dialogue by referring to episodes that correspond to the deceased's different ages and situations. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit may input episode data that corresponds to the deceased's age and situation into a generation AI, analyze the detailed information, and reflect it in the dialogue.

[0064] During the dialogue, the dialogue unit can refer to episodes with the deceased's family and friends to conduct the conversation. For example, the dialogue unit can refer to specific episodes the deceased spent with their family and conduct the dialogue. The dialogue unit can also refer to specific episodes the deceased spent with their friends to conduct the dialogue. Furthermore, the dialogue unit can refer to specific activities or events the deceased had with their family and friends to conduct the dialogue. This enables more detailed dialogue by referring to episodes with the deceased's family and friends. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data with the deceased's family and friends into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0065] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0066] The acquisition unit can also record the deceased's lifestyle habits and daily routine in detail. For example, it can record the exercise the deceased did every morning or activities they performed at specific times and reflect these in the 3D model. The acquisition unit can also record in detail the movements and facial expressions of the deceased when eating a specific meal. Furthermore, the acquisition unit can prioritize movements related to the deceased's hobbies and special skills. This allows for the generation of a more realistic 3D model by recording the deceased's lifestyle habits and daily routine in detail. Some or all of the above-mentioned processes in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the deceased's lifestyle habits data into a generation AI, which then analyzes the detailed information.

[0067] The generation unit can also reproduce movements and facial expressions according to the deceased's health condition and age. For example, it can reproduce the energetic movements of a young person and the slower movements of an older person. If the deceased had a specific illness or disability, the generation unit can also reproduce movements and facial expressions affected by that illness or disability. Furthermore, the generation unit can also reproduce in detail the movements and facial expressions of the deceased when expressing specific emotions and reflect them in the 3D model. This allows for the generation of a more realistic 3D model by reproducing movements and facial expressions according to the deceased's health condition and age. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs data related to the deceased's health condition and age into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0068] When storing episode data, the database can record in detail episodes that include specific emotional expressions of the deceased. For example, episodes in which the deceased expressed joy can be recorded in detail and stored in the database. Episodes in which the deceased expressed sadness can also be recorded in detail and stored in the database. Furthermore, episodes in which the deceased expressed anger can also be recorded in detail and stored in the database. In this way, by recording in detail episodes that include specific emotional expressions of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs emotional expression data of the deceased into a generation AI, which analyzes the detailed information and stores it as episode data.

[0069] The acquisition unit can prioritize acquiring movements related to the deceased's hobbies and special skills. For example, it prioritizes acquiring movements that the deceased performed as hobbies (e.g., drawing, playing an instrument, etc.). It can also record in detail movements that the deceased performed as special skills (e.g., sports movements) and reflect them in the 3D model. It can also prioritize acquiring facial expressions and movements of the deceased performing their hobbies or special skills. In this way, by prioritizing acquisition of movements related to the deceased's hobbies and special skills, a more detailed 3D model can be generated. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs movement data related to the deceased's hobbies and special skills into a generation AI, which analyzes the detailed information.

[0070] When storing episode data, the database can record in detail episodes with the deceased's family and friends. For example, specific episodes the deceased spent with their family can be recorded in detail and stored in the database. Specific episodes the deceased spent with their friends can also be recorded in detail and stored in the database. Specific activities and events the deceased had with their family and friends can also be recorded in detail and stored in the database. This allows for more detailed data to be stored by recording episodes with the deceased's family and friends in detail. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database inputs episode data with the deceased's family and friends into a generation AI, which analyzes the detailed information and stores it as episode data.

[0071] When generating the 3D model, the generation unit can reproduce in detail the specific facial expressions and movements of the deceased. For example, the facial expression of the deceased when smiling can be reproduced in detail and reflected in the 3D model. The generation unit can also reproduce in detail the movements of the deceased when performing a specific action (such as waving). Furthermore, the generation unit can reproduce in detail the facial expressions and movements of the deceased when expressing a specific emotion and reflect them in the 3D model. This allows for the generation of a more natural 3D model by reproducing the specific facial expressions and movements of the deceased in detail. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the facial expression and movement data of the deceased into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0072] The processing flow of the first embodiment will be briefly explained below.

[0073] Step 1: The acquisition unit acquires appearance data. Appearance data includes the deceased's facial expressions, skeletal structure, movements, and habits. For example, facial expressions are captured with a camera, the skeleton is scanned with a 3D scanner, and movements and habits are recorded using motion capture technology. Step 2: The acquisition unit acquires audio data. The audio data includes the deceased's voice, speaking style, tone, and catchphrases. For example, the voice is recorded with a microphone, and subtle changes in speaking style and tone are recorded in detail. Step 3: The generator analyzes the acquired appearance data to generate a 3D model that faithfully reproduces the appearance of the deceased, and analyzes the acquired audio data to generate audio that faithfully reproduces the voice and speaking style of the deceased. Step 4: The database stores the acquired episode data, which includes events and memories that the deceased talked about during their lifetime. Step 5: The dialogue unit references the episode data stored in the database and conducts dialogue based on past events, allowing the user to reminisce about their memories with the deceased and feel the conversation with them more realistically.

[0074] (Example 2) A system according to an embodiment of the present invention generates a 3D virtual human from data recorded and acquired during a person's lifetime, allowing for conversations with the deceased in a metaverse space after death. This system includes an acquisition unit that acquires appearance data, an acquisition unit that acquires voice data, a generation unit that generates a 3D model based on the acquired data, a database that stores episode data, and a dialogue unit that engages in dialogue by referring to the episode data stored in the database. For example, the system records and acquires the appearance and voice of the deceased during their lifetime. For example, the system records detailed data related to appearance, such as the deceased's facial expressions, skeletal structure, movements, and habits. It also acquires data related to speech, such as the deceased's voice, speaking style, tone, and catchphrases. Next, the system generates a 3D virtual human based on the acquired data. The generation AI analyzes the collected data and generates a 3D model that faithfully reproduces the appearance and voice of the deceased. Furthermore, the system stores the acquired episodes in a database, allowing for conversations with the deceased by referring to past events while utilizing the conversational AI. This allows users to enjoy conversations with the deceased in a metaverse space. For example, the conversational AI can refer to and speak stories that the deceased shared while they were alive, allowing the user to reminisce about their memories with the deceased. Also, by faithfully recreating the appearance and voice of the deceased, the user can feel as if they are having a more realistic conversation with the deceased.

[0075] The system according to the embodiment includes an acquisition unit that acquires appearance data, an acquisition unit that acquires audio data, a generation unit that generates a 3D model based on the acquired data, a database that stores episode data, and a dialogue unit that engages in dialogue by referring to the episode data stored in the database. The acquisition unit acquires the appearance data. The appearance data includes, for example, but is not limited to, the facial expressions, skeletal structure, movements, and habits of the deceased. For example, the acquisition unit captures the facial expressions of the deceased with a camera and collects data on changes in facial expressions. The acquisition unit can also scan the deceased's skeleton with a 3D scanner to acquire skeletal data. The acquisition unit can also record the movements and habits of the deceased using motion capture technology. For example, the acquisition unit records the deceased's walking patterns in detail and saves the data. The acquisition unit acquires audio data, such as the deceased's voice, speaking style, tone, and catchphrases. The audio data includes, for example, but is not limited to, the deceased's tone of voice, speaking style, and catchphrases. For example, the acquisition unit records the deceased's voice with a microphone and saves the voice data. The acquisition unit can also record the deceased's speaking style and tone in detail. For example, the acquisition unit may record in detail subtle changes in the tone and speaking style of the deceased when they speak and reflect them in the audio data. The generation unit analyzes the acquired appearance data and generates a 3D model that faithfully reproduces the appearance of the deceased. For example, the generation unit creates a 3D model based on the collected data and faithfully reproduces the appearance of the deceased. The generation unit also analyzes the acquired audio data and generates audio that faithfully reproduces the deceased's voice and speaking style. For example, the generation unit analyzes the audio data and uses an algorithm to reproduce the deceased's voice and speaking style. The database stores the acquired episode data. The episode data includes, for example, events and memories that the deceased talked about before their death, but is not limited to such examples. For example, the database records in detail specific episodes that the deceased spent with family and friends and stores them in the database. The dialogue unit refers to the episode data stored in the database and conducts a dialogue based on past events. For example, the dialogue unit refers to episodes that the deceased talked about before their death and conducts a dialogue.As a result, the system according to the embodiment generates a 3D virtual human from data recorded and acquired during a person's lifetime, and allows the user to converse with the deceased in the Metaverse space after death. For example, the user can reminisce about memories with the deceased, making conversation with the deceased feel more real.

[0076] The acquisition unit can acquire appearance data such as the deceased's facial expressions, skeletal structure, movements, and habits. For example, the acquisition unit can capture the deceased's facial expressions with a camera and collect changes in facial expressions as data. The acquisition unit can also scan the deceased's skeleton with a 3D scanner to acquire skeletal data. The acquisition unit can also record the deceased's movements and habits using motion capture technology. For example, the acquisition unit can record the deceased's walking patterns in detail and save them as data. This allows data to be acquired to faithfully reproduce the deceased's appearance. Some or all of the above-mentioned processing in the acquisition unit may be performed using, for example, AI, or may be performed without AI. For example, the acquisition unit can input facial expression data of the deceased captured with a camera into a generation AI to analyze changes in facial expressions.

[0077] The acquisition unit can acquire audio data such as the deceased's voice, speaking style, tone, and catchphrases. For example, the acquisition unit can record the deceased's voice using a microphone and save it as audio data. The acquisition unit can also record the deceased's speaking style and tone in detail. For example, the acquisition unit can record in detail subtle changes in the tone and speaking style of the deceased when they speak and reflect them in the audio data. This makes it possible to acquire data for faithfully reproducing the deceased's voice. Some or all of the above-mentioned processing in the acquisition unit may be performed using, for example, AI, or may be performed without using AI. For example, the acquisition unit can input the deceased's voice data recorded using a microphone into a generation AI and analyze changes in the voice tone and speaking style.

[0078] The generation unit can analyze the acquired appearance data and generate a 3D model that faithfully reproduces the appearance of the deceased. For example, the generation unit creates a 3D model based on the collected data and faithfully reproduces the appearance of the deceased. The generation unit also analyzes the acquired appearance data and uses an algorithm to reproduce the facial expressions and movements of the deceased. For example, the generation unit reproduces in detail the facial expression of the deceased when they smile and reflects this in the 3D model. The generation unit can also reproduce in detail the movements of the deceased when they perform a specific action (e.g., waving their hands). This makes it possible to generate a 3D model that faithfully reproduces the appearance of the deceased. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the collected appearance data into the generation AI and generates a 3D model that faithfully reproduces the appearance of the deceased.

[0079] The generation unit can analyze the acquired voice data and generate voice that faithfully reproduces the voice and speaking style of the deceased. For example, the generation unit analyzes the voice data and uses an algorithm to reproduce the voice and speaking style of the deceased. The generation unit records changes in the tone of the voice and speaking style of the deceased in detail and reflects them in the voice data. For example, the generation unit reproduces subtle changes in the tone and speaking style of the deceased when they speak in detail and reflects them in the voice data. The generation unit can also reproduce the tone of voice and speaking style of the deceased when they express specific emotions. This makes it possible to generate voice that faithfully reproduces the voice and speaking style of the deceased. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the collected voice data into a generation AI and generates voice that faithfully reproduces the voice and speaking style of the deceased.

[0080] The dialogue unit can refer to episode data stored in a database and conduct a dialogue based on past events. The dialogue unit, for example, refers to episodes that the deceased talked about while alive and conducts a dialogue. The dialogue unit searches the episode data stored in the database and uses an algorithm for conducting a dialogue based on past events. For example, the dialogue unit refers to a specific episode that the deceased spent with family or friends and conducts a dialogue. The dialogue unit can also refer to events that the deceased experienced at a specific event and conduct a dialogue. This enables a more realistic conversation by conducting a dialogue based on past events. Some or all of the above-mentioned processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data stored in a database into AI and conducts a dialogue based on past events.

[0081] The acquisition unit can estimate the user's emotions and adjust the timing of appearance data acquisition based on the estimated user emotions. For example, the acquisition unit can acquire the deceased's appearance data when the user is relaxed, thereby recording natural facial expressions and movements. The acquisition unit can also temporarily suspend acquisition of appearance data when the user is emotional and resume acquisition when the user has calmed down. Furthermore, when the user is expressing a specific emotion, the acquisition unit can prioritize acquisition of facial expressions and movements related to that emotion. This allows for more natural data to be acquired by adjusting the timing of appearance data acquisition according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the acquisition unit may be performed using AI, or without AI. For example, the acquisition unit inputs the user's facial expression data into the generation AI, estimates the emotion, and adjusts the timing of appearance data acquisition.

[0082] The acquisition unit can optimize the method of acquiring appearance data based on the living environment and activity status of the deceased. For example, the acquisition unit acquires appearance data at places where the deceased spent their daily lives and records natural movements and facial expressions. The acquisition unit can also acquire appearance data when the deceased is engaged in a specific activity and record in detail the movements and facial expressions related to that activity. Furthermore, the acquisition unit can acquire appearance data in different environments, such as indoors and outdoors, depending on the living environment of the deceased, to recreate various situations. In this way, by optimizing the method of acquiring appearance data based on the living environment and activity status of the deceased, more natural data can be acquired. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs data on the living environment of the deceased into a generation AI to optimize the method of acquiring appearance data.

[0083] When acquiring appearance data, the acquisition unit can simultaneously acquire information about the deceased's clothing and accessories. For example, the acquisition unit may record in detail the clothing and accessories that the deceased often wore and reflect this in the 3D model. The acquisition unit may also acquire the clothing and accessories that the deceased wore at a particular event and recreate that situation. Furthermore, the acquisition unit may also record in detail the color and design of the deceased's clothing and accessories and reflect this in the appearance data. In this way, by simultaneously acquiring information about the deceased's clothing and accessories, a more detailed 3D model can be generated. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit may input data about the deceased's clothing and accessories into a generation AI, which analyzes the detailed information.

[0084] When acquiring appearance data, the acquisition unit records the deceased's movement patterns in detail, thereby improving the naturalness of the movements. For example, the acquisition unit may record the deceased's walking movement patterns in detail and reflect them in the 3D model. The acquisition unit may also acquire the movement patterns of the deceased when performing specific gestures and reproduce natural movements. Furthermore, the acquisition unit may record the deceased's daily movements (e.g., sitting, standing, etc.) in detail and reflect them in the 3D model. In this way, by recording the deceased's movement patterns in detail, more natural movements can be reproduced. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit may input the deceased's movement data into a generation AI, which analyzes the movement patterns and reproduces natural movements.

[0085] The acquisition unit can estimate the user's emotions and determine the priority of appearance data to be acquired based on the estimated user emotions. For example, when the user is emotional, the acquisition unit prioritizes acquiring facial expressions and actions related to the emotion. Furthermore, when the user is relaxed, the acquisition unit can prioritize acquiring natural facial expressions and actions. Furthermore, when the user is expressing a specific emotion, the acquisition unit can prioritize acquiring appearance data related to that emotion. Thus, by prioritizing appearance data according to the user's emotions, more important data can be acquired preferentially. Emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the user's facial expression data into the generation AI, estimates the emotion, and determines the priority of the appearance data.

[0086] When acquiring appearance data, the acquisition unit can prioritize acquiring movements related to the deceased's hobbies and special skills. For example, the acquisition unit prioritizes acquiring movements that the deceased performed as hobbies (e.g., drawing, playing a musical instrument, etc.). The acquisition unit can also record in detail movements that the deceased performed as special skills (e.g., sports movements) and reflect these in the 3D model. Furthermore, the acquisition unit can prioritize acquiring facial expressions and movements of the deceased performing their hobbies or special skills. In this way, by prioritizing the acquisition of movements related to the deceased's hobbies and special skills, a more detailed 3D model can be generated. Some or all of the above-described processing by the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs movement data related to the deceased's hobbies and special skills into a generation AI, which analyzes the detailed information.

[0087] When acquiring appearance data, the acquisition unit can acquire data corresponding to the deceased's health condition and age. For example, the acquisition unit can record in detail the movements and facial expressions of the deceased corresponding to their health condition and reflect them in the 3D model. The acquisition unit can also acquire appearance data corresponding to the deceased's age and reproduce the appearance and movements corresponding to that age. Furthermore, the acquisition unit can prioritize acquiring specific movements and facial expressions related to the deceased's health condition and age. In this way, by acquiring data corresponding to the deceased's health condition and age, a more detailed 3D model can be generated. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs data related to the deceased's health condition and age into the generation AI, which analyzes the detailed information.

[0088] The acquisition unit can record the deceased's interactions with family and friends when acquiring appearance data. For example, the acquisition unit can record in detail the facial expressions and movements of the deceased while spending time with family and friends and reflect these in the 3D model. The acquisition unit can also prioritize acquiring specific activities (e.g., eating, conversation, etc.) that the deceased engaged in with family and friends. Furthermore, the acquisition unit can record the deceased's natural movements and expressions while interacting with family and friends. This allows for the generation of a more detailed 3D model by recording the deceased's interactions with family and friends. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs interaction data of the deceased with family and friends into a generation AI, which analyzes the detailed information.

[0089] The acquisition unit can estimate the user's emotions and adjust the timing of voice data acquisition based on the estimated user emotions. For example, the acquisition unit can record a natural speaking style and tone by acquiring voice data of the deceased when the user is relaxed. The acquisition unit can also temporarily suspend voice data acquisition when the user is emotional and resume it when the user has calmed down. Furthermore, if the user is expressing a specific emotion, the acquisition unit can prioritize acquisition of voice data related to that emotion. This allows for more natural data to be acquired by adjusting the timing of voice data acquisition according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the user's voice data into the generation AI, estimates the emotion, and adjusts the timing of voice data acquisition.

[0090] The acquisition unit can record in detail changes in the tone and speaking style of the deceased's voice to achieve more natural voice reproduction. For example, the acquisition unit can record in detail subtle changes in the tone and speaking style of the deceased when they speak and reflect them in the voice data. The acquisition unit can also acquire changes in the tone and speaking style of the deceased when they express different emotions to achieve natural voice reproduction. Furthermore, the acquisition unit can also record in detail changes in the tone and speaking style of the deceased when they speak in specific situations and reflect them in the voice data. This enables more natural voice reproduction by recording changes in the tone and speaking style of the deceased in detail. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI or without AI. For example, the acquisition unit inputs the deceased's voice data into a generation AI, analyzes changes in tone and speaking style, and achieves natural voice reproduction.

[0091] The acquisition unit can record in detail specific emotional expressions of the deceased (such as joy, sadness, and anger) when acquiring voice data. For example, the acquisition unit can record in detail the tone of voice and manner of speaking of the deceased when expressing joy and reflect this in the voice data. The acquisition unit can also record in detail the tone of voice and manner of speaking of the deceased when expressing sadness and reflect this in the voice data. Furthermore, the acquisition unit can also record in detail the tone of voice and manner of speaking of the deceased when expressing anger and reflect this in the voice data. In this way, by recording in detail the specific emotional expressions of the deceased, more natural voice reproduction is possible. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs the emotional expression data of the deceased into a generation AI, which analyzes the detailed information.

[0092] When acquiring voice data, the acquisition unit can focus on recording specific phrases and catchphrases of the deceased. For example, the acquisition unit can record in detail specific phrases that the deceased frequently used and reflect them in the voice data. The acquisition unit can also record in detail the catchphrases of the deceased and reflect them in the voice of the 3D model. Furthermore, the acquisition unit can focus on acquiring phrases and catchphrases that the deceased frequently used in specific situations. By focusing on recording the specific phrases and catchphrases of the deceased, more natural voice reproduction is possible. Some or all of the above-mentioned processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs data on the specific phrases and catchphrases of the deceased into a generation AI, which analyzes the detailed information.

[0093] The acquisition unit can estimate the user's emotion and determine the priority of the voice data to be acquired based on the estimated user emotion. For example, when the user is emotional, the acquisition unit prioritizes acquiring voice data related to the emotion. Furthermore, when the user is relaxed, the acquisition unit can prioritize acquiring natural speaking style and tone. Furthermore, when the user is expressing a specific emotion, the acquisition unit can prioritize acquiring voice data related to the emotion. Thus, by prioritizing the voice data according to the user's emotion, more important data can be acquired preferentially. The emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the user's voice data into the generation AI, estimates the emotion, and determines the priority of the voice data.

[0094] When acquiring audio data, the acquisition unit can acquire special audio, such as the deceased's singing voice or poetry recitation. For example, the acquisition unit can record detailed audio data of the deceased singing and reflect it in the audio of the 3D model. The acquisition unit can also acquire audio data of the deceased reciting poetry and recreate it in a special situation. Furthermore, the acquisition unit can preferentially acquire audio data of the deceased singing or reciting at a specific event. This allows for the acquisition of special audio of the deceased, thereby generating a more detailed 3D model. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the deceased's singing voice or poetry recitation data into a generation AI, which analyzes the detailed information.

[0095] When acquiring voice data, the acquisition unit can record voice changes according to the deceased's different ages and situations. For example, the acquisition unit can record in detail the deceased's voice when young and when older, and reflect this in the 3D model. The acquisition unit can also acquire changes in the deceased's tone of voice and speaking style when speaking in different situations, thereby achieving natural voice reproduction. Furthermore, the acquisition unit can also record in detail voice changes according to the deceased's age and situation, and reflect these in the voice data. This allows for the generation of a more detailed 3D model by recording voice changes according to the deceased's different ages and situations. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs data on the deceased's voice changes into a generation AI, which analyzes the detailed information.

[0096] The acquisition unit can record conversations between the deceased and family and friends when acquiring voice data. For example, the acquisition unit can record detailed voice data of the deceased talking with family and friends and reflect the voice data in the 3D model. The acquisition unit can also prioritize acquisition of specific conversations the deceased had with family and friends. Furthermore, the acquisition unit can record the natural speaking style and tone of the deceased when talking with family and friends. This allows for the generation of a more detailed 3D model by recording the deceased's conversations with family and friends. Some or all of the above-described processing in the acquisition unit can be performed using AI or without AI. For example, the acquisition unit can input conversation data between the deceased and family and friends into a generation AI and analyze the detailed information.

[0097] The generation unit can estimate the user's emotions and adjust the generation method of the 3D model based on the estimated user emotions. For example, if the user is relaxed, the generation unit generates a 3D model with relaxed movements and facial expressions. Furthermore, if the user is emotional, the generation unit can generate a 3D model with facial expressions and movements related to the emotion. Furthermore, if the user is expressing a specific emotion, the generation unit can generate a 3D model related to that emotion. This allows for the generation of a more natural 3D model by adjusting the generation method of the 3D model according to the user's emotions. The emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the generation unit is performed using the generation AI. For example, the generation unit inputs the user's emotion data into the generation AI and adjusts the generation method of the 3D model.

[0098] The generation unit can reproduce in detail specific facial expressions and movements of the deceased when generating the 3D model. For example, the generation unit can reproduce in detail the facial expression of the deceased when they smile and reflect this in the 3D model. The generation unit can also reproduce in detail the movements of the deceased when they perform a specific action (for example, waving their hands). Furthermore, the generation unit can also reproduce in detail the facial expressions and movements of the deceased when they express a specific emotion and reflect this in the 3D model. In this way, by reproducing in detail the specific facial expressions and movements of the deceased, a more natural 3D model can be generated. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs facial expression and movement data of the deceased into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0099] The generation unit can reproduce details of the deceased's clothing and accessories when generating the 3D model. For example, the generation unit can reproduce in detail the clothing and accessories that the deceased often wore and reflect them in the 3D model. The generation unit can also reproduce clothing and accessories that the deceased wore at specific events. Furthermore, the generation unit can reproduce in detail the colors and designs of the deceased's clothing and accessories and reflect them in the 3D model. This allows for the generation of a more natural-looking 3D model by reproducing the details of the deceased's clothing and accessories. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs data on the deceased's clothing and accessories into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0100] When generating the 3D model, the generation unit analyzes the movement patterns of the deceased to achieve more natural movements. For example, the generation unit analyzes the movement patterns of the deceased when walking and reflects them in the 3D model. The generation unit can also analyze the movement patterns of the deceased when performing specific gestures and reproduce natural movements. Furthermore, the generation unit can analyze the movements that the deceased performed daily (e.g., sitting, standing, etc.) and reflect them in the 3D model. In this way, by analyzing the movement patterns of the deceased, more natural movements can be reproduced. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the movement data of the deceased into the generation AI, analyzes detailed information, and reflects it in the 3D model.

[0101] The generation unit can estimate the user's emotions and adjust the display method of the 3D model based on the estimated user emotions. For example, if the user is relaxed, the generation unit displays a 3D model with relaxed movements and facial expressions. Furthermore, if the user is emotional, the generation unit can display a 3D model with facial expressions and movements related to the emotion. Furthermore, if the user is expressing a specific emotion, the generation unit can display a 3D model related to that emotion. This allows for a more natural 3D model to be displayed by adjusting the display method of the 3D model according to the user's emotions. The emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the generation unit is performed using the generation AI. For example, the generation unit inputs the user's emotion data into the generation AI and adjusts the display method of the 3D model.

[0102] When generating the 3D model, the generation unit can reproduce movements related to the deceased's hobbies and special skills. For example, the generation unit reproduces movements that the deceased performed as a hobby (e.g., drawing, playing an instrument, etc.). The generation unit can also reproduce in detail movements that the deceased performed as a special skill (e.g., sports movements) and reflect them in the 3D model. Furthermore, the generation unit can also reproduce the facial expressions and movements of the deceased while performing their hobbies or special skills. In this way, by reproducing movements related to the deceased's hobbies and special skills, a more detailed 3D model can be generated. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs movement data related to the deceased's hobbies and special skills into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0103] When generating the 3D model, the generation unit can reproduce the appearance of the deceased according to their health condition and age. For example, the generation unit can reproduce in detail the appearance and movements of the deceased according to their health condition and reflect this in the 3D model. The generation unit can also obtain appearance data according to the age of the deceased and reproduce the appearance and movements according to their age. Furthermore, the generation unit can reproduce specific movements and facial expressions related to the health condition and age of the deceased. This allows for the generation of a more detailed 3D model by reproducing the appearance according to the health condition and age of the deceased. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs data related to the health condition and age of the deceased into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0104] The generation unit can recreate the deceased's interactions with family and friends when generating the 3D model. For example, the generation unit can recreate in detail the facial expressions and movements of the deceased when spending time with family and friends and reflect them in the 3D model. The generation unit can also recreate specific activities (e.g., eating, conversation, etc.) that the deceased engaged in with family and friends. Furthermore, the generation unit can recreate the natural movements and facial expressions of the deceased when interacting with family and friends. This allows for the generation of a more detailed 3D model by recreating the deceased's interactions with family and friends. Some or all of the above-described processing in the generation unit is performed using a generation AI. For example, the generation unit inputs interaction data of the deceased with family and friends into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0105] The database can estimate the user's emotion and adjust the storage method of episode data based on the estimated user emotion. For example, if the user is emotional, the database can prioritize storing episodes related to the emotion. Furthermore, if the user is relaxed, the database can prioritize storing natural episodes. Furthermore, if the user is expressing a particular emotion, the database can prioritize storing episodes related to that emotion. This allows for more appropriate data storage by adjusting the storage method of episode data according to the user's emotion. The emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the database can be performed using AI, or without AI. For example, the database inputs the user's emotion data into the generative AI and adjusts the storage method of episode data.

[0106] When storing episode data, the database can record in detail episodes including specific emotional expressions of the deceased. For example, the database can record in detail episodes when the deceased expressed joy and store them in the database. The database can also record in detail episodes when the deceased expressed sadness and store them in the database. Furthermore, the database can also record in detail episodes when the deceased expressed anger and store them in the database. In this way, by recording in detail episodes including specific emotional expressions of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs emotional expression data of the deceased into a generation AI, which analyzes the detailed information and stores it as episode data.

[0107] When storing episode data, the database can focus on recording specific events and memories of the deceased. For example, the database can record in detail events experienced by the deceased at a specific event and store them in the database. The database can also record in detail specific memories the deceased had with family and friends and store them in the database. Furthermore, the database can also record in detail events and memories the deceased experienced in a specific place and store them in the database. This allows for more detailed data to be stored by focusing on recording specific events and memories of the deceased. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database inputs the deceased's event and memory data into a generation AI, analyzes the detailed information, and stores it as episode data.

[0108] When storing episode data, the database can record in detail episodes with the deceased's family and friends. For example, the database can record in detail specific episodes the deceased spent with their family and store them in the database. The database can also record in detail specific episodes the deceased spent with their friends and store them in the database. Furthermore, the database can also record in detail specific activities and events the deceased had with their family and friends and store them in the database. This allows for more detailed data to be stored by recording episodes with the deceased's family and friends in detail. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database inputs episode data with the deceased's family and friends into a generation AI, which analyzes the detailed information and stores it as episode data.

[0109] The database can estimate the user's emotions and prioritize episode data based on the estimated user emotions. For example, if the user is emotional, the database can prioritize storing episodes related to the emotion. Furthermore, if the user is relaxed, the database can prioritize storing natural episodes. Furthermore, if the user is expressing a particular emotion, the database can prioritize storing episodes related to that emotion. This allows for prioritized storage of more important data by prioritizing episode data according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the database can be performed using AI, or without AI. For example, the database inputs the user's emotion data into a generative AI to prioritize episode data.

[0110] When storing episode data, the database can record episodes related to the hobbies and special skills of the deceased. For example, the database can record in detail episodes related to activities that the deceased engaged in as hobbies and store them in the database. The database can also record in detail episodes related to activities that the deceased considered to be special skills and store them in the database. Furthermore, the database can also record in detail episodes of the deceased engaging in hobbies and special skills and store them in the database. In this way, by recording episodes related to the hobbies and special skills of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs episode data related to the hobbies and special skills of the deceased into a generation AI, which analyzes the detailed information and stores it as episode data.

[0111] When storing episode data, the database can record episodes according to different ages and circumstances of the deceased. For example, the database can record in detail episodes from the deceased's youth and episodes from when the deceased was older and store them in the database. The database can also record in detail episodes experienced by the deceased in different circumstances and store them in the database. Furthermore, the database can also record in detail episodes according to the deceased's age and circumstances and store them in the database. In this way, by recording episodes according to different ages and circumstances of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs episode data according to the deceased's age and circumstances into a generation AI, which analyzes the detailed information and stores it as episode data.

[0112] When storing episode data, the database can record episodes with the deceased's family and friends. For example, the database can record in detail specific episodes the deceased spent with their family and store them in the database. The database can also record in detail specific episodes the deceased spent with their friends and store them in the database. Furthermore, the database can also record in detail specific activities and events the deceased had with their family and friends and store them in the database. This allows for more detailed data to be stored by recording episodes with the deceased's family and friends. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database can input episode data with the deceased's family and friends into a generation AI, analyze the detailed information, and store it as episode data.

[0113] The dialogue unit can estimate the user's emotions and adjust the content and tone of the dialogue based on the estimated user emotions. For example, if the user is relaxed, the dialogue unit can use a calm tone. Furthermore, if the user is emotional, the dialogue unit can use content and tone that reflects the user's emotions. Furthermore, if the user is expressing a specific emotion, the dialogue unit can use content and tone that reflects the emotion. This allows for more appropriate dialogue by adjusting the content and tone of the dialogue according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without AI. For example, the dialogue unit inputs the user's emotion data into the generation AI and adjusts the content and tone of the dialogue.

[0114] During the dialogue, the dialogue unit can conduct a conversation by referring to episodes containing specific emotional expressions of the deceased. For example, the dialogue unit can conduct a dialogue by referring to episodes in which the deceased expressed joy. The dialogue unit can also conduct a dialogue by referring to episodes in which the deceased expressed sadness. Furthermore, the dialogue unit can conduct a dialogue by referring to episodes in which the deceased expressed anger. In this way, by referring to episodes containing specific emotional expressions of the deceased, more detailed dialogue is possible. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs emotional expression data of the deceased into a generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0115] During the dialogue, the dialogue unit can conduct a conversation by referring to specific events or memories of the deceased. For example, the dialogue unit can conduct a dialogue by referring to events experienced by the deceased at a specific event. The dialogue unit can also conduct a dialogue by referring to specific memories the deceased had with family and friends. Furthermore, the dialogue unit can conduct a dialogue by referring to events or memories the deceased experienced in a specific place. This enables a more detailed dialogue by referring to specific events and memories of the deceased. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs data on events and memories of the deceased into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0116] During the dialogue, the dialogue unit can refer to episodes with the deceased's family and friends to conduct the conversation. For example, the dialogue unit can refer to specific episodes the deceased spent with their family and conduct the dialogue. The dialogue unit can also refer to specific episodes the deceased spent with their friends to conduct the dialogue. Furthermore, the dialogue unit can refer to specific activities or events the deceased had with their family and friends to conduct the dialogue. This enables more detailed dialogue by referring to episodes with the deceased's family and friends. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data with the deceased's family and friends into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0117] The dialogue unit can estimate the user's emotions and determine the priority of dialogues based on the estimated user emotions. For example, when the user is emotional, the dialogue unit can prioritize dialogues related to the emotion. Furthermore, when the user is relaxed, the dialogue unit can prioritize natural dialogues. Furthermore, when the user is showing a specific emotion, the dialogue unit can prioritize dialogues related to that emotion. In this way, by determining the priority of dialogues according to the user's emotions, more important dialogues can be prioritized. Emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the dialogue unit may be performed using AI, or may be performed without AI. For example, the dialogue unit inputs the user's emotion data into the generation AI and determines the priority of dialogues.

[0118] During the dialogue, the dialogue unit can conduct the conversation by referring to episodes related to the hobbies and special skills of the deceased. For example, the dialogue unit can conduct the dialogue by referring to episodes related to activities that the deceased engaged in as hobbies. The dialogue unit can also conduct the dialogue by referring to episodes related to activities that the deceased considered to be special skills. Furthermore, the dialogue unit can conduct the dialogue by referring to episodes of the deceased engaging in hobbies and special skills. In this way, by referring to episodes related to the hobbies and special skills of the deceased, more detailed dialogue is possible. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data related to the hobbies and special skills of the deceased into the generation AI, analyzes the detailed information, and reflects it in the dialogue.

[0119] During the dialogue, the dialogue unit can conduct a conversation by referring to episodes that correspond to different ages and situations of the deceased. For example, the dialogue unit may conduct a dialogue by referring to episodes from the deceased's youth and episodes from when the deceased was older. The dialogue unit may also conduct a dialogue by referring to episodes that the deceased experienced in different situations. Furthermore, the dialogue unit may conduct a dialogue by referring to episodes that correspond to the deceased's ages and situations. This enables a more detailed dialogue by referring to episodes that correspond to the deceased's different ages and situations. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit may input episode data that corresponds to the deceased's age and situation into a generation AI, analyze the detailed information, and reflect it in the dialogue.

[0120] During the dialogue, the dialogue unit can refer to episodes with the deceased's family and friends to conduct the conversation. For example, the dialogue unit can refer to specific episodes the deceased spent with their family and conduct the dialogue. The dialogue unit can also refer to specific episodes the deceased spent with their friends to conduct the dialogue. Furthermore, the dialogue unit can refer to specific activities or events the deceased had with their family and friends to conduct the dialogue. This enables more detailed dialogue by referring to episodes with the deceased's family and friends. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without using AI. For example, the dialogue unit inputs episode data with the deceased's family and friends into the generation AI, analyzes the detailed information, and reflects it in the dialogue. === Hard Collateral 1-1 === Each of the multiple elements including the acquisition unit, generation unit, database, and dialogue unit described above is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the acquisition unit can acquire appearance data and voice data of the deceased using the camera 42 and microphone 38B of the smart device 14. For example, the generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a 3D model based on the acquired data. For example, the database is stored in the storage 32 of the data processing device 12. For example, the dialogue unit is realized by the specific processing unit 290 of the data processing device 12 and engages in dialogue by referring to episode data stored in the database. === Hard Collateral 1-2 === Each of the multiple elements, including the acquisition unit, generation unit, database, and dialogue unit, described above, is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the acquisition unit can acquire appearance data and voice data of the deceased using the camera 42 and microphone 238 of the smart glasses 214. For example, the generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a 3D model based on the acquired data. For example, the database is stored in the storage 32 of the data processing device 12. For example, the dialogue unit is realized by the specific processing unit 290 of the data processing device 12 and engages in dialogue by referring to episode data stored in the database. === Hard Collateral 1-3 === Each of the multiple elements including the acquisition unit, generation unit, database, and dialogue unit described above is realized, for example, by at least one of the headset-type terminal 314 and the data processing device 12. For example, the acquisition unit can acquire appearance data and voice data of the deceased using the camera 42 and microphone 238 of the headset-type terminal 314. For example, the generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a 3D model based on the acquired data. For example, the database is stored in the storage 32 of the data processing device 12. For example, the dialogue unit is realized by the specific processing unit 290 of the data processing device 12 and conducts a dialogue by referring to episode data stored in the database. === Hard Collateral 1-4 === Each of the multiple elements including the acquisition unit, generation unit, database, and dialogue unit described above is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the acquisition unit can acquire appearance data and voice data of the deceased using the camera 42 and microphone 238 of the robot 414. For example, the generation unit is realized by the specific processing unit 290 of the data processing device 12 and generates a 3D model based on the acquired data. For example, the database is stored in the storage 32 of the data processing device 12. For example, the dialogue unit is realized by the specific processing unit 290 of the data processing device 12 and conducts dialogue by referring to episode data stored in the database.

[0121] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0122] The acquisition unit can also record the deceased's lifestyle habits and daily routine in detail. For example, it can record the exercise the deceased did every morning or activities they performed at specific times and reflect these in the 3D model. The acquisition unit can also record in detail the movements and facial expressions of the deceased when eating a specific meal. Furthermore, the acquisition unit can prioritize movements related to the deceased's hobbies and special skills. This allows for the generation of a more realistic 3D model by recording the deceased's lifestyle habits and daily routine in detail. Some or all of the above-mentioned processes in the acquisition unit may be performed using AI, or may be performed without AI. For example, the acquisition unit inputs the deceased's lifestyle habits data into a generation AI, which then analyzes the detailed information.

[0123] The generation unit can also reproduce movements and facial expressions according to the deceased's health condition and age. For example, it can reproduce the energetic movements of a young person and the slower movements of an older person. If the deceased had a specific illness or disability, the generation unit can also reproduce movements and facial expressions affected by that illness or disability. Furthermore, the generation unit can also reproduce in detail the movements and facial expressions of the deceased when expressing specific emotions and reflect them in the 3D model. This allows for the generation of a more realistic 3D model by reproducing movements and facial expressions according to the deceased's health condition and age. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs data related to the deceased's health condition and age into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0124] The dialogue unit can estimate the user's emotions and adjust the content and tone of the dialogue based on the estimated user's emotions. For example, if the user is relaxed, the dialogue can be conducted in a calm tone. Furthermore, if the user is emotional, the dialogue can be conducted in a content and tone that is in line with the user's emotions. Furthermore, if the user is expressing a specific emotion, the dialogue can be conducted in a content and tone that is related to that emotion. This allows for more appropriate dialogue by adjusting the content and tone of the dialogue according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without AI. For example, the dialogue unit inputs the user's emotion data into the generation AI and adjusts the content and tone of the dialogue.

[0125] When storing episode data, the database can record in detail episodes that include specific emotional expressions of the deceased. For example, episodes in which the deceased expressed joy can be recorded in detail and stored in the database. Episodes in which the deceased expressed sadness can also be recorded in detail and stored in the database. Furthermore, episodes in which the deceased expressed anger can also be recorded in detail and stored in the database. In this way, by recording in detail episodes that include specific emotional expressions of the deceased, more detailed data can be stored. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without using AI. For example, the database inputs emotional expression data of the deceased into a generation AI, which analyzes the detailed information and stores it as episode data.

[0126] The acquisition unit can prioritize acquiring movements related to the deceased's hobbies and special skills. For example, it prioritizes acquiring movements that the deceased performed as hobbies (e.g., drawing, playing an instrument, etc.). It can also record in detail movements that the deceased performed as special skills (e.g., sports movements) and reflect them in the 3D model. It can also prioritize acquiring facial expressions and movements of the deceased performing their hobbies or special skills. In this way, by prioritizing acquisition of movements related to the deceased's hobbies and special skills, a more detailed 3D model can be generated. Some or all of the above-described processing in the acquisition unit may be performed using AI, or may be performed without using AI. For example, the acquisition unit inputs movement data related to the deceased's hobbies and special skills into a generation AI, which analyzes the detailed information.

[0127] The generation unit can estimate the user's emotions and adjust the generation method of the 3D model based on the estimated user emotions. For example, if the user is relaxed, a 3D model with relaxed movements and facial expressions can be generated. Also, if the user is emotional, a 3D model with facial expressions and movements related to the emotion can be generated. Furthermore, if the user is expressing a specific emotion, a 3D model related to that emotion can be generated. This allows for the generation of a more natural 3D model by adjusting the generation method of the 3D model according to the user's emotions. The emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be, for example, a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-described processing in the generation unit is performed using the generation AI. For example, the generation unit inputs the user's emotion data into the generation AI and adjusts the generation method of the 3D model.

[0128] When storing episode data, the database can record in detail episodes with the deceased's family and friends. For example, specific episodes the deceased spent with their family can be recorded in detail and stored in the database. Specific episodes the deceased spent with their friends can also be recorded in detail and stored in the database. Specific activities and events the deceased had with their family and friends can also be recorded in detail and stored in the database. This allows for more detailed data to be stored by recording episodes with the deceased's family and friends in detail. Some or all of the above-mentioned processing in the database may be performed using AI, or may be performed without AI. For example, the database inputs episode data with the deceased's family and friends into a generation AI, which analyzes the detailed information and stores it as episode data.

[0129] The dialogue unit can estimate the user's emotions and determine the priority of dialogues based on the estimated user emotions. For example, if the user is emotional, dialogues related to the emotion can be prioritized. Furthermore, if the user is relaxed, natural dialogues can be prioritized. Furthermore, if the user is showing a specific emotion, dialogues related to that emotion can be prioritized. By determining the priority of dialogues according to the user's emotions, more important dialogues can be prioritized. The emotion estimation is realized using an emotion estimation function, for example, an emotion engine or a generation AI. The generation AI can be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-described processing in the dialogue unit may be performed using AI, or may be performed without AI. For example, the dialogue unit inputs the user's emotion data into the generation AI to determine the priority of dialogues.

[0130] When generating the 3D model, the generation unit can reproduce in detail the specific facial expressions and movements of the deceased. For example, the facial expression of the deceased when smiling can be reproduced in detail and reflected in the 3D model. The generation unit can also reproduce in detail the movements of the deceased when performing a specific action (such as waving). Furthermore, the generation unit can reproduce in detail the facial expressions and movements of the deceased when expressing a specific emotion and reflect them in the 3D model. This allows for the generation of a more natural 3D model by reproducing the specific facial expressions and movements of the deceased in detail. Some or all of the above-mentioned processing in the generation unit is performed using a generation AI. For example, the generation unit inputs the facial expression and movement data of the deceased into the generation AI, analyzes the detailed information, and reflects it in the 3D model.

[0131] The database can estimate the user's emotions and adjust the storage method of episode data based on the estimated user emotions. For example, if the user is emotional, episodes related to that emotion can be stored preferentially. Alternatively, if the user is relaxed, natural episodes can be stored preferentially. Furthermore, if the user is expressing a particular emotion, episodes related to that emotion can be stored preferentially. This allows for more appropriate data storage by adjusting the storage method of episode data according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI can be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processing in the database can be performed using AI, or without AI. For example, the database inputs the user's emotion data into a generative AI and adjusts the storage method of episode data.

[0132] The processing flow of the second embodiment will be briefly explained below.

[0133] Step 1: The acquisition unit acquires appearance data. Appearance data includes the deceased's facial expressions, skeletal structure, movements, and habits. For example, facial expressions are captured with a camera, the skeleton is scanned with a 3D scanner, and movements and habits are recorded using motion capture technology. Step 2: The acquisition unit acquires audio data. The audio data includes the deceased's voice, speaking style, tone, and catchphrases. For example, the voice is recorded with a microphone, and subtle changes in speaking style and tone are recorded in detail. Step 3: The generator analyzes the acquired appearance data to generate a 3D model that faithfully reproduces the appearance of the deceased, and analyzes the acquired audio data to generate audio that faithfully reproduces the voice and speaking style of the deceased. Step 4: The database stores the acquired episode data, which includes events and memories that the deceased talked about during their lifetime. Step 5: The dialogue unit references the episode data stored in the database and conducts dialogue based on past events, allowing the user to reminisce about their memories with the deceased and feel the conversation with them more realistically.

[0134] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0135] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0136] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0137] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0138] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0139] 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0140] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0141] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0142] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0143] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0144] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0145] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0146] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0147] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0148] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0149] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0150] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0151] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0152] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0153] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0154] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0155] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0156] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0157] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0158] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0159] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0160] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0161] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0162] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0163] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0164] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0165] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0166] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0167] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0168] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0169] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0170] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0171] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0172] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0173] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0174] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0175] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0176] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0177] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0178] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0179] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0180] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0181] In the robot 414, the processor 46 performs the identification process. The storage 50 stores the identification program 60. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as the control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform the same process as the identification processing unit 290 using these models.

[0182] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0183] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0184] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0185] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0186] The correspondence between each part and the device or control part is not limited to the example described above, and various modifications are possible.

[0187] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0188] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0189] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0190] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0191] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0192] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0193] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0194] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0195] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0196] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0197] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0198] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0199] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0200] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0201] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0202] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0203] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0204] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0205] [Explanation of symbols]

[0206] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. an acquisition unit for acquiring appearance data; an acquisition unit that acquires voice data; a generation unit that generates a 3D model based on the data acquired by the acquisition unit; a database for storing episode data; A dialogue unit that carries out dialogue by referring to episode data stored in the database.

2. The acquisition unit Obtaining appearance data such as facial expressions, bone structure, movements, and habits of the deceased 2. The system of claim 1.

3. The acquisition unit Acquire audio data such as the deceased's voice, speaking style, tone, and catchphrases 2. The system of claim 1.

4. The generation unit The system analyzes the acquired appearance data and generates a 3D model that faithfully reproduces the appearance of the deceased.

5. The generation unit A system that analyzes the acquired audio data and generates audio that faithfully reproduces the voice and speaking style of the deceased.

6. The dialogue unit A system that references episode data stored in a database and conducts dialogue based on past events.

7. The acquisition unit Estimate the user's emotions and adjust the timing of acquiring appearance data based on the estimated user emotions.

3. The system of claim 2.

8. The acquisition unit Optimize the method of acquiring appearance data based on the living environment and activity status of the deceased 3. The system of claim 2.

9. The acquisition unit When acquiring appearance data, information on the deceased's clothing and accessories is also acquired at the same time.

3. The system of claim 2.

10. The acquisition unit When acquiring appearance data, the deceased's movement patterns are recorded in detail to improve the naturalness of their movements.

3. The system of claim 2.

Images