System

Abstract

A system is provided.SOLUTION: A system comprising means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for combining the generated audio data and the animation script to generate a 3D video, and means for transmitting the generated 3D video to a terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When exhibiting historical figures in museums and exhibition facilities, visitors often have difficulty forming a concrete image of the figure. In particular, when the information on display is static, it is difficult for visitors to grasp the vivid appearance and expressions of the figure. There is also the problem of it being difficult to provide effective information to visitors who speak multiple languages. [Means for solving the problem]

[0005] Therefore, the present invention provides a system including means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating a 3D video by combining the generated audio data and the animation script, and means for transmitting the generated 3D video to a terminal. This system provides visitors with a concrete image of a great figure, enabling a more visually and intuitive exhibition. Furthermore, the generated 3D video includes audio data in multiple languages, making it possible to effectively accommodate visitors from overseas. Furthermore, by displaying the generated 3D video on a display using a terminal at a museum or exhibition facility, even more interactive exhibitions can be realized.

[0006] A "user" is a person who uses the system to provide image data and basic information about a great person.

[0007] "Image data" is a digital file containing visual information, such as a photograph or illustration of a famous person.

[0008] "Basic information" is supplementary information expressed in text, such as the great person's name, date of birth, and achievements.

[0009] A "terminal" is a device operated by a user, and has the function of transmitting image data and basic information to a server.

[0010] The "server" is a central device that receives and analyzes data sent from the terminal and performs 3D modeling and 3D video generation.

[0011] An "image recognition engine" is software or an algorithm that detects feature points from image data.

[0012] "Feature points" refer to specific parts of a great figure's face or posture, and are the basic information for 3D modeling.

[0013] A "3D model" is a digital object that represents a great person in three dimensions.

[0014] "Audio data" is a digital file that represents the speech of a great person in audio format.

[0015] An "animation script" is a program that contains instructions for controlling the timing of 3D model movements and sounds.

[0016] A "3D video" is a video file created by combining a generated 3D model, audio data, and an animation script.

[0017] A "display" is a visual device used to watch 3D videos in museums and exhibition facilities.

[0018] "Museums" and "exhibition facilities" are places that display historical and cultural items and provide information to visitors.

[0019] "Multilingual" refers to the ability to provide information in more than one language. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that uses generative AI to create 3D videos based on photographs and basic information of famous people, and broadcasts these videos in museums and exhibition facilities. Below, we will explain the program processing of the system of this invention in natural language.

[0042] System Overview

[0043] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and transmit image data and basic information about the great figure. The server analyzes the transmitted data and generates a 3D model and 3D video. The terminal then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0044] Program processing

[0045] 1. Upload photos and enter data

[0046] User

[0047] The user uses the upload interface on the terminal to select and upload image data (e.g., a JPEG file) of the great figure.

[0048] The user fills in basic information such as name, date of birth, achievements, etc. in the input form and clicks the submit button.

[0049] Terminal

[0050] The terminal receives the uploaded image data and the input basic information and assembles them into a packet format.

[0051] The terminal sends this packet to the server as an HTTP request.

[0052] 2. Image analysis and 3D modeling

[0053] server

[0054] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0055] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position and contours of his nose, eyes, and mouth.

[0056] The server uses 3D modeling software to generate a 3D model of the great person based on the detected features, which is intended to realistically recreate the great person's specific appearance.

[0057] The server also uses the basic information about the great person to generate voice data for the great person (for example, a greeting such as "I am [great person's name]") using a speech synthesis engine.

[0058] 3. 3D video generation

[0059] server

[0060] The server generates an animation script based on the generated 3D model and voice data. This script contains instructions for controlling the speaking actions and facial expressions of the great figure.

[0061] The server executes the animation script and generates 3D animation by simulating movements on the 3D model.

[0062] The server encodes the generated 3D video into multiple formats (e.g., streaming format and download format).

[0063] 4. Sending and Playing Videos

[0064] server

[0065] The server sends the encoded 3D video file to the device using a secure file transfer protocol.

[0066] Terminal

[0067] The device decodes the received 3D video file and saves it in local storage.

[0068] The device will connect to the display system of the museum or exhibition facility and prepare to play 3D videos.

[0069] User (visitor)

[0070] Visitors to the museum can watch 3D videos on the display, either automatically or by pressing a play button.

[0071] Specific examples

[0072] For example, when generating a video of "Great Person A," a great historical figure in Japan, the following specific example can be considered.

[0073] User

[0074] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the terminal.

[0075] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[0076] Terminal

[0077] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[0078] server

[0079] The server uses an image recognition engine to detect the facial features of Great Person A from "A_person.jpg."

[0080] The server uses 3D modeling software to generate a 3D model of Great Person A based on the detected feature points.

[0081] The server also uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event."

[0082] The server generates an animation script based on the 3D model and audio data, and generates a 3D video of great person A speaking.

[0083] The server transmits the generated 3D video to the terminal.

[0084] Terminal

[0085] The device receives the 3D video and plays it on the museum's display.

[0086] User (visitor)

[0087] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[0088] The above is an embodiment of the present invention. This system makes it possible to give a sense of liveliness to exhibits in museums and exhibition facilities, and to provide visitors with a more concrete image.

[0089] The processing flow will be explained below.

[0090] Step 1:

[0091] User

[0092] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[0093] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[0094] Step 2:

[0095] Terminal

[0096] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[0097] The terminal sends the collected packets to the server as an HTTP request.

[0098] Step 3:

[0099] server

[0100] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0101] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[0102] Step 4:

[0103] server

[0104] Based on the detected facial features, the server uses 3D modeling software to generate a 3D model of the great figure, which realistically reproduces the shape and detailed features of the face.

[0105] The server uses a speech synthesis engine to generate voice data for the great person based on the basic information about the great person, such as "I am (great person's name). I have accomplished (achievement)."

[0106] Step 5:

[0107] server

[0108] The server combines the generated 3D model with the audio data to generate an animation script, which includes the movements and facial expressions of the great man as he speaks.

[0109] The server executes the generated animation script and generates a 3D video by simulating the movements of the great man.

[0110] Step 6:

[0111] server

[0112] The server encodes the generated 3D video into streaming and download formats.

[0113] The server then sends the encoded 3D video to the device, using a file transfer protocol to ensure security.

[0114] Step 7:

[0115] Terminal

[0116] The device decodes the 3D video file received from the server and saves it in local storage.

[0117] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[0118] Step 8:

[0119] User (visitor)

[0120] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[0121] Through the above processing steps, the system provides visitors with a concrete and vivid image of the great man, enabling a more engaging exhibition experience.

[0122] Example 1

[0123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] Traditional exhibition facilities and museums often display the achievements of great people and famous figures and historical events using only static images and text, making it difficult to provide visitors with a sense of dynamism and realism. Furthermore, paper media and static displays limit the amount of information visitors receive, making them less appealing to younger visitors in particular. To address this issue, more dynamic and interactive exhibition methods are needed.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0126] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal, means for the terminal to decode the received 3D video and save it in local storage, and means for the terminal to play the 3D video in cooperation with a display system of a museum or exhibition facility. This allows visitors to get a concrete image of historical figures and famous people through the 3D video of the people.

[0127] "Image data" is digital data in the form of still images that is uploaded by the user.

[0128] "Basic information" is text data such as the name, date of birth, and achievements of a great or famous person.

[0129] An "image recognition engine" is software or an algorithm for detecting feature points from image data.

[0130] "Feature points" are important points that refer to the positions and contours of facial features such as the nose, eyes, and mouth that are detected from image data.

[0131] A "3D model" is a three-dimensional computer graphic representation of a great or famous person, generated based on detected feature points.

[0132] "Voice data" refers to a voice file generated by a voice synthesis engine based on basic information.

[0133] An "animation script" is a program containing a series of instructions created to control the movements and expressions of a 3D model.

[0134] A "3D video" is a video file generated by executing an animation script based on a 3D model and audio data.

[0135] "Terminal" refers to a device operated by a user that receives, stores, and plays data from a server.

[0136] "Display system" refers to the device or configuration installed in a museum or exhibition facility that displays 3D videos.

[0137] This invention is a system that uses generative AI to create 3D videos based on image data and basic information of great people and famous figures, and then broadcasts these videos in museums and exhibition facilities.

[0138] System configuration

[0139] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and send image data and basic information about the great figure. The server analyzes the data and generates a 3D model and 3D video. The terminal receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0140] Photo upload and data entry

[0141] Using a web interface on the device, users select and upload image data (e.g., a JPEG file) of a great person, fill out a form with basic information such as name, date of birth, and achievements, and click the submit button.

[0142] The terminal receives the input, assembles it into a single data packet, which contains both image and text data, and sends the packet to the server as an HTTP request.

[0143] Image Analysis and 3D Modeling

[0144] The server receives the HTTP request and parses the data packet to extract the image data and basic information. The server then uses an image recognition engine (e.g., OpenCV) to detect facial features from the image data, including the location and contours of the nose, eyes, and mouth.

[0145] The server then uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person based on the detected feature points. This model realistically reproduces the great person's appearance. The server also uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the great person's basic information. For example, it generates an audio file such as, "I am the name of a great person. My achievements are..."

[0146] 3D video generation

[0147] The server creates an animation script based on the generated 3D model and audio data. The script contains instructions to control the figure's speaking movements and facial expressions. The server executes the animation script and generates 3D video by simulating the 3D model's movements. The server then encodes the generated 3D video into multiple formats (e.g., MP4, WebM).

[0148] Video transmission and playback

[0149] The server sends the generated 3D video file to the device using a secure file transfer protocol (e.g., SFTP).

[0150] The device decodes the received 3D video file and saves it in local storage. The device then connects to the display system of the museum or exhibition facility and prepares the 3D video for playback, for example, using a media player connected to the display system.

[0151] Users (visitors) visit the museum and watch 3D videos on the exhibited display. Playback is either automatic or initiated by the visitor pressing a play button.

[0152] Specific examples

[0153] For example, consider the case of generating a video of "Great Person A," a great historical figure in Japan.

[0154] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the device, and then enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to a historical event."

[0155] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[0156] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg." It then uses 3D modeling software to generate a 3D model of great person A based on the detected features. It then uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event." The server then generates an animation script based on the 3D model and voice data, generating a 3D video of great person A speaking. The 3D video is then sent to the device.

[0157] The device receives the 3D video and plays it on the museum display. Visitors can watch the 3D video of great person A on the museum display, giving them a stronger, more concrete image of him.

[0158] The above is an embodiment of the present invention. This system allows visitors to deepen their understanding of the exhibits at museums and exhibition facilities and provides a lively experience.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] The user opens the web interface of the device, selects and uploads an image file of a great person or famous person (e.g., "A_person.jpg"). The selected image file is input, and basic information (such as name, date of birth, and achievements) is entered in the text box. By clicking the send button together with these, the input data is sent to the device. The output is the data received by the device.

[0162] Step 2:

[0163] The terminal combines the image file and basic information received from the user into a single data packet. This data packet contains image data and text data. The terminal then sends this data packet to the server as an HTTP request. It receives data from the user as input, creates and sends out a data packet, and the output is sent to the server.

[0164] Step 3:

[0165] The server analyzes the HTTP request received from the terminal and extracts image data and basic information from the data packet. Based on the analysis results of the HTTP request, the extracted image data and text data are passed to the next process. The output is the structured analysis of the data.

[0166] Step 4:

[0167] The server uses an image recognition engine (e.g., OpenCV) to detect feature points (e.g., the position and contours of the nose, eyes, and mouth) from the received image data. The input is image data, and the output is data on the detected feature points. Specifically, it runs a face detection algorithm.

[0168] Step 5:

[0169] The server generates a 3D model of the great person using 3D modeling software (e.g. Blender) based on the detected feature points. The input is the feature point data, and the output is the 3D model. Specifically, the server uses Blender's API to build the 3D model.

[0170] Step 6:

[0171] The server uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the basic information about the great person. The input is text data, and the output is an audio file. Specifically, the server passes the basic information as text to the speech synthesis engine.

[0172] Step 7:

[0173] The server creates an animation script based on the generated 3D model and audio data. The input is the 3D model and audio data, and the output is the animation script. In terms of specific operations, the script constitutes instructions to control the movements and facial expressions of the 3D model.

[0174] Step 8:

[0175] The server executes the animation script and simulates the movement of the 3D model to generate a 3D video. The input is the animation script and the 3D model, and the output is a 3D video file. Specifically, the server performs the animation rendering process.

[0176] Step 9:

[0177] The server encodes the generated 3D video file into multiple formats (e.g., MP4, WebM). The input is the 3D video file, and the output is the encoded video format. This is done using video encoding software.

[0178] Step 10:

[0179] The server sends the encoded 3D video file to the device using a secure file transfer protocol (e.g., SFTP). The input is the encoded video file, and the output is the transmission to the device.

[0180] Step 11:

[0181] The device decodes the received 3D video file and saves it to local storage. The input is the video file from the server, and the output is saving to local storage. Specifically, the video file is saved in a specified folder.

[0182] Step 12:

[0183] The terminal works in conjunction with the display system of the museum or exhibition facility to prepare for 3D video playback. The input is a stored 3D video file, and the output is playback on the display system. Specifically, it launches the media player and prepares to start playback.

[0184] Step 13:

[0185] Users (visitors) watch 3D videos displayed on a display in a museum or exhibition facility. The input is the 3D video played on the display, and the output is what the visitor watches. Specifically, the visitor presses the play button or waits for automatic playback.

[0186] (Application example 1)

[0187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] In traditional factories and manufacturing sites, training new employees and promoting corporate culture required a great deal of time and effort. Furthermore, lectures and explanations to motivate employees had to be delivered in person, which was inefficient. In addition, current methods for recreating historical figures and corporate heroes often lacked a sense of vitality and diminished impact. Given these circumstances, there was a need for an efficient and effective way to train new employees, promote corporate culture, and increase employee motivation.

[0189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0190] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating a 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal and playing it on the terminal, means for installing the generated 3D video and audio data on a robot, and means for the robot to play the 3D video and audio data and perform actions specified by the user. This allows for efficient training of new employees and dissemination of corporate culture, enables the realistic reproduction of historical figures and great figures, and increases employee motivation. Furthermore, automated explanations and guidance by the robot reduce human labor and realize an efficient training process.

[0191] A "user" is a person who operates the system and is responsible for inputting image data and basic information.

[0192] "Image data" refers to still image files uploaded by users, and is the basis for recreating historical figures and great people.

[0193] "Basic information" refers to basic information such as the target person's name, date of birth, achievements, etc., and is data used when generating voice data and animation scripts.

[0194] An "image recognition engine" is a software tool for detecting feature points from uploaded image data.

[0195] "Feature points" are important points detected from image data by an image recognition engine and are used to generate 3D models.

[0196] A "3D model" is a three-dimensional digital model generated based on detected feature points to recreate the appearance of a person.

[0197] "Audio data" is an audio file generated based on basic information about the target person, and is used as part of the animation script.

[0198] An "animation script" is a set of motion instructions generated based on 3D models and audio data, and serves as the basis for generating 3D videos.

[0199] "3D video" refers to a video file generated by combining a 3D model with audio data and an animation script, and includes footage of a subject speaking and moving.

[0200] "Terminal" refers to a device operated by a user, which is used to receive and play 3D video sent from the server.

[0201] A "robot" is an autonomous mechanical device that can be installed with 3D video and audio data and can automatically move and play audio.

[0202] A "specified action" is an action or instruction that a robot performs based on instructions programmed by a user to perform a specific task.

[0203] This invention is a system for automating and streamlining new employee training and the dissemination of corporate culture in factories. The system consists of three main components: users, terminals, and servers. The roles of each component and the overall process are described in detail below.

[0204] System Overview

[0205] 1. User inputs image data and basic information

[0206] The user uses the terminal interface to upload image data (e.g., a JPEG file) and enter basic information (e.g., name, date of birth, achievements, etc.).

[0207] The terminal collects the received image data and basic information into packets and sends them to the server as an HTTP request.

[0208] 2. Data analysis and 3D modeling using a server

[0209] The server receives the HTTP request and analyzes the image data and basic information.

[0210] Specifically, an image recognition engine such as OpenCV is used to detect feature points from image data.

[0211] Then, a 3D model is generated based on the detected feature points using 3D modeling software such as Blender.

[0212] 3. Generate audio data and animation script

[0213] The server generates voice data based on the generated 3D model using a speech synthesis engine such as Google Text-to-Speech.

[0214] Furthermore, an animation script is generated based on the generated audio data and 3D model, and a 3D video is created.

[0215] 4. Data transmission and playback

[0216] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal.

[0217] The device decodes the received 3D video and stores it in local storage, preparing it for playback on displays in museums and exhibition facilities.

[0218] 5. Installing data into the robot and controlling its movements

[0219] The server installs the generated 3D video and audio data on the robot.

[0220] The robot performs specified actions based on the data it receives, automatically training new employees and promoting corporate culture within the factory.

[0221] Specific examples

[0222] For example, consider the case of using historical figures of company founders to train new employees.

[0223] The user uploads an image file of the founder (e.g., "founder.jpg") to the terminal and enters basic information such as the name "Founder A," date of birth "January 1, 1900," and achievement "starting and expanding the company."

[0224] The server uses OpenCV and other tools to detect feature points from the uploaded "founder.jpg" and generates a 3D model using Blender. It also uses Google Text-to-Speech to generate audio data saying, "I am Founder A. I contributed to the launch and expansion of the company."

[0225] The server generates an animation script based on the 3D model and audio data, creates a 3D video, encodes it, and sends it to the device.

[0226] The terminal receives the 3D video and prepares it for playback on a display in the factory.

[0227] The robot patrols the factory and gives instructions to new employees while playing installed 3D video and audio data.

[0228] Prompt Sentence Examples

[0229] Examples of prompts include:

[0230] "Users upload a historical image of 'Founder A' and enter basic information (name, achievements, date of birth). The system uses image recognition and 3D modeling technology to generate a 3D model of Founder A, and then uses voice synthesis to create audio data that explains the company's history and important events. Based on this data, a robot that patrols the factory will train new employees."

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] The user selects and uploads image data (e.g., JPEG files) of a historical figure using the upload interface on the device, and then fills in basic information such as name, date of birth, and achievements in an input form and clicks the submit button.

[0234] Input: Image data of the great person (JPEG file), basic information (name, date of birth, achievements)

[0235] Output: HTTP request containing image data and basic information

[0236] Specific actions: Select an image file of a great person and upload it in your browser. Enter basic information in the input fields and click the submit button.

[0237] Step 2:

[0238] The terminal collects the image data and basic information received from the user into packets and sends them to the server as an HTTP request.

[0239] Input: Image data and basic information sent by the user

[0240] Output: HTTP request to send to the server

[0241] Specific operation: Image data and basic information are converted into JSON format and an HTTP POST request is sent to the server.

[0242] Step 3:

[0243] The server receives the HTTP request, analyzes the image data and basic information, and uses an image recognition engine such as OpenCV to detect feature points from the image data.

[0244] Input: Image data and basic information received from the device

[0245] Output: Facial feature point data of great people

[0246] Specific operation: Analyzes the HTTP request and extracts feature points such as the nose, eyes, and mouth from the image data using OpenCV.

[0247] Step 4:

[0248] The server uses 3D modeling software such as Blender to generate a 3D model based on the detected feature points.

[0249] Input: feature point data

[0250] Output: 3D model data (.obj file, etc.)

[0251] Specific operation: Input feature point data into Blender and generate 3D models of the faces and bodies of great figures.

[0252] Step 5:

[0253] The server uses a speech synthesis engine such as Google Text-to-Speech to generate voice data based on the generated 3D model, and then generates an animation script based on the generated voice data and the 3D model to create a 3D video.

[0254] Input: 3D model data, basic information (name, date of birth, achievements)

[0255] Output: Audio data (.mp3 files, etc.), animation script, 3D video data (.mp4 files, etc.)

[0256] Specific operation: Generates voice data using a voice synthesis engine based on basic information. Generates an animation script based on the voice data and 3D model, and renders a 3D video.

[0257] Step 6:

[0258] The server encodes the generated 3D video into streaming and download formats and transmits it to the terminal.

[0259] Input: 3D video data

[0260] Output: Encoded 3D video file (MPEG format, etc.)

[0261] Specific operation: Encodes 3D video data and sends it to the device via a secure file transfer protocol (e.g., SFTP).

[0262] Step 7:

[0263] The device decodes the received 3D video, stores it in local storage, and then prepares it for playback on a display in the museum or exhibition facility.

[0264] Input: Encoded 3D video file received from the server

[0265] Output: Video data in a format that can be played on a display

[0266] Specific operation: The 3D video file is saved to local storage and prepared for playback on the display using the specified playback software.

[0267] Step 8:

[0268] The server installs the generated 3D video and audio data on the robot.

[0269] Input: 3D video file, audio data file

[0270] Output: Data to be installed on the robot

[0271] Specific operation: 3D video and audio data is transferred to the robot through the robot's software interface and installed.

[0272] Step 9:

[0273] The robot patrols the factory while playing back the received 3D video and audio data, performing designated tasks such as training new employees and promoting corporate culture.

[0274] Input: Installed 3D video and audio data

[0275] Output: Explanations and guides for new employee training and corporate culture promotion

[0276] Specific operation: The robot moves autonomously around the factory, playing videos at appropriate times to provide specified explanations and guidance.

[0277] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0278] The present invention is a system that uses a generation AI to create 3D videos based on image data and basic information of famous people provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[0279] System Overview

[0280] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and send image data and basic information about the great figure. The server analyzes the data and uses the emotion engine to generate a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0281] Program processing

[0282] 1. Upload photos and enter data

[0283] User

[0284] The user selects and uploads image data (e.g., JPEG files) of the great person using the interface on the terminal.

[0285] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[0286] Terminal

[0287] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[0288] The terminal sends this packet to the server as an HTTP request.

[0289] 2. Image analysis and 3D modeling

[0290] server

[0291] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0292] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position of the eyes, nose, mouth, and facial contours.

[0293] The server generates a 3D model based on the detected feature points, which is used to realistically recreate the specific appearance of the great person.

[0294] The server uses the basic information to generate voice data for the great person using a speech synthesis engine, such as "I am (name of great person). I have achieved (achievement)."

[0295] 3. Coordination of emotion recognition and video generation

[0296] server

[0297] The server uses an emotion engine to analyze the user's emotion data, which is obtained from input information, past user behavior history, etc.

[0298] The server adjusts the content of the 3D video it generates based on the emotion recognition results: for example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[0299] The server also changes the tone and content of the generated audio data based on the emotion, for example, it can include more instances of the word "thank you" if the emotion is strong.

[0300] The server executes the tailored animation script and simulates the movements on the 3D model, ultimately generating the 3D video.

[0301] 4. Sending and Playing Videos

[0302] server

[0303] The server encodes the generated 3D video into streaming and download formats.

[0304] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[0305] Terminal

[0306] The device decodes the 3D video file received from the server and saves it in local storage.

[0307] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[0308] User (visitor)

[0309] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[0310] Specific examples

[0311] For example, when generating a video of "Great Person A," the following specific example can be considered.

[0312] User

[0313] The user uploads an image file of "Great Person A" (e.g., "A_person.jpg") to the terminal.

[0314] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[0315] Terminal

[0316] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[0317] server

[0318] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg" and generates a 3D model using 3D modeling software.

[0319] The server generates voice data of great person A using a voice synthesis engine.

[0320] The server uses an emotion engine to analyze the user's emotional data and adjust the content of the video, for example, by adding a gentle, emotional tone to the audio if the user is emotional.

[0321] The server generates the animation script and creates the 3D video.

[0322] The server transmits the generated 3D video to the terminal.

[0323] Terminal

[0324] The device receives the 3D video and plays it on the museum's display.

[0325] User (visitor)

[0326] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[0327] The above is an embodiment of the present invention. This system allows visitors to museums and exhibition facilities to have a concrete and vivid image of great people, and also provides appropriate information according to their emotions.

[0328] The processing flow will be explained below.

[0329] Step 1:

[0330] User

[0331] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[0332] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[0333] Step 2:

[0334] Terminal

[0335] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[0336] The terminal sends this packet to the server as an HTTP request.

[0337] Step 3:

[0338] server

[0339] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0340] Step 4:

[0341] server

[0342] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[0343] Step 5:

[0344] server

[0345] The server uses 3D modeling software to generate a 3D model of the great person based on the detected feature points, which is intended to realistically recreate the specific appearance of the great person.

[0346] Step 6:

[0347] server

[0348] The server uses a speech synthesis engine based on the basic information to generate voice data for the great person, such as "I am (name of great person). I have accomplished (achievement)."

[0349] Step 7:

[0350] server

[0351] The server uses an emotion engine to analyze the user's emotion data, which is obtained from the input information and the user's past behavior history.

[0352] Step 8:

[0353] server

[0354] The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[0355] Step 9:

[0356] server

[0357] The server also adjusts the tone and content of the generated voice data based on the emotion, for example, by including more "thank you" words if the emotion is strong.

[0358] Step 10:

[0359] server

[0360] The server generates a tailored animation script and generates a 3D video by simulating the movements on the 3D model.

[0361] Step 11:

[0362] server

[0363] The server encodes the generated 3D video into streaming and download formats.

[0364] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[0365] Step 12:

[0366] Terminal

[0367] The device decodes the 3D video file received from the server and saves it in local storage.

[0368] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[0369] Step 13:

[0370] User (visitor)

[0371] Visitors to the museum watch 3D videos on the display, which either play automatically or can be started by the visitor pressing a play button.

[0372] Through these detailed processing steps, the system can provide a concrete and vivid image of the great person, and adjust the content according to the user's emotions, providing visitors with a more engaging exhibition experience.

[0373] Example 2

[0374] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0375] Previously, when generating 3D models or videos based on image data of great people or historical figures, it was difficult to provide optimal information to each individual user because the models or videos were not adjusted to reflect the user's emotions. Furthermore, no efficient method had been established for transmitting and playing the generated 3D models or videos. For this reason, there was a demand for a system that could interest visitors in museums and exhibition facilities and give them a vivid, concrete image.

[0376] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0377] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script, means for transmitting the generated 3D video to a terminal, and means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the results. This makes it possible to provide optimal information according to the user's emotions and to efficiently generate, transmit, and play 3D videos that will attract visitors' attention.

[0378] A "user" is a user who accesses the system and provides image data and basic information.

[0379] A "terminal" refers to a device operated by a user, and is a device that has an interface for performing data communication with a server.

[0380] "Server" means a central computer system for analyzing image data and basic information and generating 3D models and 3D animations.

[0381] "Image data" is data in a file format that contains images of great people or historical figures, and is mainly in JPEG or PNG format.

[0382] "Basic information" is text information about great people and historical figures, such as their names, birth dates, and achievements.

[0383] An "image recognition engine" is a software component for detecting feature points from image data.

[0384] "Feature points" are key points in image data (e.g., the positions of the eyes, nose, mouth, and facial contours).

[0385] A "three-dimensional model" is a digital model that recreates great people and historical figures in three dimensions.

[0386] "Audio data" refers to an audio file generated based on text information.

[0387] An "animation script" is a script file that contains instructions for adding movement to a three-dimensional model.

[0388] "3D video" is a video created by combining a generated 3D model, audio data, and an animation script.

[0389] "Emotion data" is information that indicates the user's emotional state, and is extracted from text input, past behavior history, and the like.

[0390] An "emotion recognition engine" is a software component that analyzes a user's emotional data and outputs the results.

[0391] "Streaming format" is a data transfer format in which data is played back while being continuously delivered.

[0392] "Download format" is a data transfer format in which data is played after being completely acquired.

[0393] The "playback means" refers to a device or software that has the function of displaying the received three-dimensional video on a display device.

[0394] The present invention is a system that uses a generative AI to create 3D videos based on image data and basic information of great people and historical figures provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[0395] System Overview

[0396] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and transmit image data and basic information about the great figure. The server analyzes the transmitted information and utilizes the emotion engine in the process of generating a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0397] Specific explanation of program processing

[0398] The user uses the device interface to select and upload image data (e.g., a JPEG file) of the great person, then fills in basic information about the great person, such as their name, date of birth, and achievements, in an input form and clicks the submit button.

[0399] The device receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and sends them to the server as an HTTP request. At this time, the data is encrypted using SSL / TLS to prevent tampering.

[0400] The server receives the HTTP request and extracts the image data and basic information from the packet. The server then analyzes the image data using an image recognition engine (for example, OpenCV or TensorFlow) to detect the facial features of the great person (such as the position of the eyes, nose, and mouth, as well as the facial contours). Based on the detected features, the server uses the API of 3D modeling software (for example, Blender) to generate a three-dimensional model. This three-dimensional model is used to realistically reproduce the specific appearance of the great person.

[0401] Next, the server uses a speech synthesis engine (for example, the Google Text-to-Speech API) to generate voice data based on the basic information. The generated voice data will say something like, "I am (name of great person). I have achieved (achievement)."

[0402] Furthermore, the server uses an emotion engine (specifically, IBM Watson Tone Analyzer) to analyze the user's emotional data. This emotional data is obtained from input information and past user behavior history. The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone. It also adjusts the tone of the audio data, so that if the user is expressing strong feelings of gratitude, it can include more instances of the word "thank you."

[0403] By executing the adjusted animation script, the server finally generates the 3D video, which is then encoded into streaming and download formats and sent to the terminal using a secure file transfer protocol (e.g., SFTP).

[0404] The terminal decodes the 3D video file received from the server and saves it in local storage. The terminal then sends the 3D video to the display system of the museum or exhibition facility and prepares it for playback. Visitors to the museum watch the 3D video displayed on the display. Playback is either automatic or initiated by the visitor pressing a play button. This allows visitors to form a concrete and vivid image of the great man and gain a deeper understanding of the exhibits.

[0405] Example prompt

[0406] Here is an example of how to input the following prompt sentence into a generative AI model:

[0407] Generate a 3D model and video of a great person using the image "A_person.jpg" and basic information "Name: Great Person A, Date of Birth: January 1, 1900, Achievement: Contributed to a historical event." Also, if the user's emotion is "emotional," adjust the audio tone and video tempo accordingly.

[0408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0409] Step 1:

[0410] The user selects and uploads image data (e.g., JPEG files) of a great person using a dedicated interface on the device. As input, the user provides the image data and basic information (such as the great person's name, date of birth, and achievements) and clicks the send button. As output, the device receives the image data and basic information.

[0411] Step 2:

[0412] The terminal receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and converts them into an HTTP request. The input image data and basic information are encrypted using SSL / TLS and sent to the server. As output, the data is converted into a format that can be sent to the server.

[0413] Step 3:

[0414] The server receives the HTTP request and extracts the image data and basic information from the packet. As input, the encrypted HTTP request is sent to the server, and as output, the decrypted image data and basic information are obtained.

[0415] Step 4:

[0416] The server analyzes the image data using an image recognition engine (e.g., OpenCV or TensorFlow). The image data is provided as input, and the facial feature points of the great figure (e.g., the positions of the eyes, nose, and mouth, and the facial contours) are detected as output. The server generates a three-dimensional model using 3D modeling software (e.g., Blender) based on these feature points.

[0417] Step 5:

[0418] The server uses a speech synthesis engine (e.g., Google Text-to-Speech API) based on the basic information to generate voice data. The basic information (e.g., name, date of birth, achievement) is used as input, and voice data such as "I am (name of great person). I have achieved (achievement)" is generated as output.

[0419] Step 6:

[0420] The server analyzes the user's emotional data using an emotion engine (e.g., IBM Watson Tone Analyzer). The user's emotional data (e.g., excitement, emotion, joy) is provided as input, and the emotion analysis results are obtained as output. The content and tone of the generated 3D video are adjusted based on the emotion recognition results. For example, if the user is excited, the tempo of the video is made faster and the tone is made more energetic.

[0421] Step 7:

[0422] The server executes the adjusted animation script and simulates the movement of the 3D model to ultimately generate a 3D video. The 3D model, audio data, and animation script are provided as input, and the adjusted 3D video is generated as output.

[0423] Step 8:

[0424] The server encodes the generated 3D video into a streaming format and a download format and transmits it to the terminal using a secure file transfer protocol (e.g., SFTP). The generated 3D video is provided as input, and the encoded 3D video is transmitted to the terminal as output.

[0425] Step 9:

[0426] The terminal decodes the 3D video file received from the server and stores it in local storage. The encoded 3D video is provided as input, and the decoded 3D video is stored in the terminal as output.

[0427] Step 10:

[0428] The terminal transmits the 3D video to the display system of the museum or exhibition facility and prepares it for playback. As input, the 3D video stored in local storage is provided, and as output, preparation for playback on the display system is completed. Visitors to the museum can watch the 3D video displayed on the display.

[0429] (Application example 2)

[0430] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] Conventional food delivery applications offer a uniform user experience and lack personalized interaction. Furthermore, the automated guidance provided on order details and delivery status makes it difficult to provide users with a sense of familiarity or enjoyment. Furthermore, the information provided is not tailored to the user's emotions, which can lead to a decline in overall satisfaction with the service.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the analysis results, means for receiving image data and basic information uploaded by the user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, and means for transmitting the generated 3D video to the terminal. This makes it possible to provide friendly and entertaining interactions that correspond to the user's emotions and realize a personalized food delivery service.

[0433] A "user" is a person who uses the system to provide image data and basic information.

[0434] "Uploaded image data" refers to image files sent from a user to a server, specifically in formats such as JPEG or PNG.

[0435] "Basic information" refers to information such as name, date of birth, position, and achievements associated with image data.

[0436] An "image recognition engine" is a software component for analyzing and detecting features from image data.

[0437] "Feature points" are important points such as the position of the eyes, nose, mouth, and facial contours that are extracted from image data of great people.

[0438] A "3D model" is a three-dimensional virtual human image generated based on extracted feature points.

[0439] "Audio data" refers to audio files synthesized based on the generated 3D model, including audio related to basic information.

[0440] An "animation script" is a set of instructions that controls the movement of a 3D model, enabling it to animate.

[0441] "3D video" refers to three-dimensional moving images generated by combining audio data and animation scripts.

[0442] A "terminal" is a device used by a user to operate the device, and includes smartphones, personal computers, etc.

[0443] "Emotion data" refers to emotional information extracted from the user's facial expressions, voice, etc.

[0444] An "emotion engine" is a software component that analyzes a user's emotional data and outputs the results.

[0445] "Tone" refers to the atmosphere and nuances of the audio and animation in a video.

[0446] The embodiment for carrying out the present invention is a system that is composed of four main components: a user, a terminal, a server, and an emotion engine. Specific embodiments of the system of the present invention will be described below.

[0447] System Overview

[0448] The user operates the device to input image data and basic information about the great person, and sends it to the server. The server analyzes the data and generates a 3D model and 3D video. During this process, an emotion engine is used to adjust the video according to the user's emotions. The generated 3D video is sent to the device and provided as a food delivery application.

[0449] 1. Upload photos and enter data

[0450] The user selects and uploads image data (e.g., JPEG files) of the famous person using the interface on the device. They also enter basic information about the famous person, such as their name, date of birth, and achievements, into an input form and send it to the server. The device receives the uploaded image data and the basic information, and sends it to the server.

[0451] 2. Image analysis and 3D modeling

[0452] The server receives the transmitted data and uses an image recognition engine (e.g., OpenCV) to detect the facial features of the great person from the image data. Based on the detected features, it uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person. Furthermore, it uses a speech synthesis engine (e.g., Amazon Polly) based on the basic information to generate voice data.

[0453] 3. Coordination of emotion recognition and video generation

[0454] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotional data and adjusts the content of the 3D video to be generated based on the results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. Based on the adjusted animation script, it simulates movement on the 3D model and finally generates the 3D video.

[0455] 4. Sending and Playing Videos

[0456] The server encodes the generated 3D video into streaming and download formats and sends it to the device. The device decodes the received 3D video file, saves it in local storage, and provides it to the user as a food delivery application. Through the application, the user can receive information about their order and delivery status while having a concrete image of the great man.

[0457] Specific examples

[0458] For example, if a user provides an image file and basic information about "Historical Figure A," the system operates as follows: The user uploads the image file of "Historical Figure A" to their device and enters the user's name ("Historical Figure A"), date of birth ("January 1, 1900"), and achievement ("Contributed to historical events") as basic information. The server receives this information and uses an image recognition engine to detect feature points from the image file and generate a 3D model. Next, it uses a speech synthesis engine to generate voice data and an emotion engine to analyze the user's emotion data. Based on these results, it adjusts the content and tone of the video, and finally generates a 3D video. The server sends the generated 3D video to the device, where the user can watch it through a food delivery application.

[0459] Example prompts for generative AI models

[0460] Name of a famous person: Historical figure A

[0461] Basic information: A patriot from the end of the Edo period, who contributed to historical events

[0462] Characteristics: Excellent negotiation and leadership skills

[0463] Expected output:

[0464] Hello, I'm Historical Figure A. As a patriot in the late Edo period, I contributed to historical events. Today, I'm here to recommend some delicious dishes. Have you ordered yet?

[0465] In this way, the present invention provides personalized interactions that respond to the user's emotions, realizing a food delivery service that is both entertaining and convenient.

[0466] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0467] Step 1:

[0468] The user uses the interface on the terminal to select and upload image data (e.g., JPEG files) of the famous person. They also fill in basic information about the famous person, such as their name, date of birth, and achievements, in an input form and click the submit button. The input data includes the image file and basic information in text format. The output is a packet containing this data.

[0469] Step 2:

[0470] The terminal receives the uploaded image data and the input basic information, and assembles this data into packets. The assembled packets are sent to the server as HTTP requests. The input is the image data and basic information from the user, and the output is a data packet in HTTP request format.

[0471] Step 3:

[0472] The server receives the HTTP request and extracts the image data and basic information from the packet. Specifically, the server analyzes the request and separates each piece of data while checking the integrity of the data. The input is a data packet in the HTTP request format, and the output is individual pieces of data: image data and basic information.

[0473] Step 4:

[0474] The server analyzes the image data using an image recognition engine (e.g., OpenCV) to detect the facial feature points of the great man. The detected feature points include the positions of the eyes, nose, and mouth, as well as the facial contours. The input is the image data obtained from the user, and the output is the coordinate data of the detected feature points.

[0475] Step 5:

[0476] The server generates a 3D model based on the detected feature points. This process uses 3D modeling software (e.g., Blender). The input is the coordinate data of the feature points, and the output is a 3D model that reproduces the specific appearance of the great figure.

[0477] Step 6:

[0478] The server uses the basic information to generate voice data of the great person using a speech synthesis engine (e.g., Amazon Polly). For example, the speech synthesis uses content such as "I am (name of great person). I have achieved (achievement)." The input is the basic information in text format, and the output is voice data.

[0479] Step 7:

[0480] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotion data. The emotion data is obtained from the user's past behavior history and interaction data. The input is the user's behavior history data and current interaction data, and the output is the emotion analysis result.

[0481] Step 8:

[0482] The server adjusts the content and tone of the generated 3D video based on the emotion recognition results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. The inputs are the emotion analysis results, 3D models, audio data, and animation scripts, and the output is the adjusted 3D video.

[0483] Step 9:

[0484] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal using a secure file transfer protocol. The input is the calibrated 3D video, and the output is the encoded 3D video file.

[0485] Step 10:

[0486] The device decodes the 3D video file received from the server, stores it in local storage, and provides it to the user as a food delivery application. The input is an encoded 3D video file, and the output is a viewable 3D video.

[0487] Step 11:

[0488] Through the application, users can watch 3D videos of famous people and receive information about their order and delivery status while getting a concrete image of the person. The input is the image data and basic information of the famous person provided by the user, and the 3D video sent from the server, and the output is an entertainment-rich food delivery service experience.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0492] [Second embodiment]

[0493] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0494] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0496] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0500] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0503] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0504] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0505] This invention is a system that uses generative AI to create 3D videos based on photographs and basic information of famous people, and broadcasts these videos in museums and exhibition facilities. Below, we will explain the program processing of the system of this invention in natural language.

[0506] System Overview

[0507] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and transmit image data and basic information about the great figure. The server analyzes the transmitted data and generates a 3D model and 3D video. The terminal then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0508] Program processing

[0509] 1. Upload photos and enter data

[0510] User

[0511] The user uses the upload interface on the terminal to select and upload image data (e.g., a JPEG file) of the great figure.

[0512] The user fills in basic information such as name, date of birth, achievements, etc. in the input form and clicks the submit button.

[0513] Terminal

[0514] The terminal receives the uploaded image data and the input basic information and assembles them into a packet format.

[0515] The terminal sends this packet to the server as an HTTP request.

[0516] 2. Image analysis and 3D modeling

[0517] server

[0518] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0519] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position and contours of his nose, eyes, and mouth.

[0520] The server uses 3D modeling software to generate a 3D model of the great person based on the detected features, which is intended to realistically recreate the great person's specific appearance.

[0521] The server also uses the basic information about the great person to generate voice data for the great person (for example, a greeting such as "I am [great person's name]") using a speech synthesis engine.

[0522] 3. 3D video generation

[0523] server

[0524] The server generates an animation script based on the generated 3D model and voice data. This script contains instructions for controlling the speaking actions and facial expressions of the great figure.

[0525] The server executes the animation script and generates 3D animation by simulating movements on the 3D model.

[0526] The server encodes the generated 3D video into multiple formats (e.g., streaming format and download format).

[0527] 4. Sending and Playing Videos

[0528] server

[0529] The server sends the encoded 3D video file to the device using a secure file transfer protocol.

[0530] Terminal

[0531] The device decodes the received 3D video file and saves it in local storage.

[0532] The device will connect to the display system of the museum or exhibition facility and prepare to play 3D videos.

[0533] User (visitor)

[0534] Visitors to the museum can watch 3D videos on the display, either automatically or by pressing a play button.

[0535] Specific examples

[0536] For example, when generating a video of "Great Person A," a great historical figure in Japan, the following specific example can be considered.

[0537] User

[0538] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the terminal.

[0539] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[0540] Terminal

[0541] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[0542] server

[0543] The server uses an image recognition engine to detect the facial features of Great Person A from "A_person.jpg."

[0544] The server uses 3D modeling software to generate a 3D model of Great Person A based on the detected feature points.

[0545] The server also uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event."

[0546] The server generates an animation script based on the 3D model and audio data, and generates a 3D video of great person A speaking.

[0547] The server transmits the generated 3D video to the terminal.

[0548] Terminal

[0549] The device receives the 3D video and plays it on the museum's display.

[0550] User (visitor)

[0551] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[0552] The above is an embodiment of the present invention. This system makes it possible to give a sense of liveliness to exhibits in museums and exhibition facilities, and to provide visitors with a more concrete image.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] User

[0556] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[0557] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[0558] Step 2:

[0559] Terminal

[0560] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[0561] The terminal sends the collected packets to the server as an HTTP request.

[0562] Step 3:

[0563] server

[0564] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0565] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[0566] Step 4:

[0567] server

[0568] Based on the detected facial features, the server uses 3D modeling software to generate a 3D model of the great figure, which realistically reproduces the shape and detailed features of the face.

[0569] The server uses a speech synthesis engine to generate voice data for the great person based on the basic information about the great person, such as "I am (great person's name). I have accomplished (achievement)."

[0570] Step 5:

[0571] server

[0572] The server combines the generated 3D model with the audio data to generate an animation script, which includes the movements and facial expressions of the great man as he speaks.

[0573] The server executes the generated animation script and generates a 3D video by simulating the movements of the great man.

[0574] Step 6:

[0575] server

[0576] The server encodes the generated 3D video into streaming and download formats.

[0577] The server then sends the encoded 3D video to the device, using a file transfer protocol to ensure security.

[0578] Step 7:

[0579] Terminal

[0580] The device decodes the 3D video file received from the server and saves it in local storage.

[0581] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[0582] Step 8:

[0583] User (visitor)

[0584] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[0585] Through the above processing steps, the system provides visitors with a concrete and vivid image of the great man, enabling a more engaging exhibition experience.

[0586] Example 1

[0587] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] Traditional exhibition facilities and museums often display the achievements of great people and famous figures and historical events using only static images and text, making it difficult to provide visitors with a sense of dynamism and realism. Furthermore, paper media and static displays limit the amount of information visitors receive, making them less appealing to younger visitors in particular. To address this issue, more dynamic and interactive exhibition methods are needed.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0590] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal, means for the terminal to decode the received 3D video and save it in local storage, and means for the terminal to play the 3D video in cooperation with a display system of a museum or exhibition facility. This allows visitors to get a concrete image of historical figures and famous people through the 3D video of the people.

[0591] "Image data" is digital data in the form of still images that is uploaded by the user.

[0592] "Basic information" is text data such as the name, date of birth, and achievements of a great or famous person.

[0593] An "image recognition engine" is software or an algorithm for detecting feature points from image data.

[0594] "Feature points" are important points that refer to the positions and contours of facial features such as the nose, eyes, and mouth that are detected from image data.

[0595] A "3D model" is a three-dimensional computer graphic representation of a great or famous person, generated based on detected feature points.

[0596] "Voice data" refers to a voice file generated by a voice synthesis engine based on basic information.

[0597] An "animation script" is a program containing a series of instructions created to control the movements and expressions of a 3D model.

[0598] A "3D video" is a video file generated by executing an animation script based on a 3D model and audio data.

[0599] "Terminal" refers to a device operated by a user that receives, stores, and plays data from a server.

[0600] "Display system" refers to the device or configuration installed in a museum or exhibition facility that displays 3D videos.

[0601] This invention is a system that uses generative AI to create 3D videos based on image data and basic information of great people and famous figures, and then broadcasts these videos in museums and exhibition facilities.

[0602] System configuration

[0603] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and send image data and basic information about the great figure. The server analyzes the data and generates a 3D model and 3D video. The terminal receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0604] Photo upload and data entry

[0605] Using a web interface on the device, users select and upload image data (e.g., a JPEG file) of a great person, fill out a form with basic information such as name, date of birth, and achievements, and click the submit button.

[0606] The terminal receives the input, assembles it into a single data packet, which contains both image and text data, and sends the packet to the server as an HTTP request.

[0607] Image Analysis and 3D Modeling

[0608] The server receives the HTTP request and parses the data packet to extract the image data and basic information. The server then uses an image recognition engine (e.g., OpenCV) to detect facial features from the image data, including the location and contours of the nose, eyes, and mouth.

[0609] The server then uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person based on the detected feature points. This model realistically reproduces the great person's appearance. The server also uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the great person's basic information. For example, it generates an audio file such as, "I am the name of a great person. My achievements are..."

[0610] 3D video generation

[0611] The server creates an animation script based on the generated 3D model and audio data. The script contains instructions to control the figure's speaking movements and facial expressions. The server executes the animation script and generates 3D video by simulating the 3D model's movements. The server then encodes the generated 3D video into multiple formats (e.g., MP4, WebM).

[0612] Video transmission and playback

[0613] The server sends the generated 3D video file to the device using a secure file transfer protocol (e.g., SFTP).

[0614] The device decodes the received 3D video file and saves it in local storage. The device then connects to the display system of the museum or exhibition facility and prepares the 3D video for playback, for example, using a media player connected to the display system.

[0615] Users (visitors) visit the museum and watch 3D videos on the exhibited display. Playback is either automatic or initiated by the visitor pressing a play button.

[0616] Specific examples

[0617] For example, consider the case of generating a video of "Great Person A," a great historical figure in Japan.

[0618] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the device, and then enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to a historical event."

[0619] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[0620] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg." It then uses 3D modeling software to generate a 3D model of great person A based on the detected features. It then uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event." The server then generates an animation script based on the 3D model and voice data, generating a 3D video of great person A speaking. The 3D video is then sent to the device.

[0621] The device receives the 3D video and plays it on the museum display. Visitors can watch the 3D video of great person A on the museum display, giving them a stronger, more concrete image of him.

[0622] The above is an embodiment of the present invention. This system allows visitors to deepen their understanding of the exhibits at museums and exhibition facilities and provides a lively experience.

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] The user opens the web interface of the device, selects and uploads an image file of a great person or famous person (e.g., "A_person.jpg"). The selected image file is input, and basic information (such as name, date of birth, and achievements) is entered in the text box. By clicking the send button together with these, the input data is sent to the device. The output is the data received by the device.

[0626] Step 2:

[0627] The terminal combines the image file and basic information received from the user into a single data packet. This data packet contains image data and text data. The terminal then sends this data packet to the server as an HTTP request. It receives data from the user as input, creates and sends out a data packet, and the output is sent to the server.

[0628] Step 3:

[0629] The server analyzes the HTTP request received from the terminal and extracts image data and basic information from the data packet. Based on the analysis results of the HTTP request, the extracted image data and text data are passed to the next process. The output is the structured analysis of the data.

[0630] Step 4:

[0631] The server uses an image recognition engine (e.g., OpenCV) to detect feature points (e.g., the position and contours of the nose, eyes, and mouth) from the received image data. The input is image data, and the output is data on the detected feature points. Specifically, it runs a face detection algorithm.

[0632] Step 5:

[0633] The server generates a 3D model of the great person using 3D modeling software (e.g. Blender) based on the detected feature points. The input is the feature point data, and the output is the 3D model. Specifically, the server uses Blender's API to build the 3D model.

[0634] Step 6:

[0635] The server uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the basic information about the great person. The input is text data, and the output is an audio file. Specifically, the server passes the basic information as text to the speech synthesis engine.

[0636] Step 7:

[0637] The server creates an animation script based on the generated 3D model and audio data. The input is the 3D model and audio data, and the output is the animation script. In terms of specific operations, the script constitutes instructions to control the movements and facial expressions of the 3D model.

[0638] Step 8:

[0639] The server executes the animation script and simulates the movement of the 3D model to generate a 3D video. The input is the animation script and the 3D model, and the output is a 3D video file. Specifically, the server performs the animation rendering process.

[0640] Step 9:

[0641] The server encodes the generated 3D video file into multiple formats (e.g., MP4, WebM). The input is the 3D video file, and the output is the encoded video format. This is done using video encoding software.

[0642] Step 10:

[0643] The server sends the encoded 3D video file to the device using a secure file transfer protocol (e.g., SFTP). The input is the encoded video file, and the output is the transmission to the device.

[0644] Step 11:

[0645] The device decodes the received 3D video file and saves it to local storage. The input is the video file from the server, and the output is saving to local storage. Specifically, the video file is saved in a specified folder.

[0646] Step 12:

[0647] The terminal works in conjunction with the display system of the museum or exhibition facility to prepare for 3D video playback. The input is a stored 3D video file, and the output is playback on the display system. Specifically, it launches the media player and prepares to start playback.

[0648] Step 13:

[0649] Users (visitors) watch 3D videos displayed on a display in a museum or exhibition facility. The input is the 3D video played on the display, and the output is what the visitor watches. Specifically, the visitor presses the play button or waits for automatic playback.

[0650] (Application example 1)

[0651] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0652] In traditional factories and manufacturing sites, training new employees and promoting corporate culture required a great deal of time and effort. Furthermore, lectures and explanations to motivate employees had to be delivered in person, which was inefficient. In addition, current methods for recreating historical figures and corporate heroes often lacked a sense of vitality and diminished impact. Given these circumstances, there was a need for an efficient and effective way to train new employees, promote corporate culture, and increase employee motivation.

[0653] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0654] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating a 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal and playing it on the terminal, means for installing the generated 3D video and audio data on a robot, and means for the robot to play the 3D video and audio data and perform actions specified by the user. This allows for efficient training of new employees and dissemination of corporate culture, enables the realistic reproduction of historical figures and great figures, and increases employee motivation. Furthermore, automated explanations and guidance by the robot reduce human labor and realize an efficient training process.

[0655] A "user" is a person who operates the system and is responsible for inputting image data and basic information.

[0656] "Image data" refers to still image files uploaded by users, and is the basis for recreating historical figures and great people.

[0657] "Basic information" refers to basic information such as the target person's name, date of birth, achievements, etc., and is data used when generating voice data and animation scripts.

[0658] An "image recognition engine" is a software tool for detecting feature points from uploaded image data.

[0659] "Feature points" are important points detected from image data by an image recognition engine and are used to generate 3D models.

[0660] A "3D model" is a three-dimensional digital model generated based on detected feature points to recreate the appearance of a person.

[0661] "Audio data" is an audio file generated based on basic information about the target person, and is used as part of the animation script.

[0662] An "animation script" is a set of motion instructions generated based on 3D models and audio data, and serves as the basis for generating 3D videos.

[0663] "3D video" refers to a video file generated by combining a 3D model with audio data and an animation script, and includes footage of a subject speaking and moving.

[0664] "Terminal" refers to a device operated by a user, which is used to receive and play 3D video sent from the server.

[0665] A "robot" is an autonomous mechanical device that can be installed with 3D video and audio data and can automatically move and play audio.

[0666] A "specified action" is an action or instruction that a robot performs based on instructions programmed by a user to perform a specific task.

[0667] This invention is a system for automating and streamlining new employee training and the dissemination of corporate culture in factories. The system consists of three main components: users, terminals, and servers. The roles of each component and the overall process are described in detail below.

[0668] System Overview

[0669] 1. User inputs image data and basic information

[0670] The user uses the terminal interface to upload image data (e.g., a JPEG file) and enter basic information (e.g., name, date of birth, achievements, etc.).

[0671] The terminal collects the received image data and basic information into packets and sends them to the server as an HTTP request.

[0672] 2. Data analysis and 3D modeling using a server

[0673] The server receives the HTTP request and analyzes the image data and basic information.

[0674] Specifically, an image recognition engine such as OpenCV is used to detect feature points from image data.

[0675] Then, a 3D model is generated based on the detected feature points using 3D modeling software such as Blender.

[0676] 3. Generate audio data and animation script

[0677] The server generates voice data based on the generated 3D model using a speech synthesis engine such as Google Text-to-Speech.

[0678] Furthermore, an animation script is generated based on the generated audio data and 3D model, and a 3D video is created.

[0679] 4. Data transmission and playback

[0680] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal.

[0681] The device decodes the received 3D video and stores it in local storage, preparing it for playback on displays in museums and exhibition facilities.

[0682] 5. Installing data into the robot and controlling its movements

[0683] The server installs the generated 3D video and audio data on the robot.

[0684] The robot performs specified actions based on the data it receives, automatically training new employees and promoting corporate culture within the factory.

[0685] Specific examples

[0686] For example, consider the case of using historical figures of company founders to train new employees.

[0687] The user uploads an image file of the founder (e.g., "founder.jpg") to the terminal and enters basic information such as the name "Founder A," date of birth "January 1, 1900," and achievement "starting and expanding the company."

[0688] The server uses OpenCV and other tools to detect feature points from the uploaded "founder.jpg" and generates a 3D model using Blender. It also uses Google Text-to-Speech to generate audio data saying, "I am Founder A. I contributed to the launch and expansion of the company."

[0689] The server generates an animation script based on the 3D model and audio data, creates a 3D video, encodes it, and sends it to the device.

[0690] The terminal receives the 3D video and prepares it for playback on a display in the factory.

[0691] The robot patrols the factory and gives instructions to new employees while playing installed 3D video and audio data.

[0692] Prompt Sentence Examples

[0693] Examples of prompts include:

[0694] "Users upload a historical image of 'Founder A' and enter basic information (name, achievements, date of birth). The system uses image recognition and 3D modeling technology to generate a 3D model of Founder A, and then uses voice synthesis to create audio data that explains the company's history and important events. Based on this data, a robot that patrols the factory will train new employees."

[0695] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0696] Step 1:

[0697] The user selects and uploads image data (e.g., JPEG files) of a historical figure using the upload interface on the device, and then fills in basic information such as name, date of birth, and achievements in an input form and clicks the submit button.

[0698] Input: Image data of the great person (JPEG file), basic information (name, date of birth, achievements)

[0699] Output: HTTP request containing image data and basic information

[0700] Specific actions: Select an image file of a great person and upload it in your browser. Enter basic information in the input fields and click the submit button.

[0701] Step 2:

[0702] The terminal collects the image data and basic information received from the user into packets and sends them to the server as an HTTP request.

[0703] Input: Image data and basic information sent by the user

[0704] Output: HTTP request to send to the server

[0705] Specific operation: Image data and basic information are converted into JSON format and an HTTP POST request is sent to the server.

[0706] Step 3:

[0707] The server receives the HTTP request, analyzes the image data and basic information, and uses an image recognition engine such as OpenCV to detect feature points from the image data.

[0708] Input: Image data and basic information received from the device

[0709] Output: Facial feature point data of great people

[0710] Specific operation: Analyzes the HTTP request and extracts feature points such as the nose, eyes, and mouth from the image data using OpenCV.

[0711] Step 4:

[0712] The server uses 3D modeling software such as Blender to generate a 3D model based on the detected feature points.

[0713] Input: feature point data

[0714] Output: 3D model data (.obj file, etc.)

[0715] Specific operation: Input feature point data into Blender and generate 3D models of the faces and bodies of great figures.

[0716] Step 5:

[0717] The server uses a speech synthesis engine such as Google Text-to-Speech to generate voice data based on the generated 3D model, and then generates an animation script based on the generated voice data and the 3D model to create a 3D video.

[0718] Input: 3D model data, basic information (name, date of birth, achievements)

[0719] Output: Audio data (.mp3 files, etc.), animation script, 3D video data (.mp4 files, etc.)

[0720] Specific operation: Generates voice data using a voice synthesis engine based on basic information. Generates an animation script based on the voice data and 3D model, and renders a 3D video.

[0721] Step 6:

[0722] The server encodes the generated 3D video into streaming and download formats and transmits it to the terminal.

[0723] Input: 3D video data

[0724] Output: Encoded 3D video file (MPEG format, etc.)

[0725] Specific operation: Encodes 3D video data and sends it to the device via a secure file transfer protocol (e.g., SFTP).

[0726] Step 7:

[0727] The device decodes the received 3D video, stores it in local storage, and then prepares it for playback on a display in the museum or exhibition facility.

[0728] Input: Encoded 3D video file received from the server

[0729] Output: Video data in a format that can be played on a display

[0730] Specific operation: The 3D video file is saved to local storage and prepared for playback on the display using the specified playback software.

[0731] Step 8:

[0732] The server installs the generated 3D video and audio data on the robot.

[0733] Input: 3D video file, audio data file

[0734] Output: Data to be installed on the robot

[0735] Specific operation: 3D video and audio data is transferred to the robot through the robot's software interface and installed.

[0736] Step 9:

[0737] The robot patrols the factory while playing back the received 3D video and audio data, performing designated tasks such as training new employees and promoting corporate culture.

[0738] Input: Installed 3D video and audio data

[0739] Output: Explanations and guides for new employee training and corporate culture promotion

[0740] Specific operation: The robot moves autonomously around the factory, playing videos at appropriate times to provide specified explanations and guidance.

[0741] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0742] The present invention is a system that uses a generation AI to create 3D videos based on image data and basic information of famous people provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[0743] System Overview

[0744] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and send image data and basic information about the great figure. The server analyzes the data and uses the emotion engine to generate a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0745] Program processing

[0746] 1. Upload photos and enter data

[0747] User

[0748] The user selects and uploads image data (e.g., JPEG files) of the great person using the interface on the terminal.

[0749] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[0750] Terminal

[0751] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[0752] The terminal sends this packet to the server as an HTTP request.

[0753] 2. Image analysis and 3D modeling

[0754] server

[0755] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0756] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position of the eyes, nose, mouth, and facial contours.

[0757] The server generates a 3D model based on the detected feature points, which is used to realistically recreate the specific appearance of the great person.

[0758] The server uses the basic information to generate voice data for the great person using a speech synthesis engine, such as "I am (name of great person). I have achieved (achievement)."

[0759] 3. Coordination of emotion recognition and video generation

[0760] server

[0761] The server uses an emotion engine to analyze the user's emotion data, which is obtained from input information, past user behavior history, etc.

[0762] The server adjusts the content of the 3D video it generates based on the emotion recognition results: for example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[0763] The server also changes the tone and content of the generated audio data based on the emotion, for example, it can include more instances of the word "thank you" if the emotion is strong.

[0764] The server executes the tailored animation script and simulates the movements on the 3D model, ultimately generating the 3D video.

[0765] 4. Sending and Playing Videos

[0766] server

[0767] The server encodes the generated 3D video into streaming and download formats.

[0768] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[0769] Terminal

[0770] The device decodes the 3D video file received from the server and saves it in local storage.

[0771] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[0772] User (visitor)

[0773] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[0774] Specific examples

[0775] For example, when generating a video of "Great Person A," the following specific example can be considered.

[0776] User

[0777] The user uploads an image file of "Great Person A" (e.g., "A_person.jpg") to the terminal.

[0778] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[0779] Terminal

[0780] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[0781] server

[0782] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg" and generates a 3D model using 3D modeling software.

[0783] The server generates voice data of great person A using a voice synthesis engine.

[0784] The server uses an emotion engine to analyze the user's emotional data and adjust the content of the video, for example, by adding a gentle, emotional tone to the audio if the user is emotional.

[0785] The server generates the animation script and creates the 3D video.

[0786] The server transmits the generated 3D video to the terminal.

[0787] Terminal

[0788] The device receives the 3D video and plays it on the museum's display.

[0789] User (visitor)

[0790] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[0791] The above is an embodiment of the present invention. This system allows visitors to museums and exhibition facilities to have a concrete and vivid image of great people, and also provides appropriate information according to their emotions.

[0792] The processing flow will be explained below.

[0793] Step 1:

[0794] User

[0795] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[0796] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[0797] Step 2:

[0798] Terminal

[0799] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[0800] The terminal sends this packet to the server as an HTTP request.

[0801] Step 3:

[0802] server

[0803] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0804] Step 4:

[0805] server

[0806] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[0807] Step 5:

[0808] server

[0809] The server uses 3D modeling software to generate a 3D model of the great person based on the detected feature points, which is intended to realistically recreate the specific appearance of the great person.

[0810] Step 6:

[0811] server

[0812] The server uses a speech synthesis engine based on the basic information to generate voice data for the great person, such as "I am (name of great person). I have accomplished (achievement)."

[0813] Step 7:

[0814] server

[0815] The server uses an emotion engine to analyze the user's emotion data, which is obtained from the input information and the user's past behavior history.

[0816] Step 8:

[0817] server

[0818] The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[0819] Step 9:

[0820] server

[0821] The server also adjusts the tone and content of the generated voice data based on the emotion, for example, by including more "thank you" words if the emotion is strong.

[0822] Step 10:

[0823] server

[0824] The server generates a tailored animation script and generates a 3D video by simulating the movements on the 3D model.

[0825] Step 11:

[0826] server

[0827] The server encodes the generated 3D video into streaming and download formats.

[0828] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[0829] Step 12:

[0830] Terminal

[0831] The device decodes the 3D video file received from the server and saves it in local storage.

[0832] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[0833] Step 13:

[0834] User (visitor)

[0835] Visitors to the museum watch 3D videos on the display, which either play automatically or can be started by the visitor pressing a play button.

[0836] Through these detailed processing steps, the system can provide a concrete and vivid image of the great person, and adjust the content according to the user's emotions, providing visitors with a more engaging exhibition experience.

[0837] Example 2

[0838] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Previously, when generating 3D models or videos based on image data of great people or historical figures, it was difficult to provide optimal information to each individual user because the models or videos were not adjusted to reflect the user's emotions. Furthermore, no efficient method had been established for transmitting and playing the generated 3D models or videos. For this reason, there was a demand for a system that could interest visitors in museums and exhibition facilities and give them a vivid, concrete image.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0841] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script, means for transmitting the generated 3D video to a terminal, and means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the results. This makes it possible to provide optimal information according to the user's emotions and to efficiently generate, transmit, and play 3D videos that will attract visitors' attention.

[0842] A "user" is a user who accesses the system and provides image data and basic information.

[0843] A "terminal" refers to a device operated by a user, and is a device that has an interface for performing data communication with a server.

[0844] "Server" means a central computer system for analyzing image data and basic information and generating 3D models and 3D animations.

[0845] "Image data" is data in a file format that contains images of great people or historical figures, and is mainly in JPEG or PNG format.

[0846] "Basic information" is text information about great people and historical figures, such as their names, birth dates, and achievements.

[0847] An "image recognition engine" is a software component for detecting feature points from image data.

[0848] "Feature points" are key points in image data (e.g., the positions of the eyes, nose, mouth, and facial contours).

[0849] A "three-dimensional model" is a digital model that recreates great people and historical figures in three dimensions.

[0850] "Audio data" refers to an audio file generated based on text information.

[0851] An "animation script" is a script file that contains instructions for adding movement to a three-dimensional model.

[0852] "3D video" is a video created by combining a generated 3D model, audio data, and an animation script.

[0853] "Emotion data" is information that indicates the user's emotional state, and is extracted from text input, past behavior history, and the like.

[0854] An "emotion recognition engine" is a software component that analyzes a user's emotional data and outputs the results.

[0855] "Streaming format" is a data transfer format in which data is played back while being continuously delivered.

[0856] "Download format" is a data transfer format in which data is played after being completely acquired.

[0857] The "playback means" refers to a device or software that has the function of displaying the received three-dimensional video on a display device.

[0858] The present invention is a system that uses a generative AI to create 3D videos based on image data and basic information of great people and historical figures provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[0859] System Overview

[0860] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and transmit image data and basic information about the great figure. The server analyzes the transmitted information and utilizes the emotion engine in the process of generating a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0861] Specific explanation of program processing

[0862] The user uses the device interface to select and upload image data (e.g., a JPEG file) of the great person, then fills in basic information about the great person, such as their name, date of birth, and achievements, in an input form and clicks the submit button.

[0863] The device receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and sends them to the server as an HTTP request. At this time, the data is encrypted using SSL / TLS to prevent tampering.

[0864] The server receives the HTTP request and extracts the image data and basic information from the packet. The server then analyzes the image data using an image recognition engine (for example, OpenCV or TensorFlow) to detect the facial features of the great person (such as the position of the eyes, nose, and mouth, as well as the facial contours). Based on the detected features, the server uses the API of 3D modeling software (for example, Blender) to generate a three-dimensional model. This three-dimensional model is used to realistically reproduce the specific appearance of the great person.

[0865] Next, the server uses a speech synthesis engine (for example, the Google Text-to-Speech API) to generate voice data based on the basic information. The generated voice data will say something like, "I am (name of great person). I have achieved (achievement)."

[0866] Furthermore, the server uses an emotion engine (specifically, IBM Watson Tone Analyzer) to analyze the user's emotional data. This emotional data is obtained from input information and past user behavior history. The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone. It also adjusts the tone of the audio data, so that if the user is expressing strong feelings of gratitude, it can include more instances of the word "thank you."

[0867] By executing the adjusted animation script, the server finally generates the 3D video, which is then encoded into streaming and download formats and sent to the terminal using a secure file transfer protocol (e.g., SFTP).

[0868] The terminal decodes the 3D video file received from the server and saves it in local storage. The terminal then sends the 3D video to the display system of the museum or exhibition facility and prepares it for playback. Visitors to the museum watch the 3D video displayed on the display. Playback is either automatic or initiated by the visitor pressing a play button. This allows visitors to form a concrete and vivid image of the great man and gain a deeper understanding of the exhibits.

[0869] Example prompt

[0870] Here is an example of how to input the following prompt sentence into a generative AI model:

[0871] Generate a 3D model and video of a great person using the image "A_person.jpg" and basic information "Name: Great Person A, Date of Birth: January 1, 1900, Achievement: Contributed to a historical event." Also, if the user's emotion is "emotional," adjust the audio tone and video tempo accordingly.

[0872] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0873] Step 1:

[0874] The user selects and uploads image data (e.g., JPEG files) of a great person using a dedicated interface on the device. As input, the user provides the image data and basic information (such as the great person's name, date of birth, and achievements) and clicks the send button. As output, the device receives the image data and basic information.

[0875] Step 2:

[0876] The terminal receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and converts them into an HTTP request. The input image data and basic information are encrypted using SSL / TLS and sent to the server. As output, the data is converted into a format that can be sent to the server.

[0877] Step 3:

[0878] The server receives the HTTP request and extracts the image data and basic information from the packet. As input, the encrypted HTTP request is sent to the server, and as output, the decrypted image data and basic information are obtained.

[0879] Step 4:

[0880] The server analyzes the image data using an image recognition engine (e.g., OpenCV or TensorFlow). The image data is provided as input, and the facial feature points of the great figure (e.g., the positions of the eyes, nose, and mouth, and the facial contours) are detected as output. The server generates a three-dimensional model using 3D modeling software (e.g., Blender) based on these feature points.

[0881] Step 5:

[0882] The server uses a speech synthesis engine (e.g., Google Text-to-Speech API) based on the basic information to generate voice data. The basic information (e.g., name, date of birth, achievement) is used as input, and voice data such as "I am (name of great person). I have achieved (achievement)" is generated as output.

[0883] Step 6:

[0884] The server analyzes the user's emotional data using an emotion engine (e.g., IBM Watson Tone Analyzer). The user's emotional data (e.g., excitement, emotion, joy) is provided as input, and the emotion analysis results are obtained as output. The content and tone of the generated 3D video are adjusted based on the emotion recognition results. For example, if the user is excited, the tempo of the video is made faster and the tone is made more energetic.

[0885] Step 7:

[0886] The server executes the adjusted animation script and simulates the movement of the 3D model to ultimately generate a 3D video. The 3D model, audio data, and animation script are provided as input, and the adjusted 3D video is generated as output.

[0887] Step 8:

[0888] The server encodes the generated 3D video into a streaming format and a download format and transmits it to the terminal using a secure file transfer protocol (e.g., SFTP). The generated 3D video is provided as input, and the encoded 3D video is transmitted to the terminal as output.

[0889] Step 9:

[0890] The terminal decodes the 3D video file received from the server and stores it in local storage. The encoded 3D video is provided as input, and the decoded 3D video is stored in the terminal as output.

[0891] Step 10:

[0892] The terminal transmits the 3D video to the display system of the museum or exhibition facility and prepares it for playback. As input, the 3D video stored in local storage is provided, and as output, preparation for playback on the display system is completed. Visitors to the museum can watch the 3D video displayed on the display.

[0893] (Application example 2)

[0894] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0895] Conventional food delivery applications offer a uniform user experience and lack personalized interaction. Furthermore, the automated guidance provided on order details and delivery status makes it difficult to provide users with a sense of familiarity or enjoyment. Furthermore, the information provided is not tailored to the user's emotions, which can lead to a decline in overall satisfaction with the service.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the analysis results, means for receiving image data and basic information uploaded by the user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, and means for transmitting the generated 3D video to the terminal. This makes it possible to provide friendly and entertaining interactions that correspond to the user's emotions and realize a personalized food delivery service.

[0897] A "user" is a person who uses the system to provide image data and basic information.

[0898] "Uploaded image data" refers to image files sent from a user to a server, specifically in formats such as JPEG or PNG.

[0899] "Basic information" refers to information such as name, date of birth, position, and achievements associated with image data.

[0900] An "image recognition engine" is a software component for analyzing and detecting features from image data.

[0901] "Feature points" are important points such as the position of the eyes, nose, mouth, and facial contours that are extracted from image data of great people.

[0902] A "3D model" is a three-dimensional virtual human image generated based on extracted feature points.

[0903] "Audio data" refers to audio files synthesized based on the generated 3D model, including audio related to basic information.

[0904] An "animation script" is a set of instructions that controls the movement of a 3D model, enabling it to animate.

[0905] "3D video" refers to three-dimensional moving images generated by combining audio data and animation scripts.

[0906] A "terminal" is a device used by a user to operate the device, and includes smartphones, personal computers, etc.

[0907] "Emotion data" refers to emotional information extracted from the user's facial expressions, voice, etc.

[0908] An "emotion engine" is a software component that analyzes a user's emotional data and outputs the results.

[0909] "Tone" refers to the atmosphere and nuances of the audio and animation in a video.

[0910] The embodiment for carrying out the present invention is a system that is composed of four main components: a user, a terminal, a server, and an emotion engine. Specific embodiments of the system of the present invention will be described below.

[0911] System Overview

[0912] The user operates the device to input image data and basic information about the great person, and sends it to the server. The server analyzes the data and generates a 3D model and 3D video. During this process, an emotion engine is used to adjust the video according to the user's emotions. The generated 3D video is sent to the device and provided as a food delivery application.

[0913] 1. Upload photos and enter data

[0914] The user selects and uploads image data (e.g., JPEG files) of the famous person using the interface on the device. They also enter basic information about the famous person, such as their name, date of birth, and achievements, into an input form and send it to the server. The device receives the uploaded image data and the basic information, and sends it to the server.

[0915] 2. Image analysis and 3D modeling

[0916] The server receives the transmitted data and uses an image recognition engine (e.g., OpenCV) to detect the facial features of the great person from the image data. Based on the detected features, it uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person. Furthermore, it uses a speech synthesis engine (e.g., Amazon Polly) based on the basic information to generate voice data.

[0917] 3. Coordination of emotion recognition and video generation

[0918] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotional data and adjusts the content of the 3D video to be generated based on the results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. Based on the adjusted animation script, it simulates movement on the 3D model and finally generates the 3D video.

[0919] 4. Sending and Playing Videos

[0920] The server encodes the generated 3D video into streaming and download formats and sends it to the device. The device decodes the received 3D video file, saves it in local storage, and provides it to the user as a food delivery application. Through the application, the user can receive information about their order and delivery status while having a concrete image of the great man.

[0921] Specific examples

[0922] For example, if a user provides an image file and basic information about "Historical Figure A," the system operates as follows: The user uploads the image file of "Historical Figure A" to their device and enters the user's name ("Historical Figure A"), date of birth ("January 1, 1900"), and achievement ("Contributed to historical events") as basic information. The server receives this information and uses an image recognition engine to detect feature points from the image file and generate a 3D model. Next, it uses a speech synthesis engine to generate voice data and an emotion engine to analyze the user's emotion data. Based on these results, it adjusts the content and tone of the video, and finally generates a 3D video. The server sends the generated 3D video to the device, where the user can watch it through a food delivery application.

[0923] Example prompts for generative AI models

[0924] Name of a famous person: Historical figure A

[0925] Basic information: A patriot from the end of the Edo period, who contributed to historical events

[0926] Characteristics: Excellent negotiation and leadership skills

[0927] Expected output:

[0928] Hello, I'm Historical Figure A. As a patriot in the late Edo period, I contributed to historical events. Today, I'm here to recommend some delicious dishes. Have you ordered yet?

[0929] In this way, the present invention provides personalized interactions that respond to the user's emotions, realizing a food delivery service that is both entertaining and convenient.

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] The user uses the interface on the terminal to select and upload image data (e.g., JPEG files) of the famous person. They also fill in basic information about the famous person, such as their name, date of birth, and achievements, in an input form and click the submit button. The input data includes the image file and basic information in text format. The output is a packet containing this data.

[0933] Step 2:

[0934] The terminal receives the uploaded image data and the input basic information, and assembles this data into packets. The assembled packets are sent to the server as HTTP requests. The input is the image data and basic information from the user, and the output is a data packet in HTTP request format.

[0935] Step 3:

[0936] The server receives the HTTP request and extracts the image data and basic information from the packet. Specifically, the server analyzes the request and separates each piece of data while checking the integrity of the data. The input is a data packet in the HTTP request format, and the output is individual pieces of data: image data and basic information.

[0937] Step 4:

[0938] The server analyzes the image data using an image recognition engine (e.g., OpenCV) to detect the facial feature points of the great man. The detected feature points include the positions of the eyes, nose, and mouth, as well as the facial contours. The input is the image data obtained from the user, and the output is the coordinate data of the detected feature points.

[0939] Step 5:

[0940] The server generates a 3D model based on the detected feature points. This process uses 3D modeling software (e.g., Blender). The input is the coordinate data of the feature points, and the output is a 3D model that reproduces the specific appearance of the great figure.

[0941] Step 6:

[0942] The server uses the basic information to generate voice data of the great person using a speech synthesis engine (e.g., Amazon Polly). For example, the speech synthesis uses content such as "I am (name of great person). I have achieved (achievement)." The input is the basic information in text format, and the output is voice data.

[0943] Step 7:

[0944] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotion data. The emotion data is obtained from the user's past behavior history and interaction data. The input is the user's behavior history data and current interaction data, and the output is the emotion analysis result.

[0945] Step 8:

[0946] The server adjusts the content and tone of the generated 3D video based on the emotion recognition results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. The inputs are the emotion analysis results, 3D models, audio data, and animation scripts, and the output is the adjusted 3D video.

[0947] Step 9:

[0948] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal using a secure file transfer protocol. The input is the calibrated 3D video, and the output is the encoded 3D video file.

[0949] Step 10:

[0950] The device decodes the 3D video file received from the server, stores it in local storage, and provides it to the user as a food delivery application. The input is an encoded 3D video file, and the output is a viewable 3D video.

[0951] Step 11:

[0952] Through the application, users can watch 3D videos of famous people and receive information about their order and delivery status while getting a concrete image of the person. The input is the image data and basic information of the famous person provided by the user, and the 3D video sent from the server, and the output is an entertainment-rich food delivery service experience.

[0953] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0954] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0955] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0956] [Third embodiment]

[0957] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0958] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0959] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0960] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0961] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0962] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0963] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0964] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0965] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0966] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0967] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0968] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0969] This invention is a system that uses generative AI to create 3D videos based on photographs and basic information of famous people, and broadcasts these videos in museums and exhibition facilities. Below, we will explain the program processing of the system of this invention in natural language.

[0970] System Overview

[0971] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and transmit image data and basic information about the great figure. The server analyzes the transmitted data and generates a 3D model and 3D video. The terminal then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[0972] Program processing

[0973] 1. Upload photos and enter data

[0974] User

[0975] The user uses the upload interface on the terminal to select and upload image data (e.g., a JPEG file) of the great figure.

[0976] The user fills in basic information such as name, date of birth, achievements, etc. in the input form and clicks the submit button.

[0977] Terminal

[0978] The terminal receives the uploaded image data and the input basic information and assembles them into a packet format.

[0979] The terminal sends this packet to the server as an HTTP request.

[0980] 2. Image analysis and 3D modeling

[0981] server

[0982] The server receives the HTTP request and extracts the image data and basic information from the packet.

[0983] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position and contours of his nose, eyes, and mouth.

[0984] The server uses 3D modeling software to generate a 3D model of the great person based on the detected features, which is intended to realistically recreate the great person's specific appearance.

[0985] The server also uses the basic information about the great person to generate voice data for the great person (for example, a greeting such as "I am [great person's name]") using a speech synthesis engine.

[0986] 3. 3D video generation

[0987] server

[0988] The server generates an animation script based on the generated 3D model and voice data. This script contains instructions for controlling the speaking actions and facial expressions of the great figure.

[0989] The server executes the animation script and generates 3D animation by simulating movements on the 3D model.

[0990] The server encodes the generated 3D video into multiple formats (e.g., streaming format and download format).

[0991] 4. Sending and Playing Videos

[0992] server

[0993] The server sends the encoded 3D video file to the device using a secure file transfer protocol.

[0994] Terminal

[0995] The device decodes the received 3D video file and saves it in local storage.

[0996] The device will connect to the display system of the museum or exhibition facility and prepare to play 3D videos.

[0997] User (visitor)

[0998] Visitors to the museum can watch 3D videos on the display, either automatically or by pressing a play button.

[0999] Specific examples

[1000] For example, when generating a video of "Great Person A," a great historical figure in Japan, the following specific example can be considered.

[1001] User

[1002] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the terminal.

[1003] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[1004] Terminal

[1005] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[1006] server

[1007] The server uses an image recognition engine to detect the facial features of Great Person A from "A_person.jpg."

[1008] The server uses 3D modeling software to generate a 3D model of Great Person A based on the detected feature points.

[1009] The server also uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event."

[1010] The server generates an animation script based on the 3D model and audio data, and generates a 3D video of great person A speaking.

[1011] The server transmits the generated 3D video to the terminal.

[1012] Terminal

[1013] The device receives the 3D video and plays it on the museum's display.

[1014] User (visitor)

[1015] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[1016] The above is an embodiment of the present invention. This system makes it possible to give a sense of liveliness to exhibits in museums and exhibition facilities, and to provide visitors with a more concrete image.

[1017] The processing flow will be explained below.

[1018] Step 1:

[1019] User

[1020] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[1021] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[1022] Step 2:

[1023] Terminal

[1024] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[1025] The terminal sends the collected packets to the server as an HTTP request.

[1026] Step 3:

[1027] server

[1028] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1029] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[1030] Step 4:

[1031] server

[1032] Based on the detected facial features, the server uses 3D modeling software to generate a 3D model of the great figure, which realistically reproduces the shape and detailed features of the face.

[1033] The server uses a speech synthesis engine to generate voice data for the great person based on the basic information about the great person, such as "I am (great person's name). I have accomplished (achievement)."

[1034] Step 5:

[1035] server

[1036] The server combines the generated 3D model with the audio data to generate an animation script, which includes the movements and facial expressions of the great man as he speaks.

[1037] The server executes the generated animation script and generates a 3D video by simulating the movements of the great man.

[1038] Step 6:

[1039] server

[1040] The server encodes the generated 3D video into streaming and download formats.

[1041] The server then sends the encoded 3D video to the device, using a file transfer protocol to ensure security.

[1042] Step 7:

[1043] Terminal

[1044] The device decodes the 3D video file received from the server and saves it in local storage.

[1045] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[1046] Step 8:

[1047] User (visitor)

[1048] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[1049] Through the above processing steps, the system provides visitors with a concrete and vivid image of the great man, enabling a more engaging exhibition experience.

[1050] Example 1

[1051] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1052] Traditional exhibition facilities and museums often display the achievements of great people and famous figures and historical events using only static images and text, making it difficult to provide visitors with a sense of dynamism and realism. Furthermore, paper media and static displays limit the amount of information visitors receive, making them less appealing to younger visitors in particular. To address this issue, more dynamic and interactive exhibition methods are needed.

[1053] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1054] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal, means for the terminal to decode the received 3D video and save it in local storage, and means for the terminal to play the 3D video in cooperation with a display system of a museum or exhibition facility. This allows visitors to get a concrete image of historical figures and famous people through the 3D video of the people.

[1055] "Image data" is digital data in the form of still images that is uploaded by the user.

[1056] "Basic information" is text data such as the name, date of birth, and achievements of a great or famous person.

[1057] An "image recognition engine" is software or an algorithm for detecting feature points from image data.

[1058] "Feature points" are important points that refer to the positions and contours of facial features such as the nose, eyes, and mouth that are detected from image data.

[1059] A "3D model" is a three-dimensional computer graphic representation of a great or famous person, generated based on detected feature points.

[1060] "Voice data" refers to a voice file generated by a voice synthesis engine based on basic information.

[1061] An "animation script" is a program containing a series of instructions created to control the movements and expressions of a 3D model.

[1062] A "3D video" is a video file generated by executing an animation script based on a 3D model and audio data.

[1063] "Terminal" refers to a device operated by a user that receives, stores, and plays data from a server.

[1064] "Display system" refers to the device or configuration installed in a museum or exhibition facility that displays 3D videos.

[1065] This invention is a system that uses generative AI to create 3D videos based on image data and basic information of great people and famous figures, and then broadcasts these videos in museums and exhibition facilities.

[1066] System configuration

[1067] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and send image data and basic information about the great figure. The server analyzes the data and generates a 3D model and 3D video. The terminal receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1068] Photo upload and data entry

[1069] Using a web interface on the device, users select and upload image data (e.g., a JPEG file) of a great person, fill out a form with basic information such as name, date of birth, and achievements, and click the submit button.

[1070] The terminal receives the input, assembles it into a single data packet, which contains both image and text data, and sends the packet to the server as an HTTP request.

[1071] Image Analysis and 3D Modeling

[1072] The server receives the HTTP request and parses the data packet to extract the image data and basic information. The server then uses an image recognition engine (e.g., OpenCV) to detect facial features from the image data, including the location and contours of the nose, eyes, and mouth.

[1073] The server then uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person based on the detected feature points. This model realistically reproduces the great person's appearance. The server also uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the great person's basic information. For example, it generates an audio file such as, "I am the name of a great person. My achievements are..."

[1074] 3D video generation

[1075] The server creates an animation script based on the generated 3D model and audio data. The script contains instructions to control the figure's speaking movements and facial expressions. The server executes the animation script and generates 3D video by simulating the 3D model's movements. The server then encodes the generated 3D video into multiple formats (e.g., MP4, WebM).

[1076] Video transmission and playback

[1077] The server sends the generated 3D video file to the device using a secure file transfer protocol (e.g., SFTP).

[1078] The device decodes the received 3D video file and saves it in local storage. The device then connects to the display system of the museum or exhibition facility and prepares the 3D video for playback, for example, using a media player connected to the display system.

[1079] Users (visitors) visit the museum and watch 3D videos on the exhibited display. Playback is either automatic or initiated by the visitor pressing a play button.

[1080] Specific examples

[1081] For example, consider the case of generating a video of "Great Person A," a great historical figure in Japan.

[1082] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the device, and then enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to a historical event."

[1083] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[1084] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg." It then uses 3D modeling software to generate a 3D model of great person A based on the detected features. It then uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event." The server then generates an animation script based on the 3D model and voice data, generating a 3D video of great person A speaking. The 3D video is then sent to the device.

[1085] The device receives the 3D video and plays it on the museum display. Visitors can watch the 3D video of great person A on the museum display, giving them a stronger, more concrete image of him.

[1086] The above is an embodiment of the present invention. This system allows visitors to deepen their understanding of the exhibits at museums and exhibition facilities and provides a lively experience.

[1087] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1088] Step 1:

[1089] The user opens the web interface of the device, selects and uploads an image file of a great person or famous person (e.g., "A_person.jpg"). The selected image file is input, and basic information (such as name, date of birth, and achievements) is entered in the text box. By clicking the send button together with these, the input data is sent to the device. The output is the data received by the device.

[1090] Step 2:

[1091] The terminal combines the image file and basic information received from the user into a single data packet. This data packet contains image data and text data. The terminal then sends this data packet to the server as an HTTP request. It receives data from the user as input, creates and sends out a data packet, and the output is sent to the server.

[1092] Step 3:

[1093] The server analyzes the HTTP request received from the terminal and extracts image data and basic information from the data packet. Based on the analysis results of the HTTP request, the extracted image data and text data are passed to the next process. The output is the structured analysis of the data.

[1094] Step 4:

[1095] The server uses an image recognition engine (e.g., OpenCV) to detect feature points (e.g., the position and contours of the nose, eyes, and mouth) from the received image data. The input is image data, and the output is data on the detected feature points. Specifically, it runs a face detection algorithm.

[1096] Step 5:

[1097] The server generates a 3D model of the great person using 3D modeling software (e.g. Blender) based on the detected feature points. The input is the feature point data, and the output is the 3D model. Specifically, the server uses Blender's API to build the 3D model.

[1098] Step 6:

[1099] The server uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the basic information about the great person. The input is text data, and the output is an audio file. Specifically, the server passes the basic information as text to the speech synthesis engine.

[1100] Step 7:

[1101] The server creates an animation script based on the generated 3D model and audio data. The input is the 3D model and audio data, and the output is the animation script. In terms of specific operations, the script constitutes instructions to control the movements and facial expressions of the 3D model.

[1102] Step 8:

[1103] The server executes the animation script and simulates the movement of the 3D model to generate a 3D video. The input is the animation script and the 3D model, and the output is a 3D video file. Specifically, the server performs the animation rendering process.

[1104] Step 9:

[1105] The server encodes the generated 3D video file into multiple formats (e.g., MP4, WebM). The input is the 3D video file, and the output is the encoded video format. This is done using video encoding software.

[1106] Step 10:

[1107] The server sends the encoded 3D video file to the device using a secure file transfer protocol (e.g., SFTP). The input is the encoded video file, and the output is the transmission to the device.

[1108] Step 11:

[1109] The device decodes the received 3D video file and saves it to local storage. The input is the video file from the server, and the output is saving to local storage. Specifically, the video file is saved in a specified folder.

[1110] Step 12:

[1111] The terminal works in conjunction with the display system of the museum or exhibition facility to prepare for 3D video playback. The input is a stored 3D video file, and the output is playback on the display system. Specifically, it launches the media player and prepares to start playback.

[1112] Step 13:

[1113] Users (visitors) watch 3D videos displayed on a display in a museum or exhibition facility. The input is the 3D video played on the display, and the output is what the visitor watches. Specifically, the visitor presses the play button or waits for automatic playback.

[1114] (Application example 1)

[1115] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1116] In traditional factories and manufacturing sites, training new employees and promoting corporate culture required a great deal of time and effort. Furthermore, lectures and explanations to motivate employees had to be delivered in person, which was inefficient. In addition, current methods for recreating historical figures and corporate heroes often lacked a sense of vitality and diminished impact. Given these circumstances, there was a need for an efficient and effective way to train new employees, promote corporate culture, and increase employee motivation.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1118] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating a 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal and playing it on the terminal, means for installing the generated 3D video and audio data on a robot, and means for the robot to play the 3D video and audio data and perform actions specified by the user. This allows for efficient training of new employees and dissemination of corporate culture, enables the realistic reproduction of historical figures and great figures, and increases employee motivation. Furthermore, automated explanations and guidance by the robot reduce human labor and realize an efficient training process.

[1119] A "user" is a person who operates the system and is responsible for inputting image data and basic information.

[1120] "Image data" refers to still image files uploaded by users, and is the basis for recreating historical figures and great people.

[1121] "Basic information" refers to basic information such as the target person's name, date of birth, achievements, etc., and is data used when generating voice data and animation scripts.

[1122] An "image recognition engine" is a software tool for detecting feature points from uploaded image data.

[1123] "Feature points" are important points detected from image data by an image recognition engine and are used to generate 3D models.

[1124] A "3D model" is a three-dimensional digital model generated based on detected feature points to recreate the appearance of a person.

[1125] "Audio data" is an audio file generated based on basic information about the target person, and is used as part of the animation script.

[1126] An "animation script" is a set of motion instructions generated based on 3D models and audio data, and serves as the basis for generating 3D videos.

[1127] "3D video" refers to a video file generated by combining a 3D model with audio data and an animation script, and includes footage of a subject speaking and moving.

[1128] "Terminal" refers to a device operated by a user, which is used to receive and play 3D video sent from the server.

[1129] A "robot" is an autonomous mechanical device that can be installed with 3D video and audio data and can automatically move and play audio.

[1130] A "specified action" is an action or instruction that a robot performs based on instructions programmed by a user to perform a specific task.

[1131] This invention is a system for automating and streamlining new employee training and the dissemination of corporate culture in factories. The system consists of three main components: users, terminals, and servers. The roles of each component and the overall process are described in detail below.

[1132] System Overview

[1133] 1. User inputs image data and basic information

[1134] The user uses the terminal interface to upload image data (e.g., a JPEG file) and enter basic information (e.g., name, date of birth, achievements, etc.).

[1135] The terminal collects the received image data and basic information into packets and sends them to the server as an HTTP request.

[1136] 2. Data analysis and 3D modeling using a server

[1137] The server receives the HTTP request and analyzes the image data and basic information.

[1138] Specifically, an image recognition engine such as OpenCV is used to detect feature points from image data.

[1139] Then, a 3D model is generated based on the detected feature points using 3D modeling software such as Blender.

[1140] 3. Generate audio data and animation script

[1141] The server generates voice data based on the generated 3D model using a speech synthesis engine such as Google Text-to-Speech.

[1142] Furthermore, an animation script is generated based on the generated audio data and 3D model, and a 3D video is created.

[1143] 4. Data transmission and playback

[1144] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal.

[1145] The device decodes the received 3D video and stores it in local storage, preparing it for playback on displays in museums and exhibition facilities.

[1146] 5. Installing data into the robot and controlling its movements

[1147] The server installs the generated 3D video and audio data on the robot.

[1148] The robot performs specified actions based on the data it receives, automatically training new employees and promoting corporate culture within the factory.

[1149] Specific examples

[1150] For example, consider the case of using historical figures of company founders to train new employees.

[1151] The user uploads an image file of the founder (e.g., "founder.jpg") to the terminal and enters basic information such as the name "Founder A," date of birth "January 1, 1900," and achievement "starting and expanding the company."

[1152] The server uses OpenCV and other tools to detect feature points from the uploaded "founder.jpg" and generates a 3D model using Blender. It also uses Google Text-to-Speech to generate audio data saying, "I am Founder A. I contributed to the launch and expansion of the company."

[1153] The server generates an animation script based on the 3D model and audio data, creates a 3D video, encodes it, and sends it to the device.

[1154] The terminal receives the 3D video and prepares it for playback on a display in the factory.

[1155] The robot patrols the factory and gives instructions to new employees while playing installed 3D video and audio data.

[1156] Prompt Sentence Examples

[1157] Examples of prompts include:

[1158] "Users upload a historical image of 'Founder A' and enter basic information (name, achievements, date of birth). The system uses image recognition and 3D modeling technology to generate a 3D model of Founder A, and then uses voice synthesis to create audio data that explains the company's history and important events. Based on this data, a robot that patrols the factory will train new employees."

[1159] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1160] Step 1:

[1161] The user selects and uploads image data (e.g., JPEG files) of a historical figure using the upload interface on the device, and then fills in basic information such as name, date of birth, and achievements in an input form and clicks the submit button.

[1162] Input: Image data of the great person (JPEG file), basic information (name, date of birth, achievements)

[1163] Output: HTTP request containing image data and basic information

[1164] Specific actions: Select an image file of a great person and upload it in your browser. Enter basic information in the input fields and click the submit button.

[1165] Step 2:

[1166] The terminal collects the image data and basic information received from the user into packets and sends them to the server as an HTTP request.

[1167] Input: Image data and basic information sent by the user

[1168] Output: HTTP request to send to the server

[1169] Specific operation: Image data and basic information are converted into JSON format and an HTTP POST request is sent to the server.

[1170] Step 3:

[1171] The server receives the HTTP request, analyzes the image data and basic information, and uses an image recognition engine such as OpenCV to detect feature points from the image data.

[1172] Input: Image data and basic information received from the device

[1173] Output: Facial feature point data of great people

[1174] Specific operation: Analyzes the HTTP request and extracts feature points such as the nose, eyes, and mouth from the image data using OpenCV.

[1175] Step 4:

[1176] The server uses 3D modeling software such as Blender to generate a 3D model based on the detected feature points.

[1177] Input: feature point data

[1178] Output: 3D model data (.obj file, etc.)

[1179] Specific operation: Input feature point data into Blender and generate 3D models of the faces and bodies of great figures.

[1180] Step 5:

[1181] The server uses a speech synthesis engine such as Google Text-to-Speech to generate voice data based on the generated 3D model, and then generates an animation script based on the generated voice data and the 3D model to create a 3D video.

[1182] Input: 3D model data, basic information (name, date of birth, achievements)

[1183] Output: Audio data (.mp3 files, etc.), animation script, 3D video data (.mp4 files, etc.)

[1184] Specific operation: Generates voice data using a voice synthesis engine based on basic information. Generates an animation script based on the voice data and 3D model, and renders a 3D video.

[1185] Step 6:

[1186] The server encodes the generated 3D video into streaming and download formats and transmits it to the terminal.

[1187] Input: 3D video data

[1188] Output: Encoded 3D video file (MPEG format, etc.)

[1189] Specific operation: Encodes 3D video data and sends it to the device via a secure file transfer protocol (e.g., SFTP).

[1190] Step 7:

[1191] The device decodes the received 3D video, stores it in local storage, and then prepares it for playback on a display in the museum or exhibition facility.

[1192] Input: Encoded 3D video file received from the server

[1193] Output: Video data in a format that can be played on a display

[1194] Specific operation: The 3D video file is saved to local storage and prepared for playback on the display using the specified playback software.

[1195] Step 8:

[1196] The server installs the generated 3D video and audio data on the robot.

[1197] Input: 3D video file, audio data file

[1198] Output: Data to be installed on the robot

[1199] Specific operation: 3D video and audio data is transferred to the robot through the robot's software interface and installed.

[1200] Step 9:

[1201] The robot patrols the factory while playing back the received 3D video and audio data, performing designated tasks such as training new employees and promoting corporate culture.

[1202] Input: Installed 3D video and audio data

[1203] Output: Explanations and guides for new employee training and corporate culture promotion

[1204] Specific operation: The robot moves autonomously around the factory, playing videos at appropriate times to provide specified explanations and guidance.

[1205] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1206] The present invention is a system that uses a generation AI to create 3D videos based on image data and basic information of famous people provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[1207] System Overview

[1208] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and send image data and basic information about the great figure. The server analyzes the data and uses the emotion engine to generate a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1209] Program processing

[1210] 1. Upload photos and enter data

[1211] User

[1212] The user selects and uploads image data (e.g., JPEG files) of the great person using the interface on the terminal.

[1213] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[1214] Terminal

[1215] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[1216] The terminal sends this packet to the server as an HTTP request.

[1217] 2. Image analysis and 3D modeling

[1218] server

[1219] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1220] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position of the eyes, nose, mouth, and facial contours.

[1221] The server generates a 3D model based on the detected feature points, which is used to realistically recreate the specific appearance of the great person.

[1222] The server uses the basic information to generate voice data for the great person using a speech synthesis engine, such as "I am (name of great person). I have achieved (achievement)."

[1223] 3. Coordination of emotion recognition and video generation

[1224] server

[1225] The server uses an emotion engine to analyze the user's emotion data, which is obtained from input information, past user behavior history, etc.

[1226] The server adjusts the content of the 3D video it generates based on the emotion recognition results: for example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[1227] The server also changes the tone and content of the generated audio data based on the emotion, for example, it can include more instances of the word "thank you" if the emotion is strong.

[1228] The server executes the tailored animation script and simulates the movements on the 3D model, ultimately generating the 3D video.

[1229] 4. Sending and Playing Videos

[1230] server

[1231] The server encodes the generated 3D video into streaming and download formats.

[1232] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[1233] Terminal

[1234] The device decodes the 3D video file received from the server and saves it in local storage.

[1235] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[1236] User (visitor)

[1237] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[1238] Specific examples

[1239] For example, when generating a video of "Great Person A," the following specific example can be considered.

[1240] User

[1241] The user uploads an image file of "Great Person A" (e.g., "A_person.jpg") to the terminal.

[1242] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[1243] Terminal

[1244] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[1245] server

[1246] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg" and generates a 3D model using 3D modeling software.

[1247] The server generates voice data of great person A using a voice synthesis engine.

[1248] The server uses an emotion engine to analyze the user's emotional data and adjust the content of the video, for example, by adding a gentle, emotional tone to the audio if the user is emotional.

[1249] The server generates the animation script and creates the 3D video.

[1250] The server transmits the generated 3D video to the terminal.

[1251] Terminal

[1252] The device receives the 3D video and plays it on the museum's display.

[1253] User (visitor)

[1254] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[1255] The above is an embodiment of the present invention. This system allows visitors to museums and exhibition facilities to have a concrete and vivid image of great people, and also provides appropriate information according to their emotions.

[1256] The processing flow will be explained below.

[1257] Step 1:

[1258] User

[1259] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[1260] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[1261] Step 2:

[1262] Terminal

[1263] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[1264] The terminal sends this packet to the server as an HTTP request.

[1265] Step 3:

[1266] server

[1267] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1268] Step 4:

[1269] server

[1270] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[1271] Step 5:

[1272] server

[1273] The server uses 3D modeling software to generate a 3D model of the great person based on the detected feature points, which is intended to realistically recreate the specific appearance of the great person.

[1274] Step 6:

[1275] server

[1276] The server uses a speech synthesis engine based on the basic information to generate voice data for the great person, such as "I am (name of great person). I have accomplished (achievement)."

[1277] Step 7:

[1278] server

[1279] The server uses an emotion engine to analyze the user's emotion data, which is obtained from the input information and the user's past behavior history.

[1280] Step 8:

[1281] server

[1282] The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[1283] Step 9:

[1284] server

[1285] The server also adjusts the tone and content of the generated voice data based on the emotion, for example, by including more "thank you" words if the emotion is strong.

[1286] Step 10:

[1287] server

[1288] The server generates a tailored animation script and generates a 3D video by simulating the movements on the 3D model.

[1289] Step 11:

[1290] server

[1291] The server encodes the generated 3D video into streaming and download formats.

[1292] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[1293] Step 12:

[1294] Terminal

[1295] The device decodes the 3D video file received from the server and saves it in local storage.

[1296] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[1297] Step 13:

[1298] User (visitor)

[1299] Visitors to the museum watch 3D videos on the display, which either play automatically or can be started by the visitor pressing a play button.

[1300] Through these detailed processing steps, the system can provide a concrete and vivid image of the great person, and adjust the content according to the user's emotions, providing visitors with a more engaging exhibition experience.

[1301] Example 2

[1302] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1303] Previously, when generating 3D models or videos based on image data of great people or historical figures, it was difficult to provide optimal information to each individual user because the models or videos were not adjusted to reflect the user's emotions. Furthermore, no efficient method had been established for transmitting and playing the generated 3D models or videos. For this reason, there was a demand for a system that could interest visitors in museums and exhibition facilities and give them a vivid, concrete image.

[1304] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1305] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script, means for transmitting the generated 3D video to a terminal, and means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the results. This makes it possible to provide optimal information according to the user's emotions and to efficiently generate, transmit, and play 3D videos that will attract visitors' attention.

[1306] A "user" is a user who accesses the system and provides image data and basic information.

[1307] A "terminal" refers to a device operated by a user, and is a device that has an interface for performing data communication with a server.

[1308] "Server" means a central computer system for analyzing image data and basic information and generating 3D models and 3D animations.

[1309] "Image data" is data in a file format that contains images of great people or historical figures, and is mainly in JPEG or PNG format.

[1310] "Basic information" is text information about great people and historical figures, such as their names, birth dates, and achievements.

[1311] An "image recognition engine" is a software component for detecting feature points from image data.

[1312] "Feature points" are key points in image data (e.g., the positions of the eyes, nose, mouth, and facial contours).

[1313] A "three-dimensional model" is a digital model that recreates great people and historical figures in three dimensions.

[1314] "Audio data" refers to an audio file generated based on text information.

[1315] An "animation script" is a script file that contains instructions for adding movement to a three-dimensional model.

[1316] "3D video" is a video created by combining a generated 3D model, audio data, and an animation script.

[1317] "Emotion data" is information that indicates the user's emotional state, and is extracted from text input, past behavior history, and the like.

[1318] An "emotion recognition engine" is a software component that analyzes a user's emotional data and outputs the results.

[1319] "Streaming format" is a data transfer format in which data is played back while being continuously delivered.

[1320] "Download format" is a data transfer format in which data is played after being completely acquired.

[1321] The "playback means" refers to a device or software that has the function of displaying the received three-dimensional video on a display device.

[1322] The present invention is a system that uses a generative AI to create 3D videos based on image data and basic information of great people and historical figures provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[1323] System Overview

[1324] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and transmit image data and basic information about the great figure. The server analyzes the transmitted information and utilizes the emotion engine in the process of generating a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1325] Specific explanation of program processing

[1326] The user uses the device interface to select and upload image data (e.g., a JPEG file) of the great person, then fills in basic information about the great person, such as their name, date of birth, and achievements, in an input form and clicks the submit button.

[1327] The device receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and sends them to the server as an HTTP request. At this time, the data is encrypted using SSL / TLS to prevent tampering.

[1328] The server receives the HTTP request and extracts the image data and basic information from the packet. The server then analyzes the image data using an image recognition engine (for example, OpenCV or TensorFlow) to detect the facial features of the great person (such as the position of the eyes, nose, and mouth, as well as the facial contours). Based on the detected features, the server uses the API of 3D modeling software (for example, Blender) to generate a three-dimensional model. This three-dimensional model is used to realistically reproduce the specific appearance of the great person.

[1329] Next, the server uses a speech synthesis engine (for example, the Google Text-to-Speech API) to generate voice data based on the basic information. The generated voice data will say something like, "I am (name of great person). I have achieved (achievement)."

[1330] Furthermore, the server uses an emotion engine (specifically, IBM Watson Tone Analyzer) to analyze the user's emotional data. This emotional data is obtained from input information and past user behavior history. The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone. It also adjusts the tone of the audio data, so that if the user is expressing strong feelings of gratitude, it can include more instances of the word "thank you."

[1331] By executing the adjusted animation script, the server finally generates the 3D video, which is then encoded into streaming and download formats and sent to the terminal using a secure file transfer protocol (e.g., SFTP).

[1332] The terminal decodes the 3D video file received from the server and saves it in local storage. The terminal then sends the 3D video to the display system of the museum or exhibition facility and prepares it for playback. Visitors to the museum watch the 3D video displayed on the display. Playback is either automatic or initiated by the visitor pressing a play button. This allows visitors to form a concrete and vivid image of the great man and gain a deeper understanding of the exhibits.

[1333] Example prompt

[1334] Here is an example of how to input the following prompt sentence into a generative AI model:

[1335] Generate a 3D model and video of a great person using the image "A_person.jpg" and basic information "Name: Great Person A, Date of Birth: January 1, 1900, Achievement: Contributed to a historical event." Also, if the user's emotion is "emotional," adjust the audio tone and video tempo accordingly.

[1336] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1337] Step 1:

[1338] The user selects and uploads image data (e.g., JPEG files) of a great person using a dedicated interface on the device. As input, the user provides the image data and basic information (such as the great person's name, date of birth, and achievements) and clicks the send button. As output, the device receives the image data and basic information.

[1339] Step 2:

[1340] The terminal receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and converts them into an HTTP request. The input image data and basic information are encrypted using SSL / TLS and sent to the server. As output, the data is converted into a format that can be sent to the server.

[1341] Step 3:

[1342] The server receives the HTTP request and extracts the image data and basic information from the packet. As input, the encrypted HTTP request is sent to the server, and as output, the decrypted image data and basic information are obtained.

[1343] Step 4:

[1344] The server analyzes the image data using an image recognition engine (e.g., OpenCV or TensorFlow). The image data is provided as input, and the facial feature points of the great figure (e.g., the positions of the eyes, nose, and mouth, and the facial contours) are detected as output. The server generates a three-dimensional model using 3D modeling software (e.g., Blender) based on these feature points.

[1345] Step 5:

[1346] The server uses a speech synthesis engine (e.g., Google Text-to-Speech API) based on the basic information to generate voice data. The basic information (e.g., name, date of birth, achievement) is used as input, and voice data such as "I am (name of great person). I have achieved (achievement)" is generated as output.

[1347] Step 6:

[1348] The server analyzes the user's emotional data using an emotion engine (e.g., IBM Watson Tone Analyzer). The user's emotional data (e.g., excitement, emotion, joy) is provided as input, and the emotion analysis results are obtained as output. The content and tone of the generated 3D video are adjusted based on the emotion recognition results. For example, if the user is excited, the tempo of the video is made faster and the tone is made more energetic.

[1349] Step 7:

[1350] The server executes the adjusted animation script and simulates the movement of the 3D model to ultimately generate a 3D video. The 3D model, audio data, and animation script are provided as input, and the adjusted 3D video is generated as output.

[1351] Step 8:

[1352] The server encodes the generated 3D video into a streaming format and a download format and transmits it to the terminal using a secure file transfer protocol (e.g., SFTP). The generated 3D video is provided as input, and the encoded 3D video is transmitted to the terminal as output.

[1353] Step 9:

[1354] The terminal decodes the 3D video file received from the server and stores it in local storage. The encoded 3D video is provided as input, and the decoded 3D video is stored in the terminal as output.

[1355] Step 10:

[1356] The terminal transmits the 3D video to the display system of the museum or exhibition facility and prepares it for playback. As input, the 3D video stored in local storage is provided, and as output, preparation for playback on the display system is completed. Visitors to the museum can watch the 3D video displayed on the display.

[1357] (Application example 2)

[1358] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1359] Conventional food delivery applications offer a uniform user experience and lack personalized interaction. Furthermore, the automated guidance provided on order details and delivery status makes it difficult to provide users with a sense of familiarity or enjoyment. Furthermore, the information provided is not tailored to the user's emotions, which can lead to a decline in overall satisfaction with the service.

[1360] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the analysis results, means for receiving image data and basic information uploaded by the user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, and means for transmitting the generated 3D video to the terminal. This makes it possible to provide friendly and entertaining interactions that correspond to the user's emotions and realize a personalized food delivery service.

[1361] A "user" is a person who uses the system to provide image data and basic information.

[1362] "Uploaded image data" refers to image files sent from a user to a server, specifically in formats such as JPEG or PNG.

[1363] "Basic information" refers to information such as name, date of birth, position, and achievements associated with image data.

[1364] An "image recognition engine" is a software component for analyzing and detecting features from image data.

[1365] "Feature points" are important points such as the position of the eyes, nose, mouth, and facial contours that are extracted from image data of great people.

[1366] A "3D model" is a three-dimensional virtual human image generated based on extracted feature points.

[1367] "Audio data" refers to audio files synthesized based on the generated 3D model, including audio related to basic information.

[1368] An "animation script" is a set of instructions that controls the movement of a 3D model, enabling it to animate.

[1369] "3D video" refers to three-dimensional moving images generated by combining audio data and animation scripts.

[1370] A "terminal" is a device used by a user to operate the device, and includes smartphones, personal computers, etc.

[1371] "Emotion data" refers to emotional information extracted from the user's facial expressions, voice, etc.

[1372] An "emotion engine" is a software component that analyzes a user's emotional data and outputs the results.

[1373] "Tone" refers to the atmosphere and nuances of the audio and animation in a video.

[1374] The embodiment for carrying out the present invention is a system that is composed of four main components: a user, a terminal, a server, and an emotion engine. Specific embodiments of the system of the present invention will be described below.

[1375] System Overview

[1376] The user operates the device to input image data and basic information about the great person, and sends it to the server. The server analyzes the data and generates a 3D model and 3D video. During this process, an emotion engine is used to adjust the video according to the user's emotions. The generated 3D video is sent to the device and provided as a food delivery application.

[1377] 1. Upload photos and enter data

[1378] The user selects and uploads image data (e.g., JPEG files) of the famous person using the interface on the device. They also enter basic information about the famous person, such as their name, date of birth, and achievements, into an input form and send it to the server. The device receives the uploaded image data and the basic information, and sends it to the server.

[1379] 2. Image analysis and 3D modeling

[1380] The server receives the transmitted data and uses an image recognition engine (e.g., OpenCV) to detect the facial features of the great person from the image data. Based on the detected features, it uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person. Furthermore, it uses a speech synthesis engine (e.g., Amazon Polly) based on the basic information to generate voice data.

[1381] 3. Coordination of emotion recognition and video generation

[1382] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotional data and adjusts the content of the 3D video to be generated based on the results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. Based on the adjusted animation script, it simulates movement on the 3D model and finally generates the 3D video.

[1383] 4. Sending and Playing Videos

[1384] The server encodes the generated 3D video into streaming and download formats and sends it to the device. The device decodes the received 3D video file, saves it in local storage, and provides it to the user as a food delivery application. Through the application, the user can receive information about their order and delivery status while having a concrete image of the great man.

[1385] Specific examples

[1386] For example, if a user provides an image file and basic information about "Historical Figure A," the system operates as follows: The user uploads the image file of "Historical Figure A" to their device and enters the user's name ("Historical Figure A"), date of birth ("January 1, 1900"), and achievement ("Contributed to historical events") as basic information. The server receives this information and uses an image recognition engine to detect feature points from the image file and generate a 3D model. Next, it uses a speech synthesis engine to generate voice data and an emotion engine to analyze the user's emotion data. Based on these results, it adjusts the content and tone of the video, and finally generates a 3D video. The server sends the generated 3D video to the device, where the user can watch it through a food delivery application.

[1387] Example prompts for generative AI models

[1388] Name of a famous person: Historical figure A

[1389] Basic information: A patriot from the end of the Edo period, who contributed to historical events

[1390] Characteristics: Excellent negotiation and leadership skills

[1391] Expected output:

[1392] Hello, I'm Historical Figure A. As a patriot in the late Edo period, I contributed to historical events. Today, I'm here to recommend some delicious dishes. Have you ordered yet?

[1393] In this way, the present invention provides personalized interactions that respond to the user's emotions, realizing a food delivery service that is both entertaining and convenient.

[1394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1395] Step 1:

[1396] The user uses the interface on the terminal to select and upload image data (e.g., JPEG files) of the famous person. They also fill in basic information about the famous person, such as their name, date of birth, and achievements, in an input form and click the submit button. The input data includes the image file and basic information in text format. The output is a packet containing this data.

[1397] Step 2:

[1398] The terminal receives the uploaded image data and the input basic information, and assembles this data into packets. The assembled packets are sent to the server as HTTP requests. The input is the image data and basic information from the user, and the output is a data packet in HTTP request format.

[1399] Step 3:

[1400] The server receives the HTTP request and extracts the image data and basic information from the packet. Specifically, the server analyzes the request and separates each piece of data while checking the integrity of the data. The input is a data packet in the HTTP request format, and the output is individual pieces of data: image data and basic information.

[1401] Step 4:

[1402] The server analyzes the image data using an image recognition engine (e.g., OpenCV) to detect the facial feature points of the great man. The detected feature points include the positions of the eyes, nose, and mouth, as well as the facial contours. The input is the image data obtained from the user, and the output is the coordinate data of the detected feature points.

[1403] Step 5:

[1404] The server generates a 3D model based on the detected feature points. This process uses 3D modeling software (e.g., Blender). The input is the coordinate data of the feature points, and the output is a 3D model that reproduces the specific appearance of the great figure.

[1405] Step 6:

[1406] The server uses the basic information to generate voice data of the great person using a speech synthesis engine (e.g., Amazon Polly). For example, the speech synthesis uses content such as "I am (name of great person). I have achieved (achievement)." The input is the basic information in text format, and the output is voice data.

[1407] Step 7:

[1408] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotion data. The emotion data is obtained from the user's past behavior history and interaction data. The input is the user's behavior history data and current interaction data, and the output is the emotion analysis result.

[1409] Step 8:

[1410] The server adjusts the content and tone of the generated 3D video based on the emotion recognition results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. The inputs are the emotion analysis results, 3D models, audio data, and animation scripts, and the output is the adjusted 3D video.

[1411] Step 9:

[1412] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal using a secure file transfer protocol. The input is the calibrated 3D video, and the output is the encoded 3D video file.

[1413] Step 10:

[1414] The device decodes the 3D video file received from the server, stores it in local storage, and provides it to the user as a food delivery application. The input is an encoded 3D video file, and the output is a viewable 3D video.

[1415] Step 11:

[1416] Through the application, users can watch 3D videos of famous people and receive information about their order and delivery status while getting a concrete image of the person. The input is the image data and basic information of the famous person provided by the user, and the 3D video sent from the server, and the output is an entertainment-rich food delivery service experience.

[1417] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1418] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1419] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1420] [Fourth embodiment]

[1421] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1422] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1423] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1424] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1425] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1427] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1428] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1429] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1430] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1431] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1432] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1433] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1434] This invention is a system that uses generative AI to create 3D videos based on photographs and basic information of famous people, and broadcasts these videos in museums and exhibition facilities. Below, we will explain the program processing of the system of this invention in natural language.

[1435] System Overview

[1436] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and transmit image data and basic information about the great figure. The server analyzes the transmitted data and generates a 3D model and 3D video. The terminal then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1437] Program processing

[1438] 1. Upload photos and enter data

[1439] User

[1440] The user uses the upload interface on the terminal to select and upload image data (e.g., a JPEG file) of the great figure.

[1441] The user fills in basic information such as name, date of birth, achievements, etc. in the input form and clicks the submit button.

[1442] Terminal

[1443] The terminal receives the uploaded image data and the input basic information and assembles them into a packet format.

[1444] The terminal sends this packet to the server as an HTTP request.

[1445] 2. Image analysis and 3D modeling

[1446] server

[1447] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1448] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position and contours of his nose, eyes, and mouth.

[1449] The server uses 3D modeling software to generate a 3D model of the great person based on the detected features, which is intended to realistically recreate the great person's specific appearance.

[1450] The server also uses the basic information about the great person to generate voice data for the great person (for example, a greeting such as "I am [great person's name]") using a speech synthesis engine.

[1451] 3. 3D video generation

[1452] server

[1453] The server generates an animation script based on the generated 3D model and voice data. This script contains instructions for controlling the speaking actions and facial expressions of the great figure.

[1454] The server executes the animation script and generates 3D animation by simulating movements on the 3D model.

[1455] The server encodes the generated 3D video into multiple formats (e.g., streaming format and download format).

[1456] 4. Sending and Playing Videos

[1457] server

[1458] The server sends the encoded 3D video file to the device using a secure file transfer protocol.

[1459] Terminal

[1460] The device decodes the received 3D video file and saves it in local storage.

[1461] The device will connect to the display system of the museum or exhibition facility and prepare to play 3D videos.

[1462] User (visitor)

[1463] Visitors to the museum can watch 3D videos on the display, either automatically or by pressing a play button.

[1464] Specific examples

[1465] For example, when generating a video of "Great Person A," a great historical figure in Japan, the following specific example can be considered.

[1466] User

[1467] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the terminal.

[1468] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[1469] Terminal

[1470] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[1471] server

[1472] The server uses an image recognition engine to detect the facial features of Great Person A from "A_person.jpg."

[1473] The server uses 3D modeling software to generate a 3D model of Great Person A based on the detected feature points.

[1474] The server also uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event."

[1475] The server generates an animation script based on the 3D model and audio data, and generates a 3D video of great person A speaking.

[1476] The server transmits the generated 3D video to the terminal.

[1477] Terminal

[1478] The device receives the 3D video and plays it on the museum's display.

[1479] User (visitor)

[1480] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[1481] The above is an embodiment of the present invention. This system makes it possible to give a sense of liveliness to exhibits in museums and exhibition facilities, and to provide visitors with a more concrete image.

[1482] The processing flow will be explained below.

[1483] Step 1:

[1484] User

[1485] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[1486] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[1487] Step 2:

[1488] Terminal

[1489] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[1490] The terminal sends the collected packets to the server as an HTTP request.

[1491] Step 3:

[1492] server

[1493] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1494] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[1495] Step 4:

[1496] server

[1497] Based on the detected facial features, the server uses 3D modeling software to generate a 3D model of the great figure, which realistically reproduces the shape and detailed features of the face.

[1498] The server uses a speech synthesis engine to generate voice data for the great person based on the basic information about the great person, such as "I am (great person's name). I have accomplished (achievement)."

[1499] Step 5:

[1500] server

[1501] The server combines the generated 3D model with the audio data to generate an animation script, which includes the movements and facial expressions of the great man as he speaks.

[1502] The server executes the generated animation script and generates a 3D video by simulating the movements of the great man.

[1503] Step 6:

[1504] server

[1505] The server encodes the generated 3D video into streaming and download formats.

[1506] The server then sends the encoded 3D video to the device, using a file transfer protocol to ensure security.

[1507] Step 7:

[1508] Terminal

[1509] The device decodes the 3D video file received from the server and saves it in local storage.

[1510] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[1511] Step 8:

[1512] User (visitor)

[1513] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[1514] Through the above processing steps, the system provides visitors with a concrete and vivid image of the great man, enabling a more engaging exhibition experience.

[1515] Example 1

[1516] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1517] Traditional exhibition facilities and museums often display the achievements of great people and famous figures and historical events using only static images and text, making it difficult to provide visitors with a sense of dynamism and realism. Furthermore, paper media and static displays limit the amount of information visitors receive, making them less appealing to younger visitors in particular. To address this issue, more dynamic and interactive exhibition methods are needed.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1519] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal, means for the terminal to decode the received 3D video and save it in local storage, and means for the terminal to play the 3D video in cooperation with a display system of a museum or exhibition facility. This allows visitors to get a concrete image of historical figures and famous people through the 3D video of the people.

[1520] "Image data" is digital data in the form of still images that is uploaded by the user.

[1521] "Basic information" is text data such as the name, date of birth, and achievements of a great or famous person.

[1522] An "image recognition engine" is software or an algorithm for detecting feature points from image data.

[1523] "Feature points" are important points that refer to the positions and contours of facial features such as the nose, eyes, and mouth that are detected from image data.

[1524] A "3D model" is a three-dimensional computer graphic representation of a great or famous person, generated based on detected feature points.

[1525] "Voice data" refers to a voice file generated by a voice synthesis engine based on basic information.

[1526] An "animation script" is a program containing a series of instructions created to control the movements and expressions of a 3D model.

[1527] A "3D video" is a video file generated by executing an animation script based on a 3D model and audio data.

[1528] "Terminal" refers to a device operated by a user that receives, stores, and plays data from a server.

[1529] "Display system" refers to the device or configuration installed in a museum or exhibition facility that displays 3D videos.

[1530] This invention is a system that uses generative AI to create 3D videos based on image data and basic information of great people and famous figures, and then broadcasts these videos in museums and exhibition facilities.

[1531] System configuration

[1532] This system consists of three main components: the user, the terminal, and the server. The user operates the terminal to input and send image data and basic information about the great figure. The server analyzes the data and generates a 3D model and 3D video. The terminal receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1533] Photo upload and data entry

[1534] Using a web interface on the device, users select and upload image data (e.g., a JPEG file) of a great person, fill out a form with basic information such as name, date of birth, and achievements, and click the submit button.

[1535] The terminal receives the input, assembles it into a single data packet, which contains both image and text data, and sends the packet to the server as an HTTP request.

[1536] Image Analysis and 3D Modeling

[1537] The server receives the HTTP request and parses the data packet to extract the image data and basic information. The server then uses an image recognition engine (e.g., OpenCV) to detect facial features from the image data, including the location and contours of the nose, eyes, and mouth.

[1538] The server then uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person based on the detected feature points. This model realistically reproduces the great person's appearance. The server also uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the great person's basic information. For example, it generates an audio file such as, "I am the name of a great person. My achievements are..."

[1539] 3D video generation

[1540] The server creates an animation script based on the generated 3D model and audio data. The script contains instructions to control the figure's speaking movements and facial expressions. The server executes the animation script and generates 3D video by simulating the 3D model's movements. The server then encodes the generated 3D video into multiple formats (e.g., MP4, WebM).

[1541] Video transmission and playback

[1542] The server sends the generated 3D video file to the device using a secure file transfer protocol (e.g., SFTP).

[1543] The device decodes the received 3D video file and saves it in local storage. The device then connects to the display system of the museum or exhibition facility and prepares the 3D video for playback, for example, using a media player connected to the display system.

[1544] Users (visitors) visit the museum and watch 3D videos on the exhibited display. Playback is either automatic or initiated by the visitor pressing a play button.

[1545] Specific examples

[1546] For example, consider the case of generating a video of "Great Person A," a great historical figure in Japan.

[1547] The user uploads an image file of "Great Person A" (for example, "A_person.jpg") to the device, and then enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to a historical event."

[1548] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[1549] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg." It then uses 3D modeling software to generate a 3D model of great person A based on the detected features. It then uses a speech synthesis engine to generate voice data such as, "I am great person A. I contributed to a historic event." The server then generates an animation script based on the 3D model and voice data, generating a 3D video of great person A speaking. The 3D video is then sent to the device.

[1550] The device receives the 3D video and plays it on the museum display. Visitors can watch the 3D video of great person A on the museum display, giving them a stronger, more concrete image of him.

[1551] The above is an embodiment of the present invention. This system allows visitors to deepen their understanding of the exhibits at museums and exhibition facilities and provides a lively experience.

[1552] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1553] Step 1:

[1554] The user opens the web interface of the device, selects and uploads an image file of a great person or famous person (e.g., "A_person.jpg"). The selected image file is input, and basic information (such as name, date of birth, and achievements) is entered in the text box. By clicking the send button together with these, the input data is sent to the device. The output is the data received by the device.

[1555] Step 2:

[1556] The terminal combines the image file and basic information received from the user into a single data packet. This data packet contains image data and text data. The terminal then sends this data packet to the server as an HTTP request. It receives data from the user as input, creates and sends out a data packet, and the output is sent to the server.

[1557] Step 3:

[1558] The server analyzes the HTTP request received from the terminal and extracts image data and basic information from the data packet. Based on the analysis results of the HTTP request, the extracted image data and text data are passed to the next process. The output is the structured analysis of the data.

[1559] Step 4:

[1560] The server uses an image recognition engine (e.g., OpenCV) to detect feature points (e.g., the position and contours of the nose, eyes, and mouth) from the received image data. The input is image data, and the output is data on the detected feature points. Specifically, it runs a face detection algorithm.

[1561] Step 5:

[1562] The server generates a 3D model of the great person using 3D modeling software (e.g. Blender) based on the detected feature points. The input is the feature point data, and the output is the 3D model. Specifically, the server uses Blender's API to build the 3D model.

[1563] Step 6:

[1564] The server uses a speech synthesis engine (e.g., Google Text-to-Speech) to generate audio data based on the basic information about the great person. The input is text data, and the output is an audio file. Specifically, the server passes the basic information as text to the speech synthesis engine.

[1565] Step 7:

[1566] The server creates an animation script based on the generated 3D model and audio data. The input is the 3D model and audio data, and the output is the animation script. In terms of specific operations, the script constitutes instructions to control the movements and facial expressions of the 3D model.

[1567] Step 8:

[1568] The server executes the animation script and simulates the movement of the 3D model to generate a 3D video. The input is the animation script and the 3D model, and the output is a 3D video file. Specifically, the server performs the animation rendering process.

[1569] Step 9:

[1570] The server encodes the generated 3D video file into multiple formats (e.g., MP4, WebM). The input is the 3D video file, and the output is the encoded video format. This is done using video encoding software.

[1571] Step 10:

[1572] The server sends the encoded 3D video file to the device using a secure file transfer protocol (e.g., SFTP). The input is the encoded video file, and the output is the transmission to the device.

[1573] Step 11:

[1574] The device decodes the received 3D video file and saves it to local storage. The input is the video file from the server, and the output is saving to local storage. Specifically, the video file is saved in a specified folder.

[1575] Step 12:

[1576] The terminal works in conjunction with the display system of the museum or exhibition facility to prepare for 3D video playback. The input is a stored 3D video file, and the output is playback on the display system. Specifically, it launches the media player and prepares to start playback.

[1577] Step 13:

[1578] Users (visitors) watch 3D videos displayed on a display in a museum or exhibition facility. The input is the 3D video played on the display, and the output is what the visitor watches. Specifically, the visitor presses the play button or waits for automatic playback.

[1579] (Application example 1)

[1580] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1581] In traditional factories and manufacturing sites, training new employees and promoting corporate culture required a great deal of time and effort. Furthermore, lectures and explanations to motivate employees had to be delivered in person, which was inefficient. In addition, current methods for recreating historical figures and corporate heroes often lacked a sense of vitality and diminished impact. Given these circumstances, there was a need for an efficient and effective way to train new employees, promote corporate culture, and increase employee motivation.

[1582] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1583] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating a 3D video by combining the generated audio data and the animation script, means for transmitting the generated 3D video to a terminal and playing it on the terminal, means for installing the generated 3D video and audio data on a robot, and means for the robot to play the 3D video and audio data and perform actions specified by the user. This allows for efficient training of new employees and dissemination of corporate culture, enables the realistic reproduction of historical figures and great figures, and increases employee motivation. Furthermore, automated explanations and guidance by the robot reduce human labor and realize an efficient training process.

[1584] A "user" is a person who operates the system and is responsible for inputting image data and basic information.

[1585] "Image data" refers to still image files uploaded by users, and is the basis for recreating historical figures and great people.

[1586] "Basic information" refers to basic information such as the target person's name, date of birth, achievements, etc., and is data used when generating voice data and animation scripts.

[1587] An "image recognition engine" is a software tool for detecting feature points from uploaded image data.

[1588] "Feature points" are important points detected from image data by an image recognition engine and are used to generate 3D models.

[1589] A "3D model" is a three-dimensional digital model generated based on detected feature points to recreate the appearance of a person.

[1590] "Audio data" is an audio file generated based on basic information about the target person, and is used as part of the animation script.

[1591] An "animation script" is a set of motion instructions generated based on 3D models and audio data, and serves as the basis for generating 3D videos.

[1592] "3D video" refers to a video file generated by combining a 3D model with audio data and an animation script, and includes footage of a subject speaking and moving.

[1593] "Terminal" refers to a device operated by a user, which is used to receive and play 3D video sent from the server.

[1594] A "robot" is an autonomous mechanical device that can be installed with 3D video and audio data and can automatically move and play audio.

[1595] A "specified action" is an action or instruction that a robot performs based on instructions programmed by a user to perform a specific task.

[1596] This invention is a system for automating and streamlining new employee training and the dissemination of corporate culture in factories. The system consists of three main components: users, terminals, and servers. The roles of each component and the overall process are described in detail below.

[1597] System Overview

[1598] 1. User inputs image data and basic information

[1599] The user uses the terminal interface to upload image data (e.g., a JPEG file) and enter basic information (e.g., name, date of birth, achievements, etc.).

[1600] The terminal collects the received image data and basic information into packets and sends them to the server as an HTTP request.

[1601] 2. Data analysis and 3D modeling using a server

[1602] The server receives the HTTP request and analyzes the image data and basic information.

[1603] Specifically, an image recognition engine such as OpenCV is used to detect feature points from image data.

[1604] Then, a 3D model is generated based on the detected feature points using 3D modeling software such as Blender.

[1605] 3. Generate audio data and animation script

[1606] The server generates voice data based on the generated 3D model using a speech synthesis engine such as Google Text-to-Speech.

[1607] Furthermore, an animation script is generated based on the generated audio data and 3D model, and a 3D video is created.

[1608] 4. Data transmission and playback

[1609] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal.

[1610] The device decodes the received 3D video and stores it in local storage, preparing it for playback on displays in museums and exhibition facilities.

[1611] 5. Installing data into the robot and controlling its movements

[1612] The server installs the generated 3D video and audio data on the robot.

[1613] The robot performs specified actions based on the data it receives, automatically training new employees and promoting corporate culture within the factory.

[1614] Specific examples

[1615] For example, consider the case of using historical figures of company founders to train new employees.

[1616] The user uploads an image file of the founder (e.g., "founder.jpg") to the terminal and enters basic information such as the name "Founder A," date of birth "January 1, 1900," and achievement "starting and expanding the company."

[1617] The server uses OpenCV and other tools to detect feature points from the uploaded "founder.jpg" and generates a 3D model using Blender. It also uses Google Text-to-Speech to generate audio data saying, "I am Founder A. I contributed to the launch and expansion of the company."

[1618] The server generates an animation script based on the 3D model and audio data, creates a 3D video, encodes it, and sends it to the device.

[1619] The terminal receives the 3D video and prepares it for playback on a display in the factory.

[1620] The robot patrols the factory and gives instructions to new employees while playing installed 3D video and audio data.

[1621] Prompt Sentence Examples

[1622] Examples of prompts include:

[1623] "Users upload a historical image of 'Founder A' and enter basic information (name, achievements, date of birth). The system uses image recognition and 3D modeling technology to generate a 3D model of Founder A, and then uses voice synthesis to create audio data that explains the company's history and important events. Based on this data, a robot that patrols the factory will train new employees."

[1624] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1625] Step 1:

[1626] The user selects and uploads image data (e.g., JPEG files) of a historical figure using the upload interface on the device, and then fills in basic information such as name, date of birth, and achievements in an input form and clicks the submit button.

[1627] Input: Image data of the great person (JPEG file), basic information (name, date of birth, achievements)

[1628] Output: HTTP request containing image data and basic information

[1629] Specific actions: Select an image file of a great person and upload it in your browser. Enter basic information in the input fields and click the submit button.

[1630] Step 2:

[1631] The terminal collects the image data and basic information received from the user into packets and sends them to the server as an HTTP request.

[1632] Input: Image data and basic information sent by the user

[1633] Output: HTTP request to send to the server

[1634] Specific operation: Image data and basic information are converted into JSON format and an HTTP POST request is sent to the server.

[1635] Step 3:

[1636] The server receives the HTTP request, analyzes the image data and basic information, and uses an image recognition engine such as OpenCV to detect feature points from the image data.

[1637] Input: Image data and basic information received from the device

[1638] Output: Facial feature point data of great people

[1639] Specific operation: Analyzes the HTTP request and extracts feature points such as the nose, eyes, and mouth from the image data using OpenCV.

[1640] Step 4:

[1641] The server uses 3D modeling software such as Blender to generate a 3D model based on the detected feature points.

[1642] Input: feature point data

[1643] Output: 3D model data (.obj file, etc.)

[1644] Specific operation: Input feature point data into Blender and generate 3D models of the faces and bodies of great figures.

[1645] Step 5:

[1646] The server uses a speech synthesis engine such as Google Text-to-Speech to generate voice data based on the generated 3D model, and then generates an animation script based on the generated voice data and the 3D model to create a 3D video.

[1647] Input: 3D model data, basic information (name, date of birth, achievements)

[1648] Output: Audio data (.mp3 files, etc.), animation script, 3D video data (.mp4 files, etc.)

[1649] Specific operation: Generates voice data using a voice synthesis engine based on basic information. Generates an animation script based on the voice data and 3D model, and renders a 3D video.

[1650] Step 6:

[1651] The server encodes the generated 3D video into streaming and download formats and transmits it to the terminal.

[1652] Input: 3D video data

[1653] Output: Encoded 3D video file (MPEG format, etc.)

[1654] Specific operation: Encodes 3D video data and sends it to the device via a secure file transfer protocol (e.g., SFTP).

[1655] Step 7:

[1656] The device decodes the received 3D video, stores it in local storage, and then prepares it for playback on a display in the museum or exhibition facility.

[1657] Input: Encoded 3D video file received from the server

[1658] Output: Video data in a format that can be played on a display

[1659] Specific operation: The 3D video file is saved to local storage and prepared for playback on the display using the specified playback software.

[1660] Step 8:

[1661] The server installs the generated 3D video and audio data on the robot.

[1662] Input: 3D video file, audio data file

[1663] Output: Data to be installed on the robot

[1664] Specific operation: 3D video and audio data is transferred to the robot through the robot's software interface and installed.

[1665] Step 9:

[1666] The robot patrols the factory while playing back the received 3D video and audio data, performing designated tasks such as training new employees and promoting corporate culture.

[1667] Input: Installed 3D video and audio data

[1668] Output: Explanations and guides for new employee training and corporate culture promotion

[1669] Specific operation: The robot moves autonomously around the factory, playing videos at appropriate times to provide specified explanations and guidance.

[1670] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1671] The present invention is a system that uses a generation AI to create 3D videos based on image data and basic information of famous people provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[1672] System Overview

[1673] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and send image data and basic information about the great figure. The server analyzes the data and uses the emotion engine to generate a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1674] Program processing

[1675] 1. Upload photos and enter data

[1676] User

[1677] The user selects and uploads image data (e.g., JPEG files) of the great person using the interface on the terminal.

[1678] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[1679] Terminal

[1680] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[1681] The terminal sends this packet to the server as an HTTP request.

[1682] 2. Image analysis and 3D modeling

[1683] server

[1684] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1685] The server uses an image recognition engine to analyze the image data and detect the facial features of the great man, including the position of the eyes, nose, mouth, and facial contours.

[1686] The server generates a 3D model based on the detected feature points, which is used to realistically recreate the specific appearance of the great person.

[1687] The server uses the basic information to generate voice data for the great person using a speech synthesis engine, such as "I am (name of great person). I have achieved (achievement)."

[1688] 3. Coordination of emotion recognition and video generation

[1689] server

[1690] The server uses an emotion engine to analyze the user's emotion data, which is obtained from input information, past user behavior history, etc.

[1691] The server adjusts the content of the 3D video it generates based on the emotion recognition results: for example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[1692] The server also changes the tone and content of the generated audio data based on the emotion, for example, it can include more instances of the word "thank you" if the emotion is strong.

[1693] The server executes the tailored animation script and simulates the movements on the 3D model, ultimately generating the 3D video.

[1694] 4. Sending and Playing Videos

[1695] server

[1696] The server encodes the generated 3D video into streaming and download formats.

[1697] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[1698] Terminal

[1699] The device decodes the 3D video file received from the server and saves it in local storage.

[1700] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[1701] User (visitor)

[1702] Visitors to the museum watch 3D videos on the display. The videos play automatically or can be started by the visitor pressing a play button. This allows visitors to get a concrete image of the great man.

[1703] Specific examples

[1704] For example, when generating a video of "Great Person A," the following specific example can be considered.

[1705] User

[1706] The user uploads an image file of "Great Person A" (e.g., "A_person.jpg") to the terminal.

[1707] The user enters basic information such as the name "Great Person A," date of birth "January 1, 1900," and achievement "contributed to historical events."

[1708] Terminal

[1709] The terminal receives the image file "A_person.jpg" and basic information and sends it to the server.

[1710] server

[1711] The server uses an image recognition engine to detect the facial features of great person A from "A_person.jpg" and generates a 3D model using 3D modeling software.

[1712] The server generates voice data of great person A using a voice synthesis engine.

[1713] The server uses an emotion engine to analyze the user's emotional data and adjust the content of the video, for example, by adding a gentle, emotional tone to the audio if the user is emotional.

[1714] The server generates the animation script and creates the 3D video.

[1715] The server transmits the generated 3D video to the terminal.

[1716] Terminal

[1717] The device receives the 3D video and plays it on the museum's display.

[1718] User (visitor)

[1719] Visitors can watch a 3D video of Great Person A on the museum display, giving them a stronger, more concrete image of him.

[1720] The above is an embodiment of the present invention. This system allows visitors to museums and exhibition facilities to have a concrete and vivid image of great people, and also provides appropriate information according to their emotions.

[1721] The processing flow will be explained below.

[1722] Step 1:

[1723] User

[1724] The user uses the interface on the device to select image data (e.g., a JPEG file) of the great person and clicks the upload button.

[1725] The user fills in basic information such as the great person's name, date of birth, and achievements in the input form and clicks the submit button.

[1726] Step 2:

[1727] Terminal

[1728] The terminal receives the uploaded image data and the input basic information, and assembles this data into a packet format.

[1729] The terminal sends this packet to the server as an HTTP request.

[1730] Step 3:

[1731] server

[1732] The server receives the HTTP request and extracts the image data and basic information from the packet.

[1733] Step 4:

[1734] server

[1735] The server uses an image recognition engine to analyze the received image data and detect the facial features of the great man, including the positions of the eyes, nose, mouth, and facial contours.

[1736] Step 5:

[1737] server

[1738] The server uses 3D modeling software to generate a 3D model of the great person based on the detected feature points, which is intended to realistically recreate the specific appearance of the great person.

[1739] Step 6:

[1740] server

[1741] The server uses a speech synthesis engine based on the basic information to generate voice data for the great person, such as "I am (name of great person). I have accomplished (achievement)."

[1742] Step 7:

[1743] server

[1744] The server uses an emotion engine to analyze the user's emotion data, which is obtained from the input information and the user's past behavior history.

[1745] Step 8:

[1746] server

[1747] The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone.

[1748] Step 9:

[1749] server

[1750] The server also adjusts the tone and content of the generated voice data based on the emotion, for example, by including more "thank you" words if the emotion is strong.

[1751] Step 10:

[1752] server

[1753] The server generates a tailored animation script and generates a 3D video by simulating the movements on the 3D model.

[1754] Step 11:

[1755] server

[1756] The server encodes the generated 3D video into streaming and download formats.

[1757] The server then sends the encoded 3D video to the device using a secure file transfer protocol.

[1758] Step 12:

[1759] Terminal

[1760] The device decodes the 3D video file received from the server and saves it in local storage.

[1761] The terminal transmits the 3D video to the display system of the museum or exhibition facility, preparing it for playback.

[1762] Step 13:

[1763] User (visitor)

[1764] Visitors to the museum watch 3D videos on the display, which either play automatically or can be started by the visitor pressing a play button.

[1765] Through these detailed processing steps, the system can provide a concrete and vivid image of the great person, and adjust the content according to the user's emotions, providing visitors with a more engaging exhibition experience.

[1766] Example 2

[1767] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1768] Previously, when generating 3D models or videos based on image data of great people or historical figures, it was difficult to provide optimal information to each individual user because the models or videos were not adjusted to reflect the user's emotions. Furthermore, no efficient method had been established for transmitting and playing the generated 3D models or videos. For this reason, there was a demand for a system that could interest visitors in museums and exhibition facilities and give them a vivid, concrete image.

[1769] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1770] In this invention, the server includes means for receiving image data and basic information uploaded by a user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script, means for transmitting the generated 3D video to a terminal, and means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the results. This makes it possible to provide optimal information according to the user's emotions and to efficiently generate, transmit, and play 3D videos that will attract visitors' attention.

[1771] A "user" is a user who accesses the system and provides image data and basic information.

[1772] A "terminal" refers to a device operated by a user, and is a device that has an interface for performing data communication with a server.

[1773] "Server" means a central computer system for analyzing image data and basic information and generating 3D models and 3D animations.

[1774] "Image data" is data in a file format that contains images of great people or historical figures, and is mainly in JPEG or PNG format.

[1775] "Basic information" is text information about great people and historical figures, such as their names, birth dates, and achievements.

[1776] An "image recognition engine" is a software component for detecting feature points from image data.

[1777] "Feature points" are key points in image data (e.g., the positions of the eyes, nose, mouth, and facial contours).

[1778] A "three-dimensional model" is a digital model that recreates great people and historical figures in three dimensions.

[1779] "Audio data" refers to an audio file generated based on text information.

[1780] An "animation script" is a script file that contains instructions for adding movement to a three-dimensional model.

[1781] "3D video" is a video created by combining a generated 3D model, audio data, and an animation script.

[1782] "Emotion data" is information that indicates the user's emotional state, and is extracted from text input, past behavior history, and the like.

[1783] An "emotion recognition engine" is a software component that analyzes a user's emotional data and outputs the results.

[1784] "Streaming format" is a data transfer format in which data is played back while being continuously delivered.

[1785] "Download format" is a data transfer format in which data is played after being completely acquired.

[1786] The "playback means" refers to a device or software that has the function of displaying the received three-dimensional video on a display device.

[1787] The present invention is a system that uses a generative AI to create 3D videos based on image data and basic information of great people and historical figures provided by the user, and further combines it with an emotion engine to adjust the content according to the user's emotions. Below, the program processing of the system of the present invention is explained in natural language.

[1788] System Overview

[1789] The system consists of four main components: the user, the device, the server, and the emotion engine. The user operates the device to input and transmit image data and basic information about the great figure. The server analyzes the transmitted information and utilizes the emotion engine in the process of generating a 3D model and 3D video. The device then receives the generated 3D video and plays it on a display in the museum or exhibition facility.

[1790] Specific explanation of program processing

[1791] The user uses the device interface to select and upload image data (e.g., a JPEG file) of the great person, then fills in basic information about the great person, such as their name, date of birth, and achievements, in an input form and clicks the submit button.

[1792] The device receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and sends them to the server as an HTTP request. At this time, the data is encrypted using SSL / TLS to prevent tampering.

[1793] The server receives the HTTP request and extracts the image data and basic information from the packet. The server then analyzes the image data using an image recognition engine (for example, OpenCV or TensorFlow) to detect the facial features of the great person (such as the position of the eyes, nose, and mouth, as well as the facial contours). Based on the detected features, the server uses the API of 3D modeling software (for example, Blender) to generate a three-dimensional model. This three-dimensional model is used to realistically reproduce the specific appearance of the great person.

[1794] Next, the server uses a speech synthesis engine (for example, the Google Text-to-Speech API) to generate voice data based on the basic information. The generated voice data will say something like, "I am (name of great person). I have achieved (achievement)."

[1795] Furthermore, the server uses an emotion engine (specifically, IBM Watson Tone Analyzer) to analyze the user's emotional data. This emotional data is obtained from input information and past user behavior history. The server adjusts the content of the 3D video it generates based on the emotion recognition results. For example, if the user is excited, it speeds up the tempo of the video and gives it an energetic tone. It also adjusts the tone of the audio data, so that if the user is expressing strong feelings of gratitude, it can include more instances of the word "thank you."

[1796] By executing the adjusted animation script, the server finally generates the 3D video, which is then encoded into streaming and download formats and sent to the terminal using a secure file transfer protocol (e.g., SFTP).

[1797] The terminal decodes the 3D video file received from the server and saves it in local storage. The terminal then sends the 3D video to the display system of the museum or exhibition facility and prepares it for playback. Visitors to the museum watch the 3D video displayed on the display. Playback is either automatic or initiated by the visitor pressing a play button. This allows visitors to form a concrete and vivid image of the great man and gain a deeper understanding of the exhibits.

[1798] Example prompt

[1799] Here is an example of how to input the following prompt sentence into a generative AI model:

[1800] Generate a 3D model and video of a great person using the image "A_person.jpg" and basic information "Name: Great Person A, Date of Birth: January 1, 1900, Achievement: Contributed to a historical event." Also, if the user's emotion is "emotional," adjust the audio tone and video tempo accordingly.

[1801] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1802] Step 1:

[1803] The user selects and uploads image data (e.g., JPEG files) of a great person using a dedicated interface on the device. As input, the user provides the image data and basic information (such as the great person's name, date of birth, and achievements) and clicks the send button. As output, the device receives the image data and basic information.

[1804] Step 2:

[1805] The terminal receives the image data uploaded by the user and the basic information entered, assembles this data into packets, and converts them into an HTTP request. The input image data and basic information are encrypted using SSL / TLS and sent to the server. As output, the data is converted into a format that can be sent to the server.

[1806] Step 3:

[1807] The server receives the HTTP request and extracts the image data and basic information from the packet. As input, the encrypted HTTP request is sent to the server, and as output, the decrypted image data and basic information are obtained.

[1808] Step 4:

[1809] The server analyzes the image data using an image recognition engine (e.g., OpenCV or TensorFlow). The image data is provided as input, and the facial feature points of the great figure (e.g., the positions of the eyes, nose, and mouth, and the facial contours) are detected as output. The server generates a three-dimensional model using 3D modeling software (e.g., Blender) based on these feature points.

[1810] Step 5:

[1811] The server uses a speech synthesis engine (e.g., Google Text-to-Speech API) based on the basic information to generate voice data. The basic information (e.g., name, date of birth, achievement) is used as input, and voice data such as "I am (name of great person). I have achieved (achievement)" is generated as output.

[1812] Step 6:

[1813] The server analyzes the user's emotional data using an emotion engine (e.g., IBM Watson Tone Analyzer). The user's emotional data (e.g., excitement, emotion, joy) is provided as input, and the emotion analysis results are obtained as output. The content and tone of the generated 3D video are adjusted based on the emotion recognition results. For example, if the user is excited, the tempo of the video is made faster and the tone is made more energetic.

[1814] Step 7:

[1815] The server executes the adjusted animation script and simulates the movement of the 3D model to ultimately generate a 3D video. The 3D model, audio data, and animation script are provided as input, and the adjusted 3D video is generated as output.

[1816] Step 8:

[1817] The server encodes the generated 3D video into a streaming format and a download format and transmits it to the terminal using a secure file transfer protocol (e.g., SFTP). The generated 3D video is provided as input, and the encoded 3D video is transmitted to the terminal as output.

[1818] Step 9:

[1819] The terminal decodes the 3D video file received from the server and stores it in local storage. The encoded 3D video is provided as input, and the decoded 3D video is stored in the terminal as output.

[1820] Step 10:

[1821] The terminal transmits the 3D video to the display system of the museum or exhibition facility and prepares it for playback. As input, the 3D video stored in local storage is provided, and as output, preparation for playback on the display system is completed. Visitors to the museum can watch the 3D video displayed on the display.

[1822] (Application example 2)

[1823] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1824] Conventional food delivery applications offer a uniform user experience and lack personalized interaction. Furthermore, the automated guidance provided on order details and delivery status makes it difficult to provide users with a sense of familiarity or enjoyment. Furthermore, the information provided is not tailored to the user's emotions, which can lead to a decline in overall satisfaction with the service.

[1825] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user emotion data and adjusting the content and tone of the 3D video based on the analysis results, means for receiving image data and basic information uploaded by the user, means for detecting feature points from the received image data using an image recognition engine, means for generating a 3D model based on the detected feature points, means for generating audio data and an animation script based on the generated 3D model, means for generating 3D video by combining the generated audio data and the animation script, and means for transmitting the generated 3D video to the terminal. This makes it possible to provide friendly and entertaining interactions that correspond to the user's emotions and realize a personalized food delivery service.

[1826] A "user" is a person who uses the system to provide image data and basic information.

[1827] "Uploaded image data" refers to image files sent from a user to a server, specifically in formats such as JPEG or PNG.

[1828] "Basic information" refers to information such as name, date of birth, position, and achievements associated with image data.

[1829] An "image recognition engine" is a software component for analyzing and detecting features from image data.

[1830] "Feature points" are important points such as the position of the eyes, nose, mouth, and facial contours that are extracted from image data of great people.

[1831] A "3D model" is a three-dimensional virtual human image generated based on extracted feature points.

[1832] "Audio data" refers to audio files synthesized based on the generated 3D model, including audio related to basic information.

[1833] An "animation script" is a set of instructions that controls the movement of a 3D model, enabling it to animate.

[1834] "3D video" refers to three-dimensional moving images generated by combining audio data and animation scripts.

[1835] A "terminal" is a device used by a user to operate the device, and includes smartphones, personal computers, etc.

[1836] "Emotion data" refers to emotional information extracted from the user's facial expressions, voice, etc.

[1837] An "emotion engine" is a software component that analyzes a user's emotional data and outputs the results.

[1838] "Tone" refers to the atmosphere and nuances of the audio and animation in a video.

[1839] The embodiment for carrying out the present invention is a system that is composed of four main components: a user, a terminal, a server, and an emotion engine. Specific embodiments of the system of the present invention will be described below.

[1840] System Overview

[1841] The user operates the device to input image data and basic information about the great person, and sends it to the server. The server analyzes the data and generates a 3D model and 3D video. During this process, an emotion engine is used to adjust the video according to the user's emotions. The generated 3D video is sent to the device and provided as a food delivery application.

[1842] 1. Upload photos and enter data

[1843] The user selects and uploads image data (e.g., JPEG files) of the famous person using the interface on the device. They also enter basic information about the famous person, such as their name, date of birth, and achievements, into an input form and send it to the server. The device receives the uploaded image data and the basic information, and sends it to the server.

[1844] 2. Image analysis and 3D modeling

[1845] The server receives the transmitted data and uses an image recognition engine (e.g., OpenCV) to detect the facial features of the great person from the image data. Based on the detected features, it uses 3D modeling software (e.g., Blender) to generate a 3D model of the great person. Furthermore, it uses a speech synthesis engine (e.g., Amazon Polly) based on the basic information to generate voice data.

[1846] 3. Coordination of emotion recognition and video generation

[1847] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotional data and adjusts the content of the 3D video to be generated based on the results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. Based on the adjusted animation script, it simulates movement on the 3D model and finally generates the 3D video.

[1848] 4. Sending and Playing Videos

[1849] The server encodes the generated 3D video into streaming and download formats and sends it to the device. The device decodes the received 3D video file, saves it in local storage, and provides it to the user as a food delivery application. Through the application, the user can receive information about their order and delivery status while having a concrete image of the great man.

[1850] Specific examples

[1851] For example, if a user provides an image file and basic information about "Historical Figure A," the system operates as follows: The user uploads the image file of "Historical Figure A" to their device and enters the user's name ("Historical Figure A"), date of birth ("January 1, 1900"), and achievement ("Contributed to historical events") as basic information. The server receives this information and uses an image recognition engine to detect feature points from the image file and generate a 3D model. Next, it uses a speech synthesis engine to generate voice data and an emotion engine to analyze the user's emotion data. Based on these results, it adjusts the content and tone of the video, and finally generates a 3D video. The server sends the generated 3D video to the device, where the user can watch it through a food delivery application.

[1852] Example prompts for generative AI models

[1853] Name of a famous person: Historical figure A

[1854] Basic information: A patriot from the end of the Edo period, who contributed to historical events

[1855] Characteristics: Excellent negotiation and leadership skills

[1856] Expected output:

[1857] Hello, I'm Historical Figure A. As a patriot in the late Edo period, I contributed to historical events. Today, I'm here to recommend some delicious dishes. Have you ordered yet?

[1858] In this way, the present invention provides personalized interactions that respond to the user's emotions, realizing a food delivery service that is both entertaining and convenient.

[1859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1860] Step 1:

[1861] The user uses the interface on the terminal to select and upload image data (e.g., JPEG files) of the famous person. They also fill in basic information about the famous person, such as their name, date of birth, and achievements, in an input form and click the submit button. The input data includes the image file and basic information in text format. The output is a packet containing this data.

[1862] Step 2:

[1863] The terminal receives the uploaded image data and the input basic information, and assembles this data into packets. The assembled packets are sent to the server as HTTP requests. The input is the image data and basic information from the user, and the output is a data packet in HTTP request format.

[1864] Step 3:

[1865] The server receives the HTTP request and extracts the image data and basic information from the packet. Specifically, the server analyzes the request and separates each piece of data while checking the integrity of the data. The input is a data packet in the HTTP request format, and the output is individual pieces of data: image data and basic information.

[1866] Step 4:

[1867] The server analyzes the image data using an image recognition engine (e.g., OpenCV) to detect the facial feature points of the great man. The detected feature points include the positions of the eyes, nose, and mouth, as well as the facial contours. The input is the image data obtained from the user, and the output is the coordinate data of the detected feature points.

[1868] Step 5:

[1869] The server generates a 3D model based on the detected feature points. This process uses 3D modeling software (e.g., Blender). The input is the coordinate data of the feature points, and the output is a 3D model that reproduces the specific appearance of the great figure.

[1870] Step 6:

[1871] The server uses the basic information to generate voice data of the great person using a speech synthesis engine (e.g., Amazon Polly). For example, the speech synthesis uses content such as "I am (name of great person). I have achieved (achievement)." The input is the basic information in text format, and the output is voice data.

[1872] Step 7:

[1873] The server uses an emotion engine (e.g., Emotion SDK) to analyze the user's emotion data. The emotion data is obtained from the user's past behavior history and interaction data. The input is the user's behavior history data and current interaction data, and the output is the emotion analysis result.

[1874] Step 8:

[1875] The server adjusts the content and tone of the generated 3D video based on the emotion recognition results. For example, if the user is in a happy mood, it generates a video with an energetic tone, and if the user is in a calm mood, it generates a video with a calm tone. The inputs are the emotion analysis results, 3D models, audio data, and animation scripts, and the output is the adjusted 3D video.

[1876] Step 9:

[1877] The server encodes the generated 3D video into streaming and download formats and sends it to the terminal using a secure file transfer protocol. The input is the calibrated 3D video, and the output is the encoded 3D video file.

[1878] Step 10:

[1879] The device decodes the 3D video file received from the server, stores it in local storage, and provides it to the user as a food delivery application. The input is an encoded 3D video file, and the output is a viewable 3D video.

[1880] Step 11:

[1881] Through the application, users can watch 3D videos of famous people and receive information about their order and delivery status while getting a concrete image of the person. The input is the image data and basic information of the famous person provided by the user, and the 3D video sent from the server, and the output is an entertainment-rich food delivery service experience.

[1882] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1883] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1884] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1885] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1886] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1887] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1888] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1889] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1890] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1891] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1892] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1893] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1894] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1895] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1896] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1897] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1898] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1899] As an example of a system configured with a single processor, first, one p...

Claims

1. means for receiving image data and basic information uploaded by a user; means for detecting feature points from received image data using an image recognition engine; means for generating a 3D model based on the detected feature points; means for generating voice data and an animation script based on the generated 3D model; A means for generating 3D video by combining the generated audio data with an animation script; A means for transmitting the generated 3D video to a terminal; A system including:

2. The system of claim 1, further comprising: generating a 3D model of the great person based on the received image data.

3. The system of claim 1 , wherein the generated 3D video is encoded in a streaming format and a download format.

4. The system of claim 1, wherein the generated 3D video includes audio data in multiple languages.

5. 2. The system according to claim 1, further comprising a terminal for displaying 3D video generated in a museum or exhibition facility on a display.

Images