System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving image data selected by a user; means for receiving text data input by the user; means for analyzing the received image data and text data; means for generating a story based on an analysis result; means for generating an animation based on the generated story; and means for transmitting the generated animation to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, it is common for smartphone and tablet users to save photos and stories as digital data. However, preserving these memories in a memorable format requires advanced technology. Preserving these memories as animations creates more moving and valuable content, but creating such content on one's own can be difficult. The objective of this invention is to provide a system that allows anyone to easily create high-quality animations based on memorable photos and stories. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including a means for receiving image data selected by a user, a means for receiving text data entered by a user, a means for analyzing the received image data and text data, a means for generating a story based on the analysis results, a means for generating an animation based on the generated story, and a means for transmitting the generated animation to a user terminal.

[0006] Specifically, the generated story is based on image data analyzed using image recognition technology and text data analyzed using natural language processing technology. The generated animation is also characterized by drawing multiple scenes based on the generated story and combining them to create a series of video files. This allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge or skills.

[0007] "Image data" refers to visual information such as photographs and drawings uploaded by users.

[0008] "Text data" refers to character information such as anecdotes and descriptions about memories entered by the user.

[0009] "Means of receiving" refers to the mechanism for obtaining data from user devices such as smartphones and tablets.

[0010] "Means for analysis" refers to the technology or algorithms used to perform detailed analysis of the received data.

[0011] "Means for generating stories" refers to a method for automatically creating a series of scenarios or stories based on the results of the analysis.

[0012] "Means for generating animation" refers to a method for drawing multiple scenes based on the generated story and creating them as a video file.

[0013] "Means of sending to user terminal" refers to the mechanism that allows the generated animation to be transferred and downloaded to the user's smartphone or tablet.

[0014] "Image recognition technology" refers to technology for analyzing image data and understanding and identifying its content.

[0015] "Natural language processing technology" refers to technology for analyzing text data and understanding its meaning and context.

[0016] A "scene" refers to an individual scene depicted in a story.

[0017] "Video File" means a digital file containing a sequence of animated footage created by stitching together successive scenes. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[0040] The program processing of this system will be explained in natural language below.

[0041] Program processing overview

[0042] 1. User selection and upload of image data

[0043] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[0044] 2. User input of text data

[0045] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[0046] 3. Data transmission by the device

[0047] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[0048] 4. Data analysis by the server

[0049] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[0050] 5. Server-generated animation

[0051] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[0052] 6. Sending animation from the server to the device

[0053] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[0054] 7. User can check, save, and share animations

[0055] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[0056] Specific examples

[0057] Example 1:

[0058] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0059] 2. The device sends this data to the server.

[0060] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0061] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[0062] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0063] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The user launches the app on their smartphone or tablet and selects a few memorable photos from their photo gallery. After selecting, they click the upload button to input the image data. This action prepares the image data on the device.

[0067] Step 2:

[0068] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[0069] Step 3:

[0070] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[0071] Step 4:

[0072] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[0073] Step 5:

[0074] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[0075] Step 6:

[0076] The server automatically generates a story using a generative AI model based on the results of image and text analysis. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[0077] Step 7:

[0078] The server uses an animation generation algorithm to draw each scene based on the generated story. Each scene in the story is graphically represented and linked together to generate a series of video files. For example, a scene of the child playing in the ocean and a scene of the child's son swimming may be drawn.

[0079] Step 8:

[0080] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[0081] Step 9:

[0082] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play the animation to check it out.

[0083] Step 10:

[0084] Users can save the downloaded animations and share them via social media or messaging apps as needed. Users can use the generated animations to share their memories with others.

[0085] The above is a detailed step-by-step explanation of the system's programming process, allowing users to easily create, save, and share high-quality animations of their memories.

[0086] Example 1

[0087] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0088] In today's world, anyone can easily create high-quality animations to preserve their memories. However, conventional animation creation methods require specialized knowledge, expensive software, and time, making them inaccessible to the average user. Furthermore, users must manually perform all processes themselves, which is inefficient. To address these issues, the present invention provides a system that allows anyone to easily automatically generate high-quality animations, thereby reducing the burden on users and enabling efficient management and sharing of memories.

[0089] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0090] In this invention, the server includes means for analyzing received image data using image recognition technology, means for analyzing received text data using natural language processing technology, and means for generating a story based on the analysis results. This makes it possible to automatically generate a story based on image data selected by the user and text data entered by the user, and to create an animation by drawing multiple scenes.

[0091] A "user" is an entity that uses the system to provide image data and text data.

[0092] "Image data" is a collection of digital information in the form of still images that a user uploads to the system.

[0093] "Text data" is a collection of information in the form of strings that a user enters into a system and that the system uses to parse.

[0094] "Image recognition technology" is a technology for extracting and analyzing specific information and features from digital images.

[0095] "Natural language processing technology" is a technology for automatically analyzing and understanding text data and extracting semantic information.

[0096] A "story" is a series of stories or scenarios generated based on the analysis of image data and text data.

[0097] An "animation" is a moving image file that is generated by drawing and continuously displaying moving images based on a generated story.

[0098] A "server" is a computer system that analyzes various data in the system, creates stories and animations, and transmits data to users.

[0099] A "terminal" is a device that a user uses to access the system, and includes devices such as smartphones and tablets.

[0100] The "download link" is connection information such as a URL that allows the user to download the generated animation moving image file.

[0101] This invention is a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: the user, the terminal, and the server. Specifically, the user accesses the system using a smartphone or tablet and operates it, automatically transmitting, analyzing, generating, and receiving various data.

[0102] Hardware and software used

[0103] Device: A user device such as a smartphone or tablet. These devices run applications and provide an interface with the user.

[0104] Server: A high-performance computer system located in the cloud that receives data, analyzes it, and generates stories and animations.

[0105] Image recognition technology: Technology for extracting specific information from image data. Examples include facial recognition technology and object detection technology.

[0106] Natural language processing technology: Technology for analyzing text data and understanding its content, such as extracting emotions and key events from an episode.

[0107] Animation generation algorithm: An algorithm for creating animations based on generated stories, including drawing techniques that utilize deep learning models.

[0108] Specific operation procedures and functions

[0109] 1. User selection and upload of image data

[0110] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[0111] 2. User input of text data

[0112] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[0113] 3. Data transmission by the device

[0114] The device sends the uploaded image data and input text data to the server in one batch. This process is performed via API, and the user is notified of the success or failure of the transmission.

[0115] 4. Data analysis by the server

[0116] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[0117] 5. Server-generated animation

[0118] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[0119] 6. Sending animation from the server to the device

[0120] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[0121] 7. User can check, save, and share animations

[0122] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[0123] Specific examples

[0124] Example 1:

[0125] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0126] 2. The device sends this data to the server.

[0127] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0128] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[0129] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0130] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[0131] Prompt Sentence Examples

[0132] "This system animates family memories based on image data selected by the user and text data entered by the user. As a concrete example, imagine the process of entering photos and stories from a family trip, sending that data to a server, and analyzing it to generate an animation based on the resulting story. Please explain in detail the process."

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] The user opens the app on their smartphone or tablet. They open the photo gallery from the image selection screen and tap to select a few memorable photos. They input the selected image data and click the upload button to send the image data to the system. If the upload is successful, the selected image data is sent to the server.

[0136] Step 2:

[0137] The user moves to the text input field of the app. A guide is displayed in the input field to help the user enter an episode and description. The user enters the episode and description and presses the confirm button to check the content. The user presses the send button to send the entered text data to the system. The text data is sent to the server.

[0138] Step 3:

[0139] The device sends the uploaded image data and entered text data to the server in one batch. This process is performed via API, and the user is notified of the success or failure of the transmission. The input is the image data and text data, and the output is the transmission status to the server.

[0140] Step 4:

[0141] The server analyzes the received image data using image recognition technology. The input is the received image data, and the output is the analysis results. For example, face recognition technology can be used to identify people in a photo and detect the background and key objects.

[0142] Step 5:

[0143] The server analyzes the received text data using natural language processing technology. The input is the received text data, and the output is the analysis results. For example, key events or emotional states can be extracted.

[0144] Step 6:

[0145] The server generates a story based on the results of analyzing the image data and text data. It uses a story generation algorithm to describe each scene in detail. The input is the results of analyzing the image data and text data, and the output is the generated story.

[0146] Step 7:

[0147] The server invokes an animation generation algorithm based on the generated story. It uses a deep learning model to draw realistic animation scenes. The input is the generated story, and the output is multiple animation scenes.

[0148] Step 8:

[0149] The server renders the characters, background, and actions for each scene and connects them to generate an animation video file. The input is multiple animation scenes, and the output is a series of animation video files.

[0150] Step 9:

[0151] The server saves the generated animation video file in cloud storage. For example, Amazon S3 or Google Cloud Storage is used. The input is the animation video file, and the output is the URL of the cloud storage destination.

[0152] Step 10:

[0153] The server sends a notification to the user's device containing a download link for the animation. This link is a URL where the user can download the animation file. The input is the destination URL on the cloud, and the output is a download link notification to the user.

[0154] Step 11:

[0155] The user clicks the provided download link to download the generated animation video file to their device. The input is the download link, and the output is the downloaded animation video file.

[0156] Step 12:

[0157] Users can play downloaded animations and check their content. While playing, they can check whether the animation quality and story are as expected. If necessary, they can save the downloaded animations to their device. They can also share them with family and friends via social media or email.

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] While conventional animation generation systems are useful in generating animations based on image and text data provided by users, they have limitations on the uses of the generated animations, making them difficult to display and operate in real time. For example, there is a need for a system that can easily generate animations that can be used for product introductions and promotions in virtual stores, and display and operate them in real time.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, and means for providing an interface for viewing and manipulating animations in real time, thereby enabling users to quickly and intuitively create product introductions and promotional animations for a virtual store, and display and manipulate them in real time.

[0163] "Image data" refers to still images or photographs selected by users and uploaded to the system.

[0164] "Text data" refers to sentences or explanations entered by a user and sent to the system.

[0165] "Image recognition technology" refers to computer vision technology that analyzes received image data and understands its contents.

[0166] "Natural language processing technology" refers to artificial intelligence technology for analyzing received text data and understanding and interpreting its content.

[0167] "Story generation" refers to the process of automatically creating a narrative or scenario based on analyzed image and text data.

[0168] "Animation generation means" refers to a device or program that draws multiple scenes based on a generated story and combines them to create a continuous animation.

[0169] A "user terminal" is a device for displaying the final generated animation, and includes smart glasses, smartphones, etc.

[0170] "Interface" refers to the graphical or voice-activated environment through which a user interacts with the system and views and controls animations in real time.

[0171] "Virtual store" refers to a virtual commercial facility operated on the Internet, where users can browse and purchase products online.

[0172] "Promotional animation" refers to dynamic visual content created to promote a particular product or service.

[0173] This invention is a system that generates and displays animation in real time based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[0174] First, the user takes a photo of a product using the smart glasses and sends the image data to the system. The user then uses the voice input function to input a description and features of the product, which is then sent as text data to the system. The image data selected by the user and the input text data are then sent to the server.

[0175] The server then analyzes the received image data using "image recognition technology" (e.g., Google Cloud Vision API) to understand the content of the image, and analyzes the received text data using "natural language processing technology" (e.g., Google Cloud Natural Language API) to understand the content of the text.

[0176] The server then generates a story based on the analysis results, and generates an animation based on the story. This animation is generated by drawing multiple scenes and combining them into a series of video files.

[0177] The generated animation is sent from the server to the smart glasses, where the user is provided with an interface to view and manipulate the animation in real time. This allows the user to view the generated promotional animation in real time and adjust its content as needed.

[0178] The real-time interface provided by this system provides an environment where users can operate quickly and intuitively, making it extremely effective for promoting and introducing products in virtual stores.

[0179] For example, consider the following prompt:

[0180] "This product is versatile, lasts for extended periods of time, and its lightweight design makes it perfect for travel."

[0181] In this way, users can experience real-time, dynamic product introductions and promotions through the smart glasses. This invention enables the rapid generation of dynamic and engaging content tailored to user needs, improving the shopping experience in virtual stores.

[0182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0183] Step 1:

[0184] A user uses smart glasses to take a photo of a product. At this time, the image taken by the smart glasses' camera is acquired as "image data." This image data is sent from the terminal to the server by pressing the upload button. The input is the image data taken by the user, and the output is the image data sent to the server.

[0185] Step 2:

[0186] The user uses the voice input function of the smart glasses to input a product description by voice. The voice is converted into text and saved in the smart glasses as "text data." The saved text data is converted into text format by voice recognition software and then sent to the server. The input is voice data, the intermediate output is text data, and the final output is text data sent to the server.

[0187] Step 3:

[0188] The image data received by the server is analyzed using image recognition technology (for example, Google Cloud Vision API). The content of the image is analyzed and specific objects and features are extracted. The input is the image data sent to the server, and the output is the analyzed image content information. Specifically, the shape, color, texture, etc. of the product are identified.

[0189] Step 4:

[0190] The text data received by the server is analyzed using natural language processing technology (for example, Google Cloud Natural Language API). The semantic content of the text is understood and relevant information is extracted. The input is the text data sent to the server, and the output is the semantic information of the analyzed text. Specifically, the product's features and uses are identified.

[0191] Step 5:

[0192] The server generates a story based on the analyzed image and text data. A story generation algorithm is used to create a coherent narrative based on the received content. The input is the analysis results, and the output is the generated story. Specifically, a scenario is constructed that highlights the product's features.

[0193] Step 6:

[0194] The server generates animation based on the generated story. Using an animation generation algorithm, each scene corresponding to the story is drawn and combined into a series of video files. The input is the generated story, and the output is an animated video file. Specifically, it includes scenes that explain the features of the product.

[0195] Step 7:

[0196] The server sends the generated animation to the user's smart glasses, which then provide an interface for viewing and viewing the animation in real time. The input is an animation video file, and the output is a real-time animation displayed on the user's smart glasses. Specifically, the user can manipulate the animation through the smart glasses and modify the content as needed.

[0197] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0198] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[0199] The program processing of this system will be explained in natural language below.

[0200] Program processing overview

[0201] 1. User selection and upload of image data

[0202] The user launches the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button to input the image data. This action prepares the image data on the device.

[0203] 2. User input of text data

[0204] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[0205] 3. Data transmission by the device

[0206] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[0207] 4. Data analysis by the server

[0208] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[0209] 5. Analysis of text data by the server

[0210] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[0211] 6. Emotion Recognition by Server

[0212] The server uses an emotion engine to analyze the user's emotions from the text data and image data. Emotions are extracted from the text data, and emotions are identified from the image data by analyzing the facial expressions of people in photos. The recognized emotion information is reflected in the story generation process and animation generation process.

[0213] 7. Story generation by the server

[0214] The server automatically generates a story using a generative AI model based on the results of image analysis, text analysis, and emotion recognition. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[0215] 8. Server-based animation generation

[0216] Based on the generated story, the server uses an animation generation algorithm to draw each scene. Each scene in the story is graphically represented and linked together to generate a series of video files. The atmosphere and direction of the animation are adjusted based on the recognized emotional information. For example, if a happy emotion is recognized, bright colors and more movement are added.

[0217] 9. Sending animation from the server to the device

[0218] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[0219] 10. User can check, save and share animations

[0220] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. If desired, the user can share the animation via social media or messaging apps.

[0221] Specific examples

[0222] Example 1:

[0223] A user uploads three photos of a family trip and enters an episode such as, "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0224] The terminal transmits this data to the server.

[0225] The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0226] The server uses an emotion engine to recognize the emotion "fun" from text data and identify smiling expressions from image data.

[0227] The server generates a story based on the analysis results and emotional information, and creates animations based on that story, including scenes of the child playing in the ocean in bright colors and the child swimming.

[0228] The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0229] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[0230] The processing flow will be explained below.

[0231] Step 1:

[0232] The user launches the app on their smartphone or tablet, selects a memorable photo from their photo gallery, and then clicks the upload button to input the image data. This action imports the image data into the device.

[0233] Step 2:

[0234] The user enters an episode or description into the text input field in the app. For example, they might enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they've finished entering the information, they press the send button to send the text data.

[0235] Step 3:

[0236] The device sends the uploaded image data and entered text data to the server. The sending process is performed via API, and the data is transferred to the server. The user is notified whether the sending was successful.

[0237] Step 4:

[0238] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects, people, and scenes in the photo and generates metadata for them. For example, it extracts information such as "ocean," "beach," and "smiling person."

[0239] Step 5:

[0240] The server analyzes the received text data using natural language processing technology. It understands the content of the text and extracts key topics and keywords. For example, it can extract phrases such as "summer vacation," "family," "ocean," "son," and "first time swimming."

[0241] Step 6:

[0242] The server uses an emotion engine to analyze the user's emotions from the text data. For example, emotions such as "fun" or "happiness" are extracted. The server also analyzes the facial expressions of people in the image data and identifies their emotions.

[0243] Step 7:

[0244] The server uses a generative AI model based on image analysis, text analysis, and emotion recognition to automatically generate a story. This creates a series of scenarios that reflect the user's memories. Emotional information influences the atmosphere and content of the story.

[0245] Step 8:

[0246] The server generates animation based on the generated story. Each scene is drawn using an animation generation algorithm, and the individual scenes are linked together to generate a series of video files. Emotional information is reflected in the color tone and direction.

[0247] Step 9:

[0248] The server saves the generated animation video file and generates a download link, which is notified to the user so that the user can access it.

[0249] Step 10:

[0250] The user downloads the animation using the provided download link. Once the download is complete, the user plays the animation to check it.

[0251] Step 11:

[0252] Users can save the animations they download and share them via social media or messaging apps as needed, allowing them to share their fun memories with others.

[0253] Example 2

[0254] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] When a user generates an animation based on memorable images and their descriptions, they are required to not only generate a story from images and text, but also to automatically generate a more personalized animation that reflects the user's emotions. Conventional technologies have not fully realized comprehensive analysis, including emotion recognition, and animation generation based on that analysis, making it difficult to automatically generate high-quality animation that faithfully reflects the user's emotions.

[0256] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis result and the emotion recognition result, means for generating animation based on the generated story, and means for transmitting the generated animation to the user terminal. This enables advanced analysis that takes into account the user's emotions and automatic generation of personalized animation based on the emotions.

[0257] A "user" is a person who uses the system to input image data and text data and generate animation.

[0258] "Image data" refers to photos and image files selected and uploaded by users.

[0259] "Text data" refers to sentence data that a user inputs to describe memories or episodes.

[0260] "Means for receiving" refers to a mechanism by which the server has the function of receiving image data and text data sent from the user.

[0261] "Means for analysis" refers to a mechanism by which the server analyzes the image data and text data it receives using image recognition technology and natural language processing technology.

[0262] "Emotion recognition technology" refers to technology that allows a server to extract and analyze emotions from a user's text data and image data.

[0263] A "generative AI model" is an artificial intelligence technology for automatically generating stories based on the results of image analysis, text analysis, and emotion recognition.

[0264] "Means for generating a story" refers to the function of constructing a series of scenarios or stories based on the analysis results and emotion recognition results.

[0265] "Means for generating animation" refers to the function of drawing each scene based on the generated story, connecting them, and generating a series of video files.

[0266] "Transmitting means" refers to the function by which the server transmits the generated animation to the user's terminal.

[0267] A "user terminal" is a device used by a user to operate the system, such as a smartphone or tablet.

[0268] "Personalized animation" refers to animation that reflects the user's emotions and individual information.

[0269] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[0270] First, the user launches the dedicated app on their smartphone, tablet, or other device and selects a memorable photo from their photo gallery. Next, the user clicks the upload button to input the image data. This operation temporarily saves the selected image data on the device.

[0271] Next, the user writes down their memories or stories in the text input field within the app and presses the send button to send the text data. For example, they could enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." This entered text data is temporarily stored on the device.

[0272] Next, the device sends the selected image data and the entered text data together to the server. The sending process is performed using an API (Application Program Interface). The device monitors the status of the data sending and notifies the user of success or failure. If successful, the device notifies the user that "Data sending was successful."

[0273] The server analyzes the received image data using image recognition technology (e.g., OpenCV or TENSORFLOW (registered trademark)). The server identifies objects and scenes in each photo and stores them as metadata. For example, in the case of a photo of the sea, the server extracts information such as "sea," "beach," and "people."

[0274] The server then analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT). The server extracts key topics and keywords, such as "last summer vacation," "family," "ocean," "son," and "first swim."

[0275] Furthermore, the server uses an emotion engine (e.g., AWS® Comprehend or Azure® Sentiment Analysis) to analyze the user's emotions from the text data and image data. The server extracts emotions from the text data and identifies emotions from image data by analyzing the facial expressions of people in photos. For example, the server performs processing such as "recognizing the emotion 'fun' from the text data and recognizing that a person is smiling from the image data."

[0276] Based on these analysis results and emotion recognition results, the server automatically generates a story using a generative AI model (e.g., GPT-3 (registered trademark) or BERT). The server combines each piece of information to create a series of scenarios or stories. For example, a scenario including "a scene where the son swims for the first time in bright colors" is determined.

[0277] The server then uses an animation generation algorithm based on the generated story to draw each scene. For example, it uses Python's manim library to graphically represent the scenes, and then concatenates them to generate a series of video files. The server then adjusts the mood and direction of the animation based on the recognized emotional information.

[0278] Finally, the server saves the generated animation video file and generates a download link for it. The server then notifies the user of the download link. For example, the server may notify the user that "The animation download has been completed. Please download it from the link below."

[0279] The user can download the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. They can also share the animation via social media or messaging apps if desired.

[0280] Specific examples

[0281] For example, a user might upload three family travel photos and enter an episode such as, "These are fun memories of our family trip to the beach last summer. My son learned to swim for the first time." The device then sends this data to a server. The server then uses image recognition technology to analyze the content of the photos and natural language processing technology to analyze the text of the episode. The server then uses an emotion engine to recognize the emotion "fun" from the text data and identifies smiling expressions from the image data. The server then uses a generative AI model based on the analysis results and emotion information to automatically generate a story, and creates an animation based on that story. For example, the story might include scenes of playing in the ocean in bright colors and of the son swimming. The server then provides a download link for the generated animation, which the user can download, play, and enjoy.

[0282] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[0283] Prompt Sentence Examples

[0284] "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time. Please generate an animation based on this episode."

[0285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0286] Step 1:

[0287] The user launches the dedicated app on their smartphone or tablet.

[0288] Input: The user launches the app and is able to interact with it.

[0289] Output: The app's home screen is displayed, allowing the user to access the photo gallery.

[0290] Step 2:

[0291] The user selects several memorable photos from the photo gallery and clicks the upload button to prepare the image data on the terminal.

[0292] Input: User selects an image from the photo gallery and clicks the upload button.

[0293] Output: The selected image data is temporarily saved on the device and ready to be sent.

[0294] Step 3:

[0295] The user writes an episode or description in the text input field within the app and presses the submit button to enter the text data.

[0296] Input: The user types text into the text field and clicks the submit button.

[0297] Output: The entered text data is temporarily saved on the device.

[0298] Step 4:

[0299] The device sends the uploaded image data and input text data to the server. This process is performed using an API.

[0300] Input: Image data and text data stored on the device.

[0301] Data processing / calculation: Image data and text data are collected and sent to the server using an API.

[0302] Output: The server receives the sent data, and the device notifies the user that the transmission was successful.

[0303] Step 5:

[0304] The image data received by the server is analyzed using image recognition technology (e.g., OpenCV or TensorFlow).

[0305] Input: Image data received by the server.

[0306] Data processing / computation: Using image recognition algorithms, objects and scenes in each photo are identified and stored as metadata.

[0307] Output: The analysis results include information such as "sea," "beach," and "people."

[0308] Step 6:

[0309] The server analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT).

[0310] Input: Text data received by the server.

[0311] Data processing / computation: Use natural language processing algorithms to extract key topics and keywords.

[0312] Output: The analysis results include information such as "last summer vacation," "family," "sea," "son," and "first time swimming."

[0313] Step 7:

[0314] The server uses an emotion engine (e.g., AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from text and image data.

[0315] Input: Parsed text and image data.

[0316] Data processing / computation: Using emotion recognition algorithms, emotions are extracted from text data, and emotions are identified from image data by analyzing people's facial expressions.

[0317] Output: Emotions such as "fun" are obtained as recognized emotional information.

[0318] Step 8:

[0319] The server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT) based on the results of image analysis, text analysis, and emotion recognition.

[0320] Input: Image analysis results, text analysis results, emotion recognition results.

[0321] Data processing / computation: Using generative AI models to combine information into a series of scenarios or stories.

[0322] Output: Auto-generated stories.

[0323] Step 9:

[0324] The server uses an animation generation algorithm based on the generated story to draw each scene.

[0325] Input: Auto-generated stories.

[0326] Data processing / computation: We use Python's manim library to create a graphical representation of each scene, which is then concatenated to generate a series of video files.

[0327] Output: The generated animation video.

[0328] Step 10:

[0329] The server stores the generated animation video file, generates a download link for it, and notifies the user of it.

[0330] Input: The generated animation video file.

[0331] Data processing / calculation: Save the file and generate and notify the download link.

[0332] Output: Download link notification.

[0333] Step 11:

[0334] The user can download the generated animation using the provided download link, play it, and share it.

[0335] Input: Download link.

[0336] Output: Downloaded animations can be played, reviewed, and shared.

[0337] (Application example 2)

[0338] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] The present invention relates to a system that automatically generates animations based on image data selected by a user and text data entered by the user, and provides a personalized animation experience that reflects the user's emotions. Conventional systems have had difficulty fully understanding the user's emotions and reflecting them in the animation, and have had limitations in generating content that is in tune with the user's emotions. This can improve user satisfaction.

[0340] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis results, means for generating animation based on the generated story, means for transmitting the generated animation to the user terminal, and means for recognizing the user's emotions using an emotion engine and reflecting the emotions in the animation. This makes it possible to automatically generate high-quality animation that reflects the user's emotions and provide the user with a personalized experience that emotionally resonates with them.

[0341] The "means for receiving image data selected by a user" is a system for capturing an image file selected by a user via an input device.

[0342] The "means for receiving text data input by a user" is a system for capturing character information input by a user via an input device.

[0343] The "means for analyzing received image data and text data" refers to a technology for analyzing the captured image files and text information and extracting their contents and characteristics.

[0344] The "means for generating a story based on the analysis results" is a system for automatically constructing a series of stories or scenarios using the analysis results of image data and text data.

[0345] "Means for generating animation based on a generated story" refers to a technique for drawing multiple scenes based on a generated story or scenario and generating them as a continuous animation.

[0346] The "means for transmitting the generated animation to the user terminal" is a system for transmitting the generated animation file to the user's device via a network so that the user can view it.

[0347] "Means of recognizing the user's emotions using an emotion engine and reflecting those emotions in animation" is a technology that determines the user's emotions from input text data or image data and reflects those emotions in the atmosphere and presentation of the animation.

[0348] This invention is a system that automatically generates animations based on image data selected by a user and text data entered by the user, providing a personalized animation experience that reflects emotions. Specific embodiments are described below.

[0349] 1. User input of image data and text data

[0350] A user launches a dedicated app using a device such as a smartphone or tablet. The app includes a means for receiving image data selected by the user and a means for receiving text data entered by the user. The user selects a memorable photo from a photo gallery and enters an episode or description in the text field. For example, the user might write, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once the input is complete, the user presses the send button, and the image data and text data are sent from the device to the server.

[0351] 2. Data analysis by the server

[0352] The server analyzes the received image data and text data. Specifically, it uses image recognition technology to identify objects and scenes in each photo, and natural language processing technology to understand the content of the text data. For example, in a photo of the sea, it recognizes information such as "sea," "beach," and "people," and extracts keywords such as "last summer vacation," "family," "sea," "son," and "first time swimming." It also uses an emotion engine to analyze the user's emotions from the text data and image data. If a happy emotion is recognized, that information is also included in the analysis results.

[0353] 3. Server-based story and animation generation

[0354] The server generates a story based on the analysis results. It uses a generative AI model to automatically build a series of scenarios and narratives. Once a story that reflects the user's emotions is generated, animation is generated based on it. The animation generation algorithm draws each scene, connects them, and saves them as a series of video files. For example, if a happy emotion is recognized, bright colors and movement are added.

[0355] 4. Sending animation from the server to the device

[0356] The generated animation video file is saved on the server and a download link is generated. The user is notified and can click the download link in the app to download the generated animation to their device. The user can play and check the downloaded animation and share it via social media or messaging apps as needed.

[0357] Specific examples

[0358] For example, if a user uploads three family travel photos and enters an episode such as, "These are fun memories of our family trip to the beach last summer vacation. My son learned to swim for the first time," the server analyzes this data and generates a story. It generates animations of scenes of playing in the ocean and of the son swimming, in bright colors that reflect happy emotions, and provides the user with a download link.

[0359] Prompt Sentence Examples

[0360] Create an animated story with the following characteristics:

[0361] Features: ['ocean', 'fun']

[0362] Emotion: Fun

[0363] In this way, it is possible to provide a high-quality personalized animation experience that reflects the user's emotions.

[0364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0365] Step 1:

[0366] The device receives image data and text data from the user. The user launches the app on their smartphone or tablet, selects and uploads an image from their photo gallery, and enters an episode and description in the text field. Once the input is complete, they press the send button, and the image data and text data are temporarily saved on the device. The input data format is JPEG or PNG for images, and UTF-8 for text.

[0367] Step 2:

[0368] The device sends the received image data and text data to the server. An HTTP POST request is used for transmission, and the multipart / form-data format is used to send the image and text together. The device checks whether the transmission was successful and provides feedback to the user.

[0369] Step 3:

[0370] The server analyzes the received image and text data. First, the server uses image recognition technology to analyze the content of the image data and identify objects and scenes within each photo. Specifically, it uses libraries such as OpenCV to extract metadata such as "ocean," "beach," and "people" from the images. Next, it uses natural language processing technology to analyze the text data and extract key topics and keywords. Specifically, it uses libraries such as TextBlob and NLTK to extract keywords such as "last summer vacation," "family," "ocean," "son," and "first time swimming."

[0371] Step 4:

[0372] The server uses an emotion engine to analyze the user's emotions from text and image data. Specifically, it uses TextBlob and an emotion analysis API to determine emotions such as "happy," "sad," and "neutral" from the text data, and uses facial expression recognition technology to analyze the person's facial expressions from the image data. Emotion information is saved in JSON format and used later to generate the story.

[0373] Step 5:

[0374] The server generates a story based on the analysis results and emotional information. It uses a generative AI model (such as GPT-3) to generate a story by inputting a prompt like the following. An example of a prompt is: "Generate an animated story with the following characteristics: Features: ['ocean', 'fun'], Emotion: fun." The server saves the generated story in JSON format.

[0375] Step 6:

[0376] The server generates animations based on the generated story. An animation generation algorithm is used to draw and connect each scene. Specifically, technologies such as WebGL and Three.js are used to create animations based on the images and text for each scene. The generated animations are saved as video files in MP4 format or similar.

[0377] Step 7:

[0378] The server sends the generated animation to the user's device. The generated animation video file is stored on the server and a download link is generated. The user receives a notification within the app and can click the download link to download the animation to their device. The download link is provided securely to the user's device using the HTTPS protocol.

[0379] Step 8:

[0380] Users can check the generated animation, save it, and share it. Users can play the downloaded animation within the app and check the content. If necessary, they can share the animation via social media or messaging apps. Sharing is done using the share function within the app.

[0381] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0382] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0383] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0384] [Second embodiment]

[0385] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0386] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0387] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0388] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0389] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0390] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0391] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0392] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0393] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0394] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0395] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0396] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0397] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[0398] The program processing of this system will be explained in natural language below.

[0399] Program processing overview

[0400] 1. User selection and upload of image data

[0401] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[0402] 2. User input of text data

[0403] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[0404] 3. Data transmission by the device

[0405] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[0406] 4. Data analysis by the server

[0407] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[0408] 5. Server-generated animation

[0409] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[0410] 6. Sending animation from the server to the device

[0411] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[0412] 7. User can check, save, and share animations

[0413] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[0414] Specific examples

[0415] Example 1:

[0416] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0417] 2. The device sends this data to the server.

[0418] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0419] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[0420] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0421] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[0422] The processing flow will be explained below.

[0423] Step 1:

[0424] The user launches the app on their smartphone or tablet and selects a few memorable photos from their photo gallery. After selecting, they click the upload button to input the image data. This action prepares the image data on the device.

[0425] Step 2:

[0426] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[0427] Step 3:

[0428] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[0429] Step 4:

[0430] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[0431] Step 5:

[0432] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[0433] Step 6:

[0434] The server automatically generates a story using a generative AI model based on the results of image and text analysis. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[0435] Step 7:

[0436] The server uses an animation generation algorithm to draw each scene based on the generated story. Each scene in the story is graphically represented and linked together to generate a series of video files. For example, a scene of the child playing in the ocean and a scene of the child's son swimming may be drawn.

[0437] Step 8:

[0438] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[0439] Step 9:

[0440] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play the animation to check it out.

[0441] Step 10:

[0442] Users can save the downloaded animations and share them via social media or messaging apps as needed. Users can use the generated animations to share their memories with others.

[0443] The above is a detailed step-by-step explanation of the system's programming process, allowing users to easily create, save, and share high-quality animations of their memories.

[0444] Example 1

[0445] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0446] In today's world, anyone can easily create high-quality animations to preserve their memories. However, conventional animation creation methods require specialized knowledge, expensive software, and time, making them inaccessible to the average user. Furthermore, users must manually perform all processes themselves, which is inefficient. To address these issues, the present invention provides a system that allows anyone to easily automatically generate high-quality animations, thereby reducing the burden on users and enabling efficient management and sharing of memories.

[0447] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0448] In this invention, the server includes means for analyzing received image data using image recognition technology, means for analyzing received text data using natural language processing technology, and means for generating a story based on the analysis results. This makes it possible to automatically generate a story based on image data selected by the user and text data entered by the user, and to create an animation by drawing multiple scenes.

[0449] A "user" is an entity that uses the system to provide image data and text data.

[0450] "Image data" is a collection of digital information in the form of still images that a user uploads to the system.

[0451] "Text data" is a collection of information in the form of strings that a user enters into a system and that the system uses to parse.

[0452] "Image recognition technology" is a technology for extracting and analyzing specific information and features from digital images.

[0453] "Natural language processing technology" is a technology for automatically analyzing and understanding text data and extracting semantic information.

[0454] A "story" is a series of stories or scenarios generated based on the analysis of image data and text data.

[0455] An "animation" is a moving image file that is generated by drawing and continuously displaying moving images based on a generated story.

[0456] A "server" is a computer system that analyzes various data in the system, creates stories and animations, and transmits data to users.

[0457] A "terminal" is a device that a user uses to access the system, and includes devices such as smartphones and tablets.

[0458] The "download link" is connection information such as a URL that allows the user to download the generated animation moving image file.

[0459] This invention is a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: the user, the terminal, and the server. Specifically, the user accesses the system using a smartphone or tablet and operates it, automatically transmitting, analyzing, generating, and receiving various data.

[0460] Hardware and software used

[0461] Device: A user device such as a smartphone or tablet. These devices run applications and provide an interface with the user.

[0462] Server: A high-performance computer system located in the cloud that receives data, analyzes it, and generates stories and animations.

[0463] Image recognition technology: Technology for extracting specific information from image data. Examples include facial recognition technology and object detection technology.

[0464] Natural language processing technology: Technology for analyzing text data and understanding its content, such as extracting emotions and key events from an episode.

[0465] Animation generation algorithm: An algorithm for creating animations based on generated stories, including drawing techniques that utilize deep learning models.

[0466] Specific operation procedures and functions

[0467] 1. User selection and upload of image data

[0468] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[0469] 2. User input of text data

[0470] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[0471] 3. Data transmission by the device

[0472] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[0473] 4. Data analysis by the server

[0474] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[0475] 5. Server-generated animation

[0476] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[0477] 6. Sending animation from the server to the device

[0478] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[0479] 7. User can check, save, and share animations

[0480] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[0481] Specific examples

[0482] Example 1:

[0483] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0484] 2. The device sends this data to the server.

[0485] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0486] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[0487] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0488] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[0489] Prompt Sentence Examples

[0490] "This system animates family memories based on image data selected by the user and text data entered by the user. As a concrete example, imagine the process of entering photos and stories from a family trip, sending that data to a server, and analyzing it to generate an animation based on the resulting story. Please explain in detail the process."

[0491] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0492] Step 1:

[0493] The user opens the app on their smartphone or tablet. They open the photo gallery from the image selection screen and tap to select a few memorable photos. They input the selected image data and click the upload button to send the image data to the system. If the upload is successful, the selected image data is sent to the server.

[0494] Step 2:

[0495] The user moves to the text input field of the app. A guide is displayed in the input field to help the user enter an episode and description. The user enters the episode and description and presses the confirm button to check the content. The user presses the send button to send the entered text data to the system. The text data is sent to the server.

[0496] Step 3:

[0497] The device sends the uploaded image data and entered text data to the server in one batch. This process is performed via API, and the user is notified of the success or failure of the transmission. The input is the image data and text data, and the output is the transmission status to the server.

[0498] Step 4:

[0499] The server analyzes the received image data using image recognition technology. The input is the received image data, and the output is the analysis results. For example, face recognition technology can be used to identify people in a photo and detect the background and key objects.

[0500] Step 5:

[0501] The server analyzes the received text data using natural language processing technology. The input is the received text data, and the output is the analysis results. For example, key events or emotional states can be extracted.

[0502] Step 6:

[0503] The server generates a story based on the results of analyzing the image data and text data. It uses a story generation algorithm to describe each scene in detail. The input is the results of analyzing the image data and text data, and the output is the generated story.

[0504] Step 7:

[0505] The server invokes an animation generation algorithm based on the generated story. It uses a deep learning model to draw realistic animation scenes. The input is the generated story, and the output is multiple animation scenes.

[0506] Step 8:

[0507] The server renders the characters, background, and actions for each scene and connects them to generate an animation video file. The input is multiple animation scenes, and the output is a series of animation video files.

[0508] Step 9:

[0509] The server saves the generated animation video file in cloud storage, for example, using Amazon S3 or Google Cloud Storage. The input is the animation video file, and the output is the URL of the cloud storage location.

[0510] Step 10:

[0511] The server sends a notification to the user's device containing a download link for the animation. This link is a URL where the user can download the animation file. The input is the destination URL on the cloud, and the output is a download link notification to the user.

[0512] Step 11:

[0513] The user clicks the provided download link to download the generated animation video file to their device. The input is the download link, and the output is the downloaded animation video file.

[0514] Step 12:

[0515] Users can play downloaded animations and check their content. While playing, they can check whether the animation quality and story are as expected. If necessary, they can save the downloaded animations to their device. They can also share them with family and friends via social media or email.

[0516] (Application example 1)

[0517] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0518] While conventional animation generation systems are useful in generating animations based on image and text data provided by users, they have limitations on the uses of the generated animations, making them difficult to display and operate in real time. For example, there is a need for a system that can easily generate animations that can be used for product introductions and promotions in virtual stores, and display and operate them in real time.

[0519] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0520] In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, and means for providing an interface for viewing and manipulating animations in real time, thereby enabling users to quickly and intuitively create product introductions and promotional animations for a virtual store, and display and manipulate them in real time.

[0521] "Image data" refers to still images or photographs selected by users and uploaded to the system.

[0522] "Text data" refers to sentences or explanations entered by a user and sent to the system.

[0523] "Image recognition technology" refers to computer vision technology that analyzes received image data and understands its contents.

[0524] "Natural language processing technology" refers to artificial intelligence technology for analyzing received text data and understanding and interpreting its content.

[0525] "Story generation" refers to the process of automatically creating a narrative or scenario based on analyzed image and text data.

[0526] "Animation generation means" refers to a device or program that draws multiple scenes based on a generated story and combines them to create a continuous animation.

[0527] A "user terminal" is a device for displaying the final generated animation, and includes smart glasses, smartphones, etc.

[0528] "Interface" refers to the graphical or voice-activated environment through which a user interacts with the system and views and controls animations in real time.

[0529] "Virtual store" refers to a virtual commercial facility operated on the Internet, where users can browse and purchase products online.

[0530] "Promotional animation" refers to dynamic visual content created to promote a particular product or service.

[0531] This invention is a system that generates and displays animation in real time based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[0532] First, the user takes a photo of a product using the smart glasses and sends the image data to the system. The user then uses the voice input function to input a description and features of the product, which is then sent as text data to the system. The image data selected by the user and the input text data are then sent to the server.

[0533] The server then analyzes the received image data using "image recognition technology" (e.g., Google Cloud Vision API) to understand the content of the image, and analyzes the received text data using "natural language processing technology" (e.g., Google Cloud Natural Language API) to understand the content of the text.

[0534] The server then generates a story based on the analysis results, and generates an animation based on the story. This animation is generated by drawing multiple scenes and combining them into a series of video files.

[0535] The generated animation is sent from the server to the smart glasses, where the user is provided with an interface to view and manipulate the animation in real time. This allows the user to view the generated promotional animation in real time and adjust its content as needed.

[0536] The real-time interface provided by this system provides an environment where users can operate quickly and intuitively, making it extremely effective for promoting and introducing products in virtual stores.

[0537] For example, consider the following prompt:

[0538] "This product is versatile, lasts for extended periods of time, and its lightweight design makes it perfect for travel."

[0539] In this way, users can experience real-time, dynamic product introductions and promotions through the smart glasses. This invention enables the rapid generation of dynamic and engaging content tailored to user needs, improving the shopping experience in virtual stores.

[0540] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0541] Step 1:

[0542] A user uses smart glasses to take a photo of a product. At this time, the image taken by the smart glasses' camera is acquired as "image data." This image data is sent from the terminal to the server by pressing the upload button. The input is the image data taken by the user, and the output is the image data sent to the server.

[0543] Step 2:

[0544] The user uses the voice input function of the smart glasses to input a product description by voice. The voice is converted into text and saved in the smart glasses as "text data." The saved text data is converted into text format by voice recognition software and then sent to the server. The input is voice data, the intermediate output is text data, and the final output is text data sent to the server.

[0545] Step 3:

[0546] The image data received by the server is analyzed using image recognition technology (for example, Google Cloud Vision API). The content of the image is analyzed and specific objects and features are extracted. The input is the image data sent to the server, and the output is the analyzed image content information. Specifically, the shape, color, texture, etc. of the product are identified.

[0547] Step 4:

[0548] The text data received by the server is analyzed using natural language processing technology (for example, Google Cloud Natural Language API). The semantic content of the text is understood and relevant information is extracted. The input is the text data sent to the server, and the output is the semantic information of the analyzed text. Specifically, the product's features and uses are identified.

[0549] Step 5:

[0550] The server generates a story based on the analyzed image and text data. A story generation algorithm is used to create a coherent narrative based on the received content. The input is the analysis results, and the output is the generated story. Specifically, a scenario is constructed that highlights the product's features.

[0551] Step 6:

[0552] The server generates animation based on the generated story. Using an animation generation algorithm, each scene corresponding to the story is drawn and combined into a series of video files. The input is the generated story, and the output is an animated video file. Specifically, it includes scenes that explain the features of the product.

[0553] Step 7:

[0554] The server sends the generated animation to the user's smart glasses, which then provide an interface for viewing and viewing the animation in real time. The input is an animation video file, and the output is a real-time animation displayed on the user's smart glasses. Specifically, the user can manipulate the animation through the smart glasses and modify the content as needed.

[0555] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0556] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[0557] The program processing of this system will be explained in natural language below.

[0558] Program processing overview

[0559] 1. User selection and upload of image data

[0560] The user launches the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button to input the image data. This action prepares the image data on the device.

[0561] 2. User input of text data

[0562] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[0563] 3. Data transmission by the device

[0564] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[0565] 4. Data analysis by the server

[0566] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[0567] 5. Analysis of text data by the server

[0568] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[0569] 6. Emotion Recognition by Server

[0570] The server uses an emotion engine to analyze the user's emotions from the text data and image data. Emotions are extracted from the text data, and emotions are identified from the image data by analyzing the facial expressions of people in photos. The recognized emotion information is reflected in the story generation process and animation generation process.

[0571] 7. Story generation by the server

[0572] The server automatically generates a story using a generative AI model based on the results of image analysis, text analysis, and emotion recognition. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[0573] 8. Server-based animation generation

[0574] Based on the generated story, the server uses an animation generation algorithm to draw each scene. Each scene in the story is graphically represented and linked together to generate a series of video files. The atmosphere and direction of the animation are adjusted based on the recognized emotional information. For example, if a happy emotion is recognized, bright colors and more movement are added.

[0575] 9. Sending animation from the server to the device

[0576] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[0577] 10. User can check, save and share animations

[0578] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. If desired, the user can share the animation via social media or messaging apps.

[0579] Specific examples

[0580] Example 1:

[0581] A user uploads three photos of a family trip and enters an episode such as, "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0582] The terminal transmits this data to the server.

[0583] The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0584] The server uses an emotion engine to recognize the emotion "fun" from text data and identify smiling expressions from image data.

[0585] The server generates a story based on the analysis results and emotional information, and creates animations based on that story, including scenes of the child playing in the ocean in bright colors and the child swimming.

[0586] The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0587] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[0588] The processing flow will be explained below.

[0589] Step 1:

[0590] The user launches the app on their smartphone or tablet, selects a memorable photo from their photo gallery, and then clicks the upload button to input the image data. This action imports the image data into the device.

[0591] Step 2:

[0592] The user enters an episode or description into the text input field in the app. For example, they might enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they've finished entering the information, they press the send button to send the text data.

[0593] Step 3:

[0594] The device sends the uploaded image data and entered text data to the server. The sending process is performed via API, and the data is transferred to the server. The user is notified whether the sending was successful.

[0595] Step 4:

[0596] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects, people, and scenes in the photo and generates metadata for them. For example, it extracts information such as "ocean," "beach," and "smiling person."

[0597] Step 5:

[0598] The server analyzes the received text data using natural language processing technology. It understands the content of the text and extracts key topics and keywords. For example, it can extract phrases such as "summer vacation," "family," "ocean," "son," and "first time swimming."

[0599] Step 6:

[0600] The server uses an emotion engine to analyze the user's emotions from the text data. For example, emotions such as "fun" or "happiness" are extracted. The server also analyzes the facial expressions of people in the image data and identifies their emotions.

[0601] Step 7:

[0602] The server uses a generative AI model based on image analysis, text analysis, and emotion recognition to automatically generate a story. This creates a series of scenarios that reflect the user's memories. Emotional information influences the atmosphere and content of the story.

[0603] Step 8:

[0604] The server generates animation based on the generated story. Each scene is drawn using an animation generation algorithm, and the individual scenes are linked together to generate a series of video files. Emotional information is reflected in the color tone and direction.

[0605] Step 9:

[0606] The server saves the generated animation video file and generates a download link, which is notified to the user so that the user can access it.

[0607] Step 10:

[0608] The user downloads the animation using the provided download link. Once the download is complete, the user plays the animation to check it.

[0609] Step 11:

[0610] Users can save the animations they download and share them via social media or messaging apps as needed, allowing them to share their fun memories with others.

[0611] Example 2

[0612] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0613] When a user generates an animation based on memorable images and their descriptions, they are required to not only generate a story from images and text, but also to automatically generate a more personalized animation that reflects the user's emotions. Conventional technologies have not fully realized comprehensive analysis, including emotion recognition, and animation generation based on that analysis, making it difficult to automatically generate high-quality animation that faithfully reflects the user's emotions.

[0614] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis result and the emotion recognition result, means for generating animation based on the generated story, and means for transmitting the generated animation to the user terminal. This enables advanced analysis that takes into account the user's emotions and automatic generation of personalized animation based on the emotions.

[0615] A "user" is a person who uses the system to input image data and text data and generate animation.

[0616] "Image data" refers to photos and image files selected and uploaded by users.

[0617] "Text data" refers to sentence data that a user inputs to describe memories or episodes.

[0618] "Means for receiving" refers to a mechanism by which the server has the function of receiving image data and text data sent from the user.

[0619] "Means for analysis" refers to a mechanism by which the server analyzes the image data and text data it receives using image recognition technology and natural language processing technology.

[0620] "Emotion recognition technology" refers to technology that allows a server to extract and analyze emotions from a user's text data and image data.

[0621] A "generative AI model" is an artificial intelligence technology for automatically generating stories based on the results of image analysis, text analysis, and emotion recognition.

[0622] "Means for generating a story" refers to the function of constructing a series of scenarios or stories based on the analysis results and emotion recognition results.

[0623] "Means for generating animation" refers to the function of drawing each scene based on the generated story, connecting them, and generating a series of video files.

[0624] "Transmitting means" refers to the function by which the server transmits the generated animation to the user's terminal.

[0625] A "user terminal" is a device used by a user to operate the system, such as a smartphone or tablet.

[0626] "Personalized animation" refers to animation that reflects the user's emotions and individual information.

[0627] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[0628] First, the user launches the dedicated app on their smartphone, tablet, or other device and selects a memorable photo from their photo gallery. Next, the user clicks the upload button to input the image data. This operation temporarily saves the selected image data on the device.

[0629] Next, the user writes down their memories or stories in the text input field within the app and presses the send button to send the text data. For example, they could enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." This entered text data is temporarily stored on the device.

[0630] Next, the device sends the selected image data and the entered text data together to the server. The sending process is performed using an API (Application Program Interface). The device monitors the status of the data sending and notifies the user of success or failure. If successful, the device notifies the user that "Data sending was successful."

[0631] The server analyzes the received image data using image recognition technology (e.g., OpenCV or TensorFlow). The server identifies objects and scenes in each photo and stores them as metadata. For example, in the case of a photo of the ocean, the server extracts information such as "ocean," "beach," and "people."

[0632] The server then analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT). The server extracts key topics and keywords, such as "last summer vacation," "family," "ocean," "son," and "first swim."

[0633] Furthermore, the server uses an emotion engine (for example, AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from the text data and image data. The server extracts emotions from the text data and identifies emotions from image data by analyzing the facial expressions of people in photos. For example, the server can recognize the emotion "fun" from the text data and recognize that a person is smiling from the image data.

[0634] Based on these analysis results and emotion recognition results, the server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT). The server combines each piece of information to create a series of scenarios or stories. For example, a scenario including "a scene where the son swims for the first time in bright colors" is determined.

[0635] The server then uses an animation generation algorithm based on the generated story to draw each scene. For example, it uses Python's manim library to graphically represent the scenes, and then concatenates them to generate a series of video files. The server then adjusts the mood and direction of the animation based on the recognized emotional information.

[0636] Finally, the server saves the generated animation video file and generates a download link for it. The server then notifies the user of the download link. For example, the server may notify the user that "The animation download has been completed. Please download it from the link below."

[0637] The user can download the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. They can also share the animation via social media or messaging apps if desired.

[0638] Specific examples

[0639] For example, a user might upload three family travel photos and enter an episode such as, "These are fun memories of our family trip to the beach last summer. My son learned to swim for the first time." The device then sends this data to a server. The server then uses image recognition technology to analyze the content of the photos and natural language processing technology to analyze the text of the episode. The server then uses an emotion engine to recognize the emotion "fun" from the text data and identifies smiling expressions from the image data. The server then uses a generative AI model based on the analysis results and emotion information to automatically generate a story, and creates an animation based on that story. For example, the story might include scenes of playing in the ocean in bright colors and of the son swimming. The server then provides a download link for the generated animation, which the user can download, play, and enjoy.

[0640] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[0641] Prompt Sentence Examples

[0642] "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time. Please generate an animation based on this episode."

[0643] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0644] Step 1:

[0645] The user launches the dedicated app on their smartphone or tablet.

[0646] Input: The user launches the app and is able to interact with it.

[0647] Output: The app's home screen is displayed, allowing the user to access the photo gallery.

[0648] Step 2:

[0649] The user selects several memorable photos from the photo gallery and clicks the upload button to prepare the image data on the terminal.

[0650] Input: User selects an image from the photo gallery and clicks the upload button.

[0651] Output: The selected image data is temporarily saved on the device and ready to be sent.

[0652] Step 3:

[0653] The user writes an episode or description in the text input field within the app and presses the submit button to enter the text data.

[0654] Input: The user types text into the text field and clicks the submit button.

[0655] Output: The entered text data is temporarily saved on the device.

[0656] Step 4:

[0657] The device sends the uploaded image data and input text data to the server. This process is performed using an API.

[0658] Input: Image data and text data stored on the device.

[0659] Data processing / calculation: Image data and text data are collected and sent to the server using an API.

[0660] Output: The server receives the sent data, and the device notifies the user that the transmission was successful.

[0661] Step 5:

[0662] The image data received by the server is analyzed using image recognition technology (e.g., OpenCV or TensorFlow).

[0663] Input: Image data received by the server.

[0664] Data processing / computation: Using image recognition algorithms, objects and scenes in each photo are identified and stored as metadata.

[0665] Output: The analysis results include information such as "sea," "beach," and "people."

[0666] Step 6:

[0667] The server analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT).

[0668] Input: Text data received by the server.

[0669] Data processing / computation: Use natural language processing algorithms to extract key topics and keywords.

[0670] Output: The analysis results include information such as "last summer vacation," "family," "sea," "son," and "first time swimming."

[0671] Step 7:

[0672] The server uses an emotion engine (e.g., AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from text and image data.

[0673] Input: Parsed text and image data.

[0674] Data processing / computation: Using emotion recognition algorithms, emotions are extracted from text data, and emotions are identified from image data by analyzing people's facial expressions.

[0675] Output: Emotions such as "fun" are obtained as recognized emotional information.

[0676] Step 8:

[0677] The server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT) based on the results of image analysis, text analysis, and emotion recognition.

[0678] Input: Image analysis results, text analysis results, emotion recognition results.

[0679] Data processing / computation: Using generative AI models to combine information into a series of scenarios or stories.

[0680] Output: Auto-generated stories.

[0681] Step 9:

[0682] The server uses an animation generation algorithm based on the generated story to draw each scene.

[0683] Input: Auto-generated stories.

[0684] Data processing / computation: We use Python's manim library to create a graphical representation of each scene, which is then concatenated to generate a series of video files.

[0685] Output: The generated animation video.

[0686] Step 10:

[0687] The server stores the generated animation video file, generates a download link for it, and notifies the user of it.

[0688] Input: The generated animation video file.

[0689] Data processing / calculation: Save the file and generate and notify the download link.

[0690] Output: Download link notification.

[0691] Step 11:

[0692] The user can download the generated animation using the provided download link, play it, and share it.

[0693] Input: Download link.

[0694] Output: Downloaded animations can be played, reviewed, and shared.

[0695] (Application example 2)

[0696] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] The present invention relates to a system that automatically generates animations based on image data selected by a user and text data entered by the user, and provides a personalized animation experience that reflects the user's emotions. Conventional systems have had difficulty fully understanding the user's emotions and reflecting them in the animation, and have had limitations in generating content that is in tune with the user's emotions. This can improve user satisfaction.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis results, means for generating animation based on the generated story, means for transmitting the generated animation to the user terminal, and means for recognizing the user's emotions using an emotion engine and reflecting the emotions in the animation. This makes it possible to automatically generate high-quality animation that reflects the user's emotions and provide the user with a personalized experience that emotionally resonates with them.

[0699] The "means for receiving image data selected by a user" is a system for capturing an image file selected by a user via an input device.

[0700] The "means for receiving text data input by a user" is a system for capturing character information input by a user via an input device.

[0701] The "means for analyzing received image data and text data" refers to a technology for analyzing the captured image files and text information and extracting their contents and characteristics.

[0702] The "means for generating a story based on the analysis results" is a system for automatically constructing a series of stories or scenarios using the analysis results of image data and text data.

[0703] "Means for generating animation based on a generated story" refers to a technique for drawing multiple scenes based on a generated story or scenario and generating them as a continuous animation.

[0704] The "means for transmitting the generated animation to the user terminal" is a system for transmitting the generated animation file to the user's device via a network so that the user can view it.

[0705] "Means of recognizing the user's emotions using an emotion engine and reflecting those emotions in animation" is a technology that determines the user's emotions from input text data or image data and reflects those emotions in the atmosphere and presentation of the animation.

[0706] This invention is a system that automatically generates animations based on image data selected by a user and text data entered by the user, providing a personalized animation experience that reflects emotions. Specific embodiments are described below.

[0707] 1. User input of image data and text data

[0708] A user launches a dedicated app using a device such as a smartphone or tablet. The app includes a means for receiving image data selected by the user and a means for receiving text data entered by the user. The user selects a memorable photo from a photo gallery and enters an episode or description in the text field. For example, the user might write, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once the input is complete, the user presses the send button, and the image data and text data are sent from the device to the server.

[0709] 2. Data analysis by the server

[0710] The server analyzes the received image data and text data. Specifically, it uses image recognition technology to identify objects and scenes in each photo, and natural language processing technology to understand the content of the text data. For example, in a photo of the sea, it recognizes information such as "sea," "beach," and "people," and extracts keywords such as "last summer vacation," "family," "sea," "son," and "first time swimming." It also uses an emotion engine to analyze the user's emotions from the text data and image data. If a happy emotion is recognized, that information is also included in the analysis results.

[0711] 3. Server-based story and animation generation

[0712] The server generates a story based on the analysis results. It uses a generative AI model to automatically build a series of scenarios and narratives. Once a story that reflects the user's emotions is generated, animation is generated based on it. The animation generation algorithm draws each scene, connects them, and saves them as a series of video files. For example, if a happy emotion is recognized, bright colors and movement are added.

[0713] 4. Sending animation from the server to the device

[0714] The generated animation video file is saved on the server and a download link is generated. The user is notified and can click the download link in the app to download the generated animation to their device. The user can play and check the downloaded animation and share it via social media or messaging apps as needed.

[0715] Specific examples

[0716] For example, if a user uploads three family travel photos and enters an episode such as, "These are fun memories of our family trip to the beach last summer vacation. My son learned to swim for the first time," the server analyzes this data and generates a story. It generates animations of scenes of playing in the ocean and of the son swimming, in bright colors that reflect happy emotions, and provides the user with a download link.

[0717] Prompt Sentence Examples

[0718] Create an animated story with the following characteristics:

[0719] Features: ['ocean', 'fun']

[0720] Emotion: Fun

[0721] In this way, it is possible to provide a high-quality personalized animation experience that reflects the user's emotions.

[0722] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0723] Step 1:

[0724] The device receives image data and text data from the user. The user launches the app on their smartphone or tablet, selects and uploads an image from their photo gallery, and enters an episode and description in the text field. Once the input is complete, they press the send button, and the image data and text data are temporarily saved on the device. The input data format is JPEG or PNG for images, and UTF-8 for text.

[0725] Step 2:

[0726] The device sends the received image data and text data to the server. An HTTP POST request is used for transmission, and the multipart / form-data format is used to send the image and text together. The device checks whether the transmission was successful and provides feedback to the user.

[0727] Step 3:

[0728] The server analyzes the received image and text data. First, the server uses image recognition technology to analyze the content of the image data and identify objects and scenes within each photo. Specifically, it uses libraries such as OpenCV to extract metadata such as "ocean," "beach," and "people" from the images. Next, it uses natural language processing technology to analyze the text data and extract key topics and keywords. Specifically, it uses libraries such as TextBlob and NLTK to extract keywords such as "last summer vacation," "family," "ocean," "son," and "first time swimming."

[0729] Step 4:

[0730] The server uses an emotion engine to analyze the user's emotions from text and image data. Specifically, it uses TextBlob and an emotion analysis API to determine emotions such as "happy," "sad," and "neutral" from the text data, and uses facial expression recognition technology to analyze the person's facial expressions from the image data. Emotion information is saved in JSON format and used later to generate the story.

[0731] Step 5:

[0732] The server generates a story based on the analysis results and emotional information. It uses a generative AI model (such as GPT-3) to generate a story by inputting a prompt like the following. An example of a prompt is: "Generate an animated story with the following characteristics: Features: ['ocean', 'fun'], Emotion: fun." The server saves the generated story in JSON format.

[0733] Step 6:

[0734] The server generates animations based on the generated story. An animation generation algorithm is used to draw and connect each scene. Specifically, technologies such as WebGL and Three.js are used to create animations based on the images and text for each scene. The generated animations are saved as video files in MP4 format or similar.

[0735] Step 7:

[0736] The server sends the generated animation to the user's device. The generated animation video file is stored on the server and a download link is generated. The user receives a notification within the app and can click the download link to download the animation to their device. The download link is provided securely to the user's device using the HTTPS protocol.

[0737] Step 8:

[0738] Users can check the generated animation, save it, and share it. Users can play the downloaded animation within the app and check the content. If necessary, they can share the animation via social media or messaging apps. Sharing is done using the share function within the app.

[0739] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0740] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0741] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0742] [Third embodiment]

[0743] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0744] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0745] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0746] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0747] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0748] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0749] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0750] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0751] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0752] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0753] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0754] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0755] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[0756] The program processing of this system will be explained in natural language below.

[0757] Program processing overview

[0758] 1. User selection and upload of image data

[0759] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[0760] 2. User input of text data

[0761] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[0762] 3. Data transmission by the device

[0763] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[0764] 4. Data analysis by the server

[0765] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[0766] 5. Server-generated animation

[0767] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[0768] 6. Sending animation from the server to the device

[0769] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[0770] 7. User can check, save, and share animations

[0771] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[0772] Specific examples

[0773] Example 1:

[0774] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0775] 2. The device sends this data to the server.

[0776] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0777] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[0778] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0779] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[0780] The processing flow will be explained below.

[0781] Step 1:

[0782] The user launches the app on their smartphone or tablet and selects a few memorable photos from their photo gallery. After selecting, they click the upload button to input the image data. This action prepares the image data on the device.

[0783] Step 2:

[0784] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[0785] Step 3:

[0786] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[0787] Step 4:

[0788] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[0789] Step 5:

[0790] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[0791] Step 6:

[0792] The server automatically generates a story using a generative AI model based on the results of image and text analysis. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[0793] Step 7:

[0794] The server uses an animation generation algorithm to draw each scene based on the generated story. Each scene in the story is graphically represented and linked together to generate a series of video files. For example, a scene of the child playing in the ocean and a scene of the child's son swimming may be drawn.

[0795] Step 8:

[0796] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[0797] Step 9:

[0798] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play the animation to check it out.

[0799] Step 10:

[0800] Users can save the downloaded animations and share them via social media or messaging apps as needed. Users can use the generated animations to share their memories with others.

[0801] The above is a detailed step-by-step explanation of the system's programming process, allowing users to easily create, save, and share high-quality animations of their memories.

[0802] Example 1

[0803] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0804] In today's world, anyone can easily create high-quality animations to preserve their memories. However, conventional animation creation methods require specialized knowledge, expensive software, and time, making them inaccessible to the average user. Furthermore, users must manually perform all processes themselves, which is inefficient. To address these issues, the present invention provides a system that allows anyone to easily automatically generate high-quality animations, thereby reducing the burden on users and enabling efficient management and sharing of memories.

[0805] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0806] In this invention, the server includes means for analyzing received image data using image recognition technology, means for analyzing received text data using natural language processing technology, and means for generating a story based on the analysis results. This makes it possible to automatically generate a story based on image data selected by the user and text data entered by the user, and to create an animation by drawing multiple scenes.

[0807] A "user" is an entity that uses the system to provide image data and text data.

[0808] "Image data" is a collection of digital information in the form of still images that a user uploads to the system.

[0809] "Text data" is a collection of information in the form of strings that a user enters into a system and that the system uses to parse.

[0810] "Image recognition technology" is a technology for extracting and analyzing specific information and features from digital images.

[0811] "Natural language processing technology" is a technology for automatically analyzing and understanding text data and extracting semantic information.

[0812] A "story" is a series of stories or scenarios generated based on the analysis of image data and text data.

[0813] An "animation" is a moving image file that is generated by drawing and continuously displaying moving images based on a generated story.

[0814] A "server" is a computer system that analyzes various data in the system, creates stories and animations, and transmits data to users.

[0815] A "terminal" is a device that a user uses to access the system, and includes devices such as smartphones and tablets.

[0816] The "download link" is connection information such as a URL that allows the user to download the generated animation moving image file.

[0817] This invention is a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: the user, the terminal, and the server. Specifically, the user accesses the system using a smartphone or tablet and operates it, automatically transmitting, analyzing, generating, and receiving various data.

[0818] Hardware and software used

[0819] Device: A user device such as a smartphone or tablet. These devices run applications and provide an interface with the user.

[0820] Server: A high-performance computer system located in the cloud that receives data, analyzes it, and generates stories and animations.

[0821] Image recognition technology: Technology for extracting specific information from image data. Examples include facial recognition technology and object detection technology.

[0822] Natural language processing technology: Technology for analyzing text data and understanding its content, such as extracting emotions and key events from an episode.

[0823] Animation generation algorithm: An algorithm for creating animations based on generated stories, including drawing techniques that utilize deep learning models.

[0824] Specific operation procedures and functions

[0825] 1. User selection and upload of image data

[0826] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[0827] 2. User input of text data

[0828] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[0829] 3. Data transmission by the device

[0830] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[0831] 4. Data analysis by the server

[0832] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[0833] 5. Server-generated animation

[0834] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[0835] 6. Sending animation from the server to the device

[0836] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[0837] 7. User can check, save, and share animations

[0838] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[0839] Specific examples

[0840] Example 1:

[0841] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0842] 2. The device sends this data to the server.

[0843] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0844] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[0845] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0846] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[0847] Prompt Sentence Examples

[0848] "This system animates family memories based on image data selected by the user and text data entered by the user. As a concrete example, imagine the process of entering photos and stories from a family trip, sending that data to a server, and analyzing it to generate an animation based on the resulting story. Please explain in detail the process."

[0849] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0850] Step 1:

[0851] The user opens the app on their smartphone or tablet. They open the photo gallery from the image selection screen and tap to select a few memorable photos. They input the selected image data and click the upload button to send the image data to the system. If the upload is successful, the selected image data is sent to the server.

[0852] Step 2:

[0853] The user moves to the text input field of the app. A guide is displayed in the input field to help the user enter an episode and description. The user enters the episode and description and presses the confirm button to check the content. The user presses the send button to send the entered text data to the system. The text data is sent to the server.

[0854] Step 3:

[0855] The device sends the uploaded image data and entered text data to the server in one batch. This process is performed via API, and the user is notified of the success or failure of the transmission. The input is the image data and text data, and the output is the transmission status to the server.

[0856] Step 4:

[0857] The server analyzes the received image data using image recognition technology. The input is the received image data, and the output is the analysis results. For example, face recognition technology can be used to identify people in a photo and detect the background and key objects.

[0858] Step 5:

[0859] The server analyzes the received text data using natural language processing technology. The input is the received text data, and the output is the analysis results. For example, key events or emotional states can be extracted.

[0860] Step 6:

[0861] The server generates a story based on the results of analyzing the image data and text data. It uses a story generation algorithm to describe each scene in detail. The input is the results of analyzing the image data and text data, and the output is the generated story.

[0862] Step 7:

[0863] The server invokes an animation generation algorithm based on the generated story. It uses a deep learning model to draw realistic animation scenes. The input is the generated story, and the output is multiple animation scenes.

[0864] Step 8:

[0865] The server renders the characters, background, and actions for each scene and connects them to generate an animation video file. The input is multiple animation scenes, and the output is a series of animation video files.

[0866] Step 9:

[0867] The server saves the generated animation video file in cloud storage, for example, using Amazon S3 or Google Cloud Storage. The input is the animation video file, and the output is the URL of the cloud storage location.

[0868] Step 10:

[0869] The server sends a notification to the user's device containing a download link for the animation. This link is a URL where the user can download the animation file. The input is the destination URL on the cloud, and the output is a download link notification to the user.

[0870] Step 11:

[0871] The user clicks the provided download link to download the generated animation video file to their device. The input is the download link, and the output is the downloaded animation video file.

[0872] Step 12:

[0873] Users can play downloaded animations and check their content. While playing, they can check whether the animation quality and story are as expected. If necessary, they can save the downloaded animations to their device. They can also share them with family and friends via social media or email.

[0874] (Application example 1)

[0875] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0876] While conventional animation generation systems are useful in generating animations based on image and text data provided by users, they have limitations on the uses of the generated animations, making them difficult to display and operate in real time. For example, there is a need for a system that can easily generate animations that can be used for product introductions and promotions in virtual stores, and display and operate them in real time.

[0877] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0878] In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, and means for providing an interface for viewing and manipulating animations in real time, thereby enabling users to quickly and intuitively create product introductions and promotional animations for a virtual store, and display and manipulate them in real time.

[0879] "Image data" refers to still images or photographs selected by users and uploaded to the system.

[0880] "Text data" refers to sentences or explanations entered by a user and sent to the system.

[0881] "Image recognition technology" refers to computer vision technology that analyzes received image data and understands its contents.

[0882] "Natural language processing technology" refers to artificial intelligence technology for analyzing received text data and understanding and interpreting its content.

[0883] "Story generation" refers to the process of automatically creating a narrative or scenario based on analyzed image and text data.

[0884] "Animation generation means" refers to a device or program that draws multiple scenes based on a generated story and combines them to create a continuous animation.

[0885] A "user terminal" is a device for displaying the final generated animation, and includes smart glasses, smartphones, etc.

[0886] "Interface" refers to the graphical or voice-activated environment through which a user interacts with the system and views and controls animations in real time.

[0887] "Virtual store" refers to a virtual commercial facility operated on the Internet, where users can browse and purchase products online.

[0888] "Promotional animation" refers to dynamic visual content created to promote a particular product or service.

[0889] This invention is a system that generates and displays animation in real time based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[0890] First, the user takes a photo of a product using the smart glasses and sends the image data to the system. The user then uses the voice input function to input a description and features of the product, which is then sent as text data to the system. The image data selected by the user and the input text data are then sent to the server.

[0891] The server then analyzes the received image data using "image recognition technology" (e.g., Google Cloud Vision API) to understand the content of the image, and analyzes the received text data using "natural language processing technology" (e.g., Google Cloud Natural Language API) to understand the content of the text.

[0892] The server then generates a story based on the analysis results, and generates an animation based on the story. This animation is generated by drawing multiple scenes and combining them into a series of video files.

[0893] The generated animation is sent from the server to the smart glasses, where the user is provided with an interface to view and manipulate the animation in real time. This allows the user to view the generated promotional animation in real time and adjust its content as needed.

[0894] The real-time interface provided by this system provides an environment where users can operate quickly and intuitively, making it extremely effective for promoting and introducing products in virtual stores.

[0895] For example, consider the following prompt:

[0896] "This product is versatile, lasts for extended periods of time, and its lightweight design makes it perfect for travel."

[0897] In this way, users can experience real-time, dynamic product introductions and promotions through the smart glasses. This invention enables the rapid generation of dynamic and engaging content tailored to user needs, improving the shopping experience in virtual stores.

[0898] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0899] Step 1:

[0900] A user uses smart glasses to take a photo of a product. At this time, the image taken by the smart glasses' camera is acquired as "image data." This image data is sent from the terminal to the server by pressing the upload button. The input is the image data taken by the user, and the output is the image data sent to the server.

[0901] Step 2:

[0902] The user uses the voice input function of the smart glasses to input a product description by voice. The voice is converted into text and saved in the smart glasses as "text data." The saved text data is converted into text format by voice recognition software and then sent to the server. The input is voice data, the intermediate output is text data, and the final output is text data sent to the server.

[0903] Step 3:

[0904] The image data received by the server is analyzed using image recognition technology (for example, Google Cloud Vision API). The content of the image is analyzed and specific objects and features are extracted. The input is the image data sent to the server, and the output is the analyzed image content information. Specifically, the shape, color, texture, etc. of the product are identified.

[0905] Step 4:

[0906] The text data received by the server is analyzed using natural language processing technology (for example, Google Cloud Natural Language API). The semantic content of the text is understood and relevant information is extracted. The input is the text data sent to the server, and the output is the semantic information of the analyzed text. Specifically, the product's features and uses are identified.

[0907] Step 5:

[0908] The server generates a story based on the analyzed image and text data. A story generation algorithm is used to create a coherent narrative based on the received content. The input is the analysis results, and the output is the generated story. Specifically, a scenario is constructed that highlights the product's features.

[0909] Step 6:

[0910] The server generates animation based on the generated story. Using an animation generation algorithm, each scene corresponding to the story is drawn and combined into a series of video files. The input is the generated story, and the output is an animated video file. Specifically, it includes scenes that explain the features of the product.

[0911] Step 7:

[0912] The server sends the generated animation to the user's smart glasses, which then provide an interface for viewing and viewing the animation in real time. The input is an animation video file, and the output is a real-time animation displayed on the user's smart glasses. Specifically, the user can manipulate the animation through the smart glasses and modify the content as needed.

[0913] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0914] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[0915] The program processing of this system will be explained in natural language below.

[0916] Program processing overview

[0917] 1. User selection and upload of image data

[0918] The user launches the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button to input the image data. This action prepares the image data on the device.

[0919] 2. User input of text data

[0920] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[0921] 3. Data transmission by the device

[0922] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[0923] 4. Data analysis by the server

[0924] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[0925] 5. Analysis of text data by the server

[0926] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[0927] 6. Emotion Recognition by Server

[0928] The server uses an emotion engine to analyze the user's emotions from the text data and image data. Emotions are extracted from the text data, and emotions are identified from the image data by analyzing the facial expressions of people in photos. The recognized emotion information is reflected in the story generation process and animation generation process.

[0929] 7. Story generation by the server

[0930] The server automatically generates a story using a generative AI model based on the results of image analysis, text analysis, and emotion recognition. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[0931] 8. Server-based animation generation

[0932] Based on the generated story, the server uses an animation generation algorithm to draw each scene. Each scene in the story is graphically represented and linked together to generate a series of video files. The atmosphere and direction of the animation are adjusted based on the recognized emotional information. For example, if a happy emotion is recognized, bright colors and more movement are added.

[0933] 9. Sending animation from the server to the device

[0934] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[0935] 10. User can check, save and share animations

[0936] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. If desired, the user can share the animation via social media or messaging apps.

[0937] Specific examples

[0938] Example 1:

[0939] A user uploads three photos of a family trip and enters an episode such as, "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[0940] The terminal transmits this data to the server.

[0941] The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[0942] The server uses an emotion engine to recognize the emotion "fun" from text data and identify smiling expressions from image data.

[0943] The server generates a story based on the analysis results and emotional information, and creates animations based on that story, including scenes of the child playing in the ocean in bright colors and the child swimming.

[0944] The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[0945] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[0946] The processing flow will be explained below.

[0947] Step 1:

[0948] The user launches the app on their smartphone or tablet, selects a memorable photo from their photo gallery, and then clicks the upload button to input the image data. This action imports the image data into the device.

[0949] Step 2:

[0950] The user enters an episode or description into the text input field in the app. For example, they might enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they've finished entering the information, they press the send button to send the text data.

[0951] Step 3:

[0952] The device sends the uploaded image data and entered text data to the server. The sending process is performed via API, and the data is transferred to the server. The user is notified whether the sending was successful.

[0953] Step 4:

[0954] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects, people, and scenes in the photo and generates metadata for them. For example, it extracts information such as "ocean," "beach," and "smiling person."

[0955] Step 5:

[0956] The server analyzes the received text data using natural language processing technology. It understands the content of the text and extracts key topics and keywords. For example, it can extract phrases such as "summer vacation," "family," "ocean," "son," and "first time swimming."

[0957] Step 6:

[0958] The server uses an emotion engine to analyze the user's emotions from the text data. For example, emotions such as "fun" or "happiness" are extracted. The server also analyzes the facial expressions of people in the image data and identifies their emotions.

[0959] Step 7:

[0960] The server uses a generative AI model based on image analysis, text analysis, and emotion recognition to automatically generate a story. This creates a series of scenarios that reflect the user's memories. Emotional information influences the atmosphere and content of the story.

[0961] Step 8:

[0962] The server generates animation based on the generated story. Each scene is drawn using an animation generation algorithm, and the individual scenes are linked together to generate a series of video files. Emotional information is reflected in the color tone and direction.

[0963] Step 9:

[0964] The server saves the generated animation video file and generates a download link, which is notified to the user so that the user can access it.

[0965] Step 10:

[0966] The user downloads the animation using the provided download link. Once the download is complete, the user plays the animation to check it.

[0967] Step 11:

[0968] Users can save the animations they download and share them via social media or messaging apps as needed, allowing them to share their fun memories with others.

[0969] Example 2

[0970] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0971] When a user generates an animation based on memorable images and their descriptions, they are required to not only generate a story from images and text, but also to automatically generate a more personalized animation that reflects the user's emotions. Conventional technologies have not fully realized comprehensive analysis, including emotion recognition, and animation generation based on that analysis, making it difficult to automatically generate high-quality animation that faithfully reflects the user's emotions.

[0972] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis result and the emotion recognition result, means for generating animation based on the generated story, and means for transmitting the generated animation to the user terminal. This enables advanced analysis that takes into account the user's emotions and automatic generation of personalized animation based on the emotions.

[0973] A "user" is a person who uses the system to input image data and text data and generate animation.

[0974] "Image data" refers to photos and image files selected and uploaded by users.

[0975] "Text data" refers to sentence data that a user inputs to describe memories or episodes.

[0976] "Means for receiving" refers to a mechanism by which the server has the function of receiving image data and text data sent from the user.

[0977] "Means for analysis" refers to a mechanism by which the server analyzes the image data and text data it receives using image recognition technology and natural language processing technology.

[0978] "Emotion recognition technology" refers to technology that allows a server to extract and analyze emotions from a user's text data and image data.

[0979] A "generative AI model" is an artificial intelligence technology for automatically generating stories based on the results of image analysis, text analysis, and emotion recognition.

[0980] "Means for generating a story" refers to the function of constructing a series of scenarios or stories based on the analysis results and emotion recognition results.

[0981] "Means for generating animation" refers to the function of drawing each scene based on the generated story, connecting them, and generating a series of video files.

[0982] "Transmitting means" refers to the function by which the server transmits the generated animation to the user's terminal.

[0983] A "user terminal" is a device used by a user to operate the system, such as a smartphone or tablet.

[0984] "Personalized animation" refers to animation that reflects the user's emotions and individual information.

[0985] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[0986] First, the user launches the dedicated app on their smartphone, tablet, or other device and selects a memorable photo from their photo gallery. Next, the user clicks the upload button to input the image data. This operation temporarily saves the selected image data on the device.

[0987] Next, the user writes down their memories or stories in the text input field within the app and presses the send button to send the text data. For example, they could enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." This entered text data is temporarily stored on the device.

[0988] Next, the device sends the selected image data and the entered text data together to the server. The sending process is performed using an API (Application Program Interface). The device monitors the status of the data sending and notifies the user of success or failure. If successful, the device notifies the user that "Data sending was successful."

[0989] The server analyzes the received image data using image recognition technology (e.g., OpenCV or TensorFlow). The server identifies objects and scenes in each photo and stores them as metadata. For example, in the case of a photo of the ocean, the server extracts information such as "ocean," "beach," and "people."

[0990] The server then analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT). The server extracts key topics and keywords, such as "last summer vacation," "family," "ocean," "son," and "first swim."

[0991] Furthermore, the server uses an emotion engine (for example, AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from the text data and image data. The server extracts emotions from the text data and identifies emotions from image data by analyzing the facial expressions of people in photos. For example, the server can recognize the emotion "fun" from the text data and recognize that a person is smiling from the image data.

[0992] Based on these analysis results and emotion recognition results, the server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT). The server combines each piece of information to create a series of scenarios or stories. For example, a scenario including "a scene where the son swims for the first time in bright colors" is determined.

[0993] The server then uses an animation generation algorithm based on the generated story to draw each scene. For example, it uses Python's manim library to graphically represent the scenes, and then concatenates them to generate a series of video files. The server then adjusts the mood and direction of the animation based on the recognized emotional information.

[0994] Finally, the server saves the generated animation video file and generates a download link for it. The server then notifies the user of the download link. For example, the server may notify the user that "The animation download has been completed. Please download it from the link below."

[0995] The user can download the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. They can also share the animation via social media or messaging apps if desired.

[0996] Specific examples

[0997] For example, a user might upload three family travel photos and enter an episode such as, "These are fun memories of our family trip to the beach last summer. My son learned to swim for the first time." The device then sends this data to a server. The server then uses image recognition technology to analyze the content of the photos and natural language processing technology to analyze the text of the episode. The server then uses an emotion engine to recognize the emotion "fun" from the text data and identifies smiling expressions from the image data. The server then uses a generative AI model based on the analysis results and emotion information to automatically generate a story, and creates an animation based on that story. For example, the story might include scenes of playing in the ocean in bright colors and of the son swimming. The server then provides a download link for the generated animation, which the user can download, play, and enjoy.

[0998] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[0999] Prompt Sentence Examples

[1000] "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time. Please generate an animation based on this episode."

[1001] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1002] Step 1:

[1003] The user launches the dedicated app on their smartphone or tablet.

[1004] Input: The user launches the app and is able to interact with it.

[1005] Output: The app's home screen is displayed, allowing the user to access the photo gallery.

[1006] Step 2:

[1007] The user selects several memorable photos from the photo gallery and clicks the upload button to prepare the image data on the terminal.

[1008] Input: User selects an image from the photo gallery and clicks the upload button.

[1009] Output: The selected image data is temporarily saved on the device and ready to be sent.

[1010] Step 3:

[1011] The user writes an episode or description in the text input field within the app and presses the submit button to enter the text data.

[1012] Input: The user types text into the text field and clicks the submit button.

[1013] Output: The entered text data is temporarily saved on the device.

[1014] Step 4:

[1015] The device sends the uploaded image data and input text data to the server. This process is performed using an API.

[1016] Input: Image data and text data stored on the device.

[1017] Data processing / calculation: Image data and text data are collected and sent to the server using an API.

[1018] Output: The server receives the sent data, and the device notifies the user that the transmission was successful.

[1019] Step 5:

[1020] The image data received by the server is analyzed using image recognition technology (e.g., OpenCV or TensorFlow).

[1021] Input: Image data received by the server.

[1022] Data processing / computation: Using image recognition algorithms, objects and scenes in each photo are identified and stored as metadata.

[1023] Output: The analysis results include information such as "sea," "beach," and "people."

[1024] Step 6:

[1025] The server analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT).

[1026] Input: Text data received by the server.

[1027] Data processing / computation: Use natural language processing algorithms to extract key topics and keywords.

[1028] Output: The analysis results include information such as "last summer vacation," "family," "sea," "son," and "first time swimming."

[1029] Step 7:

[1030] The server uses an emotion engine (e.g., AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from text and image data.

[1031] Input: Parsed text and image data.

[1032] Data processing / computation: Using emotion recognition algorithms, emotions are extracted from text data, and emotions are identified from image data by analyzing people's facial expressions.

[1033] Output: Emotions such as "fun" are obtained as recognized emotional information.

[1034] Step 8:

[1035] The server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT) based on the results of image analysis, text analysis, and emotion recognition.

[1036] Input: Image analysis results, text analysis results, emotion recognition results.

[1037] Data processing / computation: Using generative AI models to combine information into a series of scenarios or stories.

[1038] Output: Auto-generated stories.

[1039] Step 9:

[1040] The server uses an animation generation algorithm based on the generated story to draw each scene.

[1041] Input: Auto-generated stories.

[1042] Data processing / computation: We use Python's manim library to create a graphical representation of each scene, which is then concatenated to generate a series of video files.

[1043] Output: The generated animation video.

[1044] Step 10:

[1045] The server stores the generated animation video file, generates a download link for it, and notifies the user of it.

[1046] Input: The generated animation video file.

[1047] Data processing / calculation: Save the file and generate and notify the download link.

[1048] Output: Download link notification.

[1049] Step 11:

[1050] The user can download the generated animation using the provided download link, play it, and share it.

[1051] Input: Download link.

[1052] Output: Downloaded animations can be played, reviewed, and shared.

[1053] (Application example 2)

[1054] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1055] The present invention relates to a system that automatically generates animations based on image data selected by a user and text data entered by the user, and provides a personalized animation experience that reflects the user's emotions. Conventional systems have had difficulty fully understanding the user's emotions and reflecting them in the animation, and have had limitations in generating content that is in tune with the user's emotions. This can improve user satisfaction.

[1056] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis results, means for generating animation based on the generated story, means for transmitting the generated animation to the user terminal, and means for recognizing the user's emotions using an emotion engine and reflecting the emotions in the animation. This makes it possible to automatically generate high-quality animation that reflects the user's emotions and provide the user with a personalized experience that emotionally resonates with them.

[1057] The "means for receiving image data selected by a user" is a system for capturing an image file selected by a user via an input device.

[1058] The "means for receiving text data input by a user" is a system for capturing character information input by a user via an input device.

[1059] The "means for analyzing received image data and text data" refers to a technology for analyzing the captured image files and text information and extracting their contents and characteristics.

[1060] The "means for generating a story based on the analysis results" is a system for automatically constructing a series of stories or scenarios using the analysis results of image data and text data.

[1061] "Means for generating animation based on a generated story" refers to a technique for drawing multiple scenes based on a generated story or scenario and generating them as a continuous animation.

[1062] The "means for transmitting the generated animation to the user terminal" is a system for transmitting the generated animation file to the user's device via a network so that the user can view it.

[1063] "Means of recognizing the user's emotions using an emotion engine and reflecting those emotions in animation" is a technology that determines the user's emotions from input text data or image data and reflects those emotions in the atmosphere and presentation of the animation.

[1064] This invention is a system that automatically generates animations based on image data selected by a user and text data entered by the user, providing a personalized animation experience that reflects emotions. Specific embodiments are described below.

[1065] 1. User input of image data and text data

[1066] A user launches a dedicated app using a device such as a smartphone or tablet. The app includes a means for receiving image data selected by the user and a means for receiving text data entered by the user. The user selects a memorable photo from a photo gallery and enters an episode or description in the text field. For example, the user might write, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once the input is complete, the user presses the send button, and the image data and text data are sent from the device to the server.

[1067] 2. Data analysis by the server

[1068] The server analyzes the received image data and text data. Specifically, it uses image recognition technology to identify objects and scenes in each photo, and natural language processing technology to understand the content of the text data. For example, in a photo of the sea, it recognizes information such as "sea," "beach," and "people," and extracts keywords such as "last summer vacation," "family," "sea," "son," and "first time swimming." It also uses an emotion engine to analyze the user's emotions from the text data and image data. If a happy emotion is recognized, that information is also included in the analysis results.

[1069] 3. Server-based story and animation generation

[1070] The server generates a story based on the analysis results. It uses a generative AI model to automatically build a series of scenarios and narratives. Once a story that reflects the user's emotions is generated, animation is generated based on it. The animation generation algorithm draws each scene, connects them, and saves them as a series of video files. For example, if a happy emotion is recognized, bright colors and movement are added.

[1071] 4. Sending animation from the server to the device

[1072] The generated animation video file is saved on the server and a download link is generated. The user is notified and can click the download link in the app to download the generated animation to their device. The user can play and check the downloaded animation and share it via social media or messaging apps as needed.

[1073] Specific examples

[1074] For example, if a user uploads three family travel photos and enters an episode such as, "These are fun memories of our family trip to the beach last summer vacation. My son learned to swim for the first time," the server analyzes this data and generates a story. It generates animations of scenes of playing in the ocean and of the son swimming, in bright colors that reflect happy emotions, and provides the user with a download link.

[1075] Prompt Sentence Examples

[1076] Create an animated story with the following characteristics:

[1077] Features: ['ocean', 'fun']

[1078] Emotion: Fun

[1079] In this way, it is possible to provide a high-quality personalized animation experience that reflects the user's emotions.

[1080] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1081] Step 1:

[1082] The device receives image data and text data from the user. The user launches the app on their smartphone or tablet, selects and uploads an image from their photo gallery, and enters an episode and description in the text field. Once the input is complete, they press the send button, and the image data and text data are temporarily saved on the device. The input data format is JPEG or PNG for images, and UTF-8 for text.

[1083] Step 2:

[1084] The device sends the received image data and text data to the server. An HTTP POST request is used for transmission, and the multipart / form-data format is used to send the image and text together. The device checks whether the transmission was successful and provides feedback to the user.

[1085] Step 3:

[1086] The server analyzes the received image and text data. First, the server uses image recognition technology to analyze the content of the image data and identify objects and scenes within each photo. Specifically, it uses libraries such as OpenCV to extract metadata such as "ocean," "beach," and "people" from the images. Next, it uses natural language processing technology to analyze the text data and extract key topics and keywords. Specifically, it uses libraries such as TextBlob and NLTK to extract keywords such as "last summer vacation," "family," "ocean," "son," and "first time swimming."

[1087] Step 4:

[1088] The server uses an emotion engine to analyze the user's emotions from text and image data. Specifically, it uses TextBlob and an emotion analysis API to determine emotions such as "happy," "sad," and "neutral" from the text data, and uses facial expression recognition technology to analyze the person's facial expressions from the image data. Emotion information is saved in JSON format and used later to generate the story.

[1089] Step 5:

[1090] The server generates a story based on the analysis results and emotional information. It uses a generative AI model (such as GPT-3) to generate a story by inputting a prompt like the following. An example of a prompt is: "Generate an animated story with the following characteristics: Features: ['ocean', 'fun'], Emotion: fun." The server saves the generated story in JSON format.

[1091] Step 6:

[1092] The server generates animations based on the generated story. An animation generation algorithm is used to draw and connect each scene. Specifically, technologies such as WebGL and Three.js are used to create animations based on the images and text for each scene. The generated animations are saved as video files in MP4 format or similar.

[1093] Step 7:

[1094] The server sends the generated animation to the user's device. The generated animation video file is stored on the server and a download link is generated. The user receives a notification within the app and can click the download link to download the animation to their device. The download link is provided securely to the user's device using the HTTPS protocol.

[1095] Step 8:

[1096] Users can check the generated animation, save it, and share it. Users can play the downloaded animation within the app and check the content. If necessary, they can share the animation via social media or messaging apps. Sharing is done using the share function within the app.

[1097] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1098] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1099] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1100] [Fourth embodiment]

[1101] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1102] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1103] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1104] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1105] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1106] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1107] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1108] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1109] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1110] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1111] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1112] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1113] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1114] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[1115] The program processing of this system will be explained in natural language below.

[1116] Program processing overview

[1117] 1. User selection and upload of image data

[1118] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[1119] 2. User input of text data

[1120] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[1121] 3. Data transmission by the device

[1122] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[1123] 4. Data analysis by the server

[1124] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[1125] 5. Server-generated animation

[1126] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[1127] 6. Sending animation from the server to the device

[1128] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[1129] 7. User can check, save, and share animations

[1130] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[1131] Specific examples

[1132] Example 1:

[1133] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[1134] 2. The device sends this data to the server.

[1135] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[1136] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[1137] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[1138] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[1139] The processing flow will be explained below.

[1140] Step 1:

[1141] The user launches the app on their smartphone or tablet and selects a few memorable photos from their photo gallery. After selecting, they click the upload button to input the image data. This action prepares the image data on the device.

[1142] Step 2:

[1143] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[1144] Step 3:

[1145] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[1146] Step 4:

[1147] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[1148] Step 5:

[1149] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[1150] Step 6:

[1151] The server automatically generates a story using a generative AI model based on the results of image and text analysis. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[1152] Step 7:

[1153] The server uses an animation generation algorithm to draw each scene based on the generated story. Each scene in the story is graphically represented and linked together to generate a series of video files. For example, a scene of the child playing in the ocean and a scene of the child's son swimming may be drawn.

[1154] Step 8:

[1155] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[1156] Step 9:

[1157] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play the animation to check it out.

[1158] Step 10:

[1159] Users can save the downloaded animations and share them via social media or messaging apps as needed. Users can use the generated animations to share their memories with others.

[1160] The above is a detailed step-by-step explanation of the system's programming process, allowing users to easily create, save, and share high-quality animations of their memories.

[1161] Example 1

[1162] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1163] In today's world, anyone can easily create high-quality animations to preserve their memories. However, conventional animation creation methods require specialized knowledge, expensive software, and time, making them inaccessible to the average user. Furthermore, users must manually perform all processes themselves, which is inefficient. To address these issues, the present invention provides a system that allows anyone to easily automatically generate high-quality animations, thereby reducing the burden on users and enabling efficient management and sharing of memories.

[1164] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1165] In this invention, the server includes means for analyzing received image data using image recognition technology, means for analyzing received text data using natural language processing technology, and means for generating a story based on the analysis results. This makes it possible to automatically generate a story based on image data selected by the user and text data entered by the user, and to create an animation by drawing multiple scenes.

[1166] A "user" is an entity that uses the system to provide image data and text data.

[1167] "Image data" is a collection of digital information in the form of still images that a user uploads to the system.

[1168] "Text data" is a collection of information in the form of strings that a user enters into a system and that the system uses to parse.

[1169] "Image recognition technology" is a technology for extracting and analyzing specific information and features from digital images.

[1170] "Natural language processing technology" is a technology for automatically analyzing and understanding text data and extracting semantic information.

[1171] A "story" is a series of stories or scenarios generated based on the analysis of image data and text data.

[1172] An "animation" is a moving image file that is generated by drawing and continuously displaying moving images based on a generated story.

[1173] A "server" is a computer system that analyzes various data in the system, creates stories and animations, and transmits data to users.

[1174] A "terminal" is a device that a user uses to access the system, and includes devices such as smartphones and tablets.

[1175] The "download link" is connection information such as a URL that allows the user to download the generated animation moving image file.

[1176] This invention is a system that automatically generates animations based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: the user, the terminal, and the server. Specifically, the user accesses the system using a smartphone or tablet and operates it, automatically transmitting, analyzing, generating, and receiving various data.

[1177] Hardware and software used

[1178] Device: A user device such as a smartphone or tablet. These devices run applications and provide an interface with the user.

[1179] Server: A high-performance computer system located in the cloud that receives data, analyzes it, and generates stories and animations.

[1180] Image recognition technology: Technology for extracting specific information from image data. Examples include facial recognition technology and object detection technology.

[1181] Natural language processing technology: Technology for analyzing text data and understanding its content, such as extracting emotions and key events from an episode.

[1182] Animation generation algorithm: An algorithm for creating animations based on generated stories, including drawing techniques that utilize deep learning models.

[1183] Specific operation procedures and functions

[1184] 1. User selection and upload of image data

[1185] The user opens the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button in the app to send the selected image data to the system.

[1186] 2. User input of text data

[1187] The user writes an episode or description in the text input field of the app and presses the send button to send the text data to the system.

[1188] 3. Data transmission by the device

[1189] The device sends the uploaded image data and input text data to the server in bulk. This process is performed via API, and the user is notified of the success or failure of the transmission.

[1190] 4. Data analysis by the server

[1191] The server uses image recognition technology to analyze the image data it receives, and natural language processing technology to analyze the text data to understand the content of the episode. This allows it to link the information obtained from the images with the text information and generate a story.

[1192] 5. Server-generated animation

[1193] Based on the generated story, the server uses an animation generation algorithm to draw multiple scenes, each of which is detailed based on the story, and then combines them to generate a series of animated video files.

[1194] 6. Sending animation from the server to the device

[1195] The server saves the generated animation video file and provides a download link to the user's device, through which the user can download the generated animation and play it for viewing.

[1196] 7. User can check, save, and share animations

[1197] The user downloads the animation video file using the provided download link. The user can then play and check the downloaded animation, and save it or share it on social media as needed.

[1198] Specific examples

[1199] Example 1:

[1200] 1. A user uploads three photos of a family trip and enters the following story: "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[1201] 2. The device sends this data to the server.

[1202] 3. The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[1203] 4. The server generates a story based on the analysis results and generates animations based on that story, including scenes of the child playing in the ocean and the child's son swimming.

[1204] 5. The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[1205] In this way, a system is realized that allows users to easily create, save, and share high-quality animations of their memories, even without specialized knowledge.

[1206] Prompt Sentence Examples

[1207] "This system animates family memories based on image data selected by the user and text data entered by the user. As a concrete example, imagine the process of entering photos and stories from a family trip, sending that data to a server, and analyzing it to generate an animation based on the resulting story. Please explain in detail the process."

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Step 1:

[1210] The user opens the app on their smartphone or tablet. They open the photo gallery from the image selection screen and tap to select a few memorable photos. They input the selected image data and click the upload button to send the image data to the system. If the upload is successful, the selected image data is sent to the server.

[1211] Step 2:

[1212] The user moves to the text input field of the app. A guide is displayed in the input field to help the user enter an episode and description. The user enters the episode and description and presses the confirm button to check the content. The user presses the send button to send the entered text data to the system. The text data is sent to the server.

[1213] Step 3:

[1214] The device sends the uploaded image data and entered text data to the server in one batch. This process is performed via API, and the user is notified of the success or failure of the transmission. The input is the image data and text data, and the output is the transmission status to the server.

[1215] Step 4:

[1216] The server analyzes the received image data using image recognition technology. The input is the received image data, and the output is the analysis results. For example, face recognition technology can be used to identify people in a photo and detect the background and key objects.

[1217] Step 5:

[1218] The server analyzes the received text data using natural language processing technology. The input is the received text data, and the output is the analysis results. For example, key events or emotional states can be extracted.

[1219] Step 6:

[1220] The server generates a story based on the results of analyzing the image data and text data. It uses a story generation algorithm to describe each scene in detail. The input is the results of analyzing the image data and text data, and the output is the generated story.

[1221] Step 7:

[1222] The server invokes an animation generation algorithm based on the generated story. It uses a deep learning model to draw realistic animation scenes. The input is the generated story, and the output is multiple animation scenes.

[1223] Step 8:

[1224] The server renders the characters, background, and actions for each scene and connects them to generate an animation video file. The input is multiple animation scenes, and the output is a series of animation video files.

[1225] Step 9:

[1226] The server saves the generated animation video file in cloud storage, for example, using Amazon S3 or Google Cloud Storage. The input is the animation video file, and the output is the URL of the cloud storage location.

[1227] Step 10:

[1228] The server sends a notification to the user's device containing a download link for the animation. This link is a URL where the user can download the animation file. The input is the destination URL on the cloud, and the output is a download link notification to the user.

[1229] Step 11:

[1230] The user clicks the provided download link to download the generated animation video file to their device. The input is the download link, and the output is the downloaded animation video file.

[1231] Step 12:

[1232] Users can play downloaded animations and check their content. While playing, they can check whether the animation quality and story are as expected. If necessary, they can save the downloaded animations to their device. They can also share them with family and friends via social media or email.

[1233] (Application example 1)

[1234] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1235] While conventional animation generation systems are useful in generating animations based on image and text data provided by users, they have limitations on the uses of the generated animations, making them difficult to display and operate in real time. For example, there is a need for a system that can easily generate animations that can be used for product introductions and promotions in virtual stores, and display and operate them in real time.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1237] In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, and means for providing an interface for viewing and manipulating animations in real time, thereby enabling users to quickly and intuitively create product introductions and promotional animations for a virtual store, and display and manipulate them in real time.

[1238] "Image data" refers to still images or photographs selected by users and uploaded to the system.

[1239] "Text data" refers to sentences or explanations entered by a user and sent to the system.

[1240] "Image recognition technology" refers to computer vision technology that analyzes received image data and understands its contents.

[1241] "Natural language processing technology" refers to artificial intelligence technology for analyzing received text data and understanding and interpreting its content.

[1242] "Story generation" refers to the process of automatically creating a narrative or scenario based on analyzed image and text data.

[1243] "Animation generation means" refers to a device or program that draws multiple scenes based on a generated story and combines them to create a continuous animation.

[1244] A "user terminal" is a device for displaying the final generated animation, and includes smart glasses, smartphones, etc.

[1245] "Interface" refers to the graphical or voice-activated environment through which a user interacts with the system and views and controls animations in real time.

[1246] "Virtual store" refers to a virtual commercial facility operated on the Internet, where users can browse and purchase products online.

[1247] "Promotional animation" refers to dynamic visual content created to promote a particular product or service.

[1248] This invention is a system that generates and displays animation in real time based on image data selected by the user and text data entered by the user. This system functions by being divided into three components: a server, a terminal, and a user.

[1249] First, the user takes a photo of a product using the smart glasses and sends the image data to the system. The user then uses the voice input function to input a description and features of the product, which is then sent as text data to the system. The image data selected by the user and the input text data are then sent to the server.

[1250] The server then analyzes the received image data using "image recognition technology" (e.g., Google Cloud Vision API) to understand the content of the image, and analyzes the received text data using "natural language processing technology" (e.g., Google Cloud Natural Language API) to understand the content of the text.

[1251] The server then generates a story based on the analysis results, and generates an animation based on the story. This animation is generated by drawing multiple scenes and combining them into a series of video files.

[1252] The generated animation is sent from the server to the smart glasses, where the user is provided with an interface to view and manipulate the animation in real time. This allows the user to view the generated promotional animation in real time and adjust its content as needed.

[1253] The real-time interface provided by this system provides an environment where users can operate quickly and intuitively, making it extremely effective for promoting and introducing products in virtual stores.

[1254] For example, consider the following prompt:

[1255] "This product is versatile, lasts for extended periods of time, and its lightweight design makes it perfect for travel."

[1256] In this way, users can experience real-time, dynamic product introductions and promotions through the smart glasses. This invention enables the rapid generation of dynamic and engaging content tailored to user needs, improving the shopping experience in virtual stores.

[1257] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1258] Step 1:

[1259] A user uses smart glasses to take a photo of a product. At this time, the image taken by the smart glasses' camera is acquired as "image data." This image data is sent from the terminal to the server by pressing the upload button. The input is the image data taken by the user, and the output is the image data sent to the server.

[1260] Step 2:

[1261] The user uses the voice input function of the smart glasses to input a product description by voice. The voice is converted into text and saved in the smart glasses as "text data." The saved text data is converted into text format by voice recognition software and then sent to the server. The input is voice data, the intermediate output is text data, and the final output is text data sent to the server.

[1262] Step 3:

[1263] The image data received by the server is analyzed using image recognition technology (for example, Google Cloud Vision API). The content of the image is analyzed and specific objects and features are extracted. The input is the image data sent to the server, and the output is the analyzed image content information. Specifically, the shape, color, texture, etc. of the product are identified.

[1264] Step 4:

[1265] The text data received by the server is analyzed using natural language processing technology (for example, Google Cloud Natural Language API). The semantic content of the text is understood and relevant information is extracted. The input is the text data sent to the server, and the output is the semantic information of the analyzed text. Specifically, the product's features and uses are identified.

[1266] Step 5:

[1267] The server generates a story based on the analyzed image and text data. A story generation algorithm is used to create a coherent narrative based on the received content. The input is the analysis results, and the output is the generated story. Specifically, a scenario is constructed that highlights the product's features.

[1268] Step 6:

[1269] The server generates animation based on the generated story. Using an animation generation algorithm, each scene corresponding to the story is drawn and combined into a series of video files. The input is the generated story, and the output is an animated video file. Specifically, it includes scenes that explain the features of the product.

[1270] Step 7:

[1271] The server sends the generated animation to the user's smart glasses, which then provide an interface for viewing and viewing the animation in real time. The input is an animation video file, and the output is a real-time animation displayed on the user's smart glasses. Specifically, the user can manipulate the animation through the smart glasses and modify the content as needed.

[1272] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1273] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[1274] The program processing of this system will be explained in natural language below.

[1275] Program processing overview

[1276] 1. User selection and upload of image data

[1277] The user launches the app on their smartphone or tablet, selects a few memorable photos from their photo gallery, and then clicks the upload button to input the image data. This action prepares the image data on the device.

[1278] 2. User input of text data

[1279] The user writes an episode or explanation in the text input field within the app. For example, they can enter a sentence such as, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they have finished entering the information, they press the send button to send the text data.

[1280] 3. Data transmission by the device

[1281] The device sends the uploaded image data and entered text data to the server. This process is performed using an API, which sends multiple images and text to the server in bulk. It also includes a function to notify the user whether the transmission was successful.

[1282] 4. Data analysis by the server

[1283] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects and scenes in each photo and stores them as metadata. For example, in a photo of the ocean, it recognizes information such as "ocean," "beach," and "people."

[1284] 5. Analysis of text data by the server

[1285] The server analyzes the received text data using natural language processing technology. It understands the content of the input text and extracts key topics and keywords. For example, it extracts information such as "last summer vacation," "family," "the sea," "son," and "first time swimming."

[1286] 6. Emotion Recognition by Server

[1287] The server uses an emotion engine to analyze the user's emotions from the text data and image data. Emotions are extracted from the text data, and emotions are identified from the image data by analyzing the facial expressions of people in photos. The recognized emotion information is reflected in the story generation process and animation generation process.

[1288] 7. Story generation by the server

[1289] The server automatically generates a story using a generative AI model based on the results of image analysis, text analysis, and emotion recognition. The analyzed information is combined to create a series of scenarios and narratives. At this stage, the content of each scene is determined.

[1290] 8. Server-based animation generation

[1291] Based on the generated story, the server uses an animation generation algorithm to draw each scene. Each scene in the story is graphically represented and linked together to generate a series of video files. The atmosphere and direction of the animation are adjusted based on the recognized emotional information. For example, if a happy emotion is recognized, bright colors and more movement are added.

[1292] 9. Sending animation from the server to the device

[1293] The server saves the generated animation video file and generates a download link for it. It also generates and notifies the download URL to provide to the user.

[1294] 10. User can check, save and share animations

[1295] The user downloads the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. If desired, the user can share the animation via social media or messaging apps.

[1296] Specific examples

[1297] Example 1:

[1298] A user uploads three photos of a family trip and enters an episode such as, "These are fun memories of our family going to the beach last summer vacation. My son learned to swim for the first time."

[1299] The terminal transmits this data to the server.

[1300] The server uses image recognition technology to analyze the content of the photo and natural language processing technology to analyze the text of the episode.

[1301] The server uses an emotion engine to recognize the emotion "fun" from text data and identify smiling expressions from image data.

[1302] The server generates a story based on the analysis results and emotional information, and creates animations based on that story, including scenes of the child playing in the ocean in bright colors and the child swimming.

[1303] The server provides a download link for the generated animation, which the user can then download, play and enjoy.

[1304] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[1305] The processing flow will be explained below.

[1306] Step 1:

[1307] The user launches the app on their smartphone or tablet, selects a memorable photo from their photo gallery, and then clicks the upload button to input the image data. This action imports the image data into the device.

[1308] Step 2:

[1309] The user enters an episode or description into the text input field in the app. For example, they might enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once they've finished entering the information, they press the send button to send the text data.

[1310] Step 3:

[1311] The device sends the uploaded image data and entered text data to the server. The sending process is performed via API, and the data is transferred to the server. The user is notified whether the sending was successful.

[1312] Step 4:

[1313] The server analyzes the received image data using image recognition technology. Specifically, it identifies objects, people, and scenes in the photo and generates metadata for them. For example, it extracts information such as "ocean," "beach," and "smiling person."

[1314] Step 5:

[1315] The server analyzes the received text data using natural language processing technology. It understands the content of the text and extracts key topics and keywords. For example, it can extract phrases such as "summer vacation," "family," "ocean," "son," and "first time swimming."

[1316] Step 6:

[1317] The server uses an emotion engine to analyze the user's emotions from the text data. For example, emotions such as "fun" or "happiness" are extracted. The server also analyzes the facial expressions of people in the image data and identifies their emotions.

[1318] Step 7:

[1319] The server uses a generative AI model based on image analysis, text analysis, and emotion recognition to automatically generate a story. This creates a series of scenarios that reflect the user's memories. Emotional information influences the atmosphere and content of the story.

[1320] Step 8:

[1321] The server generates animation based on the generated story. Each scene is drawn using an animation generation algorithm, and the individual scenes are linked together to generate a series of video files. Emotional information is reflected in the color tone and direction.

[1322] Step 9:

[1323] The server saves the generated animation video file and generates a download link, which is notified to the user so that the user can access it.

[1324] Step 10:

[1325] The user downloads the animation using the provided download link. Once the download is complete, the user plays the animation to check it.

[1326] Step 11:

[1327] Users can save the animations they download and share them via social media or messaging apps as needed, allowing them to share their fun memories with others.

[1328] Example 2

[1329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1330] When a user generates an animation based on memorable images and their descriptions, they are required to not only generate a story from images and text, but also to automatically generate a more personalized animation that reflects the user's emotions. Conventional technologies have not fully realized comprehensive analysis, including emotion recognition, and animation generation based on that analysis, making it difficult to automatically generate high-quality animation that faithfully reflects the user's emotions.

[1331] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis result and the emotion recognition result, means for generating animation based on the generated story, and means for transmitting the generated animation to the user terminal. This enables advanced analysis that takes into account the user's emotions and automatic generation of personalized animation based on the emotions.

[1332] A "user" is a person who uses the system to input image data and text data and generate animation.

[1333] "Image data" refers to photos and image files selected and uploaded by users.

[1334] "Text data" refers to sentence data that a user inputs to describe memories or episodes.

[1335] "Means for receiving" refers to a mechanism by which the server has the function of receiving image data and text data sent from the user.

[1336] "Means for analysis" refers to a mechanism by which the server analyzes the image data and text data it receives using image recognition technology and natural language processing technology.

[1337] "Emotion recognition technology" refers to technology that allows a server to extract and analyze emotions from a user's text data and image data.

[1338] A "generative AI model" is an artificial intelligence technology for automatically generating stories based on the results of image analysis, text analysis, and emotion recognition.

[1339] "Means for generating a story" refers to the function of constructing a series of scenarios or stories based on the analysis results and emotion recognition results.

[1340] "Means for generating animation" refers to the function of drawing each scene based on the generated story, connecting them, and generating a series of video files.

[1341] "Transmitting means" refers to the function by which the server transmits the generated animation to the user's terminal.

[1342] A "user terminal" is a device used by a user to operate the system, such as a smartphone or tablet.

[1343] "Personalized animation" refers to animation that reflects the user's emotions and individual information.

[1344] This invention realizes a system that automatically generates animations based on image data selected by the user and text data entered by the user, and further recognizes the user's emotions and reflects them in the animation. This system functions as three components: a server, a terminal, and a user, and by combining an emotion engine, it provides a more personalized experience.

[1345] First, the user launches the dedicated app on their smartphone, tablet, or other device and selects a memorable photo from their photo gallery. Next, the user clicks the upload button to input the image data. This operation temporarily saves the selected image data on the device.

[1346] Next, the user writes down their memories or stories in the text input field within the app and presses the send button to send the text data. For example, they could enter a sentence like, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." This entered text data is temporarily stored on the device.

[1347] Next, the device sends the selected image data and the entered text data together to the server. The sending process is performed using an API (Application Program Interface). The device monitors the status of the data sending and notifies the user of success or failure. If successful, the device notifies the user that "Data sending was successful."

[1348] The server analyzes the received image data using image recognition technology (e.g., OpenCV or TensorFlow). The server identifies objects and scenes in each photo and stores them as metadata. For example, in the case of a photo of the ocean, the server extracts information such as "ocean," "beach," and "people."

[1349] The server then analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT). The server extracts key topics and keywords, such as "last summer vacation," "family," "ocean," "son," and "first swim."

[1350] Furthermore, the server uses an emotion engine (for example, AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from the text data and image data. The server extracts emotions from the text data and identifies emotions from image data by analyzing the facial expressions of people in photos. For example, the server can recognize the emotion "fun" from the text data and recognize that a person is smiling from the image data.

[1351] Based on these analysis results and emotion recognition results, the server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT). The server combines each piece of information to create a series of scenarios or stories. For example, a scenario including "a scene where the son swims for the first time in bright colors" is determined.

[1352] The server then uses an animation generation algorithm based on the generated story to draw each scene. For example, it uses Python's manim library to graphically represent the scenes, and then concatenates them to generate a series of video files. The server then adjusts the mood and direction of the animation based on the recognized emotional information.

[1353] Finally, the server saves the generated animation video file and generates a download link for it. The server then notifies the user of the download link. For example, the server may notify the user that "The animation download has been completed. Please download it from the link below."

[1354] The user can download the generated animation using the download link provided. Once the download is complete, the user can play and check the animation. They can also share the animation via social media or messaging apps if desired.

[1355] Specific examples

[1356] For example, a user might upload three family travel photos and enter an episode such as, "These are fun memories of our family trip to the beach last summer. My son learned to swim for the first time." The device then sends this data to a server. The server then uses image recognition technology to analyze the content of the photos and natural language processing technology to analyze the text of the episode. The server then uses an emotion engine to recognize the emotion "fun" from the text data and identifies smiling expressions from the image data. The server then uses a generative AI model based on the analysis results and emotion information to automatically generate a story, and creates an animation based on that story. For example, the story might include scenes of playing in the ocean in bright colors and of the son swimming. The server then provides a download link for the generated animation, which the user can download, play, and enjoy.

[1357] In this way, a system is realized that provides high-quality animation of memories that reflect the user's emotions.

[1358] Prompt Sentence Examples

[1359] "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time. Please generate an animation based on this episode."

[1360] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1361] Step 1:

[1362] The user launches the dedicated app on their smartphone or tablet.

[1363] Input: The user launches the app and is able to interact with it.

[1364] Output: The app's home screen is displayed, allowing the user to access the photo gallery.

[1365] Step 2:

[1366] The user selects several memorable photos from the photo gallery and clicks the upload button to prepare the image data on the terminal.

[1367] Input: User selects an image from the photo gallery and clicks the upload button.

[1368] Output: The selected image data is temporarily saved on the device and ready to be sent.

[1369] Step 3:

[1370] The user writes an episode or description in the text input field within the app and presses the submit button to enter the text data.

[1371] Input: The user types text into the text field and clicks the submit button.

[1372] Output: The entered text data is temporarily saved on the device.

[1373] Step 4:

[1374] The device sends the uploaded image data and input text data to the server. This process is performed using an API.

[1375] Input: Image data and text data stored on the device.

[1376] Data processing / calculation: Image data and text data are collected and sent to the server using an API.

[1377] Output: The server receives the sent data, and the device notifies the user that the transmission was successful.

[1378] Step 5:

[1379] The image data received by the server is analyzed using image recognition technology (e.g., OpenCV or TensorFlow).

[1380] Input: Image data received by the server.

[1381] Data processing / computation: Using image recognition algorithms, objects and scenes in each photo are identified and stored as metadata.

[1382] Output: The analysis results include information such as "sea," "beach," and "people."

[1383] Step 6:

[1384] The server analyzes the received text data using natural language processing techniques (e.g., spaCy or BERT).

[1385] Input: Text data received by the server.

[1386] Data processing / computation: Use natural language processing algorithms to extract key topics and keywords.

[1387] Output: The analysis results include information such as "last summer vacation," "family," "sea," "son," and "first time swimming."

[1388] Step 7:

[1389] The server uses an emotion engine (e.g., AWS Comprehend or Azure Sentiment Analysis) to analyze the user's emotions from text and image data.

[1390] Input: Parsed text and image data.

[1391] Data processing / computation: Using emotion recognition algorithms, emotions are extracted from text data, and emotions are identified from image data by analyzing people's facial expressions.

[1392] Output: Emotions such as "fun" are obtained as recognized emotional information.

[1393] Step 8:

[1394] The server automatically generates a story using a generative AI model (e.g., GPT-3 or BERT) based on the results of image analysis, text analysis, and emotion recognition.

[1395] Input: Image analysis results, text analysis results, emotion recognition results.

[1396] Data processing / computation: Using generative AI models to combine information into a series of scenarios or stories.

[1397] Output: Auto-generated stories.

[1398] Step 9:

[1399] The server uses an animation generation algorithm based on the generated story to draw each scene.

[1400] Input: Auto-generated stories.

[1401] Data processing / computation: We use Python's manim library to create a graphical representation of each scene, which is then concatenated to generate a series of video files.

[1402] Output: The generated animation video.

[1403] Step 10:

[1404] The server stores the generated animation video file, generates a download link for it, and notifies the user of it.

[1405] Input: The generated animation video file.

[1406] Data processing / calculation: Save the file and generate and notify the download link.

[1407] Output: Download link notification.

[1408] Step 11:

[1409] The user can download the generated animation using the provided download link, play it, and share it.

[1410] Input: Download link.

[1411] Output: Downloaded animations can be played, reviewed, and shared.

[1412] (Application example 2)

[1413] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1414] The present invention relates to a system that automatically generates animations based on image data selected by a user and text data entered by the user, and provides a personalized animation experience that reflects the user's emotions. Conventional systems have had difficulty fully understanding the user's emotions and reflecting them in the animation, and have had limitations in generating content that is in tune with the user's emotions. This can improve user satisfaction.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data selected by the user, means for receiving text data entered by the user, means for analyzing the received image data and text data, means for generating a story based on the analysis results, means for generating animation based on the generated story, means for transmitting the generated animation to the user terminal, and means for recognizing the user's emotions using an emotion engine and reflecting the emotions in the animation. This makes it possible to automatically generate high-quality animation that reflects the user's emotions and provide the user with a personalized experience that emotionally resonates with them.

[1416] The "means for receiving image data selected by a user" is a system for capturing an image file selected by a user via an input device.

[1417] The "means for receiving text data input by a user" is a system for capturing character information input by a user via an input device.

[1418] The "means for analyzing received image data and text data" refers to a technology for analyzing the captured image files and text information and extracting their contents and characteristics.

[1419] The "means for generating a story based on the analysis results" is a system for automatically constructing a series of stories or scenarios using the analysis results of image data and text data.

[1420] "Means for generating animation based on a generated story" refers to a technique for drawing multiple scenes based on a generated story or scenario and generating them as a continuous animation.

[1421] The "means for transmitting the generated animation to the user terminal" is a system for transmitting the generated animation file to the user's device via a network so that the user can view it.

[1422] "Means of recognizing the user's emotions using an emotion engine and reflecting those emotions in animation" is a technology that determines the user's emotions from input text data or image data and reflects those emotions in the atmosphere and presentation of the animation.

[1423] This invention is a system that automatically generates animations based on image data selected by a user and text data entered by the user, providing a personalized animation experience that reflects emotions. Specific embodiments are described below.

[1424] 1. User input of image data and text data

[1425] A user launches a dedicated app using a device such as a smartphone or tablet. The app includes a means for receiving image data selected by the user and a means for receiving text data entered by the user. The user selects a memorable photo from a photo gallery and enters an episode or description in the text field. For example, the user might write, "This is a fun memory of my family going to the beach last summer vacation. My son learned to swim for the first time." Once the input is complete, the user presses the send button, and the image data and text data are sent from the device to the server.

[1426] 2. Data analysis by the server

[1427] The server analyzes the received image data and text data. Specifically, it uses image recognition technology to identify objects and scenes in each photo, and natural language processing technology to understand the content of the text data. For example, in a photo of the sea, it recognizes information such as "sea," "beach," and "people," and extracts keywords such as "last summer vacation," "family," "sea," "son," and "first time swimming." It also uses an emotion engine to analyze the user's emotions from the text data and image data. If a happy emotion is recognized, that information is also included in the analysis results.

[1428] 3. Server-based story and animation generation

[1429] The server generates a story based on the analysis results. It uses a generative AI model to automatically build a series of scenarios and narratives. Once a story that reflects the user's emotions is generated, animation is generated based on it. The animation generation algorithm draws each scene, connects them, and saves them as a series of video files. For example, if a happy emotion is recognized, bright colors and movement are added.

[1430] 4. Sending animation from the server to the device

[1431] The generated animation video file is saved on the server and a download link is generated. The user is notified and can click the download link in the app to download the generated animation to their device. The user can play and check the downloaded animation and share it via social media or messaging apps as needed.

[1432] Specific examples

[1433] For example, if a user uploads three family travel photos and enters an episode such as, "These are fun memories of our family trip to the beach last summer vacation. My son learned to swim for the first time," the server analyzes this data and generates a story. It generates animations of scenes of playing in the ocean and of the son swimming, in bright colors that reflect happy emotions, and provides the user with a download link.

[1434] Prompt Sentence Examples

[1435] Create an animated story with the following characteristics:

[1436] Features: ['ocean', 'fun']

[1437] Emotion: Fun

[1438] In this way, it is possible to provide a high-quality personalized animation experience that reflects the user's emotions.

[1439] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1440] Step 1:

[1441] The device receives image data and text data from the user. The user launches the app on their smartphone or tablet, selects and uploads an image from their photo gallery, and enters an episode and description in the text field. Once the input is complete, they press the send button, and the image data and text data are temporarily saved on the device. The input data format is JPEG or PNG for images, and UTF-8 for text.

[1442] Step 2:

[1443] The device sends the received image data and text data to the server. An HTTP POST request is used for transmission, and the multipart / form-data format is used to send the image and text together. The device checks whether the transmission was successful and provides feedback to the user.

[1444] Step 3:

[1445] The server analyzes the received image and text data. First, the server uses image recognition technology to analyze the content of the image data and identify objects and scenes within each photo. Specifically, it uses libraries such as OpenCV to extract metadata such as "ocean," "beach," and "people" from the images. Next, it uses natural language processing technology to analyze the text data and extract key topics and keywords. Specifically, it uses libraries such as TextBlob and NLTK to extract keywords such as "last summer vacation," "family," "ocean," "son," and "first time swimming."

[1446] Step 4:

[1447] The server uses an emotion engine to analyze the user's emotions from text and image data. Specifically, it uses TextBlob and an emotion analysis API to determine emotions such as "happy," "sad," and "neutral" from the text data, and uses facial expression recognition technology to analyze the person's facial expressions from the image data. Emotion information is saved in JSON format and used later to generate the story.

[1448] Step 5:

[1449] The server generates a story based on the analysis results and emotional information. It uses a generative AI model (such as GPT-3) to generate a story by inputting a prompt like the following. An example of a prompt is: "Generate an animated story with the following characteristics: Features: ['ocean', 'fun'], Emotion: fun." The server saves the generated story in JSON format.

[1450] Step 6:

[1451] The server generates animations based on the generated story. An animation generation algorithm is used to draw and connect each scene. Specifically, technologies such as WebGL and Three.js are used to create animations based on the images and text for each scene. The generated animations are saved as video files in MP4 format or similar.

[1452] Step 7:

[1453] The server sends the generated animation to the user's device. The generated animation video file is stored on the server and a download link is generated. The user receives a notification within the app and can click the download link to download the animation to their device. The download link is provided securely to the user's device using the HTTPS protocol.

[1454] Step 8:

[1455] Users can check the generated animation, save it, and share it. Users can play the downloaded animation within the app and check the content. If necessary, they can share the animation via social media or messaging apps. Sharing is done using the share function within the app.

[1456] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1457] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1458] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1459] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1460] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1461] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1462] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1463] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1464] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1465] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1466] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1467] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1468] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1469] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1470] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1471] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1472] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1473] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1474] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1475] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1476] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1477] The following is further disclosed regarding the above embodiment.

[1478] (Claim 1)

[1479] means for receiving user-selected image data;

[1480] means for receiving text data entered by a user;

[1481] means for analyzing the received image data and text data;

[1482] a means for generating a story based on the analysis results;

[1483] A means for generating animation based on the generated story;

[1484] means for transmitting the generated animation to a user terminal;

[1485] A system including:

[1486] (Claim 2)

[1487] 2. The system of claim 1, wherein the generated story is generated based on image data analyzed using image recognition technology and text data analyzed using natural language processing technology.

[1488] (Claim 3)

[1489] 2. The system according to claim 1, wherein the generated animation is generated by drawing multiple scenes based on the generated story and combining them to form a series of video files.

[1490] "Example 1"

[1491] (Claim 1)

[1492] means for receiving user-selected image data;

[1493] means for receiving text data entered by a user;

[1494] means for transmitting the received image data and text data to a server in a batch;

[1495] means for analyzing the received image data using image recognition technology;

[1496] A means for analyzing the received text data using natural language processing technology;

[1497] a means for generating a story based on the analysis results;

[1498] A method for drawing multiple scenes based on the generated story and combining them to generate animation;

[1499] means for transmitting the generated animation to a user terminal;

[1500] A system including:

[1501] (Claim 2)

[1502] 2. The system of claim 1, wherein the generated story is generated based on image data analyzed using image recognition technology and text data analyzed using natural language processing technology.

[1503] (Claim 3)

[1504] 2. The system according to claim 1, wherein the generated animation is generated by drawing multiple scenes based on the generated story and combining them to form a series of video files.

[1505] "Application Example 1"

[1506] (Claim 1)

[1507] means for receiving user-selected image data;

[1508] means for receiving text data entered by a user;

[1509] means for analyzing the received image data and text data;

[1510] a means for generating a story based on the analysis results;

[1511] A means for generating animation based on the generated story;

[1512] means for transmitting the generated animation to a display device;

[1513] A means to provide an interface for viewing and manipulating the animation in real time; and

[1514] A system including:

[1515] (Claim 2)

[1516] 2. The system of claim 1, wherein the generated story is generated based on image data analyzed using image recognition technology and text data analyzed using natural language processing technology.

[1517] (Claim 3)

[1518] 2. The system according to claim 1, wherein the generated animation is generated by drawing multiple scenes based on the generated story and combining them to form a series of video files.

[1519] (Claim 4)

[1520] 10. The system of claim 1, wherein the promotional animation is generated and displayed to the user in real time via a display device.

[1521] (Claim 5)

[1522] 10. The system of claim 1, wherein the user provides a description using a voice input function, and the description is analyzed as text data.

[1523] (Claim 6)

[1524] 2. The system of claim 1, wherein the generated animation includes content related to product introductions and promotions.

[1525] "Example 2: Combining Emotion Engines"

[1526] (Claim 1)

[1527] means for receiving user-selected image data;

[1528] means for receiving text data entered by a user;

[1529] means for analyzing the received image data and text data;

[1530] a means for generating a story based on the analysis results and the emotion recognition results;

[1531] A means for generating animation based on the generated story;

[1532] means for transmitting the generated animation to a user terminal;

[1533] A system including:

[1534] (Claim 2)

[1535] 2. The system of claim 1, wherein the generated story is generated based on data analyzed using image recognition technology and natural language processing technology, and emotional data analyzed using emotion recognition technology.

[1536] (Claim 3)

[1537] 2. The system according to claim 1, wherein the generated animation is generated by drawing multiple scenes based on the generated story and combining them to form a series of video files.

[1538] "Application example 2 when combining emotion engines"

[1539] (Claim 1)

[1540] means for receiving user-selected image data;

[1541] means for receiving text data entered by a user;

[1542] means for analyzing the received image data and text data;

[1543] a means for generating a story based on the analysis results;

[1544] A means for generating animation based on the generated story;

[1545] means for transmitting the generated animation to a user terminal;

[1546] a means for recognizing a user's emotion using an emotion engine and reflecting the emotion in an animation;

[1547] A system including:

[1548] (Claim 2)

[1549] 2. The system of claim 1, wherein the generated story is generated based on image data analyzed using image recognition technology, text data analyzed using natural language processing technology, and recognized emotional information.

[1550] (Claim 3)

[1551] The system of claim 1, wherein the generated animation is generated by drawing multiple scenes based on the generated story, combining them to create a series of video files, and adjusting the atmosphere and presentation of the animation based on the recognized emotional information. [Explanation of symbols]

[1552] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving user-selected image data; means for receiving text data entered by a user; means for analyzing the received image data and text data; a means for generating a story based on the analysis results; A means for generating animation based on the generated story; means for transmitting the generated animation to a user terminal; A system including:

2. 2. The system according to claim 1, wherein the generated story is generated based on image data analyzed using image recognition technology and text data analyzed using natural language processing technology.

3. 2. The system according to claim 1, wherein the generated animation is generated by drawing a plurality of scenes based on the generated story and combining them to form a series of moving image files.

Images