System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading images and videos captured by a user; means for storing the uploaded images and videos; means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by the user; means for generating a movie based on the selected images and videos; and means for providing the generated movie.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, many people treasure their memories and store large amounts of photos and videos. However, these photos and videos are simply stored and are rarely reviewed. Furthermore, cloud storage capacity limitations often force users to delete photos and videos they need. Furthermore, it can be difficult to effectively organize these photos and videos and enjoy them as a single work. The present invention aims to solve these problems and provide a system that allows users to easily save and enjoy their own memories as movies. [Means for solving the problem]

[0005] The present invention provides the following means: A system including a means for uploading images and videos taken by a user. The system also includes a means for saving the uploaded images and videos and a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user. The system further includes a means for generating a movie based on the selected images and videos and a means for providing the generated movie, allowing users to easily enjoy their memories as a movie. In particular, the image and video analysis means analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm, enabling more appropriate material to be selected. The movie generation means determines the order of the selected images and videos and adds transition effects and background music to create a highly polished movie.

[0006] An "image" is a still image taken or saved by a user.

[0007] "Video" refers to moving images captured or saved by a user.

[0008] A "script" is a text provided by a user that describes the story and structure of a movie.

[0009] "Upload" refers to the act of a user sending data from a terminal to a cloud server.

[0010] "Storage" refers to keeping uploaded data on a cloud server.

[0011] "Analysis" refers to the process in which AI understands the content of images and videos and selects appropriate material based on the script.

[0012] "Selection" means choosing the most suitable images and videos based on the analysis results.

[0013] A "film" is a series of visual works created from selected images and videos.

[0014] "Providing" refers to providing the generated movie to a user in a form that can be viewed or downloaded.

[0015] An "AI model" is an algorithm that uses artificial intelligence to analyze images and videos and generate movies.

[0016] "Natural language processing technology" is one of the artificial intelligence techniques used to analyze the content of a script.

[0017] An "image classification algorithm" is an artificial intelligence technique for identifying and classifying the content of an image.

[0018] A "transition effect" is a visual effect used to smoothly connect scenes in a movie.

[0019] "Background music" refers to music in a film that complements the visual information and enhances emotions. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The purpose of this system is to automatically select and edit the most suitable images and videos based on a script provided by the user, and provide them as a movie.

[0042] System configuration

[0043] 1. Image and video upload feature:

[0044] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[0045] The device sends the images, videos, and script selected by the user to a cloud server.

[0046] 2. Data storage function:

[0047] The server stores the received images, videos, and scripts, organized by user, for further processing.

[0048] 3. Image and video analysis features:

[0049] The server then passes the stored images and videos to an AI for analysis, using natural language processing technology and image classification algorithms.

[0050] The AI ​​model analyzes the provided script and selects the most suitable images and videos for the content, and sends the results of this selection back to the server.

[0051] 4. Movie generation function:

[0052] The server then passes the selected images and videos back to the AI ​​model to generate the movie, which then determines the order of the images and videos, adds transition effects and background music, and edits the movie.

[0053] The completed movie is stored on a server and made accessible to users.

[0054] 5. Movie features:

[0055] The user watches or downloads the generated movie from the server through the terminal, and the server provides the movie data in response to the user's request.

[0056] Specific examples

[0057] For example, consider the case where User A has photos and videos from his / her graduation ceremony and wants to create a movie based on them. User A uploads these photos, videos, and a simple script from his / her device to the cloud server. The server stores the received data.

[0058] The server then analyzes the data using an AI model, which uses natural language processing to interpret the script and uses image classification algorithms to select appropriate photos and videos. The selected data is then sent back to the server for storage.

[0059] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[0060] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy his or her memories as a single movie.

[0061] The processing flow will be explained below.

[0062] Specific processing steps of the program

[0063] Step 1:

[0064] The user launches the application on their device and selects the images, videos, and scripts they want to use. These files are then uploaded as a bundle to the cloud server.

[0065] Step 2:

[0066] The device sends the files (images, videos, scripts) selected by the user to the cloud server along with the user's identification information.

[0067] Step 3:

[0068] The server stores the received images, videos, and scripts in the appropriate databases, organizing them for identification by user.

[0069] Step 4:

[0070] The server inputs the stored images and videos into an AI model, which begins the process of selecting the best material based on the script.

[0071] Step 5:

[0072] The AI ​​model uses natural language processing technology to analyze the content of the script and then analyzes images and videos that fit that content. Specifically, it uses an image classification algorithm to understand the content of the images and videos and select materials that match the scenes in the script.

[0073] Step 6:

[0074] The AI ​​model sends the selected images and videos back to the server, which temporarily stores these selected materials.

[0075] Step 7:

[0076] The server again inputs the selected images and videos into the AI ​​model to begin generating the movie. The AI ​​model then determines the sequence of the footage, adds transition effects, background music, and executes the editing process to edit the movie.

[0077] Step 8:

[0078] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[0079] Step 9:

[0080] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[0081] The above are the specific processing steps of the system.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] In today's world, individuals are increasingly taking large amounts of images and videos, but it is difficult to effectively organize this data and edit their memories into a movie. Furthermore, conventional video editing systems require advanced technology and a great deal of time and effort, making them unattainable for average users. This invention aims to provide a system that automatically selects and edits optimal images and videos based on a script provided by the user, allowing users to easily enjoy their memories as a movie.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes means for uploading images and videos taken by users, means for saving the uploaded images and videos, means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, means for generating a movie based on the selected images and videos, means for providing the generated movie, means for analyzing the script provided by the user using natural language processing technology and selecting images and videos that are optimal for the content of the script, and means for determining the order of images and videos using the movie generation means, adding transition effects and background music, and editing the movie. This enables users to easily generate and enjoy movies based on images and videos they have taken, without requiring advanced skills or a great deal of time.

[0087] "User" means any person or entity that uses the System.

[0088] "Terminal" refers to an electronic device that a user uses to connect to and operate the system.

[0089] "Server" refers to the computer system that stores and processes data provided by users and manages the entire system.

[0090] "Images and videos" refers to still image data and video data captured by the user.

[0091] "Upload" refers to the act of sending data from a terminal to a server.

[0092] "Cloud storage" refers to online storage for storing and managing data via the Internet.

[0093] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, interpret, and generate human language.

[0094] "Image classification algorithm" refers to a computer algorithm used to analyze and classify image data.

[0095] A "generative video model" refers to an artificial intelligence model for generating new videos based on image and video data.

[0096] "Transition effect" refers to an effect used to smoothly switch scenes when editing videos.

[0097] "Background music" refers to music tracks that are added to videos or movies to enhance the viewing experience.

[0098] "Script" refers to a document containing a scenario or screenplay provided by a user.

[0099] A "prompt sentence" refers to a sentence of instructions or questions given by a user to a system.

[0100] This invention is a system that effectively organizes images and videos taken by a user and edits them into a movie. In particular, it aims to enable users to easily create a movie by automatically selecting and editing the most suitable images and videos based on a script provided by the user.

[0101] The system has the following main functions:

[0102] 1. Image and video upload feature:

[0103] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[0104] The hardware used includes smartphones and PCs, which transmit data to a server via a dedicated application or web interface.

[0105] 2. Data storage function:

[0106] The server stores the received images, videos, and scripts, organized by user, for further processing. The software used includes cloud storage services (e.g., Amazon S3).

[0107] 3. Image and video analysis features:

[0108] The server then passes the stored images and videos to an AI for analysis. Natural language processing technology (e.g., GPT-4 (registered trademark)) and image classification algorithms (e.g., ResNet-50) are used for the analysis. The AI ​​model analyzes the script and selects appropriate images and videos.

[0109] 4. Movie generation function:

[0110] The server then passes the selected images and videos back to the AI ​​model to generate the movie. Specifically, it determines the order of the images and videos, adds transition effects and background music, and edits the movie. This process uses a generative animation model.

[0111] 5. Movie features:

[0112] Users watch or download movies generated from the server through their devices. The server provides movie data in response to user requests. Users access movies through a web browser or app.

[0113] Specific examples

[0114] For example, consider a case where a user wants to create a movie based on graduation photos and videos. The user uploads these photos, videos, and a simple script from their device to a cloud server. The server stores the received data and passes it to an AI model (GPT-4 and ResNet-50) for analysis.

[0115] The AI ​​model performs natural language analysis on the script to extract appropriate scenes and uses an image classification algorithm to select related photos and videos. Based on the selected data, the server again uses the AI ​​model to generate the movie. The AI ​​model (generative video model) determines the order of scenes, applies transition effects, and adds background music, editing the movie into a complete film. The completed movie is stored on the server, and users can watch or download it using their devices.

[0116] Prompt Sentence Examples

[0117] "I would like to make a movie using photos and videos from the graduation ceremony. My script includes a scene of the graduates entering first, then memories with their friends, and finally a scene of the diploma presentation. Please edit the movie based on these contents."

[0118] This system allows users to easily create and enjoy movies from images and videos they have taken themselves, without requiring advanced skills or a great deal of time.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] Users access the system from their own devices and select and upload images, videos, and scripts. The input is the image data and video data taken by the user, and the script they create. The output is the transfer of these files to the cloud server.

[0122] Specific operation: The user uses the interface of a smartphone or PC to select images, videos, and script files from the file selection screen and presses the upload button. The device then sends these files to the cloud server.

[0123] Step 2:

[0124] The server stores the received images, videos, and scripts. The input content is the image, video, and script data sent from the user's device, and the output is the state in which that data is stored in cloud storage.

[0125] Specific operation: The server organizes each file based on the user ID and stores it in cloud storage (e.g., Amazon S3). At this stage, the data is structured (e.g., by adding metadata).

[0126] Step 3:

[0127] The server sends the stored image and video data to the AI ​​processing server for analysis. The input is the image and video data retrieved from cloud storage, and the output is a list of selected images and videos as the analysis results.

[0128] How it works: The server uses natural language processing technology (e.g., GPT-4) to analyze the script and categorize the script content based on keywords and context. It then uses an image classification algorithm (e.g., ResNet-50) to analyze the stored image and video data. It selects images and videos that match the script content and generates a list of them.

[0129] Step 4:

[0130] The server then passes the selected images and video list to the AI ​​generation model to begin generating the movie. The input is the selected images and video list as the analysis results, and the output is the generated movie file.

[0131] Specific operation: The server uses the generative video model to determine the scene order of the input data, adds transition effects and background music, and edits each scene into a movie. Once the movie generation is complete, the server generates the finished movie file.

[0132] Step 5:

[0133] The server stores the generated movie files in cloud storage and provides a link or download option for users to access. The input is the generated movie file and the output is a link that users can access.

[0134] Specific operation: The server saves the generated movie file to the cloud storage again and generates an access link for it. The user can access this link using their device to watch or download the movie.

[0135] These steps allow users to enjoy their memories as a movie automatically generated from a large number of images and videos.

[0136] (Application example 1)

[0137] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] In the past, editing a large number of images and videos taken by a user into a single movie required a great deal of time and effort, and the task of selecting appropriate scenes based on a script was particularly difficult without specialized knowledge. Furthermore, there were limited ways to easily view and download the resulting movie on a smartphone. A system that solves these problems is needed.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0140] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for inputting the script and media data uploaded by the user into a generative AI model, and a means for delivering the generated movie data to a smartphone, thereby enabling users to easily generate and watch high-quality movies while saving time and effort.

[0141] "Means for uploading images and videos taken by users" refers to a function that allows users to send images and videos from their own devices to a cloud server.

[0142] "Means for storing the uploaded images and videos" refers to the function of the cloud server to safely store the images and videos received in a database or storage.

[0143] "Means for analyzing the stored images and videos and selecting appropriate images and videos based on the script provided by the user" refers to a function that uses AI technology and algorithms to analyze the content of a script and automatically select the images and videos that best fit that content.

[0144] The "means for generating a movie from the selected images and videos" refers to the function of arranging the selected images and videos in a sequence, adding transition effects and background music, and editing them into a single movie.

[0145] The "means for providing the generated movie" refers to a function that makes the completed movie data accessible to users, allowing them to download or stream the movie.

[0146] "Means for inputting script and media data uploaded by users into the generative AI model" refers to the function of passing the script and image / video data sent by users to the AI ​​model for analysis and use in generating the movie.

[0147] "Means for delivering the generated movie data to a smartphone" refers to the function of providing edited and generated movie files in a downloadable format for smartphone devices.

[0148] This invention is a system that effectively organizes a large number of images and videos taken by a user and edits them into a single movie. This system is designed especially for smartphones, allowing users to easily create, view, and download high-quality movies. Specific embodiments of this system are described below.

[0149] 1. Image and video upload function

[0150] Users can upload images and videos taken with their smartphones to the cloud server. When uploading, users also upload the script. The script specifies the storyline and scene order of the movie and is submitted in text file format.

[0151] 2. Data storage function

[0152] The server securely stores uploaded images, videos, and scripts. The stored data is organized by user and prepared for subsequent processing. This function allows for efficient management of large amounts of data.

[0153] 3. Image and video analysis function

[0154] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model first analyzes the content of the script using natural language processing technology. It then uses an image classification algorithm to analyze the content of the uploaded images and videos and selects the scenes that best fit the script. Generative AI models used include, for example, "text-davinci-003" from OpenAI (registered trademark).

[0155] 4. Movie generation function

[0156] The server generates a movie from the selected images and videos. Specifically, it determines the order of the selected images and videos and adds transition effects and background music. This results in a high-quality movie. For example, the Python "moviepy" library can be used to generate the movie.

[0157] 5. Movie provision function

[0158] The generated movie is stored on the server and made available for users to access. Users can watch or download the movie from their smartphones. This feature allows users to easily enjoy the movie they have created.

[0159] Specific examples

[0160] As a concrete example, consider the case where User A has photos and videos from a graduation ceremony and wants to use them to create a movie. User A uploads these photos and videos from his smartphone to a cloud server, along with a simple script such as "I want to include the graduation scene first, followed by photos with friends." The server stores the data it receives. The server then analyzes the data using a generative AI model. The AI ​​model analyzes the contents of the script using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. The selected data is then sent back to the server and stored.

[0161] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[0162] Finally, User A can watch or download movies from the server using his / her smartphone. The server provides movie data in response to the user's request, allowing User A to enjoy his / her memories as a single movie.

[0163] Prompt Sentence Examples

[0164] The following are examples of prompt sentences:

[0165] Script: I want to include the graduation scene first, followed by photos with friends.

[0166] Please select the most suitable images and videos.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] Users upload images and videos taken from their smartphones to the cloud server. In addition, users also upload a script. The inputs include image files, video files, and the script in text file format. The cloud server receives and stores these files. The output is data confirming that the uploaded files are stored on the cloud server.

[0170] Specifically, the user taps the "Upload" button in the application, selects a file, and sends it. The server receives the HTTP request and saves the file in the specified directory.

[0171] Step 2:

[0172] The server stores the uploaded images, videos, and scripts, storing the data in a database organized by user. Inputs include the files received in step 1. Outputs include organized folders and confirmation that the files have been saved to the database.

[0173] Specifically, the server performs a database operation to store files in separate directories for each user.

[0174] Step 3:

[0175] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model uses natural language processing technology to analyze the content of the script and then uses an image classification algorithm to select images and videos based on that analysis. The input includes the script's text data and image / video files. The output is a list of selected images and videos.

[0176] Specifically, the server generates a prompt for the AI ​​model and makes an API call to obtain the analysis results. Examples of prompts include the following:

[0177] Script: I want to include the graduation scene first, followed by photos with friends.

[0178] Please select the most suitable images and videos.

[0179] Step 4:

[0180] The server generates a movie from the selected images and videos. It uses the Python "moviepy" library to determine the order of the selected images and videos and add transition effects and background music. The input includes the list of selected images and videos obtained from the AI ​​model in step 3. The output is an edited movie file.

[0181] Specifically, the server uses the "moviepy" library to combine each media file and add specified effects and music.

[0182] Step 5:

[0183] The resulting movie is stored on a server and made available for user access. The input includes the completed movie file. The output is a movie file ready for users to watch or download.

[0184] Specifically, the server saves the completed movie file in a specific directory and generates an access link for the user, who taps the link through the application interface to watch or download the movie.

[0185] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0186] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The system aims to select and edit the optimal images and videos based on a script provided by the user, and provide them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized movie generation becomes possible.

[0187] System configuration

[0188] 1. Image and video upload feature:

[0189] Users upload captured images and videos from their own devices to the cloud server, and at the same time, they also upload the script.

[0190] The device sends the images, videos, and script selected by the user to a cloud server.

[0191] 2. Data storage function:

[0192] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[0193] 3. Emotion recognition function:

[0194] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[0195] 4. Image and video analysis features:

[0196] The server inputs the stored images and videos into an AI and emotion engine, and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also uses the results of the emotion engine to select the most emotionally appropriate material.

[0197] 5. Movie generation function:

[0198] The server then passes the selected images and videos back to the AI ​​model to generate the movie. The AI ​​model then determines the sequence of the footage, adds transition effects and background music, and executes the editing process for the movie. Scene placement and effects are also adjusted based on the emotional data.

[0199] 6. Movie features:

[0200] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[0201] Users can watch or download movies generated from the server through their terminals, and the server provides movie data in response to user requests.

[0202] Specific examples

[0203] For example, consider the case where User A has wedding photos and videos and wants to create a movie based on them. User A uploads the photos, videos, and a script depicting the touching moments of the wedding from his / her device to the cloud server. The server stores the received data.

[0204] Next, the server uses an emotion engine to analyze User A's facial expressions and voice from images and videos to obtain emotional data. The AI ​​model then analyzes the script content using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. At the same time, it also refers to the results of the emotion engine to select the most emotionally appropriate material.

[0205] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, adds background music, and edits it into a movie, making adjustments based on the emotional data. The finished movie is then stored on the server.

[0206] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy a movie that reflects his or her memories in a way that is emotionally relevant.

[0207] The processing flow will be explained below.

[0208] Specific processing flow

[0209] Step 1:

[0210] Users launch the application from their device, select the images, videos, and scripts they want to use, and upload the selected files as a bundle to the cloud server.

[0211] Step 2:

[0212] The device sends the images, videos, and scripts selected by the user to a cloud server, along with the user's identification information.

[0213] Step 3:

[0214] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[0215] Step 4:

[0216] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[0217] Step 5:

[0218] The emotion engine analyzes the user's facial expressions and voice from images and videos, and sends the recognized emotion data back to the server, where it is stored.

[0219] Step 6:

[0220] The server inputs the stored images and videos into an AI model and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also refers to the emotional data recognized by the emotion engine to select emotionally appropriate material.

[0221] Step 7:

[0222] The AI ​​model sends the selected images and videos back to the server, which temporarily stores the selected materials.

[0223] Step 8:

[0224] The server then inputs the selected images and videos back into the AI ​​model, which then begins generating the movie. The AI ​​model then determines the sequence of the footage and edits the movie, adding transition effects and background music. Scene placement and effects are also adjusted based on the emotional data.

[0225] Step 9:

[0226] The completed movie is stored on a server and made accessible to users, and the server links the completed movie file to a user interface.

[0227] Step 10:

[0228] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[0229] Example 2

[0230] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] When effectively organizing a large number of images and videos taken by a user and editing them into a single movie, it is difficult to automatically generate a personalized work that is emotionally relevant. To address this issue, it is necessary to detect emotions from the user's facial expressions and voice, select appropriate materials, and reduce the effort and time required to compile them into a movie.

[0232] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading images and videos taken by the user, a means for saving the uploaded images and videos, a means for inputting the saved images and videos into an emotion analysis engine and recognizing emotion data, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for generating a movie based on the selected images and videos, and a means for providing the generated movie. This makes it possible to automatically generate a movie that is in line with the user's emotions, significantly reducing time and effort.

[0233] "Uploading" refers to the act of sending images and videos taken by a user from a terminal to a server.

[0234] "Storage" refers to the act of storing the images, videos, and scripts received by the server in a database or storage device in preparation for subsequent processing.

[0235] An "emotion analysis engine" is software or hardware that analyzes a user's facial expressions and voice from stored images and videos to recognize emotional data.

[0236] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0237] An "image classification algorithm" is an algorithm that analyzes the content of images and videos and classifies them into specific categories or features.

[0238] "Movie creation means" means a means for editing a movie based on selected images and videos by determining the order of scenes, adding transition effects, background music, and effects.

[0239] "Providing" is the act of making the generated movie available for viewing or download by users.

[0240] A "script" is a document that indicates the guidelines and scenario of a video work provided by a user.

[0241] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. This system has the function of selecting and editing the optimal images and videos based on a script provided by the user, and providing them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, personalized movie generation becomes possible.

[0242] A specific embodiment for carrying out the present invention will be described.

[0243] System configuration

[0244] 1. Server and terminal hardware and software

[0245] The devices used by users include general devices such as smartphones and PCs, which have the function of uploading images and videos.

[0246] The server needs storage and processing power to store and analyze the data, and a NoSQL database (e.g., MongoDB) may be used as the database.

[0247] 2. Sentiment Analysis Engine

[0248] The emotion analysis engine used by the server is software that can analyze the user's facial expressions and voice (e.g., facial recognition API or voice analysis API). Specific examples include "Face API" and "Text Analytics."

[0249] 3. Natural language processing engine and image classification algorithm

[0250] The server analyzes the script using natural language processing technology, such as Cloud Natural Language API and IBM Watson (registered trademark).

[0251] Image classification algorithms (e.g., ResNet) are used to classify images and videos.

[0252] 4. AI Model

[0253] AI models that could be used to generate the movie include DALL-E and GPT-4, which support the editing process by determining the order of scenes, applying transition effects, and adding background music.

[0254] Specific examples

[0255] For example, let us consider the case where user A wants to create a movie based on wedding photos and videos.

[0256] User A uses the terminal to upload wedding photos, videos, and a script depicting touching moments from the wedding to the cloud server.

[0257] The server receives this data and stores it in a database.

[0258] Next, the server inputs the saved images and videos into an emotion analysis engine to obtain emotion data from User A's facial expressions and voice.

[0259] The saved script is input into a natural language processing engine, which analyzes the content of the script and extracts important scenes and keywords.

[0260] The server uses an image classification algorithm to select the most suitable images and videos based on the emotional data and script.

[0261] The selected materials are input into the AI ​​model, which determines the order of scenes, applies transition effects, adds background music, and adjusts scene placement and effects based on emotional data.

[0262] Finally, the generated movie is stored on the server and made available for user A to view or download.

[0263] Prompt Sentence Examples

[0264] "We want to create a moving movie based on photos and videos taken at a user's wedding. Below is the script and emotional data. Please analyze them, select the best footage, determine the scene order, apply background music and transition effects, and edit the movie."

[0265] Using this system, users can automatically generate movies that evoke their emotions, allowing them to enjoy their memories in a deeper way.

[0266] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0267] Step 1:

[0268] The user selects images and videos on the device

[0269] Users select the images and videos they want to upload from their smartphone or PC, and also provide a script to use as a guide for the movie.

[0270] Input: User-selected images, videos, and scripts.

[0271] Output: The selected data saved on the device.

[0272] Step 2:

[0273] The device sends data to the cloud server

[0274] The device compresses the selected images, videos, and scripts and sends them to a cloud server using a secure communication protocol (e.g., HTTPS).

[0275] Input: Selected images, videos, scripts.

[0276] Output: Send data to cloud server.

[0277] Step 3:

[0278] The server receives the data

[0279] The server receives the images, videos, and scripts sent from the terminals.

[0280] Input: Images, videos, and scripts sent to the cloud server.

[0281] Output: The received data.

[0282] Step 4:

[0283] The server saves the data to a database

[0284] The server identifies the received data for each user and stores it in an appropriate database, such as a NoSQL database like MongoDB.

[0285] Input: Received images, videos, scripts.

[0286] Output: Data stored in the database.

[0287] Step 5:

[0288] The server inputs the images and videos into the sentiment analysis engine

[0289] The server inputs the stored images and videos into an emotion analysis engine, for example, using a facial recognition API or a voice analysis API.

[0290] Input: Images and videos stored in a database.

[0291] Output: Analysis results of emotion data.

[0292] Step 6:

[0293] The sentiment analysis engine generates sentiment data

[0294] The emotion analysis engine recognizes emotion data from the user's facial expressions and voice data and returns it to the server.

[0295] Input: Images and videos.

[0296] Output: Emotion data.

[0297] Step 7:

[0298] The server inputs the script into a natural language processing engine

[0299] The server inputs the uploaded script into a natural language processing engine, for example, using the Cloud Natural Language API or IBM Watson.

[0300] Input: script.

[0301] Output: Parsed script content.

[0302] Step 8:

[0303] A natural language processing engine analyzes the script

[0304] The natural language processing engine analyzes the contents of the script and extracts important keywords and scenes.

[0305] Input: script.

[0306] Output: Parsed keywords and scenes.

[0307] Step 9:

[0308] The server uses image classification algorithms to select the best images and videos.

[0309] The server uses an image classification algorithm, such as ResNet, to select the most suitable images and videos based on the analyzed script content and emotional data.

[0310] Input: Script analysis results, emotion data.

[0311] Output: Selected images and videos.

[0312] Step 10:

[0313] The server inputs the selected data into the AI ​​model

[0314] The server inputs the selected images and videos into the AI ​​model and begins generating the movie. The AI ​​model used is a generative AI model.

[0315] Input: Selected images and videos.

[0316] Output: The generated film material.

[0317] Step 11:

[0318] AI model generates movies

[0319] The AI ​​model determines the order of images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[0320] Input: Images and videos, emotion data.

[0321] Output: The generated movie.

[0322] Step 12:

[0323] The server stores the finished movie

[0324] The server stores the generated movies in a database and prepares them for distribution to users.

[0325] Input: The generated movie.

[0326] Output: Saved movie.

[0327] Step 13:

[0328] The user watches or downloads the movie through the device

[0329] Users can use their terminals to access the server and watch or download the completed movie, and the server provides a link to the movie file through a user interface.

[0330] Input: Saved movies, user requests.

[0331] Output: Movies available to watch or download.

[0332] (Application example 2)

[0333] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] Conventional image and video editing systems make it difficult for users to create personalized movies based on their own emotions and scenarios. Manually organizing and editing large amounts of material is laborious and time-consuming. Furthermore, since advanced editing using emotion analysis is not possible, it is not possible to generate content that is in tune with the user's emotions. Therefore, there is a need for a system that can easily generate and provide personalized movies based on the user's emotions and scenarios.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0336] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for adjusting scene placement and effects based on the user's emotions using an emotion recognition engine with a generative artificial intelligence model, and a means for providing the generated movie, thereby enabling easy generation of personalized movies based on the user's emotions and scenarios.

[0337] A "user" is an individual or corporation that uses this system and wishes to upload images and videos they have taken and a script to create a movie.

[0338] "Images and videos" are still images and videos taken by users, and are digital files used as material for movies.

[0339] A "script" is a document provided by a user that describes the content and story of a movie, and serves as a guideline for generating the movie.

[0340] An "emotion recognition engine" is software or hardware that analyzes a user's emotions from images and videos and generates emotion data.

[0341] "Storage means" refers to a combination of software and hardware for storing uploaded images and videos in a storage device such as a database.

[0342] The "analyzing means" refers to algorithms and processing devices that perform data analysis based on stored images and videos, as well as scripts, and select appropriate material.

[0343] The "generating means" refers to the algorithms and processing devices that utilize the selected images and video to create a movie.

[0344] The "means for providing" refers to the software and hardware configuration for providing the generated movie to users and enabling viewing and downloading.

[0345] This invention is a system that edits a large number of images and videos taken by a user into a single movie based on a script, and in particular, by combining an emotion recognition engine and a generative artificial intelligence model, it enables the creation of more personalized movies. This system is realized using a server, a terminal, and the following hardware and software.

[0346] Hardware and software:

[0347] Smartphone: A device that allows users to take and upload images and videos.

[0348] Cloud Server: The central location for data storage, analysis, and movie generation.

[0349] Emotion recognition engine (e.g., EmotionEngine): Software that analyzes the user's emotions.

[0350] Natural language processing technology (e.g., NLTK, SpaCy): Software for analyzing the content of scripts.

[0351] Image classification algorithms (e.g., OpenCV): Software for analyzing the content of images and videos.

[0352] Video editing software (e.g., MoviePy): Software for editing a selection of images and videos to create a movie.

[0353] Web frameworks (e.g., Flask, Django): A framework for cloud servers to process and serve data.

[0354] Overview of what the system does:

[0355] 1. A way for users to upload images and videos

[0356] Users upload images and videos taken with their smartphones, as well as scripts, to a cloud server, where the uploaded data is stored.

[0357] 2. Means of data storage

[0358] The cloud server organizes and stores the uploaded images, videos, and scripts in a database.

[0359] 3. Image and video analysis methods

[0360] The server uses an emotion recognition engine to obtain user emotion data from uploaded images and videos, analyzes the script content using natural language processing technology, and selects appropriate images and videos using an image classification algorithm.

[0361] 4. A means of generating a film based on selected material

[0362] The server inputs the selected images and videos into video editing software, which uses generative AI models to determine the order of scenes, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[0363] 5. Means of providing the generated movie

[0364] Users can access, watch, or download completed movies stored on the cloud server from devices such as smartphones.

[0365] Examples:

[0366] Users upload photos and videos from their family trips and are provided with a script based on "fun" and "emotional" elements. The server uses an emotion recognition engine to select happy expressions and touching scenes from the images and videos. It then analyzes the script using natural language processing technology and selects appropriate material using an image classification algorithm. A generative AI model is used to create a movie based on the selected material. Finally, users can watch or download the generated movie from their device.

[0367] Example prompt sentence:

[0368] "Using user-uploaded family vacation photos and videos, use an emotion engine to generate a movie highlighting the fun moments."

[0369] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0370] Step 1:

[0371] This is the stage where users upload images and videos.

[0372] Input: User-taken images, video, and script

[0373] How it works: A user selects images, videos, and scripts using a smartphone application and uploads them to a cloud server. The application then sends these files to the cloud server.

[0374] Output: Images, videos, and script files uploaded to the cloud server

[0375] Step 2:

[0376] This is the stage where the server stores the data.

[0377] Input: Uploaded images, videos, and script files

[0378] Specific operations: The server receives these files and stores each in a database, organizing the database based on user identification information for subsequent processing.

[0379] Output: Images, videos, and script files stored in a database

[0380] Step 3:

[0381] This is the stage where the server acquires emotion data using an emotion recognition engine.

[0382] Input: Image and video files stored in a database

[0383] Specific operation: Using an emotion recognition engine (EmotionEngine), the server analyzes the user's facial expressions and voice from images and videos, and generates emotion data. The emotion data is associated with each file.

[0384] Output: Image and video files associated with emotion data

[0385] Step 4:

[0386] This is the stage where the server analyzes the script, images and videos and selects appropriate materials.

[0387] Input: scripts, images, videos, and emotion data stored in a database

[0388] Specific operation: Analyze the script using natural language processing technology (NLTK, SpaCy) and extract selection criteria. Then, analyze images and videos using an image classification algorithm (OpenCV) and select appropriate materials based on the script and emotion data.

[0389] Output: Selected image and video files

[0390] Step 5:

[0391] This is the stage where the server generates the movie based on the selected material.

[0392] Input: Selected image and video files

[0393] How it works: Using video editing software (MoviePy) and a generative AI model, the server determines the order based on the selected images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[0394] Output: Generated movie file

[0395] Step 6:

[0396] The server now serves the generated movie.

[0397] Input: Generated movie file

[0398] Specific operation: The cloud server stores the generated movie in a database and generates a link that users can access. Users can access the server from their own devices and watch or download the generated movie.

[0399] Output: Movie file that users can watch or download

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The purpose of this system is to automatically select and edit the most suitable images and videos based on a script provided by the user, and provide them as a movie.

[0417] System configuration

[0418] 1. Image and video upload feature:

[0419] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[0420] The device sends the images, videos, and script selected by the user to a cloud server.

[0421] 2. Data storage function:

[0422] The server stores the received images, videos, and scripts, organized by user, for further processing.

[0423] 3. Image and video analysis features:

[0424] The server then passes the stored images and videos to an AI for analysis, using natural language processing technology and image classification algorithms.

[0425] The AI ​​model analyzes the provided script and selects the most suitable images and videos for the content, and sends the results of this selection back to the server.

[0426] 4. Movie generation function:

[0427] The server then passes the selected images and videos back to the AI ​​model to generate the movie, which then determines the order of the images and videos, adds transition effects and background music, and edits the movie.

[0428] The completed movie is stored on a server and made accessible to users.

[0429] 5. Movie features:

[0430] The user watches or downloads the generated movie from the server through the terminal, and the server provides the movie data in response to the user's request.

[0431] Specific examples

[0432] For example, consider the case where User A has photos and videos from his / her graduation ceremony and wants to create a movie based on them. User A uploads these photos, videos, and a simple script from his / her device to the cloud server. The server stores the received data.

[0433] The server then analyzes the data using an AI model, which uses natural language processing to interpret the script and uses image classification algorithms to select appropriate photos and videos. The selected data is then sent back to the server for storage.

[0434] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[0435] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy his or her memories as a single movie.

[0436] The processing flow will be explained below.

[0437] Specific processing steps of the program

[0438] Step 1:

[0439] The user launches the application on their device and selects the images, videos, and scripts they want to use. These files are then uploaded as a bundle to the cloud server.

[0440] Step 2:

[0441] The device sends the files (images, videos, scripts) selected by the user to the cloud server along with the user's identification information.

[0442] Step 3:

[0443] The server stores the received images, videos, and scripts in the appropriate databases, organizing them for identification by user.

[0444] Step 4:

[0445] The server inputs the stored images and videos into an AI model, which begins the process of selecting the best material based on the script.

[0446] Step 5:

[0447] The AI ​​model uses natural language processing technology to analyze the content of the script and then analyzes images and videos that fit that content. Specifically, it uses an image classification algorithm to understand the content of the images and videos and select materials that match the scenes in the script.

[0448] Step 6:

[0449] The AI ​​model sends the selected images and videos back to the server, which temporarily stores these selected materials.

[0450] Step 7:

[0451] The server again inputs the selected images and videos into the AI ​​model to begin generating the movie. The AI ​​model then determines the sequence of the footage, adds transition effects, background music, and executes the editing process to edit the movie.

[0452] Step 8:

[0453] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[0454] Step 9:

[0455] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[0456] The above are the specific processing steps of the system.

[0457] Example 1

[0458] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0459] In today's world, individuals are increasingly taking large amounts of images and videos, but it is difficult to effectively organize this data and edit their memories into a movie. Furthermore, conventional video editing systems require advanced technology and a great deal of time and effort, making them unattainable for average users. This invention aims to provide a system that automatically selects and edits optimal images and videos based on a script provided by the user, allowing users to easily enjoy their memories as a movie.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0461] In this invention, the server includes means for uploading images and videos taken by users, means for saving the uploaded images and videos, means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, means for generating a movie based on the selected images and videos, means for providing the generated movie, means for analyzing the script provided by the user using natural language processing technology and selecting images and videos that are optimal for the content of the script, and means for determining the order of images and videos using the movie generation means, adding transition effects and background music, and editing the movie. This enables users to easily generate and enjoy movies based on images and videos they have taken, without requiring advanced skills or a great deal of time.

[0462] "User" means any person or entity that uses the System.

[0463] "Terminal" refers to an electronic device that a user uses to connect to and operate the system.

[0464] "Server" refers to the computer system that stores and processes data provided by users and manages the entire system.

[0465] "Images and videos" refers to still image data and video data captured by the user.

[0466] "Upload" refers to the act of sending data from a terminal to a server.

[0467] "Cloud storage" refers to online storage for storing and managing data via the Internet.

[0468] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, interpret, and generate human language.

[0469] "Image classification algorithm" refers to a computer algorithm used to analyze and classify image data.

[0470] A "generative video model" refers to an artificial intelligence model for generating new videos based on image and video data.

[0471] "Transition effect" refers to an effect used to smoothly switch scenes when editing videos.

[0472] "Background music" refers to music tracks that are added to videos or movies to enhance the viewing experience.

[0473] "Script" refers to a document containing a scenario or screenplay provided by a user.

[0474] A "prompt sentence" refers to a sentence of instructions or questions given by a user to a system.

[0475] This invention is a system that effectively organizes images and videos taken by a user and edits them into a movie. In particular, it aims to enable users to easily create a movie by automatically selecting and editing the most suitable images and videos based on a script provided by the user.

[0476] The system has the following main functions:

[0477] 1. Image and video upload feature:

[0478] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[0479] The hardware used includes smartphones and PCs, which transmit data to a server via a dedicated application or web interface.

[0480] 2. Data storage function:

[0481] The server stores the received images, videos, and scripts, organized by user, for further processing. The software used includes cloud storage services (e.g., Amazon S3).

[0482] 3. Image and video analysis features:

[0483] The server then passes the stored images and videos to an AI for analysis. Natural language processing techniques (e.g., GPT-4) and image classification algorithms (e.g., ResNet-50) are used for the analysis. The AI ​​model analyzes the script and selects appropriate images and videos.

[0484] 4. Movie generation function:

[0485] The server then passes the selected images and videos back to the AI ​​model to generate the movie. Specifically, it determines the order of the images and videos, adds transition effects and background music, and edits the movie. This process uses a generative animation model.

[0486] 5. Movie features:

[0487] Users watch or download movies generated from the server through their devices. The server provides movie data in response to user requests. Users access movies through a web browser or app.

[0488] Specific examples

[0489] For example, consider a case where a user wants to create a movie based on graduation photos and videos. The user uploads these photos, videos, and a simple script from their device to a cloud server. The server stores the received data and passes it to an AI model (GPT-4 and ResNet-50) for analysis.

[0490] The AI ​​model performs natural language analysis on the script to extract appropriate scenes and uses an image classification algorithm to select related photos and videos. Based on the selected data, the server again uses the AI ​​model to generate the movie. The AI ​​model (generative video model) determines the order of scenes, applies transition effects, and adds background music, editing the movie into a complete film. The completed movie is stored on the server, and users can watch or download it using their devices.

[0491] Prompt Sentence Examples

[0492] "I would like to make a movie using photos and videos from the graduation ceremony. My script includes a scene of the graduates entering first, then memories with their friends, and finally a scene of the diploma presentation. Please edit the movie based on these contents."

[0493] This system allows users to easily create and enjoy movies from images and videos they have taken themselves, without requiring advanced skills or a great deal of time.

[0494] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0495] Step 1:

[0496] Users access the system from their own devices and select and upload images, videos, and scripts. The input is the image data and video data taken by the user, and the script they create. The output is the transfer of these files to the cloud server.

[0497] Specific operation: The user uses the interface of a smartphone or PC to select images, videos, and script files from the file selection screen and presses the upload button. The device then sends these files to the cloud server.

[0498] Step 2:

[0499] The server stores the received images, videos, and scripts. The input content is the image, video, and script data sent from the user's device, and the output is the state in which that data is stored in cloud storage.

[0500] Specific operation: The server organizes each file based on the user ID and stores it in cloud storage (e.g., Amazon S3). At this stage, the data is structured (e.g., by adding metadata).

[0501] Step 3:

[0502] The server sends the stored image and video data to the AI ​​processing server for analysis. The input is the image and video data retrieved from cloud storage, and the output is a list of selected images and videos as the analysis results.

[0503] How it works: The server uses natural language processing technology (e.g., GPT-4) to analyze the script and categorize the script content based on keywords and context. It then uses an image classification algorithm (e.g., ResNet-50) to analyze the stored image and video data. It selects images and videos that match the script content and generates a list of them.

[0504] Step 4:

[0505] The server then passes the selected images and video list to the AI ​​generation model to begin generating the movie. The input is the selected images and video list as the analysis results, and the output is the generated movie file.

[0506] Specific operation: The server uses the generative video model to determine the scene order of the input data, adds transition effects and background music, and edits each scene into a movie. Once the movie generation is complete, the server generates the finished movie file.

[0507] Step 5:

[0508] The server stores the generated movie files in cloud storage and provides a link or download option for users to access. The input is the generated movie file and the output is a link that users can access.

[0509] Specific operation: The server saves the generated movie file to the cloud storage again and generates an access link for it. The user can access this link using their device to watch or download the movie.

[0510] These steps allow users to enjoy their memories as a movie automatically generated from a large number of images and videos.

[0511] (Application example 1)

[0512] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0513] In the past, editing a large number of images and videos taken by a user into a single movie required a great deal of time and effort, and the task of selecting appropriate scenes based on a script was particularly difficult without specialized knowledge. Furthermore, there were limited ways to easily view and download the resulting movie on a smartphone. A system that solves these problems is needed.

[0514] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0515] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for inputting the script and media data uploaded by the user into a generative AI model, and a means for delivering the generated movie data to a smartphone, thereby enabling users to easily generate and watch high-quality movies while saving time and effort.

[0516] "Means for uploading images and videos taken by users" refers to a function that allows users to send images and videos from their own devices to a cloud server.

[0517] "Means for storing the uploaded images and videos" refers to the function of the cloud server to safely store the images and videos received in a database or storage.

[0518] "Means for analyzing the stored images and videos and selecting appropriate images and videos based on the script provided by the user" refers to a function that uses AI technology and algorithms to analyze the content of a script and automatically select the images and videos that best fit that content.

[0519] The "means for generating a movie from the selected images and videos" refers to the function of arranging the selected images and videos in a sequence, adding transition effects and background music, and editing them into a single movie.

[0520] The "means for providing the generated movie" refers to a function that makes the completed movie data accessible to users, allowing them to download or stream the movie.

[0521] "Means for inputting script and media data uploaded by users into the generative AI model" refers to the function of passing the script and image / video data sent by users to the AI ​​model for analysis and use in generating the movie.

[0522] "Means for delivering the generated movie data to a smartphone" refers to the function of providing edited and generated movie files in a downloadable format for smartphone devices.

[0523] This invention is a system that effectively organizes a large number of images and videos taken by a user and edits them into a single movie. This system is designed especially for smartphones, allowing users to easily create, view, and download high-quality movies. Specific embodiments of this system are described below.

[0524] 1. Image and video upload function

[0525] Users can upload images and videos taken with their smartphones to the cloud server. When uploading, users also upload the script. The script specifies the storyline and scene order of the movie and is submitted in text file format.

[0526] 2. Data storage function

[0527] The server securely stores uploaded images, videos, and scripts. The stored data is organized by user and prepared for subsequent processing. This function allows for efficient management of large amounts of data.

[0528] 3. Image and video analysis function

[0529] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model first analyzes the content of the script using natural language processing technology. It then uses an image classification algorithm to analyze the content of the uploaded images and videos and selects the scenes that best fit the script. Generative AI models used include OpenAI's "text-davinci-003."

[0530] 4. Movie generation function

[0531] The server generates a movie from the selected images and videos. Specifically, it determines the order of the selected images and videos and adds transition effects and background music. This results in a high-quality movie. For example, the Python "moviepy" library can be used to generate the movie.

[0532] 5. Movie provision function

[0533] The generated movie is stored on the server and made available for users to access. Users can watch or download the movie from their smartphones. This feature allows users to easily enjoy the movie they have created.

[0534] Specific examples

[0535] As a concrete example, consider the case where User A has photos and videos from a graduation ceremony and wants to use them to create a movie. User A uploads these photos and videos from his smartphone to a cloud server, along with a simple script such as "I want to include the graduation scene first, followed by photos with friends." The server stores the data it receives. The server then analyzes the data using a generative AI model. The AI ​​model analyzes the contents of the script using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. The selected data is then sent back to the server and stored.

[0536] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[0537] Finally, User A can watch or download movies from the server using his / her smartphone. The server provides movie data in response to the user's request, allowing User A to enjoy his / her memories as a single movie.

[0538] Prompt Sentence Examples

[0539] The following are examples of prompt sentences:

[0540] Script: I want to include the graduation scene first, followed by photos with friends.

[0541] Please select the most suitable images and videos.

[0542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0543] Step 1:

[0544] Users upload images and videos taken from their smartphones to the cloud server. In addition, users also upload a script. The inputs include image files, video files, and the script in text file format. The cloud server receives and stores these files. The output is data confirming that the uploaded files are stored on the cloud server.

[0545] Specifically, the user taps the "Upload" button in the application, selects a file, and sends it. The server receives the HTTP request and saves the file in the specified directory.

[0546] Step 2:

[0547] The server stores the uploaded images, videos, and scripts, storing the data in a database organized by user. Inputs include the files received in step 1. Outputs include organized folders and confirmation that the files have been saved to the database.

[0548] Specifically, the server performs a database operation to store files in separate directories for each user.

[0549] Step 3:

[0550] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model uses natural language processing technology to analyze the content of the script and then uses an image classification algorithm to select images and videos based on that analysis. The input includes the script's text data and image / video files. The output is a list of selected images and videos.

[0551] Specifically, the server generates a prompt for the AI ​​model and makes an API call to obtain the analysis results. Examples of prompts include the following:

[0552] Script: I want to include the graduation scene first, followed by photos with friends.

[0553] Please select the most suitable images and videos.

[0554] Step 4:

[0555] The server generates a movie from the selected images and videos. It uses the Python "moviepy" library to determine the order of the selected images and videos and add transition effects and background music. The input includes the list of selected images and videos obtained from the AI ​​model in step 3. The output is an edited movie file.

[0556] Specifically, the server uses the "moviepy" library to combine each media file and add specified effects and music.

[0557] Step 5:

[0558] The resulting movie is stored on a server and made available for user access. The input includes the completed movie file. The output is a movie file ready for users to watch or download.

[0559] Specifically, the server saves the completed movie file in a specific directory and generates an access link for the user, who taps the link through the application interface to watch or download the movie.

[0560] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0561] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The system aims to select and edit the optimal images and videos based on a script provided by the user, and provide them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized movie generation becomes possible.

[0562] System configuration

[0563] 1. Image and video upload feature:

[0564] Users upload captured images and videos from their own devices to the cloud server, and at the same time, they also upload the script.

[0565] The device sends the images, videos, and script selected by the user to a cloud server.

[0566] 2. Data storage function:

[0567] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[0568] 3. Emotion recognition function:

[0569] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[0570] 4. Image and video analysis features:

[0571] The server inputs the stored images and videos into an AI and emotion engine, and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also uses the results of the emotion engine to select the most emotionally appropriate material.

[0572] 5. Movie generation function:

[0573] The server then passes the selected images and videos back to the AI ​​model to generate the movie. The AI ​​model then determines the sequence of the footage, adds transition effects and background music, and executes the editing process for the movie. Scene placement and effects are also adjusted based on the emotional data.

[0574] 6. Movie features:

[0575] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[0576] Users can watch or download movies generated from the server through their terminals, and the server provides movie data in response to user requests.

[0577] Specific examples

[0578] For example, consider the case where User A has wedding photos and videos and wants to create a movie based on them. User A uploads the photos, videos, and a script depicting the touching moments of the wedding from his / her device to the cloud server. The server stores the received data.

[0579] Next, the server uses an emotion engine to analyze User A's facial expressions and voice from images and videos to obtain emotional data. The AI ​​model then analyzes the script content using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. At the same time, it also refers to the results of the emotion engine to select the most emotionally appropriate material.

[0580] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, adds background music, and edits it into a movie, making adjustments based on the emotional data. The finished movie is then stored on the server.

[0581] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy a movie that reflects his or her memories in a way that is emotionally relevant.

[0582] The processing flow will be explained below.

[0583] Specific processing flow

[0584] Step 1:

[0585] Users launch the application from their device, select the images, videos, and scripts they want to use, and upload the selected files as a bundle to the cloud server.

[0586] Step 2:

[0587] The device sends the images, videos, and scripts selected by the user to a cloud server, along with the user's identification information.

[0588] Step 3:

[0589] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[0590] Step 4:

[0591] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[0592] Step 5:

[0593] The emotion engine analyzes the user's facial expressions and voice from images and videos, and sends the recognized emotion data back to the server, where it is stored.

[0594] Step 6:

[0595] The server inputs the stored images and videos into an AI model and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also refers to the emotional data recognized by the emotion engine to select emotionally appropriate material.

[0596] Step 7:

[0597] The AI ​​model sends the selected images and videos back to the server, which temporarily stores the selected materials.

[0598] Step 8:

[0599] The server then inputs the selected images and videos back into the AI ​​model, which then begins generating the movie. The AI ​​model then determines the sequence of the footage and edits the movie, adding transition effects and background music. Scene placement and effects are also adjusted based on the emotional data.

[0600] Step 9:

[0601] The completed movie is stored on a server and made accessible to users, and the server links the completed movie file to a user interface.

[0602] Step 10:

[0603] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[0604] Example 2

[0605] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0606] When effectively organizing a large number of images and videos taken by a user and editing them into a single movie, it is difficult to automatically generate a personalized work that is emotionally relevant. To address this issue, it is necessary to detect emotions from the user's facial expressions and voice, select appropriate materials, and reduce the effort and time required to compile them into a movie.

[0607] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading images and videos taken by the user, a means for saving the uploaded images and videos, a means for inputting the saved images and videos into an emotion analysis engine and recognizing emotion data, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for generating a movie based on the selected images and videos, and a means for providing the generated movie. This makes it possible to automatically generate a movie that is in line with the user's emotions, significantly reducing time and effort.

[0608] "Uploading" refers to the act of sending images and videos taken by a user from a terminal to a server.

[0609] "Storage" refers to the act of storing the images, videos, and scripts received by the server in a database or storage device in preparation for subsequent processing.

[0610] An "emotion analysis engine" is software or hardware that analyzes a user's facial expressions and voice from stored images and videos to recognize emotional data.

[0611] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0612] An "image classification algorithm" is an algorithm that analyzes the content of images and videos and classifies them into specific categories or features.

[0613] "Movie creation means" means a means for editing a movie based on selected images and videos by determining the order of scenes, adding transition effects, background music, and effects.

[0614] "Providing" is the act of making the generated movie available for viewing or download by users.

[0615] A "script" is a document that indicates the guidelines and scenario of a video work provided by a user.

[0616] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. This system has the function of selecting and editing the optimal images and videos based on a script provided by the user, and providing them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, personalized movie generation becomes possible.

[0617] A specific embodiment for carrying out the present invention will be described.

[0618] System configuration

[0619] 1. Server and terminal hardware and software

[0620] The devices used by users include general devices such as smartphones and PCs, which have the function of uploading images and videos.

[0621] The server needs storage and processing power to store and analyze the data, and a NoSQL database (e.g., MongoDB) may be used as the database.

[0622] 2. Sentiment Analysis Engine

[0623] The emotion analysis engine used by the server is software that can analyze the user's facial expressions and voice (e.g., facial recognition API or voice analysis API). Specific examples include "Face API" and "Text Analytics."

[0624] 3. Natural language processing engine and image classification algorithm

[0625] The server analyzes the script using natural language processing technology, such as Cloud Natural Language API and IBM Watson.

[0626] Image classification algorithms (e.g., ResNet) are used to classify images and videos.

[0627] 4. AI Model

[0628] AI models that could be used to generate the movie include DALL-E and GPT-4, which support the editing process by determining the order of scenes, applying transition effects, and adding background music.

[0629] Specific examples

[0630] For example, let us consider the case where user A wants to create a movie based on wedding photos and videos.

[0631] User A uses the terminal to upload wedding photos, videos, and a script depicting touching moments from the wedding to the cloud server.

[0632] The server receives this data and stores it in a database.

[0633] Next, the server inputs the saved images and videos into an emotion analysis engine to obtain emotion data from User A's facial expressions and voice.

[0634] The saved script is input into a natural language processing engine, which analyzes the content of the script and extracts important scenes and keywords.

[0635] The server uses an image classification algorithm to select the most suitable images and videos based on the emotional data and script.

[0636] The selected materials are input into the AI ​​model, which determines the order of scenes, applies transition effects, adds background music, and adjusts scene placement and effects based on emotional data.

[0637] Finally, the generated movie is stored on the server and made available for user A to view or download.

[0638] Prompt Sentence Examples

[0639] "We want to create a moving movie based on photos and videos taken at a user's wedding. Below is the script and emotional data. Please analyze them, select the best footage, determine the scene order, apply background music and transition effects, and edit the movie."

[0640] Using this system, users can automatically generate movies that evoke their emotions, allowing them to enjoy their memories in a deeper way.

[0641] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0642] Step 1:

[0643] The user selects images and videos on the device

[0644] Users select the images and videos they want to upload from their smartphone or PC, and also provide a script to use as a guide for the movie.

[0645] Input: User-selected images, videos, and scripts.

[0646] Output: The selected data saved on the device.

[0647] Step 2:

[0648] The device sends data to the cloud server

[0649] The device compresses the selected images, videos, and scripts and sends them to a cloud server using a secure communication protocol (e.g., HTTPS).

[0650] Input: Selected images, videos, scripts.

[0651] Output: Send data to cloud server.

[0652] Step 3:

[0653] The server receives the data

[0654] The server receives the images, videos, and scripts sent from the terminals.

[0655] Input: Images, videos, and scripts sent to the cloud server.

[0656] Output: The received data.

[0657] Step 4:

[0658] The server saves the data to a database

[0659] The server identifies the received data for each user and stores it in an appropriate database, such as a NoSQL database like MongoDB.

[0660] Input: Received images, videos, scripts.

[0661] Output: Data stored in the database.

[0662] Step 5:

[0663] The server inputs the images and videos into the sentiment analysis engine

[0664] The server inputs the stored images and videos into an emotion analysis engine, for example, using a facial recognition API or a voice analysis API.

[0665] Input: Images and videos stored in a database.

[0666] Output: Analysis results of emotion data.

[0667] Step 6:

[0668] The sentiment analysis engine generates sentiment data

[0669] The emotion analysis engine recognizes emotion data from the user's facial expressions and voice data and returns it to the server.

[0670] Input: Images and videos.

[0671] Output: Emotion data.

[0672] Step 7:

[0673] The server inputs the script into a natural language processing engine

[0674] The server inputs the uploaded script into a natural language processing engine, for example, using the Cloud Natural Language API or IBM Watson.

[0675] Input: script.

[0676] Output: Parsed script content.

[0677] Step 8:

[0678] A natural language processing engine analyzes the script

[0679] The natural language processing engine analyzes the contents of the script and extracts important keywords and scenes.

[0680] Input: script.

[0681] Output: Parsed keywords and scenes.

[0682] Step 9:

[0683] The server uses image classification algorithms to select the best images and videos.

[0684] The server uses an image classification algorithm, such as ResNet, to select the most suitable images and videos based on the analyzed script content and emotional data.

[0685] Input: Script analysis results, emotion data.

[0686] Output: Selected images and videos.

[0687] Step 10:

[0688] The server inputs the selected data into the AI ​​model

[0689] The server inputs the selected images and videos into the AI ​​model and begins generating the movie. The AI ​​model used is a generative AI model.

[0690] Input: Selected images and videos.

[0691] Output: The generated film material.

[0692] Step 11:

[0693] AI model generates movies

[0694] The AI ​​model determines the order of images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[0695] Input: Images and videos, emotion data.

[0696] Output: The generated movie.

[0697] Step 12:

[0698] The server stores the finished movie

[0699] The server stores the generated movies in a database and prepares them for distribution to users.

[0700] Input: The generated movie.

[0701] Output: Saved movie.

[0702] Step 13:

[0703] The user watches or downloads the movie through the device

[0704] Users can use their terminals to access the server and watch or download the completed movie, and the server provides a link to the movie file through a user interface.

[0705] Input: Saved movies, user requests.

[0706] Output: Movies available to watch or download.

[0707] (Application example 2)

[0708] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0709] Conventional image and video editing systems make it difficult for users to create personalized movies based on their own emotions and scenarios. Manually organizing and editing large amounts of material is laborious and time-consuming. Furthermore, since advanced editing using emotion analysis is not possible, it is not possible to generate content that is in tune with the user's emotions. Therefore, there is a need for a system that can easily generate and provide personalized movies based on the user's emotions and scenarios.

[0710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0711] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for adjusting scene placement and effects based on the user's emotions using an emotion recognition engine with a generative artificial intelligence model, and a means for providing the generated movie, thereby enabling easy generation of personalized movies based on the user's emotions and scenarios.

[0712] A "user" is an individual or corporation that uses this system and wishes to upload images and videos they have taken and a script to create a movie.

[0713] "Images and videos" are still images and videos taken by users, and are digital files used as material for movies.

[0714] A "script" is a document provided by a user that describes the content and story of a movie, and serves as a guideline for generating the movie.

[0715] An "emotion recognition engine" is software or hardware that analyzes a user's emotions from images and videos and generates emotion data.

[0716] "Storage means" refers to a combination of software and hardware for storing uploaded images and videos in a storage device such as a database.

[0717] The "analyzing means" refers to algorithms and processing devices that perform data analysis based on stored images and videos, as well as scripts, and select appropriate material.

[0718] The "generating means" refers to the algorithms and processing devices that utilize the selected images and video to create a movie.

[0719] The "means for providing" refers to the software and hardware configuration for providing the generated movie to users and enabling viewing and downloading.

[0720] This invention is a system that edits a large number of images and videos taken by a user into a single movie based on a script, and in particular, by combining an emotion recognition engine and a generative artificial intelligence model, it enables the creation of more personalized movies. This system is realized using a server, a terminal, and the following hardware and software.

[0721] Hardware and software:

[0722] Smartphone: A device that allows users to take and upload images and videos.

[0723] Cloud Server: The central location for data storage, analysis, and movie generation.

[0724] Emotion recognition engine (e.g., EmotionEngine): Software that analyzes the user's emotions.

[0725] Natural language processing technology (e.g., NLTK, SpaCy): Software for analyzing the content of scripts.

[0726] Image classification algorithms (e.g., OpenCV): Software for analyzing the content of images and videos.

[0727] Video editing software (e.g., MoviePy): Software for editing a selection of images and videos to create a movie.

[0728] Web frameworks (e.g., Flask, Django): A framework for cloud servers to process and serve data.

[0729] Overview of what the system does:

[0730] 1. A way for users to upload images and videos

[0731] Users upload images and videos taken with their smartphones, as well as scripts, to a cloud server, where the uploaded data is stored.

[0732] 2. Means of data storage

[0733] The cloud server organizes and stores the uploaded images, videos, and scripts in a database.

[0734] 3. Image and video analysis methods

[0735] The server uses an emotion recognition engine to obtain user emotion data from uploaded images and videos, analyzes the script content using natural language processing technology, and selects appropriate images and videos using an image classification algorithm.

[0736] 4. A means of generating a film based on selected material

[0737] The server inputs the selected images and videos into video editing software, which uses generative AI models to determine the order of scenes, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[0738] 5. Means of providing the generated movie

[0739] Users can access, watch, or download completed movies stored on the cloud server from devices such as smartphones.

[0740] Examples:

[0741] Users upload photos and videos from their family trips and are provided with a script based on "fun" and "emotional" elements. The server uses an emotion recognition engine to select happy expressions and touching scenes from the images and videos. It then analyzes the script using natural language processing technology and selects appropriate material using an image classification algorithm. A generative AI model is used to create a movie based on the selected material. Finally, users can watch or download the generated movie from their device.

[0742] Example prompt sentence:

[0743] "Using user-uploaded family vacation photos and videos, use an emotion engine to generate a movie highlighting the fun moments."

[0744] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0745] Step 1:

[0746] This is the stage where users upload images and videos.

[0747] Input: User-taken images, video, and script

[0748] How it works: A user selects images, videos, and scripts using a smartphone application and uploads them to a cloud server. The application then sends these files to the cloud server.

[0749] Output: Images, videos, and script files uploaded to the cloud server

[0750] Step 2:

[0751] This is the stage where the server stores the data.

[0752] Input: Uploaded images, videos, and script files

[0753] Specific operations: The server receives these files and stores each in a database, organizing the database based on user identification information for subsequent processing.

[0754] Output: Images, videos, and script files stored in a database

[0755] Step 3:

[0756] This is the stage where the server acquires emotion data using an emotion recognition engine.

[0757] Input: Image and video files stored in a database

[0758] Specific operation: Using an emotion recognition engine (EmotionEngine), the server analyzes the user's facial expressions and voice from images and videos, and generates emotion data. The emotion data is associated with each file.

[0759] Output: Image and video files associated with emotion data

[0760] Step 4:

[0761] This is the stage where the server analyzes the script, images and videos and selects appropriate materials.

[0762] Input: scripts, images, videos, and emotion data stored in a database

[0763] Specific operation: Analyze the script using natural language processing technology (NLTK, SpaCy) and extract selection criteria. Then, analyze images and videos using an image classification algorithm (OpenCV) and select appropriate materials based on the script and emotion data.

[0764] Output: Selected image and video files

[0765] Step 5:

[0766] This is the stage where the server generates the movie based on the selected material.

[0767] Input: Selected image and video files

[0768] How it works: Using video editing software (MoviePy) and a generative AI model, the server determines the order based on the selected images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[0769] Output: Generated movie file

[0770] Step 6:

[0771] The server now serves the generated movie.

[0772] Input: Generated movie file

[0773] Specific operation: The cloud server stores the generated movie in a database and generates a link that users can access. Users can access the server from their own devices and watch or download the generated movie.

[0774] Output: Movie file that users can watch or download

[0775] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0776] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0777] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0778] [Third embodiment]

[0779] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0780] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0781] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0782] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0783] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0784] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0785] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0786] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0787] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0788] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0789] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0790] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0791] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The purpose of this system is to automatically select and edit the most suitable images and videos based on a script provided by the user, and provide them as a movie.

[0792] System configuration

[0793] 1. Image and video upload feature:

[0794] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[0795] The device sends the images, videos, and script selected by the user to a cloud server.

[0796] 2. Data storage function:

[0797] The server stores the received images, videos, and scripts, organized by user, for further processing.

[0798] 3. Image and video analysis features:

[0799] The server then passes the stored images and videos to an AI for analysis, using natural language processing technology and image classification algorithms.

[0800] The AI ​​model analyzes the provided script and selects the most suitable images and videos for the content, and sends the results of this selection back to the server.

[0801] 4. Movie generation function:

[0802] The server then passes the selected images and videos back to the AI ​​model to generate the movie, which then determines the order of the images and videos, adds transition effects and background music, and edits the movie.

[0803] The completed movie is stored on a server and made accessible to users.

[0804] 5. Movie features:

[0805] The user watches or downloads the generated movie from the server through the terminal, and the server provides the movie data in response to the user's request.

[0806] Specific examples

[0807] For example, consider the case where User A has photos and videos from his / her graduation ceremony and wants to create a movie based on them. User A uploads these photos, videos, and a simple script from his / her device to the cloud server. The server stores the received data.

[0808] The server then analyzes the data using an AI model, which uses natural language processing to interpret the script and uses image classification algorithms to select appropriate photos and videos. The selected data is then sent back to the server for storage.

[0809] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[0810] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy his or her memories as a single movie.

[0811] The processing flow will be explained below.

[0812] Specific processing steps of the program

[0813] Step 1:

[0814] The user launches the application on their device and selects the images, videos, and scripts they want to use. These files are then uploaded as a bundle to the cloud server.

[0815] Step 2:

[0816] The device sends the files (images, videos, scripts) selected by the user to the cloud server along with the user's identification information.

[0817] Step 3:

[0818] The server stores the received images, videos, and scripts in the appropriate databases, organizing them for identification by user.

[0819] Step 4:

[0820] The server inputs the stored images and videos into an AI model, which begins the process of selecting the best material based on the script.

[0821] Step 5:

[0822] The AI ​​model uses natural language processing technology to analyze the content of the script and then analyzes images and videos that fit that content. Specifically, it uses an image classification algorithm to understand the content of the images and videos and select materials that match the scenes in the script.

[0823] Step 6:

[0824] The AI ​​model sends the selected images and videos back to the server, which temporarily stores these selected materials.

[0825] Step 7:

[0826] The server again inputs the selected images and videos into the AI ​​model to begin generating the movie. The AI ​​model then determines the sequence of the footage, adds transition effects, background music, and executes the editing process to edit the movie.

[0827] Step 8:

[0828] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[0829] Step 9:

[0830] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[0831] The above are the specific processing steps of the system.

[0832] Example 1

[0833] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0834] In today's world, individuals are increasingly taking large amounts of images and videos, but it is difficult to effectively organize this data and edit their memories into a movie. Furthermore, conventional video editing systems require advanced technology and a great deal of time and effort, making them unattainable for average users. This invention aims to provide a system that automatically selects and edits optimal images and videos based on a script provided by the user, allowing users to easily enjoy their memories as a movie.

[0835] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0836] In this invention, the server includes means for uploading images and videos taken by users, means for saving the uploaded images and videos, means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, means for generating a movie based on the selected images and videos, means for providing the generated movie, means for analyzing the script provided by the user using natural language processing technology and selecting images and videos that are optimal for the content of the script, and means for determining the order of images and videos using the movie generation means, adding transition effects and background music, and editing the movie. This enables users to easily generate and enjoy movies based on images and videos they have taken, without requiring advanced skills or a great deal of time.

[0837] "User" means any person or entity that uses the System.

[0838] "Terminal" refers to an electronic device that a user uses to connect to and operate the system.

[0839] "Server" refers to the computer system that stores and processes data provided by users and manages the entire system.

[0840] "Images and videos" refers to still image data and video data captured by the user.

[0841] "Upload" refers to the act of sending data from a terminal to a server.

[0842] "Cloud storage" refers to online storage for storing and managing data via the Internet.

[0843] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, interpret, and generate human language.

[0844] "Image classification algorithm" refers to a computer algorithm used to analyze and classify image data.

[0845] A "generative video model" refers to an artificial intelligence model for generating new videos based on image and video data.

[0846] "Transition effect" refers to an effect used to smoothly switch scenes when editing videos.

[0847] "Background music" refers to music tracks that are added to videos or movies to enhance the viewing experience.

[0848] "Script" refers to a document containing a scenario or screenplay provided by a user.

[0849] A "prompt sentence" refers to a sentence of instructions or questions given by a user to a system.

[0850] This invention is a system that effectively organizes images and videos taken by a user and edits them into a movie. In particular, it aims to enable users to easily create a movie by automatically selecting and editing the most suitable images and videos based on a script provided by the user.

[0851] The system has the following main functions:

[0852] 1. Image and video upload feature:

[0853] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[0854] The hardware used includes smartphones and PCs, which transmit data to a server via a dedicated application or web interface.

[0855] 2. Data storage function:

[0856] The server stores the received images, videos, and scripts, organized by user, for further processing. The software used includes cloud storage services (e.g., Amazon S3).

[0857] 3. Image and video analysis features:

[0858] The server then passes the stored images and videos to an AI for analysis. Natural language processing techniques (e.g., GPT-4) and image classification algorithms (e.g., ResNet-50) are used for the analysis. The AI ​​model analyzes the script and selects appropriate images and videos.

[0859] 4. Movie generation function:

[0860] The server then passes the selected images and videos back to the AI ​​model to generate the movie. Specifically, it determines the order of the images and videos, adds transition effects and background music, and edits the movie. This process uses a generative animation model.

[0861] 5. Movie features:

[0862] Users watch or download movies generated from the server through their devices. The server provides movie data in response to user requests. Users access movies through a web browser or app.

[0863] Specific examples

[0864] For example, consider a case where a user wants to create a movie based on graduation photos and videos. The user uploads these photos, videos, and a simple script from their device to a cloud server. The server stores the received data and passes it to an AI model (GPT-4 and ResNet-50) for analysis.

[0865] The AI ​​model performs natural language analysis on the script to extract appropriate scenes and uses an image classification algorithm to select related photos and videos. Based on the selected data, the server again uses the AI ​​model to generate the movie. The AI ​​model (generative video model) determines the order of scenes, applies transition effects, and adds background music, editing the movie into a complete film. The completed movie is stored on the server, and users can watch or download it using their devices.

[0866] Prompt Sentence Examples

[0867] "I would like to make a movie using photos and videos from the graduation ceremony. My script includes a scene of the graduates entering first, then memories with their friends, and finally a scene of the diploma presentation. Please edit the movie based on these contents."

[0868] This system allows users to easily create and enjoy movies from images and videos they have taken themselves, without requiring advanced skills or a great deal of time.

[0869] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0870] Step 1:

[0871] Users access the system from their own devices and select and upload images, videos, and scripts. The input is the image data and video data taken by the user, and the script they create. The output is the transfer of these files to the cloud server.

[0872] Specific operation: The user uses the interface of a smartphone or PC to select images, videos, and script files from the file selection screen and presses the upload button. The device then sends these files to the cloud server.

[0873] Step 2:

[0874] The server stores the received images, videos, and scripts. The input content is the image, video, and script data sent from the user's device, and the output is the state in which that data is stored in cloud storage.

[0875] Specific operation: The server organizes each file based on the user ID and stores it in cloud storage (e.g., Amazon S3). At this stage, the data is structured (e.g., by adding metadata).

[0876] Step 3:

[0877] The server sends the stored image and video data to the AI ​​processing server for analysis. The input is the image and video data retrieved from cloud storage, and the output is a list of selected images and videos as the analysis results.

[0878] How it works: The server uses natural language processing technology (e.g., GPT-4) to analyze the script and categorize the script content based on keywords and context. It then uses an image classification algorithm (e.g., ResNet-50) to analyze the stored image and video data. It selects images and videos that match the script content and generates a list of them.

[0879] Step 4:

[0880] The server then passes the selected images and video list to the AI ​​generation model to begin generating the movie. The input is the selected images and video list as the analysis results, and the output is the generated movie file.

[0881] Specific operation: The server uses the generative video model to determine the scene order of the input data, adds transition effects and background music, and edits each scene into a movie. Once the movie generation is complete, the server generates the finished movie file.

[0882] Step 5:

[0883] The server stores the generated movie files in cloud storage and provides a link or download option for users to access. The input is the generated movie file and the output is a link that users can access.

[0884] Specific operation: The server saves the generated movie file to the cloud storage again and generates an access link for it. The user can access this link using their device to watch or download the movie.

[0885] These steps allow users to enjoy their memories as a movie automatically generated from a large number of images and videos.

[0886] (Application example 1)

[0887] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0888] In the past, editing a large number of images and videos taken by a user into a single movie required a great deal of time and effort, and the task of selecting appropriate scenes based on a script was particularly difficult without specialized knowledge. Furthermore, there were limited ways to easily view and download the resulting movie on a smartphone. A system that solves these problems is needed.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0890] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for inputting the script and media data uploaded by the user into a generative AI model, and a means for delivering the generated movie data to a smartphone, thereby enabling users to easily generate and watch high-quality movies while saving time and effort.

[0891] "Means for uploading images and videos taken by users" refers to a function that allows users to send images and videos from their own devices to a cloud server.

[0892] "Means for storing the uploaded images and videos" refers to the function of the cloud server to safely store the images and videos received in a database or storage.

[0893] "Means for analyzing the stored images and videos and selecting appropriate images and videos based on the script provided by the user" refers to a function that uses AI technology and algorithms to analyze the content of a script and automatically select the images and videos that best fit that content.

[0894] The "means for generating a movie from the selected images and videos" refers to the function of arranging the selected images and videos in a sequence, adding transition effects and background music, and editing them into a single movie.

[0895] The "means for providing the generated movie" refers to a function that makes the completed movie data accessible to users, allowing them to download or stream the movie.

[0896] "Means for inputting script and media data uploaded by users into the generative AI model" refers to the function of passing the script and image / video data sent by users to the AI ​​model for analysis and use in generating the movie.

[0897] "Means for delivering the generated movie data to a smartphone" refers to the function of providing edited and generated movie files in a downloadable format for smartphone devices.

[0898] This invention is a system that effectively organizes a large number of images and videos taken by a user and edits them into a single movie. This system is designed especially for smartphones, allowing users to easily create, view, and download high-quality movies. Specific embodiments of this system are described below.

[0899] 1. Image and video upload function

[0900] Users can upload images and videos taken with their smartphones to the cloud server. When uploading, users also upload the script. The script specifies the storyline and scene order of the movie and is submitted in text file format.

[0901] 2. Data storage function

[0902] The server securely stores uploaded images, videos, and scripts. The stored data is organized by user and prepared for subsequent processing. This function allows for efficient management of large amounts of data.

[0903] 3. Image and video analysis function

[0904] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model first analyzes the content of the script using natural language processing technology. It then uses an image classification algorithm to analyze the content of the uploaded images and videos and selects the scenes that best fit the script. Generative AI models used include OpenAI's "text-davinci-003."

[0905] 4. Movie generation function

[0906] The server generates a movie from the selected images and videos. Specifically, it determines the order of the selected images and videos and adds transition effects and background music. This results in a high-quality movie. For example, the Python "moviepy" library can be used to generate the movie.

[0907] 5. Movie provision function

[0908] The generated movie is stored on the server and made available for users to access. Users can watch or download the movie from their smartphones. This feature allows users to easily enjoy the movie they have created.

[0909] Specific examples

[0910] As a concrete example, consider the case where User A has photos and videos from a graduation ceremony and wants to use them to create a movie. User A uploads these photos and videos from his smartphone to a cloud server, along with a simple script such as "I want to include the graduation scene first, followed by photos with friends." The server stores the data it receives. The server then analyzes the data using a generative AI model. The AI ​​model analyzes the contents of the script using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. The selected data is then sent back to the server and stored.

[0911] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[0912] Finally, User A can watch or download movies from the server using his / her smartphone. The server provides movie data in response to the user's request, allowing User A to enjoy his / her memories as a single movie.

[0913] Prompt Sentence Examples

[0914] The following are examples of prompt sentences:

[0915] Script: I want to include the graduation scene first, followed by photos with friends.

[0916] Please select the most suitable images and videos.

[0917] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0918] Step 1:

[0919] Users upload images and videos taken from their smartphones to the cloud server. In addition, users also upload a script. The inputs include image files, video files, and the script in text file format. The cloud server receives and stores these files. The output is data confirming that the uploaded files are stored on the cloud server.

[0920] Specifically, the user taps the "Upload" button in the application, selects a file, and sends it. The server receives the HTTP request and saves the file in the specified directory.

[0921] Step 2:

[0922] The server stores the uploaded images, videos, and scripts, storing the data in a database organized by user. Inputs include the files received in step 1. Outputs include organized folders and confirmation that the files have been saved to the database.

[0923] Specifically, the server performs a database operation to store files in separate directories for each user.

[0924] Step 3:

[0925] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model uses natural language processing technology to analyze the content of the script and then uses an image classification algorithm to select images and videos based on that analysis. The input includes the script's text data and image / video files. The output is a list of selected images and videos.

[0926] Specifically, the server generates a prompt for the AI ​​model and makes an API call to obtain the analysis results. Examples of prompts include the following:

[0927] Script: I want to include the graduation scene first, followed by photos with friends.

[0928] Please select the most suitable images and videos.

[0929] Step 4:

[0930] The server generates a movie from the selected images and videos. It uses the Python "moviepy" library to determine the order of the selected images and videos and add transition effects and background music. The input includes the list of selected images and videos obtained from the AI ​​model in step 3. The output is an edited movie file.

[0931] Specifically, the server uses the "moviepy" library to combine each media file and add specified effects and music.

[0932] Step 5:

[0933] The resulting movie is stored on a server and made available for user access. The input includes the completed movie file. The output is a movie file ready for users to watch or download.

[0934] Specifically, the server saves the completed movie file in a specific directory and generates an access link for the user, who taps the link through the application interface to watch or download the movie.

[0935] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0936] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The system aims to select and edit the optimal images and videos based on a script provided by the user, and provide them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized movie generation becomes possible.

[0937] System configuration

[0938] 1. Image and video upload feature:

[0939] Users upload captured images and videos from their own devices to the cloud server, and at the same time, they also upload the script.

[0940] The device sends the images, videos, and script selected by the user to a cloud server.

[0941] 2. Data storage function:

[0942] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[0943] 3. Emotion recognition function:

[0944] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[0945] 4. Image and video analysis features:

[0946] The server inputs the stored images and videos into an AI and emotion engine, and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also uses the results of the emotion engine to select the most emotionally appropriate material.

[0947] 5. Movie generation function:

[0948] The server then passes the selected images and videos back to the AI ​​model to generate the movie. The AI ​​model then determines the sequence of the footage, adds transition effects and background music, and executes the editing process for the movie. Scene placement and effects are also adjusted based on the emotional data.

[0949] 6. Movie features:

[0950] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[0951] Users can watch or download movies generated from the server through their terminals, and the server provides movie data in response to user requests.

[0952] Specific examples

[0953] For example, consider the case where User A has wedding photos and videos and wants to create a movie based on them. User A uploads the photos, videos, and a script depicting the touching moments of the wedding from his / her device to the cloud server. The server stores the received data.

[0954] Next, the server uses an emotion engine to analyze User A's facial expressions and voice from images and videos to obtain emotional data. The AI ​​model then analyzes the script content using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. At the same time, it also refers to the results of the emotion engine to select the most emotionally appropriate material.

[0955] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, adds background music, and edits it into a movie, making adjustments based on the emotional data. The finished movie is then stored on the server.

[0956] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy a movie that reflects his or her memories in a way that is emotionally relevant.

[0957] The processing flow will be explained below.

[0958] Specific processing flow

[0959] Step 1:

[0960] Users launch the application from their device, select the images, videos, and scripts they want to use, and upload the selected files as a bundle to the cloud server.

[0961] Step 2:

[0962] The device sends the images, videos, and scripts selected by the user to a cloud server, along with the user's identification information.

[0963] Step 3:

[0964] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[0965] Step 4:

[0966] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[0967] Step 5:

[0968] The emotion engine analyzes the user's facial expressions and voice from images and videos, and sends the recognized emotion data back to the server, where it is stored.

[0969] Step 6:

[0970] The server inputs the stored images and videos into an AI model and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also refers to the emotional data recognized by the emotion engine to select emotionally appropriate material.

[0971] Step 7:

[0972] The AI ​​model sends the selected images and videos back to the server, which temporarily stores the selected materials.

[0973] Step 8:

[0974] The server then inputs the selected images and videos back into the AI ​​model, which then begins generating the movie. The AI ​​model then determines the sequence of the footage and edits the movie, adding transition effects and background music. Scene placement and effects are also adjusted based on the emotional data.

[0975] Step 9:

[0976] The completed movie is stored on a server and made accessible to users, and the server links the completed movie file to a user interface.

[0977] Step 10:

[0978] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] When effectively organizing a large number of images and videos taken by a user and editing them into a single movie, it is difficult to automatically generate a personalized work that is emotionally relevant. To address this issue, it is necessary to detect emotions from the user's facial expressions and voice, select appropriate materials, and reduce the effort and time required to compile them into a movie.

[0982] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading images and videos taken by the user, a means for saving the uploaded images and videos, a means for inputting the saved images and videos into an emotion analysis engine and recognizing emotion data, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for generating a movie based on the selected images and videos, and a means for providing the generated movie. This makes it possible to automatically generate a movie that is in line with the user's emotions, significantly reducing time and effort.

[0983] "Uploading" refers to the act of sending images and videos taken by a user from a terminal to a server.

[0984] "Storage" refers to the act of storing the images, videos, and scripts received by the server in a database or storage device in preparation for subsequent processing.

[0985] An "emotion analysis engine" is software or hardware that analyzes a user's facial expressions and voice from stored images and videos to recognize emotional data.

[0986] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0987] An "image classification algorithm" is an algorithm that analyzes the content of images and videos and classifies them into specific categories or features.

[0988] "Movie creation means" means a means for editing a movie based on selected images and videos by determining the order of scenes, adding transition effects, background music, and effects.

[0989] "Providing" is the act of making the generated movie available for viewing or download by users.

[0990] A "script" is a document that indicates the guidelines and scenario of a video work provided by a user.

[0991] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. This system has the function of selecting and editing the optimal images and videos based on a script provided by the user, and providing them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, personalized movie generation becomes possible.

[0992] A specific embodiment for carrying out the present invention will be described.

[0993] System configuration

[0994] 1. Server and terminal hardware and software

[0995] The devices used by users include general devices such as smartphones and PCs, which have the function of uploading images and videos.

[0996] The server needs storage and processing power to store and analyze the data, and a NoSQL database (e.g., MongoDB) may be used as the database.

[0997] 2. Sentiment Analysis Engine

[0998] The emotion analysis engine used by the server is software that can analyze the user's facial expressions and voice (e.g., facial recognition API or voice analysis API). Specific examples include "Face API" and "Text Analytics."

[0999] 3. Natural language processing engine and image classification algorithm

[1000] The server analyzes the script using natural language processing technology, such as Cloud Natural Language API and IBM Watson.

[1001] Image classification algorithms (e.g., ResNet) are used to classify images and videos.

[1002] 4. AI Model

[1003] AI models that could be used to generate the movie include DALL-E and GPT-4, which support the editing process by determining the order of scenes, applying transition effects, and adding background music.

[1004] Specific examples

[1005] For example, let us consider the case where user A wants to create a movie based on wedding photos and videos.

[1006] User A uses the terminal to upload wedding photos, videos, and a script depicting touching moments from the wedding to the cloud server.

[1007] The server receives this data and stores it in a database.

[1008] Next, the server inputs the saved images and videos into an emotion analysis engine to obtain emotion data from User A's facial expressions and voice.

[1009] The saved script is input into a natural language processing engine, which analyzes the content of the script and extracts important scenes and keywords.

[1010] The server uses an image classification algorithm to select the most suitable images and videos based on the emotional data and script.

[1011] The selected materials are input into the AI ​​model, which determines the order of scenes, applies transition effects, adds background music, and adjusts scene placement and effects based on emotional data.

[1012] Finally, the generated movie is stored on the server and made available for user A to view or download.

[1013] Prompt Sentence Examples

[1014] "We want to create a moving movie based on photos and videos taken at a user's wedding. Below is the script and emotional data. Please analyze them, select the best footage, determine the scene order, apply background music and transition effects, and edit the movie."

[1015] Using this system, users can automatically generate movies that evoke their emotions, allowing them to enjoy their memories in a deeper way.

[1016] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1017] Step 1:

[1018] The user selects images and videos on the device

[1019] Users select the images and videos they want to upload from their smartphone or PC, and also provide a script to use as a guide for the movie.

[1020] Input: User-selected images, videos, and scripts.

[1021] Output: The selected data saved on the device.

[1022] Step 2:

[1023] The device sends data to the cloud server

[1024] The device compresses the selected images, videos, and scripts and sends them to a cloud server using a secure communication protocol (e.g., HTTPS).

[1025] Input: Selected images, videos, scripts.

[1026] Output: Send data to cloud server.

[1027] Step 3:

[1028] The server receives the data

[1029] The server receives the images, videos, and scripts sent from the terminals.

[1030] Input: Images, videos, and scripts sent to the cloud server.

[1031] Output: The received data.

[1032] Step 4:

[1033] The server saves the data to a database

[1034] The server identifies the received data for each user and stores it in an appropriate database, such as a NoSQL database like MongoDB.

[1035] Input: Received images, videos, scripts.

[1036] Output: Data stored in the database.

[1037] Step 5:

[1038] The server inputs the images and videos into the sentiment analysis engine

[1039] The server inputs the stored images and videos into an emotion analysis engine, for example, using a facial recognition API or a voice analysis API.

[1040] Input: Images and videos stored in a database.

[1041] Output: Analysis results of emotion data.

[1042] Step 6:

[1043] The sentiment analysis engine generates sentiment data

[1044] The emotion analysis engine recognizes emotion data from the user's facial expressions and voice data and returns it to the server.

[1045] Input: Images and videos.

[1046] Output: Emotion data.

[1047] Step 7:

[1048] The server inputs the script into a natural language processing engine

[1049] The server inputs the uploaded script into a natural language processing engine, for example, using the Cloud Natural Language API or IBM Watson.

[1050] Input: script.

[1051] Output: Parsed script content.

[1052] Step 8:

[1053] A natural language processing engine analyzes the script

[1054] The natural language processing engine analyzes the contents of the script and extracts important keywords and scenes.

[1055] Input: script.

[1056] Output: Parsed keywords and scenes.

[1057] Step 9:

[1058] The server uses image classification algorithms to select the best images and videos.

[1059] The server uses an image classification algorithm, such as ResNet, to select the most suitable images and videos based on the analyzed script content and emotional data.

[1060] Input: Script analysis results, emotion data.

[1061] Output: Selected images and videos.

[1062] Step 10:

[1063] The server inputs the selected data into the AI ​​model

[1064] The server inputs the selected images and videos into the AI ​​model and begins generating the movie. The AI ​​model used is a generative AI model.

[1065] Input: Selected images and videos.

[1066] Output: The generated film material.

[1067] Step 11:

[1068] AI model generates movies

[1069] The AI ​​model determines the order of images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[1070] Input: Images and videos, emotion data.

[1071] Output: The generated movie.

[1072] Step 12:

[1073] The server stores the finished movie

[1074] The server stores the generated movies in a database and prepares them for distribution to users.

[1075] Input: The generated movie.

[1076] Output: Saved movie.

[1077] Step 13:

[1078] The user watches or downloads the movie through the device

[1079] Users can use their terminals to access the server and watch or download the completed movie, and the server provides a link to the movie file through a user interface.

[1080] Input: Saved movies, user requests.

[1081] Output: Movies available to watch or download.

[1082] (Application example 2)

[1083] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1084] Conventional image and video editing systems make it difficult for users to create personalized movies based on their own emotions and scenarios. Manually organizing and editing large amounts of material is laborious and time-consuming. Furthermore, since advanced editing using emotion analysis is not possible, it is not possible to generate content that is in tune with the user's emotions. Therefore, there is a need for a system that can easily generate and provide personalized movies based on the user's emotions and scenarios.

[1085] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1086] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for adjusting scene placement and effects based on the user's emotions using an emotion recognition engine with a generative artificial intelligence model, and a means for providing the generated movie, thereby enabling easy generation of personalized movies based on the user's emotions and scenarios.

[1087] A "user" is an individual or corporation that uses this system and wishes to upload images and videos they have taken and a script to create a movie.

[1088] "Images and videos" are still images and videos taken by users, and are digital files used as material for movies.

[1089] A "script" is a document provided by a user that describes the content and story of a movie, and serves as a guideline for generating the movie.

[1090] An "emotion recognition engine" is software or hardware that analyzes a user's emotions from images and videos and generates emotion data.

[1091] "Storage means" refers to a combination of software and hardware for storing uploaded images and videos in a storage device such as a database.

[1092] The "analyzing means" refers to algorithms and processing devices that perform data analysis based on stored images and videos, as well as scripts, and select appropriate material.

[1093] The "generating means" refers to the algorithms and processing devices that utilize the selected images and video to create a movie.

[1094] The "means for providing" refers to the software and hardware configuration for providing the generated movie to users and enabling viewing and downloading.

[1095] This invention is a system that edits a large number of images and videos taken by a user into a single movie based on a script, and in particular, by combining an emotion recognition engine and a generative artificial intelligence model, it enables the creation of more personalized movies. This system is realized using a server, a terminal, and the following hardware and software.

[1096] Hardware and software:

[1097] Smartphone: A device that allows users to take and upload images and videos.

[1098] Cloud Server: The central location for data storage, analysis, and movie generation.

[1099] Emotion recognition engine (e.g., EmotionEngine): Software that analyzes the user's emotions.

[1100] Natural language processing technology (e.g., NLTK, SpaCy): Software for analyzing the content of scripts.

[1101] Image classification algorithms (e.g., OpenCV): Software for analyzing the content of images and videos.

[1102] Video editing software (e.g., MoviePy): Software for editing a selection of images and videos to create a movie.

[1103] Web frameworks (e.g., Flask, Django): A framework for cloud servers to process and serve data.

[1104] Overview of what the system does:

[1105] 1. A way for users to upload images and videos

[1106] Users upload images and videos taken with their smartphones, as well as scripts, to a cloud server, where the uploaded data is stored.

[1107] 2. Means of data storage

[1108] The cloud server organizes and stores the uploaded images, videos, and scripts in a database.

[1109] 3. Image and video analysis methods

[1110] The server uses an emotion recognition engine to obtain user emotion data from uploaded images and videos, analyzes the script content using natural language processing technology, and selects appropriate images and videos using an image classification algorithm.

[1111] 4. A means of generating a film based on selected material

[1112] The server inputs the selected images and videos into video editing software, which uses generative AI models to determine the order of scenes, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[1113] 5. Means of providing the generated movie

[1114] Users can access, watch, or download completed movies stored on the cloud server from devices such as smartphones.

[1115] Examples:

[1116] Users upload photos and videos from their family trips and are provided with a script based on "fun" and "emotional" elements. The server uses an emotion recognition engine to select happy expressions and touching scenes from the images and videos. It then analyzes the script using natural language processing technology and selects appropriate material using an image classification algorithm. A generative AI model is used to create a movie based on the selected material. Finally, users can watch or download the generated movie from their device.

[1117] Example prompt sentence:

[1118] "Using user-uploaded family vacation photos and videos, use an emotion engine to generate a movie highlighting the fun moments."

[1119] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1120] Step 1:

[1121] This is the stage where users upload images and videos.

[1122] Input: User-taken images, video, and script

[1123] How it works: A user selects images, videos, and scripts using a smartphone application and uploads them to a cloud server. The application then sends these files to the cloud server.

[1124] Output: Images, videos, and script files uploaded to the cloud server

[1125] Step 2:

[1126] This is the stage where the server stores the data.

[1127] Input: Uploaded images, videos, and script files

[1128] Specific operations: The server receives these files and stores each in a database, organizing the database based on user identification information for subsequent processing.

[1129] Output: Images, videos, and script files stored in a database

[1130] Step 3:

[1131] This is the stage where the server acquires emotion data using an emotion recognition engine.

[1132] Input: Image and video files stored in a database

[1133] Specific operation: Using an emotion recognition engine (EmotionEngine), the server analyzes the user's facial expressions and voice from images and videos, and generates emotion data. The emotion data is associated with each file.

[1134] Output: Image and video files associated with emotion data

[1135] Step 4:

[1136] This is the stage where the server analyzes the script, images and videos and selects appropriate materials.

[1137] Input: scripts, images, videos, and emotion data stored in a database

[1138] Specific operation: Analyze the script using natural language processing technology (NLTK, SpaCy) and extract selection criteria. Then, analyze images and videos using an image classification algorithm (OpenCV) and select appropriate materials based on the script and emotion data.

[1139] Output: Selected image and video files

[1140] Step 5:

[1141] This is the stage where the server generates the movie based on the selected material.

[1142] Input: Selected image and video files

[1143] How it works: Using video editing software (MoviePy) and a generative AI model, the server determines the order based on the selected images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[1144] Output: Generated movie file

[1145] Step 6:

[1146] The server now serves the generated movie.

[1147] Input: Generated movie file

[1148] Specific operation: The cloud server stores the generated movie in a database and generates a link that users can access. Users can access the server from their own devices and watch or download the generated movie.

[1149] Output: Movie file that users can watch or download

[1150] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1151] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1152] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1153] [Fourth embodiment]

[1154] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1155] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1156] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1157] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1158] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1159] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1160] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1161] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1162] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1163] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1164] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1165] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1166] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1167] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The purpose of this system is to automatically select and edit the most suitable images and videos based on a script provided by the user, and provide them as a movie.

[1168] System configuration

[1169] 1. Image and video upload feature:

[1170] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[1171] The device sends the images, videos, and script selected by the user to a cloud server.

[1172] 2. Data storage function:

[1173] The server stores the received images, videos, and scripts, organized by user, for further processing.

[1174] 3. Image and video analysis features:

[1175] The server then passes the stored images and videos to an AI for analysis, using natural language processing technology and image classification algorithms.

[1176] The AI ​​model analyzes the provided script and selects the most suitable images and videos for the content, and sends the results of this selection back to the server.

[1177] 4. Movie generation function:

[1178] The server then passes the selected images and videos back to the AI ​​model to generate the movie, which then determines the order of the images and videos, adds transition effects and background music, and edits the movie.

[1179] The completed movie is stored on a server and made accessible to users.

[1180] 5. Movie features:

[1181] The user watches or downloads the generated movie from the server through the terminal, and the server provides the movie data in response to the user's request.

[1182] Specific examples

[1183] For example, consider the case where User A has photos and videos from his / her graduation ceremony and wants to create a movie based on them. User A uploads these photos, videos, and a simple script from his / her device to the cloud server. The server stores the received data.

[1184] The server then analyzes the data using an AI model, which uses natural language processing to interpret the script and uses image classification algorithms to select appropriate photos and videos. The selected data is then sent back to the server for storage.

[1185] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[1186] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy his or her memories as a single movie.

[1187] The processing flow will be explained below.

[1188] Specific processing steps of the program

[1189] Step 1:

[1190] The user launches the application on their device and selects the images, videos, and scripts they want to use. These files are then uploaded as a bundle to the cloud server.

[1191] Step 2:

[1192] The device sends the files (images, videos, scripts) selected by the user to the cloud server along with the user's identification information.

[1193] Step 3:

[1194] The server stores the received images, videos, and scripts in the appropriate databases, organizing them for identification by user.

[1195] Step 4:

[1196] The server inputs the stored images and videos into an AI model, which begins the process of selecting the best material based on the script.

[1197] Step 5:

[1198] The AI ​​model uses natural language processing technology to analyze the content of the script and then analyzes images and videos that fit that content. Specifically, it uses an image classification algorithm to understand the content of the images and videos and select materials that match the scenes in the script.

[1199] Step 6:

[1200] The AI ​​model sends the selected images and videos back to the server, which temporarily stores these selected materials.

[1201] Step 7:

[1202] The server again inputs the selected images and videos into the AI ​​model to begin generating the movie. The AI ​​model then determines the sequence of the footage, adds transition effects, background music, and executes the editing process to edit the movie.

[1203] Step 8:

[1204] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[1205] Step 9:

[1206] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[1207] The above are the specific processing steps of the system.

[1208] Example 1

[1209] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1210] In today's world, individuals are increasingly taking large amounts of images and videos, but it is difficult to effectively organize this data and edit their memories into a movie. Furthermore, conventional video editing systems require advanced technology and a great deal of time and effort, making them unattainable for average users. This invention aims to provide a system that automatically selects and edits optimal images and videos based on a script provided by the user, allowing users to easily enjoy their memories as a movie.

[1211] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1212] In this invention, the server includes means for uploading images and videos taken by users, means for saving the uploaded images and videos, means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, means for generating a movie based on the selected images and videos, means for providing the generated movie, means for analyzing the script provided by the user using natural language processing technology and selecting images and videos that are optimal for the content of the script, and means for determining the order of images and videos using the movie generation means, adding transition effects and background music, and editing the movie. This enables users to easily generate and enjoy movies based on images and videos they have taken, without requiring advanced skills or a great deal of time.

[1213] "User" means any person or entity that uses the System.

[1214] "Terminal" refers to an electronic device that a user uses to connect to and operate the system.

[1215] "Server" refers to the computer system that stores and processes data provided by users and manages the entire system.

[1216] "Images and videos" refers to still image data and video data captured by the user.

[1217] "Upload" refers to the act of sending data from a terminal to a server.

[1218] "Cloud storage" refers to online storage for storing and managing data via the Internet.

[1219] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, interpret, and generate human language.

[1220] "Image classification algorithm" refers to a computer algorithm used to analyze and classify image data.

[1221] A "generative video model" refers to an artificial intelligence model for generating new videos based on image and video data.

[1222] "Transition effect" refers to an effect used to smoothly switch scenes when editing videos.

[1223] "Background music" refers to music tracks that are added to videos or movies to enhance the viewing experience.

[1224] "Script" refers to a document containing a scenario or screenplay provided by a user.

[1225] A "prompt sentence" refers to a sentence of instructions or questions given by a user to a system.

[1226] This invention is a system that effectively organizes images and videos taken by a user and edits them into a movie. In particular, it aims to enable users to easily create a movie by automatically selecting and editing the most suitable images and videos based on a script provided by the user.

[1227] The system has the following main functions:

[1228] 1. Image and video upload feature:

[1229] Users upload the images and videos they have taken from their own devices to the cloud server, and at the same time, they also upload the script.

[1230] The hardware used includes smartphones and PCs, which transmit data to a server via a dedicated application or web interface.

[1231] 2. Data storage function:

[1232] The server stores the received images, videos, and scripts, organized by user, for further processing. The software used includes cloud storage services (e.g., Amazon S3).

[1233] 3. Image and video analysis features:

[1234] The server then passes the stored images and videos to an AI for analysis. Natural language processing techniques (e.g., GPT-4) and image classification algorithms (e.g., ResNet-50) are used for the analysis. The AI ​​model analyzes the script and selects appropriate images and videos.

[1235] 4. Movie generation function:

[1236] The server then passes the selected images and videos back to the AI ​​model to generate the movie. Specifically, it determines the order of the images and videos, adds transition effects and background music, and edits the movie. This process uses a generative animation model.

[1237] 5. Movie features:

[1238] Users watch or download movies generated from the server through their devices. The server provides movie data in response to user requests. Users access movies through a web browser or app.

[1239] Specific examples

[1240] For example, consider a case where a user wants to create a movie based on graduation photos and videos. The user uploads these photos, videos, and a simple script from their device to a cloud server. The server stores the received data and passes it to an AI model (GPT-4 and ResNet-50) for analysis.

[1241] The AI ​​model performs natural language analysis on the script to extract appropriate scenes and uses an image classification algorithm to select related photos and videos. Based on the selected data, the server again uses the AI ​​model to generate the movie. The AI ​​model (generative video model) determines the order of scenes, applies transition effects, and adds background music, editing the movie into a complete film. The completed movie is stored on the server, and users can watch or download it using their devices.

[1242] Prompt Sentence Examples

[1243] "I would like to make a movie using photos and videos from the graduation ceremony. My script includes a scene of the graduates entering first, then memories with their friends, and finally a scene of the diploma presentation. Please edit the movie based on these contents."

[1244] This system allows users to easily create and enjoy movies from images and videos they have taken themselves, without requiring advanced skills or a great deal of time.

[1245] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1246] Step 1:

[1247] Users access the system from their own devices and select and upload images, videos, and scripts. The input is the image data and video data taken by the user, and the script they create. The output is the transfer of these files to the cloud server.

[1248] Specific operation: The user uses the interface of a smartphone or PC to select images, videos, and script files from the file selection screen and presses the upload button. The device then sends these files to the cloud server.

[1249] Step 2:

[1250] The server stores the received images, videos, and scripts. The input content is the image, video, and script data sent from the user's device, and the output is the state in which that data is stored in cloud storage.

[1251] Specific operation: The server organizes each file based on the user ID and stores it in cloud storage (e.g., Amazon S3). At this stage, the data is structured (e.g., by adding metadata).

[1252] Step 3:

[1253] The server sends the stored image and video data to the AI ​​processing server for analysis. The input is the image and video data retrieved from cloud storage, and the output is a list of selected images and videos as the analysis results.

[1254] How it works: The server uses natural language processing technology (e.g., GPT-4) to analyze the script and categorize the script content based on keywords and context. It then uses an image classification algorithm (e.g., ResNet-50) to analyze the stored image and video data. It selects images and videos that match the script content and generates a list of them.

[1255] Step 4:

[1256] The server then passes the selected images and video list to the AI ​​generation model to begin generating the movie. The input is the selected images and video list as the analysis results, and the output is the generated movie file.

[1257] Specific operation: The server uses the generative video model to determine the scene order of the input data, adds transition effects and background music, and edits each scene into a movie. Once the movie generation is complete, the server generates the finished movie file.

[1258] Step 5:

[1259] The server stores the generated movie files in cloud storage and provides a link or download option for users to access. The input is the generated movie file and the output is a link that users can access.

[1260] Specific operation: The server saves the generated movie file to the cloud storage again and generates an access link for it. The user can access this link using their device to watch or download the movie.

[1261] These steps allow users to enjoy their memories as a movie automatically generated from a large number of images and videos.

[1262] (Application example 1)

[1263] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1264] In the past, editing a large number of images and videos taken by a user into a single movie required a great deal of time and effort, and the task of selecting appropriate scenes based on a script was particularly difficult without specialized knowledge. Furthermore, there were limited ways to easily view and download the resulting movie on a smartphone. A system that solves these problems is needed.

[1265] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1266] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for inputting the script and media data uploaded by the user into a generative AI model, and a means for delivering the generated movie data to a smartphone, thereby enabling users to easily generate and watch high-quality movies while saving time and effort.

[1267] "Means for uploading images and videos taken by users" refers to a function that allows users to send images and videos from their own devices to a cloud server.

[1268] "Means for storing the uploaded images and videos" refers to the function of the cloud server to safely store the images and videos received in a database or storage.

[1269] "Means for analyzing the stored images and videos and selecting appropriate images and videos based on the script provided by the user" refers to a function that uses AI technology and algorithms to analyze the content of a script and automatically select the images and videos that best fit that content.

[1270] The "means for generating a movie from the selected images and videos" refers to the function of arranging the selected images and videos in a sequence, adding transition effects and background music, and editing them into a single movie.

[1271] The "means for providing the generated movie" refers to a function that makes the completed movie data accessible to users, allowing them to download or stream the movie.

[1272] "Means for inputting script and media data uploaded by users into the generative AI model" refers to the function of passing the script and image / video data sent by users to the AI ​​model for analysis and use in generating the movie.

[1273] "Means for delivering the generated movie data to a smartphone" refers to the function of providing edited and generated movie files in a downloadable format for smartphone devices.

[1274] This invention is a system that effectively organizes a large number of images and videos taken by a user and edits them into a single movie. This system is designed especially for smartphones, allowing users to easily create, view, and download high-quality movies. Specific embodiments of this system are described below.

[1275] 1. Image and video upload function

[1276] Users can upload images and videos taken with their smartphones to the cloud server. When uploading, users also upload the script. The script specifies the storyline and scene order of the movie and is submitted in text file format.

[1277] 2. Data storage function

[1278] The server securely stores uploaded images, videos, and scripts. The stored data is organized by user and prepared for subsequent processing. This function allows for efficient management of large amounts of data.

[1279] 3. Image and video analysis function

[1280] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model first analyzes the content of the script using natural language processing technology. It then uses an image classification algorithm to analyze the content of the uploaded images and videos and selects the scenes that best fit the script. Generative AI models used include OpenAI's "text-davinci-003."

[1281] 4. Movie generation function

[1282] The server generates a movie from the selected images and videos. Specifically, it determines the order of the selected images and videos and adds transition effects and background music. This results in a high-quality movie. For example, the Python "moviepy" library can be used to generate the movie.

[1283] 5. Movie provision function

[1284] The generated movie is stored on the server and made available for users to access. Users can watch or download the movie from their smartphones. This feature allows users to easily enjoy the movie they have created.

[1285] Specific examples

[1286] As a concrete example, consider the case where User A has photos and videos from a graduation ceremony and wants to use them to create a movie. User A uploads these photos and videos from his smartphone to a cloud server, along with a simple script such as "I want to include the graduation scene first, followed by photos with friends." The server stores the data it receives. The server then analyzes the data using a generative AI model. The AI ​​model analyzes the contents of the script using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. The selected data is then sent back to the server and stored.

[1287] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, and adds background music, and compiles the movie into a complete movie. The finished movie is then stored on the server.

[1288] Finally, User A can watch or download movies from the server using his / her smartphone. The server provides movie data in response to the user's request, allowing User A to enjoy his / her memories as a single movie.

[1289] Prompt Sentence Examples

[1290] The following are examples of prompt sentences:

[1291] Script: I want to include the graduation scene first, followed by photos with friends.

[1292] Please select the most suitable images and videos.

[1293] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1294] Step 1:

[1295] Users upload images and videos taken from their smartphones to the cloud server. In addition, users also upload a script. The inputs include image files, video files, and the script in text file format. The cloud server receives and stores these files. The output is data confirming that the uploaded files are stored on the cloud server.

[1296] Specifically, the user taps the "Upload" button in the application, selects a file, and sends it. The server receives the HTTP request and saves the file in the specified directory.

[1297] Step 2:

[1298] The server stores the uploaded images, videos, and scripts, storing the data in a database organized by user. Inputs include the files received in step 1. Outputs include organized folders and confirmation that the files have been saved to the database.

[1299] Specifically, the server performs a database operation to store files in separate directories for each user.

[1300] Step 3:

[1301] The server passes the stored images and videos to a generative AI model for analysis. The AI ​​model uses natural language processing technology to analyze the content of the script and then uses an image classification algorithm to select images and videos based on that analysis. The input includes the script's text data and image / video files. The output is a list of selected images and videos.

[1302] Specifically, the server generates a prompt for the AI ​​model and makes an API call to obtain the analysis results. Examples of prompts include the following:

[1303] Script: I want to include the graduation scene first, followed by photos with friends.

[1304] Please select the most suitable images and videos.

[1305] Step 4:

[1306] The server generates a movie from the selected images and videos. It uses the Python "moviepy" library to determine the order of the selected images and videos and add transition effects and background music. The input includes the list of selected images and videos obtained from the AI ​​model in step 3. The output is an edited movie file.

[1307] Specifically, the server uses the "moviepy" library to combine each media file and add specified effects and music.

[1308] Step 5:

[1309] The resulting movie is stored on a server and made available for user access. The input includes the completed movie file. The output is a movie file ready for users to watch or download.

[1310] Specifically, the server saves the completed movie file in a specific directory and generates an access link for the user, who taps the link through the application interface to watch or download the movie.

[1311] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1312] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. The system aims to select and edit the optimal images and videos based on a script provided by the user, and provide them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized movie generation becomes possible.

[1313] System configuration

[1314] 1. Image and video upload feature:

[1315] Users upload captured images and videos from their own devices to the cloud server, and at the same time, they also upload the script.

[1316] The device sends the images, videos, and script selected by the user to a cloud server.

[1317] 2. Data storage function:

[1318] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[1319] 3. Emotion recognition function:

[1320] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[1321] 4. Image and video analysis features:

[1322] The server inputs the stored images and videos into an AI and emotion engine, and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also uses the results of the emotion engine to select the most emotionally appropriate material.

[1323] 5. Movie generation function:

[1324] The server then passes the selected images and videos back to the AI ​​model to generate the movie. The AI ​​model then determines the sequence of the footage, adds transition effects and background music, and executes the editing process for the movie. Scene placement and effects are also adjusted based on the emotional data.

[1325] 6. Movie features:

[1326] The completed movie is stored on a server and made accessible to users, and the server links the movie file to a user interface.

[1327] Users can watch or download movies generated from the server through their terminals, and the server provides movie data in response to user requests.

[1328] Specific examples

[1329] For example, consider the case where User A has wedding photos and videos and wants to create a movie based on them. User A uploads the photos, videos, and a script depicting the touching moments of the wedding from his / her device to the cloud server. The server stores the received data.

[1330] Next, the server uses an emotion engine to analyze User A's facial expressions and voice from images and videos to obtain emotional data. The AI ​​model then analyzes the script content using natural language processing technology and selects appropriate photos and videos using an image classification algorithm. At the same time, it also refers to the results of the emotion engine to select the most emotionally appropriate material.

[1331] The server then begins generating the movie based on this selected data. The AI ​​model determines the order of scenes, applies transition effects, adds background music, and edits it into a movie, making adjustments based on the emotional data. The finished movie is then stored on the server.

[1332] Finally, User A uses a terminal to watch or download a movie from the server. The server provides movie data in response to the user's request, allowing User A to enjoy a movie that reflects his or her memories in a way that is emotionally relevant.

[1333] The processing flow will be explained below.

[1334] Specific processing flow

[1335] Step 1:

[1336] Users launch the application from their device, select the images, videos, and scripts they want to use, and upload the selected files as a bundle to the cloud server.

[1337] Step 2:

[1338] The device sends the images, videos, and scripts selected by the user to a cloud server, along with the user's identification information.

[1339] Step 3:

[1340] The server stores the received images, videos, and scripts in the appropriate database, organized by user, for further processing.

[1341] Step 4:

[1342] The server inputs the stored images and videos into an emotion engine and recognizes emotion data from the user's facial expressions and voice.

[1343] Step 5:

[1344] The emotion engine analyzes the user's facial expressions and voice from images and videos, and sends the recognized emotion data back to the server, where it is stored.

[1345] Step 6:

[1346] The server inputs the stored images and videos into an AI model and selects the most appropriate material based on the script and emotional data. Specifically, it analyzes the content of the script using natural language processing technology and the content of the images and videos using an image classification algorithm. It also refers to the emotional data recognized by the emotion engine to select emotionally appropriate material.

[1347] Step 7:

[1348] The AI ​​model sends the selected images and videos back to the server, which temporarily stores the selected materials.

[1349] Step 8:

[1350] The server then inputs the selected images and videos back into the AI ​​model, which then begins generating the movie. The AI ​​model then determines the sequence of the footage and edits the movie, adding transition effects and background music. Scene placement and effects are also adjusted based on the emotional data.

[1351] Step 9:

[1352] The completed movie is stored on a server and made accessible to users, and the server links the completed movie file to a user interface.

[1353] Step 10:

[1354] The user accesses the server through a terminal to view or download the generated movie, and the server provides the movie file in response to the user's request.

[1355] Example 2

[1356] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1357] When effectively organizing a large number of images and videos taken by a user and editing them into a single movie, it is difficult to automatically generate a personalized work that is emotionally relevant. To address this issue, it is necessary to detect emotions from the user's facial expressions and voice, select appropriate materials, and reduce the effort and time required to compile them into a movie.

[1358] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading images and videos taken by the user, a means for saving the uploaded images and videos, a means for inputting the saved images and videos into an emotion analysis engine and recognizing emotion data, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for generating a movie based on the selected images and videos, and a means for providing the generated movie. This makes it possible to automatically generate a movie that is in line with the user's emotions, significantly reducing time and effort.

[1359] "Uploading" refers to the act of sending images and videos taken by a user from a terminal to a server.

[1360] "Storage" refers to the act of storing the images, videos, and scripts received by the server in a database or storage device in preparation for subsequent processing.

[1361] An "emotion analysis engine" is software or hardware that analyzes a user's facial expressions and voice from stored images and videos to recognize emotional data.

[1362] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[1363] An "image classification algorithm" is an algorithm that analyzes the content of images and videos and classifies them into specific categories or features.

[1364] "Movie creation means" means a means for editing a movie based on selected images and videos by determining the order of scenes, adding transition effects, background music, and effects.

[1365] "Providing" is the act of making the generated movie available for viewing or download by users.

[1366] A "script" is a document that indicates the guidelines and scenario of a video work provided by a user.

[1367] This invention is a system that effectively organizes a large number of images and videos taken by users and edits them into a single movie. This system has the function of selecting and editing the optimal images and videos based on a script provided by the user, and providing them as a movie. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, personalized movie generation becomes possible.

[1368] A specific embodiment for carrying out the present invention will be described.

[1369] System configuration

[1370] 1. Server and terminal hardware and software

[1371] The devices used by users include general devices such as smartphones and PCs, which have the function of uploading images and videos.

[1372] The server needs storage and processing power to store and analyze the data, and a NoSQL database (e.g., MongoDB) may be used as the database.

[1373] 2. Sentiment Analysis Engine

[1374] The emotion analysis engine used by the server is software that can analyze the user's facial expressions and voice (e.g., facial recognition API or voice analysis API). Specific examples include "Face API" and "Text Analytics."

[1375] 3. Natural language processing engine and image classification algorithm

[1376] The server analyzes the script using natural language processing technology, such as Cloud Natural Language API and IBM Watson.

[1377] Image classification algorithms (e.g., ResNet) are used to classify images and videos.

[1378] 4. AI Model

[1379] AI models that could be used to generate the movie include DALL-E and GPT-4, which support the editing process by determining the order of scenes, applying transition effects, and adding background music.

[1380] Specific examples

[1381] For example, let us consider the case where user A wants to create a movie based on wedding photos and videos.

[1382] User A uses the terminal to upload wedding photos, videos, and a script depicting touching moments from the wedding to the cloud server.

[1383] The server receives this data and stores it in a database.

[1384] Next, the server inputs the saved images and videos into an emotion analysis engine to obtain emotion data from User A's facial expressions and voice.

[1385] The saved script is input into a natural language processing engine, which analyzes the content of the script and extracts important scenes and keywords.

[1386] The server uses an image classification algorithm to select the most suitable images and videos based on the emotional data and script.

[1387] The selected materials are input into the AI ​​model, which determines the order of scenes, applies transition effects, adds background music, and adjusts scene placement and effects based on emotional data.

[1388] Finally, the generated movie is stored on the server and made available for user A to view or download.

[1389] Prompt Sentence Examples

[1390] "We want to create a moving movie based on photos and videos taken at a user's wedding. Below is the script and emotional data. Please analyze them, select the best footage, determine the scene order, apply background music and transition effects, and edit the movie."

[1391] Using this system, users can automatically generate movies that evoke their emotions, allowing them to enjoy their memories in a deeper way.

[1392] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1393] Step 1:

[1394] The user selects images and videos on the device

[1395] Users select the images and videos they want to upload from their smartphone or PC, and also provide a script to use as a guide for the movie.

[1396] Input: User-selected images, videos, and scripts.

[1397] Output: The selected data saved on the device.

[1398] Step 2:

[1399] The device sends data to the cloud server

[1400] The device compresses the selected images, videos, and scripts and sends them to a cloud server using a secure communication protocol (e.g., HTTPS).

[1401] Input: Selected images, videos, scripts.

[1402] Output: Send data to cloud server.

[1403] Step 3:

[1404] The server receives the data

[1405] The server receives the images, videos, and scripts sent from the terminals.

[1406] Input: Images, videos, and scripts sent to the cloud server.

[1407] Output: The received data.

[1408] Step 4:

[1409] The server saves the data to a database

[1410] The server identifies the received data for each user and stores it in an appropriate database, such as a NoSQL database like MongoDB.

[1411] Input: Received images, videos, scripts.

[1412] Output: Data stored in the database.

[1413] Step 5:

[1414] The server inputs the images and videos into the sentiment analysis engine

[1415] The server inputs the stored images and videos into an emotion analysis engine, for example, using a facial recognition API or a voice analysis API.

[1416] Input: Images and videos stored in a database.

[1417] Output: Analysis results of emotion data.

[1418] Step 6:

[1419] The sentiment analysis engine generates sentiment data

[1420] The emotion analysis engine recognizes emotion data from the user's facial expressions and voice data and returns it to the server.

[1421] Input: Images and videos.

[1422] Output: Emotion data.

[1423] Step 7:

[1424] The server inputs the script into a natural language processing engine

[1425] The server inputs the uploaded script into a natural language processing engine, for example, using the Cloud Natural Language API or IBM Watson.

[1426] Input: script.

[1427] Output: Parsed script content.

[1428] Step 8:

[1429] A natural language processing engine analyzes the script

[1430] The natural language processing engine analyzes the contents of the script and extracts important keywords and scenes.

[1431] Input: script.

[1432] Output: Parsed keywords and scenes.

[1433] Step 9:

[1434] The server uses image classification algorithms to select the best images and videos.

[1435] The server uses an image classification algorithm, such as ResNet, to select the most suitable images and videos based on the analyzed script content and emotional data.

[1436] Input: Script analysis results, emotion data.

[1437] Output: Selected images and videos.

[1438] Step 10:

[1439] The server inputs the selected data into the AI ​​model

[1440] The server inputs the selected images and videos into the AI ​​model and begins generating the movie. The AI ​​model used is a generative AI model.

[1441] Input: Selected images and videos.

[1442] Output: The generated film material.

[1443] Step 11:

[1444] AI model generates movies

[1445] The AI ​​model determines the order of images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[1446] Input: Images and videos, emotion data.

[1447] Output: The generated movie.

[1448] Step 12:

[1449] The server stores the finished movie

[1450] The server stores the generated movies in a database and prepares them for distribution to users.

[1451] Input: The generated movie.

[1452] Output: Saved movie.

[1453] Step 13:

[1454] The user watches or downloads the movie through the device

[1455] Users can use their terminals to access the server and watch or download the completed movie, and the server provides a link to the movie file through a user interface.

[1456] Input: Saved movies, user requests.

[1457] Output: Movies available to watch or download.

[1458] (Application example 2)

[1459] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1460] Conventional image and video editing systems make it difficult for users to create personalized movies based on their own emotions and scenarios. Manually organizing and editing large amounts of material is laborious and time-consuming. Furthermore, since advanced editing using emotion analysis is not possible, it is not possible to generate content that is in tune with the user's emotions. Therefore, there is a need for a system that can easily generate and provide personalized movies based on the user's emotions and scenarios.

[1461] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1462] In this invention, the server includes a means for uploading images and videos taken by a user, a means for saving the uploaded images and videos, a means for analyzing the saved images and videos and selecting appropriate images and videos based on a script provided by the user, a means for adjusting scene placement and effects based on the user's emotions using an emotion recognition engine with a generative artificial intelligence model, and a means for providing the generated movie, thereby enabling easy generation of personalized movies based on the user's emotions and scenarios.

[1463] A "user" is an individual or corporation that uses this system and wishes to upload images and videos they have taken and a script to create a movie.

[1464] "Images and videos" are still images and videos taken by users, and are digital files used as material for movies.

[1465] A "script" is a document provided by a user that describes the content and story of a movie, and serves as a guideline for generating the movie.

[1466] An "emotion recognition engine" is software or hardware that analyzes a user's emotions from images and videos and generates emotion data.

[1467] "Storage means" refers to a combination of software and hardware for storing uploaded images and videos in a storage device such as a database.

[1468] The "analyzing means" refers to algorithms and processing devices that perform data analysis based on stored images and videos, as well as scripts, and select appropriate material.

[1469] The "generating means" refers to the algorithms and processing devices that utilize the selected images and video to create a movie.

[1470] The "means for providing" refers to the software and hardware configuration for providing the generated movie to users and enabling viewing and downloading.

[1471] This invention is a system that edits a large number of images and videos taken by a user into a single movie based on a script, and in particular, by combining an emotion recognition engine and a generative artificial intelligence model, it enables the creation of more personalized movies. This system is realized using a server, a terminal, and the following hardware and software.

[1472] Hardware and software:

[1473] Smartphone: A device that allows users to take and upload images and videos.

[1474] Cloud Server: The central location for data storage, analysis, and movie generation.

[1475] Emotion recognition engine (e.g., EmotionEngine): Software that analyzes the user's emotions.

[1476] Natural language processing technology (e.g., NLTK, SpaCy): Software for analyzing the content of scripts.

[1477] Image classification algorithms (e.g., OpenCV): Software for analyzing the content of images and videos.

[1478] Video editing software (e.g., MoviePy): Software for editing a selection of images and videos to create a movie.

[1479] Web frameworks (e.g., Flask, Django): A framework for cloud servers to process and serve data.

[1480] Overview of what the system does:

[1481] 1. A way for users to upload images and videos

[1482] Users upload images and videos taken with their smartphones, as well as scripts, to a cloud server, where the uploaded data is stored.

[1483] 2. Means of data storage

[1484] The cloud server organizes and stores the uploaded images, videos, and scripts in a database.

[1485] 3. Image and video analysis methods

[1486] The server uses an emotion recognition engine to obtain user emotion data from uploaded images and videos, analyzes the script content using natural language processing technology, and selects appropriate images and videos using an image classification algorithm.

[1487] 4. A means of generating a film based on selected material

[1488] The server inputs the selected images and videos into video editing software, which uses generative AI models to determine the order of scenes, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[1489] 5. Means of providing the generated movie

[1490] Users can access, watch, or download completed movies stored on the cloud server from devices such as smartphones.

[1491] Examples:

[1492] Users upload photos and videos from their family trips and are provided with a script based on "fun" and "emotional" elements. The server uses an emotion recognition engine to select happy expressions and touching scenes from the images and videos. It then analyzes the script using natural language processing technology and selects appropriate material using an image classification algorithm. A generative AI model is used to create a movie based on the selected material. Finally, users can watch or download the generated movie from their device.

[1493] Example prompt sentence:

[1494] "Using user-uploaded family vacation photos and videos, use an emotion engine to generate a movie highlighting the fun moments."

[1495] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1496] Step 1:

[1497] This is the stage where users upload images and videos.

[1498] Input: User-taken images, video, and script

[1499] How it works: A user selects images, videos, and scripts using a smartphone application and uploads them to a cloud server. The application then sends these files to the cloud server.

[1500] Output: Images, videos, and script files uploaded to the cloud server

[1501] Step 2:

[1502] This is the stage where the server stores the data.

[1503] Input: Uploaded images, videos, and script files

[1504] Specific operations: The server receives these files and stores each in a database, organizing the database based on user identification information for subsequent processing.

[1505] Output: Images, videos, and script files stored in a database

[1506] Step 3:

[1507] This is the stage where the server acquires emotion data using an emotion recognition engine.

[1508] Input: Image and video files stored in a database

[1509] Specific operation: Using an emotion recognition engine (EmotionEngine), the server analyzes the user's facial expressions and voice from images and videos, and generates emotion data. The emotion data is associated with each file.

[1510] Output: Image and video files associated with emotion data

[1511] Step 4:

[1512] This is the stage where the server analyzes the script, images and videos and selects appropriate materials.

[1513] Input: scripts, images, videos, and emotion data stored in a database

[1514] Specific operation: Analyze the script using natural language processing technology (NLTK, SpaCy) and extract selection criteria. Then, analyze images and videos using an image classification algorithm (OpenCV) and select appropriate materials based on the script and emotion data.

[1515] Output: Selected image and video files

[1516] Step 5:

[1517] This is the stage where the server generates the movie based on the selected material.

[1518] Input: Selected image and video files

[1519] How it works: Using video editing software (MoviePy) and a generative AI model, the server determines the order based on the selected images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data.

[1520] Output: Generated movie file

[1521] Step 6:

[1522] The server now serves the generated movie.

[1523] Input: Generated movie file

[1524] Specific operation: The cloud server stores the generated movie in a database and generates a link that users can access. Users can access the server from their own devices and watch or download the generated movie.

[1525] Output: Movie file that users can watch or download

[1526] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1527] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1528] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1529] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1530] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1531] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1532] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1533] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1534] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1535] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1536] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1537] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1538] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1539] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1540] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1541] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1542] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1543] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1544] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1545] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1546] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1547] The following is further disclosed regarding the above embodiment.

[1548] (Claim 1)

[1549] a means for uploading images and videos taken by the user;

[1550] means for storing the uploaded images and videos;

[1551] means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by a user;

[1552] means for generating a movie from the selected images and videos;

[1553] means for providing said generated movie;

[1554] A system including:

[1555] (Claim 2)

[1556] 2. The system of claim 1, wherein the image and video analysis means analyzes the content of the script using natural language processing techniques and analyzes the content of the images and videos using image classification algorithms.

[1557] (Claim 3)

[1558] 10. The system of claim 1, wherein the means for generating the movie determines the sequence of the selected images and videos and adds transition effects and background music.

[1559] "Example 1"

[1560] (Claim 1)

[1561] a means for uploading images and videos taken by the user;

[1562] means for storing the uploaded images and videos;

[1563] means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by a user;

[1564] means for generating a movie from the selected images and videos;

[1565] means for providing said generated movie;

[1566] A means for analyzing the script provided by the user using natural language processing technology and selecting images and videos that are optimal for the content of the script;

[1567] a means for determining the order of images and videos by the movie generating means, and editing the images and videos into a movie by adding transition effects and background music;

[1568] A system including:

[1569] (Claim 2)

[1570] 2. The system of claim 1, wherein the image and video analysis means analyzes the content of the script using natural language processing techniques and analyzes the content of the images and videos using image classification algorithms.

[1571] (Claim 3)

[1572] 10. The system of claim 1, wherein the means for generating the movie determines the sequence of the selected images and videos and adds transition effects and background music.

[1573] "Application Example 1"

[1574] (Claim 1)

[1575] a means for uploading images and videos taken by the user;

[1576] means for storing the uploaded images and videos;

[1577] means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by a user;

[1578] means for generating a movie from the selected images and videos;

[1579] means for providing said generated movie;

[1580] a means for inputting user-uploaded script and media data into a generative AI model;

[1581] means for delivering the generated movie data to a smartphone;

[1582] A system including:

[1583] (Claim 2)

[1584] 2. The system of claim 1, wherein the image and video analysis means analyzes the content of the script using natural language processing techniques and analyzes the content of the images and videos using image classification algorithms.

[1585] (Claim 3)

[1586] 10. The system of claim 1, wherein the means for generating the movie determines the sequence of the selected images and videos and adds transition effects and background music.

[1587] "Example 2: Combining Emotion Engines"

[1588] Claiming a new invention

[1589] (Claim 1)

[1590] a means for uploading images and videos taken by the user;

[1591] means for storing the uploaded images and videos;

[1592] means for inputting the stored images and videos into an emotion analysis engine to recognize emotion data;

[1593] means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by a user;

[1594] means for generating a movie from the selected images and videos;

[1595] means for providing said generated movie;

[1596] A system including:

[1597] (Claim 2)

[1598] 2. The system of claim 1, wherein the image and video analysis means analyzes the content of the script using natural language processing techniques and analyzes the content of the images and videos using image classification algorithms.

[1599] (Claim 3)

[1600] 10. The system of claim 1, wherein the means for generating the movie determines the order of the selected images and videos, adds transition effects and background music, and adjusts them based on emotional data.

[1601] "Application example 2 when combining emotion engines"

[1602] (Claim 1)

[1603] a means for uploading images and videos taken by the user;

[1604] means for storing the uploaded images and videos;

[1605] means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by a user;

[1606] means for generating a movie from the selected images and videos;

[1607] a means for adjusting scene placement and effects based on a user's emotions using an emotion recognition engine using a generative artificial intelligence model;

[1608] means for providing said generated movie;

[1609] A system including:

[1610] (Claim 2)

[1611] The system of claim 1, wherein the image and video analysis means analyzes the content of the script using natural language processing technology, analyzes the content of the images and videos using an image classification algorithm, and selects appropriate material by referring to the results of an emotion recognition engine.

[1612] (Claim 3)

[1613] 2. The system of claim 1, wherein the means for generating the movie determines the order of the selected images and videos, adds transition effects and background music, and adjusts scene placement and effects based on emotional data. [Explanation of symbols]

[1614] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for uploading images and videos taken by the user; means for storing the uploaded images and videos; means for analyzing the stored images and videos and selecting appropriate images and videos based on a script provided by a user; means for generating a movie from the selected images and videos; means for providing said generated movie; A system including:

2. 2. The system of claim 1, wherein the image and video analysis means analyzes the content of the script using natural language processing techniques and analyzes the content of the images and videos using image classification algorithms.

3. 2. The system of claim 1, wherein said means for generating a movie determines the sequence of said selected images and videos and adds transition effects and background music.

Images