system

Abstract

Provide a system. A method for receiving a photo or video file from a user; A means for saving received photo or video files to cloud storage; A means for analyzing stored photo or video files to recognize people and obtain their age information; a means for generating a chronological event list based on the acquired age information and metadata; A means for automatically generating a scenario for a history video based on an event list; A means for generating a history video using an automatically generated scenario; A means for notifying a user of the generated history video and providing a download link; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, it has become common to take photos and videos to record personal growth and memories. However, it remains difficult to efficiently organize these photos and videos and automatically generate a unified history video. Furthermore, there is no easy-to-use editing function available for users who wish to manually edit the generated video. Therefore, there is a need for a system that can centrally manage individual records, automatically generate high-quality history videos, and provide a function that allows users to manually edit them as needed. [Means for solving the problem]

[0005] The present invention provides a system that receives photo or video files uploaded by users and stores them in cloud storage, analyzes the stored files to identify people and acquire age information, generates a chronological event list based on the acquired age information and metadata, and automatically generates a scenario for a video history based on the event list. The system then generates a video history based on the automatically generated scenario, notifies the user, and provides a download link, thereby solving the above-mentioned problems.

[0006] Furthermore, a manual editing function for the generated history video is provided as a paid option, and a means for rendering the final history video based on the user's edits is also provided. By allowing the user to change the order of the videos, correct the text, and change the background music through the operation screen of the editing tool, and by including a means for saving the final edited video in cloud storage and notifying the user, it is possible to improve user convenience and increase profitability.

[0007] "Photo or video file" refers to still image or video data intended to be uploaded by a user.

[0008] "User" refers to an individual or legal entity that uses the system to upload photos or video files and generate history videos.

[0009] "Cloud storage" refers to a collection of remote servers for storing data over the Internet.

[0010] "Analysis" refers to the process of recognizing people, obtaining age information, extracting metadata, etc. from received photo or video files.

[0011] "Person recognition" refers to identifying the faces of people contained in a photo or video file and extracting their features.

[0012] "Age Information" refers to information regarding the estimated age of a person appearing in a photograph or video file.

[0013] "Metadata" refers to additional information associated with a photo or video file, such as the date and time it was taken or its location.

[0014] An "event list" refers to a list of occurrences or events organized chronologically based on analyzed age information and metadata.

[0015] A "scenario" is a component of the history video to be generated, and refers to a detailed plan including the sequence, text, and transition effects.

[0016] "History videos" refer to video content that summarizes a person's growth and events, automatically generated based on analyzed photos and video files.

[0017] "Download Link" refers to the URL or resource address from which a User can download the generated history video.

[0018] The "manual editing function" refers to a function that allows a user to change the content and order of the history video that has been generated.

[0019] "Paid options" refer to extensions and services that are offered in addition to basic functions at an additional cost.

[0020] "Rendering" refers to the process of generating the final video file based on the edited scenario and data.

[0021] "Notification" refers to the act of informing a user of specific information or the occurrence of an event. [Brief explanation of the drawings]

[0022] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[0044] 1. Uploading photo and video files

[0045] User: The user registers as a member and logs into the system through the system's web interface or a dedicated application, then selects the photo and video files and clicks the upload button.

[0046] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[0047] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[0048] 2. Analyzing the file and generating time series data

[0049] Server: Analyzes the stored files using image processing algorithms, including facial recognition, age estimation, and extraction of metadata such as photo date and location.

[0050] Server: Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and shooting date and time.

[0051] 3. Automatic generation of history videos

[0052] Server: Automatically generate a historical video scenario based on time-series data, including the file display order, inserted text, transition effects, background music, etc.

[0053] Server: Edits the history video using the automatically generated scenario and generates the final video file.

[0054] 4. User Notification and Download

[0055] Server: Saves the generated history video to cloud storage and generates a download link.

[0056] Server: Notifies the user when the history video is complete and provides a download link.

[0057] 5. Manual editing function (paid option)

[0058] User: If you are not satisfied with the generated history video, you can select the manual editing function, which is available as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[0059] Server: Allows users to reorder videos, modify text, and change background music through editing tools.

[0060] User: Use the editing tools to make any necessary corrections.

[0061] On your device: Send your edits to the server.

[0062] 6. Edited video generation and notification

[0063] Server: Renders and generates the final history video based on the received edits.

[0064] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0065] Specific examples

[0066] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server then stores these files in cloud storage and begins analysis. Based on age estimation and metadata extraction, time-series data is generated, and a history video scenario is automatically created based on this data.

[0067] The generated history video is automatically edited and a download link is provided to the user. Furthermore, if the user wishes to make further edits to the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0068] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos. Furthermore, by providing a manual editing function as a paid option, it is possible to improve both user convenience and profitability.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] User: Logs in to the system's web interface or a dedicated application.

[0072] Step 2:

[0073] Server: Authenticates the user's login information and displays the upload screen.

[0074] Step 3:

[0075] User: Select the photo or video file you want to upload and click the upload button.

[0076] Step 4:

[0077] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[0078] Step 5:

[0079] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[0080] Step 6:

[0081] Server: Analyzes stored photo and video files, using facial recognition technology to identify people and estimate their age, as well as extract metadata such as the date and time of the photo and location.

[0082] Step 7:

[0083] Server: Generates time series data for photos and video files based on the analysis results, including creating an event list based on age information and shooting date and time.

[0084] Step 8:

[0085] Server: Automatically generate a scenario for a historical video based on time-series data, including the display order of each file, inserted text, transition effects, and background music selection.

[0086] Step 9:

[0087] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[0088] Step 10:

[0089] Server: Stores the generated history video in cloud storage and provides users with a notification and a download link.

[0090] Step 11:

[0091] User: After viewing the generated history video, if they feel they need to edit it manually, they can select the manual editing feature as a paid option.

[0092] Step 12:

[0093] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[0094] Step 13:

[0095] User: Edit videos by changing the order, modifying text, changing background music, etc.

[0096] Step 14:

[0097] On your device: Send your edits to the server.

[0098] Step 15:

[0099] Server: Re-renders and generates the final history video based on the submitted edits.

[0100] Step 16:

[0101] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] Conventional photo and video management systems simply store files and lack the ability to generate meaningful content along a timeline. This makes it difficult for users to effectively relive their memories. Furthermore, if users are dissatisfied with the automatically generated content, they often need specialized knowledge to manually edit it, leaving many users unsatisfied. The purpose of this invention is to solve these problems.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for receiving image or video data from a user, means for storing the received image or video data in a remote storage device, means for analyzing the stored image or video data and acquiring person and age information using facial recognition technology, means for generating a timeline-based event list based on the acquired age information and metadata, means for automatically generating a scenario based on the event list, means for generating a video file using the automatically generated scenario, and means for notifying the user of the generated video file and providing a download link. This allows the user to receive automatically generated high-quality history videos and also provides the option to manually edit them, thereby improving user satisfaction.

[0107] "User" refers to an individual or organization that uses the system to upload image and video data and generate history videos based on that data.

[0108] "Image or video data" refers to all photos and video files uploaded by users.

[0109] "Server" refers to the central computer device of this system, which receives, analyzes, stores, and processes data, and generates videos.

[0110] "Remote storage device" refers to a data storage device located in a physically separate location, such as cloud storage, that is used to store image and video data received from a server.

[0111] "Facial recognition technology" refers to technology that identifies a person's face from image or video data and identifies their age and individuality.

[0112] "Metadata" refers to additional information embedded in image and video data, such as the date and time of shooting, location information, and camera settings.

[0113] A "chronological event list" refers to a list of images and video data organized chronologically based on facial recognition technology and metadata.

[0114] A "scenario" refers to a plan for generating a chronological storyboard based on a user's image or video data, and setting the display order, including inserted text and transition effects.

[0115] "Video file" refers to video content that is generated based on a scenario and can be downloaded by users.

[0116] "Manual editing function" refers to the function that allows users to modify and edit the content of the history video they have created themselves.

[0117] "Progress display" refers to a feature that allows users to check the upload and video generation process in real time.

[0118] The "operation screen" refers to an interface that allows a user to use the manual editing function to change the order of videos, modify text, change background music, and so on.

[0119] MODE FOR CARRYING OUT THE INVENTION

[0120] The present invention is a system for automatically generating a history video along a time axis based on image or video data uploaded by a user. Specific embodiments of the present invention will be described below.

[0121] 1. System Overview

[0122] Users can use the system's web interface or a dedicated application to register and log in to upload images and video data. A file selection area and a progress bar are provided, allowing users to check the upload progress in real time.

[0123] Terminal: Sends the file selected by the user to the server. The sent file is temporarily stored and then transferred to cloud storage (e.g., AWS (registered trademark) S3).

[0124] Server: Uses facial recognition and metadata analysis technologies to analyze received image and video data. Specifically, it uses OpenCV and TENSORFLOW (registered trademark) to perform facial recognition and age estimation, and extracts metadata such as the shooting date and time and location information embedded in the image and video data. Based on this, it generates a timeline-based event list.

[0125] 2. Automatic generation of history videos

[0126] Server: Based on the event list, the server automatically generates a scenario for the history video. The scenario includes the display order of images and video data, the text to be inserted, transition effects, background music, etc. This forms a storyboard.

[0127] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg. The generated video file is then saved in cloud storage.

[0128] Server: Notifies the user of the download link for the generated history video. The user can download the video from the provided link.

[0129] 3. Manual editing function

[0130] Users: If you are not satisfied with the history video you have created, you can use the manual editing feature as a paid option. After successful payment, you will be provided with a video editing tool interface where you can change the order, edit the text, change the background music, etc.

[0131] Server: The final video is generated again based on the edits sent by the user. The generated final video is saved in cloud storage, and the download link is sent to the user again.

[0132] Specific examples

[0133] For example, consider the case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs in to the system's web interface, selects the files on the upload screen, and sends them. The device sends the files to the server while displaying the progress in real time. The server temporarily saves the files, then transfers them to cloud storage and begins analysis. The analysis involves estimating the age and extracting metadata, which is then used to generate time-series data. A history video scenario is automatically created based on this data, and the video is edited using FFmpeg. The final history video is saved on the server, and the user is notified of a download link.

[0134] Prompt Sentence Examples

[0135] For example, you can use the following prompt:

[0136] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[0137] This system allows users to easily create high-quality history videos, and manual editing functions can be used to improve satisfaction.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Program processing flow

[0140] Step 1: User registration and login

[0141] Input: Information entered by the user when registering as a member (email address, password, username).

[0142] Server: The user registers as a member using the system's web interface or a dedicated application. The registration information is saved in the database and an email is sent to the user confirming registration.

[0143] Output: Registration success message and login information.

[0144] Input: The information the user enters when logging in (email address, password).

[0145] Server: Authenticate the login information, and if authentication is successful, display the dashboard screen.

[0146] Output: A dashboard screen for the user.

[0147] Step 2: Upload your photo and video files

[0148] Input: Image or video data uploaded by the user.

[0149] Users: After logging in, click the "Upload" button on the dashboard and select the file.

[0150] Terminal: Prepares the selected file to be sent to the server. A progress bar is displayed to show the upload progress in real time.

[0151] Output: Progress of file submission.

[0152] Input: Image or video data sent from the device.

[0153] Server: Temporarily stores received files and eventually transfers them to cloud storage (e.g., AWS S3).

[0154] Output: Image or video data saved in cloud storage, upload completion notification.

[0155] Step 3: Parse the file and generate time series data

[0156] Input: Image or video data stored in cloud storage.

[0157] Server: Uses facial recognition and metadata analysis technologies (OpenCV, TensorFlow, etc.) to analyze the stored files. Extracts metadata such as facial recognition, age estimation, and photo date and location information.

[0158] Output: Face recognition results, age information, shooting date and location information.

[0159] Input: Analysis results (face recognition results, age information, photo date and time, and location information).

[0160] Server: Based on the analysis results, a timeline-based event list is generated for each file. Facial recognition and age information are used to organize the developmental process and events of people in chronological order.

[0161] Output: A chronological list of events.

[0162] Step 4: Automatic generation of history videos

[0163] Input: Event list.

[0164] Server: Automatically generates a scenario based on the event list, including the display order, inserted text, transition effects, background music, etc.

[0165] Output: The automatically generated scenario.

[0166] Input: An automatically generated scenario.

[0167] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg.

[0168] Output: The final history video file.

[0169] Step 5: Notify users and download

[0170] Input: The generated history video file.

[0171] Server: Saves the generated video file in cloud storage and generates a download link.

[0172] Output: Download link.

[0173] Input: Download link.

[0174] Server: Notify the user of the download link for the generated history video.

[0175] Output: A notification message with a download link.

[0176] Step 6: Manual Editing (paid option)

[0177] Input: User edit request.

[0178] User: If you are not satisfied with the history video, you can use the manual editing function as a paid option and make an online payment.

[0179] Server: After successful payment, the server provides the editing tool operation interface, allowing users to change the order, edit the text, change the background music, etc.

[0180] Output: Editing tool operation screen.

[0181] Input: The modifications made by the user in the editing tools.

[0182] On your device: Send your edits to the server.

[0183] Server: The final video is generated again based on the received edits. The generated final video is saved in cloud storage and the download link is sent to the user again.

[0184] Output: Final history video file, notification message with download link.

[0185] Specific examples

[0186] For example, if a user wants to upload 10 photos and 2 videos documenting their child's development, they would go through the following specific steps:

[0187] Step 1: The user logs in to the system and selects a file on the upload screen. The server displays the progress in real time and saves the file to cloud storage.

[0188] Step 2: The server analyzes the stored files, performs facial recognition and metadata analysis, and generates time series data based on this.

[0189] Step 3: The server automatically generates a scenario based on the generated time series data and generates the final history video using FFmpeg.

[0190] Step 4: The server saves the generated video to cloud storage and notifies the user of the download link.

[0191] Step 5: The user can manually edit the video as needed, generate the final version, and receive a notification again.

[0192] Prompt Sentence Examples

[0193] For example, you can use the following prompt:

[0194] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] In modern society, the act of reminiscing about past events using photos and videos is becoming increasingly important. However, manually organizing these photos and videos and generating a chronological history video requires a great deal of time and effort. Furthermore, the means to enjoy the generated history video as a visually rich experience are limited, which can lead to a monotonous user experience. In particular, there is a need for a means to relive past events through an experience in a virtual reality (VR) environment. Therefore, there is a need for a system that can automatically analyze photo and video files, generate high-quality history videos, and simultaneously convert them into a format that can be played in 3D space, providing an experience in a VR environment.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes means for receiving photo or video files from a user, means for storing the received photo or video files in cloud storage, means for analyzing the stored photo or video files to recognize people and acquire age information, means for generating a chronological event list based on the acquired age information and metadata, means for automatically generating a scenario for a history video based on the event list, means for generating the history video using the automatically generated scenario, means for notifying the user of the generated history video and providing a download link, and means for converting the generated history video into a format playable in 3D space so that the user can view it in a virtual reality environment. This allows users to not only easily view automatically generated high-quality history videos, but also to enjoy an immersive experience in a VR environment.

[0200] The "means for receiving photo or video files" is an interface for sending photos or video files uploaded by users to the server.

[0201] "Means for storing in cloud storage" refers to a system that provides server space and storage services for storing received photo and video files online.

[0202] "Means for analyzing photo or video files" refers to technology that analyzes the contents of stored photos and videos to recognize people and obtain age information.

[0203] "Means for recognizing people" are algorithms or technologies that identify faces in images or videos and identify who they are.

[0204] "Means for acquiring age information" refers to algorithms or technologies that estimate a person's age from a recognized facial image of the person.

[0205] "Metadata" is information that accompanies photos and videos, such as the date and time the photo was taken, location information, and camera settings.

[0206] The "means for generating a chronological event list" is a function that organizes related events in chronological order based on acquired age information and metadata.

[0207] "Means for automatically generating scenarios for history videos" is a technology that automatically determines the display order of videos, the text to be inserted, transition effects, etc. based on a chronological event list.

[0208] The "means for generating history videos" is a technology that uses an automatically generated scenario to create the final video file.

[0209] The "means for providing a download link" is a mechanism for generating and notifying a user of a link that allows the user to download the generated video.

[0210] "Means for converting into a format that can be played in 3D space" refers to a technology that converts the generated history video into a format that can be displayed as a 3D image.

[0211] A "virtual reality environment" is a computer-generated environment in which users can experience three-dimensional images and virtual spaces using devices such as VR headsets.

[0212] The "manual editing function" is a function that allows the user to tweak and customize the generated history video.

[0213] "Means of providing as a paid option" refers to a system that allows users to use specific functions or services by paying an additional fee.

[0214] "Means for displaying a preview in real time" is a display function that allows the user to instantly check the video being edited.

[0215] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[0216] 1. Uploading photo and video files

[0217] Users register and log in through the system's web interface or a dedicated application. They then select photo and video files and click the upload button. The device sends the files selected by the user to the server, displaying the progress in real time as they are sent. The server then saves the received photo and video files in cloud storage such as AWS S3 and notifies the user when the files have been saved.

[0218] 2. Analyzing the file and generating time series data

[0219] The server analyzes the stored files using image processing algorithms such as OpenCV and TensorFlow. This analysis includes facial recognition, age estimation, and the extraction of metadata such as the date and time of the photo and location. Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and photo date and time.

[0220] 3. Automatic generation of history videos

[0221] The server automatically generates a scenario for the history video based on the time-series data. This scenario includes the display order of the files, the text to be inserted, transition effects, background music, etc. The server then edits the history video using the automatically generated scenario to generate the final video file. These files are then converted into a format that can be played in 3D space, allowing users to view them in a virtual reality (VR) environment.

[0222] 4. User Notification and Download

[0223] The server stores the generated history video in cloud storage and generates a download link. It notifies the user that the history video has been generated and provides the download link. The user can then watch and experience the video using a smartphone or a virtual reality device.

[0224] 5. Manual editing function (paid option)

[0225] If the user is not satisfied with the generated history video, he or she can select the manual editing function, which is offered as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed. The server allows the user to change the order of the videos, edit the text, and change the background music through the editing tool. The user uses the editing tool to make the necessary corrections and sends the edited content to the server.

[0226] 6. Edited video generation and notification

[0227] The server renders and generates the final history video based on the received editing content. It also provides a function to preview the editing results in real time in the VR environment. The final history video is saved in cloud storage and a notification and download link are provided to the user. The user can then download the final video again and enjoy it in the VR environment.

[0228] Specific examples

[0229] For example, suppose a user uploads 50 photos and 10 videos documenting their child's growth from birth to the present. The system analyzes these and generates a history video that records the child's growth and milestones in chronological order. The user can then watch the video in a VR environment, experiencing it as if they were back in time.

[0230] Example prompt sentence:

[0231] Upload 50 photos and 10 videos to generate a history video showing your child's developmental history.

[0232] The above is a specific embodiment of the present invention, which is a system that not only allows users to easily watch automatically generated high-quality history videos, but also provides an immersive experience in a VR environment.

[0233] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0234] Step 1:

[0235] A user logs in to the system's web interface or a dedicated application, selects photo and video files, and clicks the upload button. The input is the photo and video files selected by the user, and the output is that these files are sent to the server via the device. Specifically, the user uses the file selection dialog on the device and presses the upload button, which causes the device to send the files to the server.

[0236] Step 2:

[0237] The device sends the photo and video files selected by the user to the server. In this case, the input is the photo and video files selected by the user, and the output is the files received by the server. The device also has the function of displaying the progress of the transmission in real time. Specifically, the device uses an HTTP request to upload the files to the server.

[0238] Step 3:

[0239] The server stores the received photo and video files in cloud storage such as AWS S3. The input is the photo and video files sent from the device, and the output is the files stored in the cloud storage. Specifically, the server receives the files and uploads them to the cloud storage using the AWS SDK. A notification is sent to the user when the files have been saved.

[0240] Step 4:

[0241] The server analyzes stored photo and video files, recognizes people, and obtains age information. The input is the photo and video files stored in cloud storage, and the output is age information and metadata as the analysis results. Specifically, the server uses OpenCV and TensorFlow to perform image analysis, recognize faces, and estimate ages. It also extracts the date and time of the photo and location information as metadata.

[0242] Step 5:

[0243] The server generates a chronological event list based on the acquired age information and metadata. The input is the analyzed age information and metadata, and the output is a chronological event list. Specifically, the server organizes the time-series data and builds the event list.

[0244] Step 6:

[0245] The server automatically generates a scenario for a historical video based on the event list. The input is a chronologically organized event list, and the output is a scenario for the historical video. Specifically, the server uses a scenario generation algorithm to automate the file display order, text insertion, transition effects, and background music application.

[0246] Step 7:

[0247] The server generates a history video using the automatically generated scenario. The input is the history video scenario, and the output is the completed history video file. Specifically, the server uses a video editing algorithm to combine the files to create the final video.

[0248] Step 8:

[0249] The server converts the generated history video into a format that can be played in 3D space. The input is the completed history video file, and the output is a video file converted into 3D format. Specifically, the server converts the video into a format that supports stereoscopic viewing.

[0250] Step 9:

[0251] The server saves the generated history video in cloud storage and generates a download link. The input is the video file converted to 3D format, and the output is a download link provided to the user. Specifically, the server uploads the video file to AWS S3, generates a link, and notifies the user.

[0252] Step 10:

[0253] Users can select the paid manual editing option as needed. The input is the history video the user wants to edit, and the output is the edited content created using the editing tool. Specifically, the user changes the order of the video, edits the text, and changes the background music, and the edited content is sent to the server.

[0254] Step 11:

[0255] The server renders the final history video based on the received edits. The input is the edits sent by the user, and the output is the final history video. Specifically, the server applies the video editing algorithm again to reflect the edits.

[0256] Step 12:

[0257] The server saves the edited final history video in cloud storage and notifies the user. The input is the final history video file, and the output is a download link provided to the user. Specifically, the server uploads the final video file to AWS S3, generates a link, and notifies the user.

[0258] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0259] The present invention relates to a system that automatically generates a chronological history video based on photos or video files uploaded by users. By combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[0260] 1. Uploading photo and video files

[0261] User: Log in to the system's web interface or dedicated application, then select the photo and video files and click the upload button.

[0262] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[0263] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[0264] 2. Analyzing files and obtaining emotional information

[0265] Server: Analyzes stored photos and video files based on image processing algorithms and an emotion engine. This analysis involves identifying people and estimating their ages using facial recognition technology, and extracting emotional information from their facial expressions using the emotion engine.

[0266] Server: Extracts the acquired age and emotion information, as well as metadata such as the date and time of the photo and location information.

[0267] 3. Scenario generation for history videos

[0268] Server: Generates time-series data for photos and video files based on age, emotion, and metadata. Based on this data, it creates a chronological event list.

[0269] Server: Automatically generates a scenario for a historical video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. Based on emotional information, it optimizes the transition effects and background music to emphasize emotionally appealing scenes.

[0270] 4. History video generation and notification

[0271] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video. The progress is displayed in real time during the generation process.

[0272] Server: Saves the completed history video to cloud storage and generates a download link.

[0273] Server: Notifies the user when generation is complete and provides a download link.

[0274] 5. Manual editing function (paid option)

[0275] User: If you are not satisfied with the generated history video, you can select the manual editing feature as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[0276] Server: Allows users to change the order of videos, edit text, change background music, etc. through editing tools.

[0277] User: Make any necessary corrections in a video editing tool.

[0278] On your device: Send your edits to the server.

[0279] 6. Edited video generation and notification

[0280] Server: Re-renders and generates the final history video based on the submitted edits.

[0281] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0282] Specific examples

[0283] For example, consider a case where a user wants to upload 10 photos and 2 videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in cloud storage and uses an emotion engine to recognize facial expressions and emotions. It also estimates ages and extracts metadata to generate time-series data.

[0284] Based on the analysis results, a scenario that takes emotional information into account is automatically generated and the history video is edited. The generated history video is then automatically edited and a download link is provided to the user. Furthermore, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0285] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos based on emotional information. Furthermore, a manual editing function can be provided as a paid option, improving both user convenience and profitability.

[0286] The processing flow will be explained below.

[0287] Step 1:

[0288] User: Logs in to the system's web interface or a dedicated application.

[0289] Step 2:

[0290] Server: Authenticates the user's login information and displays the upload screen.

[0291] Step 3:

[0292] User: Select the photo or video file you want to upload and click the upload button.

[0293] Step 4:

[0294] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[0295] Step 5:

[0296] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[0297] Step 6:

[0298] Server: Analyzes stored photo and video files using image processing algorithms and emotion engines. This analysis includes identifying people using facial recognition technology and obtaining their age and emotion information.

[0299] Step 7:

[0300] Server: Extracts the acquired age information, emotional information, and metadata such as the date and time of the photo and location information.

[0301] Step 8:

[0302] Server: Generates time series data for photos and video files based on age, emotion, and metadata, including creating an event list.

[0303] Step 9:

[0304] Server: Automatically generates a scenario for a history video based on the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. In particular, it optimizes the scenario by taking into account emotional information.

[0305] Step 10:

[0306] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[0307] Step 11:

[0308] Server: Displays the progress in real time during generation.

[0309] Step 12:

[0310] Server: Saves the completed history video to cloud storage and generates a download link.

[0311] Step 13:

[0312] Server: Notifies the user when generation is complete and provides a download link.

[0313] Step 14:

[0314] User: After viewing the generated history video, if they feel the need to edit it manually, they can select the manual editing feature as a paid option.

[0315] Step 15:

[0316] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[0317] Step 16:

[0318] User: Edit videos by changing the order, modifying text, changing background music, etc.

[0319] Step 17:

[0320] On your device: Send your edits to the server.

[0321] Step 18:

[0322] Server: Re-renders and generates the final history video based on the submitted edits.

[0323] Step 19:

[0324] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0325] Example 2

[0326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] It is technically difficult, time-consuming, and laborious to not only simply store photo and video files, but also to organize them chronologically and automatically generate history videos that take emotional information into account. Furthermore, there is a problem that when users want to manually edit the generated videos, there are limited ways to efficiently perform this editing process. Furthermore, there is a lack of systems that allow users to easily generate and quickly use the final version of a video that reflects their own edits.

[0328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0329] In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in a data management device, means for analyzing the saved photo or video files, recognizing people, and acquiring emotional and age information, means for extracting metadata such as shooting date and time and location information, means for generating a chronological event list based on the acquired age, emotional, and metadata information, means for automatically generating a scenario for the history video based on the event list, means for generating the history video using the automatically generated scenario, means for displaying progress in real time during generation, means for saving the generated history video in the data management device and generating a download link, and means for notifying the user of completion of generation and providing the download link. This enables users to automatically analyze their own photo or video files and quickly generate and edit high-quality history videos based on emotional information.

[0330] "User" refers to a person who uses the System to upload photos and video files and to create and edit history videos.

[0331] "Server" refers to the computer system that manages the overall processing of the system, receives, stores, analyzes photos and video files from users, and generates and notifies users of history videos.

[0332] "Photo or video files" refers to still images or video data uploaded by users to the system.

[0333] "Data management device" refers to a device that stores and manages data such as photos, video files, and generated history videos, including cloud storage and databases.

[0334] "Analysis" refers to the process by which the server examines and processes photo and video files to recognize people, obtain age and emotional information, and extract metadata.

[0335] "Emotional information" refers to emotional information obtained from the facial expressions of people appearing in photos and videos using emotion engines and algorithms.

[0336] "Age information" refers to information about the age of a person that the server recognizes from a photo or video file.

[0337] "Metadata" refers to supplementary information such as the date and time of the photo or video file, location information, etc.

[0338] "Time series data" refers to data that is ordered based on the date and time that a photo or video file was taken.

[0339] An "event list" refers to a list of events organized chronologically based on acquired information.

[0340] A "scenario" refers to a plan that includes elements such as the display order, inserted text, transition effects, and background music when generating a history video.

[0341] "History videos" refer to videos that are automatically generated based on photos and video files uploaded by users and are organized in chronological order.

[0342] The "manual editing function" refers to a function that allows a user to edit the automatically generated history video by changing the order, correcting the text, changing the background music, etc.

[0343] "Paid Option" refers to a service for which a user must pay a separate fee to use special features such as manual editing functionality.

[0344] "Download link" refers to the URL that allows a user to download the generated history video via the Internet.

[0345] "Notification" refers to the act of informing the user of important information, such as the completion of video generation or the provision of a download link.

[0346] "Real-time progress display" refers to a function that notifies the user of the progress of history video generation and editing in real time.

[0347] MODE FOR CARRYING OUT THE INVENTION

[0348] The present invention relates to a system that automatically generates a time-series history video based on photos or video files uploaded by users. Furthermore, by combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[0349] Uploading photo and video files

[0350] The user begins by logging in to the system's web interface or dedicated application. After logging in, the user selects a photo or video file and clicks the upload button. The device then sends the selected file to the server, displaying the progress in real time. The server then saves the received file to cloud storage (e.g., a data management device) and notifies the user when the file has been saved.

[0351] Analyzing files and extracting emotional information

[0352] The server analyzes the stored photos and video files using an image processing algorithm (e.g., an image processing library) to recognize people and obtain their age information. It also uses an emotion engine (e.g., an emotion recognition service) to obtain emotional information from people's facial expressions. It also extracts metadata such as the date and time of shooting and location information from the photos and video files.

[0353] History video scenario generation

[0354] The server generates time-series data for photos and video files based on the acquired age information, emotion information, and metadata. It then creates a chronological event list based on the generated time-series data. The server then automatically generates a scenario for the history video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, and more. In particular, it optimizes transition effects and background music based on the emotion information to emphasize emotionally appealing scenes.

[0355] History video generation and notification

[0356] The server applies a video editing algorithm (e.g., a video editing library) based on the automatically generated scenario to generate a history video. It also includes a function to display the progress in real time during generation. The completed history video is saved in cloud storage, and a download link is generated. The server notifies the user when generation is complete and provides the download link.

[0357] Manual editing function (paid option)

[0358] If users are dissatisfied with the generated history video, they can select the manual editing feature as a paid option. After completing the payment, the video editing tool operation screen will appear, allowing users to change the order, correct the text, change the background music, etc. The device will send the edited content to the server, and the server will re-render the final history video based on the content.

[0359] Edited video generation and notification

[0360] The server generates the final history video and saves it in cloud storage, and the user is notified and provided with a download link.

[0361] Specific examples

[0362] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in a data management device and uses an emotion recognition service to recognize facial expressions and emotions. Furthermore, it uses an image processing library to estimate age and extract metadata, generating time-series data.

[0363] Based on the analysis results, a scenario that takes emotional information into account is automatically generated, and the history video is edited using a video editing library. The generated history video is automatically edited, and a download link is provided to the user. If the user wishes to revise the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0364] Prompt Sentence Examples

[0365] Below are some examples of specific prompts that can be fed into a generative AI model:

[0366] "I uploaded 10 photos and 2 videos to record my family memories. I want to generate a history video that summarises the important moments based on their emotions."

[0367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0368] Step 1: User Login

[0369] Input: User ID, Password

[0370] How it works: A user logs in to the system's web interface or a dedicated application. The user enters their user ID and password, which are then sent to the authentication server. The server then authenticates them by checking the information against an existing database.

[0371] Output: Authentication result (login success or failure)

[0372] Step 2: Select and upload files

[0373] Input: Photo or video file

[0374] Specific operation: After logging in, the user selects photo and video files on the upload screen and clicks the upload button. The device will send the selected files to the server and display the progress in real time.

[0375] Output: File saved in cloud storage

[0376] Step 3: Save the file

[0377] Input: Photo or video files sent from your device

[0378] Specific operation: The server saves the received photo and video files to cloud storage (data management device). Once the saving is complete, the server sends a notification to the user.

[0379] Output: Save completion notification

[0380] Step 4: Image and video analysis

[0381] Input: Photo and video files stored in cloud storage

[0382] Specific operation: The server analyzes these files using an image processing algorithm (image processing library) to recognize people. It then uses an emotion engine (emotion recognition service) to obtain emotional information from the people's facial expressions. In parallel, it extracts metadata such as the date and time of the photo and location information.

[0383] Output: Age information, emotion information, metadata

[0384] Step 5: Generate time series data

[0385] Input: Age information, Emotion information, Metadata

[0386] Specific operation: The server generates time-series data of photos and video files based on the acquired age information, emotion information, and metadata. The order is determined based on the shooting date and time of each file.

[0387] Output: Time series data

[0388] Step 6: Create an Event List

[0389] Input: Time series data

[0390] Specific operation: The server creates a chronological event list based on the generated time series data. Each event contains detailed information such as display order, inserted text, transition effects, and background music.

[0391] Output: Event List

[0392] Step 7: Automatic generation of historical video scenarios

[0393] Input: Event List

[0394] How it works: The server automatically generates a scenario for a history video based on the event list, including transition effects and background music optimized based on emotional information.

[0395] Output: History video scenario

[0396] Step 8: Generate History Video

[0397] Input: History video scenario

[0398] Specific operation: The server generates history videos using a video editing algorithm (video editing library), including a function to display the progress in real time during generation.

[0399] Output: Generated history video

[0400] Step 9: Save history video and notification

[0401] Input: Generated history video

[0402] Specific operation: The server saves the completed history video to cloud storage, generates a download link, and provides this download link to the user along with a notification that the video has been generated.

[0403] Output: Download link, notification

[0404] Step 10: Manual Editing (paid option)

[0405] Input: User edit request, payment information

[0406] Specific operation: If the user is dissatisfied with the generated history video, he / she can select the manual editing function as a paid option and complete the payment. The server provides an operation interface for the editing tool, allowing the user to change the video order, edit the text, change the background music, etc.

[0407] Output: Editing tool operation screen

[0408] Step 11: Submit your edits

[0409] Input:Edit

[0410] Specific operation: The user makes the necessary corrections through the editing tool and sends the content to the server. The terminal sends the edited content to the server, and the server receives it.

[0411] Output: Edited content

[0412] Step 12: Generate the final history video

[0413] Input:Edited content

[0414] Specific operation: The server re-renders and generates the final history video based on the submitted edits.

[0415] Output: Final history video

[0416] Step 13: Save and notify the final history video

[0417] Input: Final history video

[0418] Specific operation: The server saves the final history video in cloud storage and provides the user with a notification and a download link.

[0419] Output: Notification, download link

[0420] (Application example 2)

[0421] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0422] When creating history videos from photos and videos taken by users, manual editing and organization is cumbersome, making it difficult to efficiently generate high-quality videos that reflect emotions. Another issue is the lack of functionality to share the generated videos or check the progress.

[0423] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in cloud storage, means for analyzing the saved photo or video files to recognize people and acquire age information and emotional information, means for generating a chronological event list based on the acquired age information, emotional information, and metadata, means for automatically generating a scenario for a history video based on the event list and emotional information, means for generating an emotionally appealing history video using the automatically generated scenario, and means for notifying the user of the generated history video and providing a download link. This enables users to automatically and effectively generate history videos that take emotional information into consideration and easily share the videos.

[0424] A "photo or video file" is a media file that records a still image or sequence of images that a user provides to the system.

[0425] "Cloud storage" is an online storage service that stores data over the Internet.

[0426] "Emotional information" is data on psychological states and emotions extracted from the facial expressions and behavior of people included in photographs or video files.

[0427] "Metadata" is auxiliary information that accompanies a photo or video file, such as the date and time the photo was taken, location information, and camera settings.

[0428] An "event list" is a chronological list of events or activities that serves as the basis for generating history videos.

[0429] "History videos" are automatically generated video clips edited in chronological order based on photos and video files provided by users.

[0430] "Automatic scenario generation" is the process of automatically creating a storyline and structure for a video using captured data and emotional information.

[0431] A "download link" is a URL that allows users to download the completed history video on the Internet.

[0432] The "manual editing function" is a function that allows the user to customize the generated history video by changing the order, correcting text, changing background music, etc.

[0433] "Rendering" is the process of converting edited video and audio into the final video format.

[0434] "SNS" is an abbreviation for social networking service, an online platform for users to share information with each other.

[0435] "Real-time progress display" is a function that instantly notifies the user of the current processing status while generating or editing a history video.

[0436] A "transition effect" is a visual effect that provides a smooth transition between different scenes or images.

[0437] System Overview

[0438] This system automatically generates a history video based on emotion recognition and age estimation using photos or video files uploaded by users, generating a chronological event list, and then creating a history video based on that list. If users prefer to edit the list manually, a manual editing function is also available as a paid option. The main components of this system and their functions are described in detail below.

[0439] System Components

[0440] 1. Uploading photo and video files

[0441] Users use a device (smartphone or PC) to log in to the system's web interface or dedicated application, select and upload photo and video files, and the device sends the files selected by the user to the server, displaying the transfer progress in real time. The server then stores the received photo and video files in cloud storage.

[0442] 2. Analyzing files and obtaining emotional information

[0443] The server analyzes the stored photo and video files. The analysis algorithm uses facial recognition technology, age estimation models, and an emotion engine to recognize people in the photos and videos and obtain their age and emotion information.

[0444] 3. Scenario generation for history videos

[0445] The server uses the collected age information, emotional information, and metadata to generate a chronological event list. Based on this, it automatically generates a scenario for the history video. This scenario includes the file display order, inserted text, transition effects, background music, and more. It takes into account the emotional information in particular and optimizes the video to emphasize emotionally appealing scenes.

[0446] 4. History video generation and notification

[0447] The server uses the automatically generated scenario to apply a video editing algorithm to generate a history video. The progress is displayed in real time during the process, and once the video is complete, it is saved in cloud storage and a download link is provided to the user.

[0448] 5. Manual editing function (paid option)

[0449] If the user wishes to make edits to the generated history video, they can select the manual editing function as a paid option. After completing the payment, the server will provide the operation screen of the video editing tool, where the user can change the order of the videos, edit the text, change the background music, etc. The edited content will be sent to the server, which will then render the final version of the history video.

[0450] Specific examples

[0451] For example, suppose a user uploads ten photos and two videos documenting their child's growth to the system. The server stores these files and uses facial recognition technology and an emotion engine to recognize the person's facial expressions and emotions. It then extracts age information and metadata to generate time-series data. Based on the analysis results, it automatically generates a scenario that takes emotional information into account and edits the history video. The generated history video is automatically edited and a download link is provided to the user. In addition, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0452] Prompt Sentence Examples

[0453] "Analyze uploaded photos and videos, extract the emotional information from each media, and generate a moving chronological history video."

[0454] This allows users to automatically generate high-quality history videos that take emotional information into account, and also makes it easy to edit and share the videos.

[0455] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0456] Step 1:

[0457] Users use their devices to log in to the system's web interface or a dedicated application to select and upload photo and video files. As input, photo and video files are provided. These files are sent from the user's device to the server. As output, the progress of file selection and sending is displayed in real time.

[0458] Step 2:

[0459] The server stores the photo and video files received from the user in cloud storage. The received photo and video files are provided as input. These files are stored in cloud storage and the server notifies the user that the storage is complete. As output, a notification of the completion of the storage is sent to the user.

[0460] Step 3:

[0461] The server analyzes the photo and video files stored in the cloud storage. The stored photo and video files are provided as input. It uses face recognition technology to identify the person, and then uses an age estimation model and an emotion engine to obtain age and emotion information. This is done using an image processing algorithm and an emotion recognition model. The analyzed age and emotion information is obtained as output.

[0462] Step 4:

[0463] The server uses the acquired age information, emotion information, and metadata to generate a time-ordered event list. Age information, emotion information, and metadata are provided as input. Data analysis algorithms are used to integrate this information and generate a time-ordered event list. The output is a time-ordered event list.

[0464] Step 5:

[0465] The server automatically generates a scenario for the history video based on the generated event list and emotion information. The server receives the chronological event list and emotion information as input. It uses a scenario generation algorithm to create a scenario including elements such as display order, inserted text, transition effects, and background music. The server outputs the automatically generated scenario.

[0466] Step 6:

[0467] The server generates a history video using an automatically generated scenario. The scenario is provided as input. A video editing algorithm is used to generate a video based on the scenario. The progress is displayed in real time during the generation process. The generated history video is obtained as output.

[0468] Step 7:

[0469] The server saves the generated history video in cloud storage and generates a download link. The server receives the generated history video as input, saves the video file in cloud storage and provides the download link to the user, and provides the download link to the user as output.

[0470] Step 8:

[0471] If the user requests a manual editing function, the server provides the manual editing function as a paid option. The user's request is provided as input. After payment is completed, the server provides the operation screen of the video editing tool, allowing the user to change the video order, edit text, change background music, etc. The operation screen of the editing tool is provided to the user as output.

[0472] Step 9:

[0473] The user sends the edited content to the server. The edited content is provided as input. The server renders the final history video based on the edited content and saves it back to cloud storage. The final history video is obtained as output.

[0474] Step 10:

[0475] The server saves the edited final history video in cloud storage and notifies the user. The final history video is provided as input. After saving is complete, the server notifies the user and provides a download link again. As output, a download link for the final history video is provided to the user.

[0476] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0477] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0478] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0479] [Second embodiment]

[0480] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0481] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0482] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0483] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0484] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0485] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0486] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0487] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0488] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0489] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0490] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0491] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0492] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[0493] 1. Uploading photo and video files

[0494] User: The user registers as a member and logs into the system through the system's web interface or a dedicated application, then selects the photo and video files and clicks the upload button.

[0495] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[0496] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[0497] 2. Analyzing the file and generating time series data

[0498] Server: Analyzes the stored files using image processing algorithms, including facial recognition, age estimation, and extraction of metadata such as photo date and location.

[0499] Server: Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and shooting date and time.

[0500] 3. Automatic generation of history videos

[0501] Server: Automatically generate a historical video scenario based on time-series data, including the file display order, inserted text, transition effects, background music, etc.

[0502] Server: Edits the history video using the automatically generated scenario and generates the final video file.

[0503] 4. User Notification and Download

[0504] Server: Saves the generated history video to cloud storage and generates a download link.

[0505] Server: Notifies the user when the history video is complete and provides a download link.

[0506] 5. Manual editing function (paid option)

[0507] User: If you are not satisfied with the generated history video, you can select the manual editing function, which is available as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[0508] Server: Allows users to reorder videos, modify text, and change background music through editing tools.

[0509] User: Use the editing tools to make any necessary corrections.

[0510] On your device: Send your edits to the server.

[0511] 6. Edited video generation and notification

[0512] Server: Renders and generates the final history video based on the received edits.

[0513] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0514] Specific examples

[0515] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server then stores these files in cloud storage and begins analysis. Based on age estimation and metadata extraction, time-series data is generated, and a history video scenario is automatically created based on this data.

[0516] The generated history video is automatically edited and a download link is provided to the user. Furthermore, if the user wishes to make further edits to the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0517] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos. Furthermore, by providing a manual editing function as a paid option, it is possible to improve both user convenience and profitability.

[0518] The processing flow will be explained below.

[0519] Step 1:

[0520] User: Logs in to the system's web interface or a dedicated application.

[0521] Step 2:

[0522] Server: Authenticates the user's login information and displays the upload screen.

[0523] Step 3:

[0524] User: Select the photo or video file you want to upload and click the upload button.

[0525] Step 4:

[0526] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[0527] Step 5:

[0528] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[0529] Step 6:

[0530] Server: Analyzes stored photo and video files, using facial recognition technology to identify people and estimate their age, as well as extract metadata such as the date and time of the photo and location.

[0531] Step 7:

[0532] Server: Generates time series data for photos and video files based on the analysis results, including creating an event list based on age information and shooting date and time.

[0533] Step 8:

[0534] Server: Automatically generate a scenario for a historical video based on time-series data, including the display order of each file, inserted text, transition effects, and background music selection.

[0535] Step 9:

[0536] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[0537] Step 10:

[0538] Server: Stores the generated history video in cloud storage and provides users with a notification and a download link.

[0539] Step 11:

[0540] User: After viewing the generated history video, if they feel they need to edit it manually, they can select the manual editing feature as a paid option.

[0541] Step 12:

[0542] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[0543] Step 13:

[0544] User: Edit videos by changing the order, modifying text, changing background music, etc.

[0545] Step 14:

[0546] On your device: Send your edits to the server.

[0547] Step 15:

[0548] Server: Re-renders and generates the final history video based on the submitted edits.

[0549] Step 16:

[0550] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0551] Example 1

[0552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Conventional photo and video management systems simply store files and lack the ability to generate meaningful content along a timeline. This makes it difficult for users to effectively relive their memories. Furthermore, if users are dissatisfied with the automatically generated content, they often need specialized knowledge to manually edit it, leaving many users unsatisfied. The purpose of this invention is to solve these problems.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0555] In this invention, the server includes means for receiving image or video data from a user, means for storing the received image or video data in a remote storage device, means for analyzing the stored image or video data and acquiring person and age information using facial recognition technology, means for generating a timeline-based event list based on the acquired age information and metadata, means for automatically generating a scenario based on the event list, means for generating a video file using the automatically generated scenario, and means for notifying the user of the generated video file and providing a download link. This allows the user to receive automatically generated high-quality history videos and also provides the option to manually edit them, thereby improving user satisfaction.

[0556] "User" refers to an individual or organization that uses the system to upload image and video data and generate history videos based on that data.

[0557] "Image or video data" refers to all photos and video files uploaded by users.

[0558] "Server" refers to the central computer device of this system, which receives, analyzes, stores, and processes data, and generates videos.

[0559] "Remote storage device" refers to a data storage device located in a physically separate location, such as cloud storage, that is used to store image and video data received from a server.

[0560] "Facial recognition technology" refers to technology that identifies a person's face from image or video data and identifies their age and individuality.

[0561] "Metadata" refers to additional information embedded in image and video data, such as the date and time of shooting, location information, and camera settings.

[0562] A "chronological event list" refers to a list of images and video data organized chronologically based on facial recognition technology and metadata.

[0563] A "scenario" refers to a plan for generating a chronological storyboard based on a user's image or video data, and setting the display order, including inserted text and transition effects.

[0564] "Video file" refers to video content that is generated based on a scenario and can be downloaded by users.

[0565] "Manual editing function" refers to the function that allows users to modify and edit the content of the history video they have created themselves.

[0566] "Progress display" refers to a feature that allows users to check the upload and video generation process in real time.

[0567] The "operation screen" refers to an interface that allows a user to use the manual editing function to change the order of videos, modify text, change background music, and so on.

[0568] MODE FOR CARRYING OUT THE INVENTION

[0569] The present invention is a system for automatically generating a history video along a time axis based on image or video data uploaded by a user. Specific embodiments of the present invention will be described below.

[0570] 1. System Overview

[0571] Users can use the system's web interface or a dedicated application to register and log in to upload images and video data. A file selection area and a progress bar are provided, allowing users to check the upload progress in real time.

[0572] Terminal: Sends the file selected by the user to the server. The file is temporarily stored and then transferred to cloud storage (e.g., AWS S3).

[0573] Server: Uses facial recognition and metadata analysis technologies to analyze received image and video data. Specifically, it uses OpenCV, TensorFlow, etc. to perform facial recognition and age estimation, and extracts metadata such as the shooting date and time and location information embedded in the image and video data. Based on this, it generates a timeline-based event list.

[0574] 2. Automatic generation of history videos

[0575] Server: Based on the event list, the server automatically generates a scenario for the history video. The scenario includes the display order of images and video data, the text to be inserted, transition effects, background music, etc. This forms a storyboard.

[0576] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg. The generated video file is then saved in cloud storage.

[0577] Server: Notifies the user of the download link for the generated history video. The user can download the video from the provided link.

[0578] 3. Manual editing function

[0579] Users: If you are not satisfied with the history video you have created, you can use the manual editing feature as a paid option. After successful payment, you will be provided with a video editing tool interface where you can change the order, edit the text, change the background music, etc.

[0580] Server: The final video is generated again based on the edits sent by the user. The generated final video is saved in cloud storage, and the download link is sent to the user again.

[0581] Specific examples

[0582] For example, consider the case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs in to the system's web interface, selects the files on the upload screen, and sends them. The device sends the files to the server while displaying the progress in real time. The server temporarily saves the files, then transfers them to cloud storage and begins analysis. The analysis involves estimating the age and extracting metadata, which is then used to generate time-series data. A history video scenario is automatically created based on this data, and the video is edited using FFmpeg. The final history video is saved on the server, and the user is notified of a download link.

[0583] Prompt Sentence Examples

[0584] For example, you can use the following prompt:

[0585] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[0586] This system allows users to easily create high-quality history videos, and manual editing functions can be used to improve satisfaction.

[0587] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0588] Program processing flow

[0589] Step 1: User registration and login

[0590] Input: Information entered by the user when registering as a member (email address, password, username).

[0591] Server: The user registers as a member using the system's web interface or a dedicated application. The registration information is saved in the database and an email is sent to the user confirming registration.

[0592] Output: Registration success message and login information.

[0593] Input: The information the user enters when logging in (email address, password).

[0594] Server: Authenticate the login information, and if authentication is successful, display the dashboard screen.

[0595] Output: A dashboard screen for the user.

[0596] Step 2: Upload your photo and video files

[0597] Input: Image or video data uploaded by the user.

[0598] Users: After logging in, click the "Upload" button on the dashboard and select the file.

[0599] Terminal: Prepares the selected file to be sent to the server. A progress bar is displayed to show the upload progress in real time.

[0600] Output: Progress of file submission.

[0601] Input: Image or video data sent from the device.

[0602] Server: Temporarily stores received files and eventually transfers them to cloud storage (e.g., AWS S3).

[0603] Output: Image or video data saved in cloud storage, upload completion notification.

[0604] Step 3: Parse the file and generate time series data

[0605] Input: Image or video data stored in cloud storage.

[0606] Server: Uses facial recognition and metadata analysis technologies (OpenCV, TensorFlow, etc.) to analyze the stored files. Extracts metadata such as facial recognition, age estimation, and photo date and location information.

[0607] Output: Face recognition results, age information, shooting date and location information.

[0608] Input: Analysis results (face recognition results, age information, photo date and time, and location information).

[0609] Server: Based on the analysis results, a timeline-based event list is generated for each file. Facial recognition and age information are used to organize the developmental process and events of people in chronological order.

[0610] Output: A chronological list of events.

[0611] Step 4: Automatic generation of history videos

[0612] Input: Event list.

[0613] Server: Automatically generates a scenario based on the event list, including the display order, inserted text, transition effects, background music, etc.

[0614] Output: The automatically generated scenario.

[0615] Input: An automatically generated scenario.

[0616] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg.

[0617] Output: The final history video file.

[0618] Step 5: Notify users and download

[0619] Input: The generated history video file.

[0620] Server: Saves the generated video file in cloud storage and generates a download link.

[0621] Output: Download link.

[0622] Input: Download link.

[0623] Server: Notify the user of the download link for the generated history video.

[0624] Output: A notification message with a download link.

[0625] Step 6: Manual Editing (paid option)

[0626] Input: User edit request.

[0627] User: If you are not satisfied with the history video, you can use the manual editing function as a paid option and make an online payment.

[0628] Server: After successful payment, the server provides the editing tool operation interface, allowing users to change the order, edit the text, change the background music, etc.

[0629] Output: Editing tool operation screen.

[0630] Input: The modifications made by the user in the editing tools.

[0631] On your device: Send your edits to the server.

[0632] Server: The final video is generated again based on the received edits. The generated final video is saved in cloud storage and the download link is sent to the user again.

[0633] Output: Final history video file, notification message with download link.

[0634] Specific examples

[0635] For example, if a user wants to upload 10 photos and 2 videos documenting their child's development, they would go through the following specific steps:

[0636] Step 1: The user logs in to the system and selects a file on the upload screen. The server displays the progress in real time and saves the file to cloud storage.

[0637] Step 2: The server analyzes the stored files, performs facial recognition and metadata analysis, and generates time series data based on this.

[0638] Step 3: The server automatically generates a scenario based on the generated time series data and generates the final history video using FFmpeg.

[0639] Step 4: The server saves the generated video to cloud storage and notifies the user of the download link.

[0640] Step 5: The user can manually edit the video as needed, generate the final version, and receive a notification again.

[0641] Prompt Sentence Examples

[0642] For example, you can use the following prompt:

[0643] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[0644] (Application example 1)

[0645] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] In modern society, the act of reminiscing about past events using photos and videos is becoming increasingly important. However, manually organizing these photos and videos and generating a chronological history video requires a great deal of time and effort. Furthermore, the means to enjoy the generated history video as a visually rich experience are limited, which can lead to a monotonous user experience. In particular, there is a need for a means to relive past events through an experience in a virtual reality (VR) environment. Therefore, there is a need for a system that can automatically analyze photo and video files, generate high-quality history videos, and simultaneously convert them into a format that can be played in 3D space, providing an experience in a VR environment.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0648] In this invention, the server includes means for receiving photo or video files from a user, means for storing the received photo or video files in cloud storage, means for analyzing the stored photo or video files to recognize people and acquire age information, means for generating a chronological event list based on the acquired age information and metadata, means for automatically generating a scenario for a history video based on the event list, means for generating the history video using the automatically generated scenario, means for notifying the user of the generated history video and providing a download link, and means for converting the generated history video into a format playable in 3D space so that the user can view it in a virtual reality environment. This allows users to not only easily view automatically generated high-quality history videos, but also to enjoy an immersive experience in a VR environment.

[0649] The "means for receiving photo or video files" is an interface for sending photos or video files uploaded by users to the server.

[0650] "Means for storing in cloud storage" refers to a system that provides server space and storage services for storing received photo and video files online.

[0651] "Means for analyzing photo or video files" refers to technology that analyzes the contents of stored photos and videos to recognize people and obtain age information.

[0652] "Means for recognizing people" are algorithms or technologies that identify faces in images or videos and identify who they are.

[0653] "Means for acquiring age information" refers to algorithms or technologies that estimate a person's age from a recognized facial image of the person.

[0654] "Metadata" is information that accompanies photos and videos, such as the date and time the photo was taken, location information, and camera settings.

[0655] The "means for generating a chronological event list" is a function that organizes related events in chronological order based on acquired age information and metadata.

[0656] "Means for automatically generating scenarios for history videos" is a technology that automatically determines the display order of videos, the text to be inserted, transition effects, etc. based on a chronological event list.

[0657] The "means for generating history videos" is a technology that uses an automatically generated scenario to create the final video file.

[0658] The "means for providing a download link" is a mechanism for generating and notifying a user of a link that allows the user to download the generated video.

[0659] "Means for converting into a format that can be played in 3D space" refers to a technology that converts the generated history video into a format that can be displayed as a 3D image.

[0660] A "virtual reality environment" is a computer-generated environment in which users can experience three-dimensional images and virtual spaces using devices such as VR headsets.

[0661] The "manual editing function" is a function that allows the user to tweak and customize the generated history video.

[0662] "Means of providing as a paid option" refers to a system that allows users to use specific functions or services by paying an additional fee.

[0663] "Means for displaying a preview in real time" is a display function that allows the user to instantly check the video being edited.

[0664] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[0665] 1. Uploading photo and video files

[0666] Users register and log in through the system's web interface or a dedicated application. They then select photo and video files and click the upload button. The device sends the files selected by the user to the server, displaying the progress in real time as they are sent. The server then saves the received photo and video files in cloud storage such as AWS S3 and notifies the user when the files have been saved.

[0667] 2. Analyzing the file and generating time series data

[0668] The server analyzes the stored files using image processing algorithms such as OpenCV and TensorFlow. This analysis includes facial recognition, age estimation, and the extraction of metadata such as the date and time of the photo and location. Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and photo date and time.

[0669] 3. Automatic generation of history videos

[0670] The server automatically generates a scenario for the history video based on the time-series data. This scenario includes the display order of the files, the text to be inserted, transition effects, background music, etc. The server then edits the history video using the automatically generated scenario to generate the final video file. These files are then converted into a format that can be played in 3D space, allowing users to view them in a virtual reality (VR) environment.

[0671] 4. User Notification and Download

[0672] The server stores the generated history video in cloud storage and generates a download link. It notifies the user that the history video has been generated and provides the download link. The user can then watch and experience the video using a smartphone or a virtual reality device.

[0673] 5. Manual editing function (paid option)

[0674] If the user is not satisfied with the generated history video, he or she can select the manual editing function, which is offered as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed. The server allows the user to change the order of the videos, edit the text, and change the background music through the editing tool. The user uses the editing tool to make the necessary corrections and sends the edited content to the server.

[0675] 6. Edited video generation and notification

[0676] The server renders and generates the final history video based on the received editing content. It also provides a function to preview the editing results in real time in the VR environment. The final history video is saved in cloud storage and a notification and download link are provided to the user. The user can then download the final video again and enjoy it in the VR environment.

[0677] Specific examples

[0678] For example, suppose a user uploads 50 photos and 10 videos documenting their child's growth from birth to the present. The system analyzes these and generates a history video that records the child's growth and milestones in chronological order. The user can then watch the video in a VR environment, experiencing it as if they were back in time.

[0679] Example prompt sentence:

[0680] Upload 50 photos and 10 videos to generate a history video showing your child's developmental history.

[0681] The above is a specific embodiment of the present invention, which is a system that not only allows users to easily watch automatically generated high-quality history videos, but also provides an immersive experience in a VR environment.

[0682] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0683] Step 1:

[0684] A user logs in to the system's web interface or a dedicated application, selects photo and video files, and clicks the upload button. The input is the photo and video files selected by the user, and the output is that these files are sent to the server via the device. Specifically, the user uses the file selection dialog on the device and presses the upload button, which causes the device to send the files to the server.

[0685] Step 2:

[0686] The device sends the photo and video files selected by the user to the server. In this case, the input is the photo and video files selected by the user, and the output is the files received by the server. The device also has the function of displaying the progress of the transmission in real time. Specifically, the device uses an HTTP request to upload the files to the server.

[0687] Step 3:

[0688] The server stores the received photo and video files in cloud storage such as AWS S3. The input is the photo and video files sent from the device, and the output is the files stored in the cloud storage. Specifically, the server receives the files and uploads them to the cloud storage using the AWS SDK. A notification is sent to the user when the files have been saved.

[0689] Step 4:

[0690] The server analyzes stored photo and video files, recognizes people, and obtains age information. The input is the photo and video files stored in cloud storage, and the output is age information and metadata as the analysis results. Specifically, the server uses OpenCV and TensorFlow to perform image analysis, recognize faces, and estimate ages. It also extracts the date and time of the photo and location information as metadata.

[0691] Step 5:

[0692] The server generates a chronological event list based on the acquired age information and metadata. The input is the analyzed age information and metadata, and the output is a chronological event list. Specifically, the server organizes the time-series data and builds the event list.

[0693] Step 6:

[0694] The server automatically generates a scenario for a historical video based on the event list. The input is a chronologically organized event list, and the output is a scenario for the historical video. Specifically, the server uses a scenario generation algorithm to automate the file display order, text insertion, transition effects, and background music application.

[0695] Step 7:

[0696] The server generates a history video using the automatically generated scenario. The input is the history video scenario, and the output is the completed history video file. Specifically, the server uses a video editing algorithm to combine the files to create the final video.

[0697] Step 8:

[0698] The server converts the generated history video into a format that can be played in 3D space. The input is the completed history video file, and the output is a video file converted into 3D format. Specifically, the server converts the video into a format that supports stereoscopic viewing.

[0699] Step 9:

[0700] The server saves the generated history video in cloud storage and generates a download link. The input is the video file converted to 3D format, and the output is a download link provided to the user. Specifically, the server uploads the video file to AWS S3, generates a link, and notifies the user.

[0701] Step 10:

[0702] Users can select the paid manual editing option as needed. The input is the history video the user wants to edit, and the output is the edited content created using the editing tool. Specifically, the user changes the order of the video, edits the text, and changes the background music, and the edited content is sent to the server.

[0703] Step 11:

[0704] The server renders the final history video based on the received edits. The input is the edits sent by the user, and the output is the final history video. Specifically, the server applies the video editing algorithm again to reflect the edits.

[0705] Step 12:

[0706] The server saves the edited final history video in cloud storage and notifies the user. The input is the final history video file, and the output is a download link provided to the user. Specifically, the server uploads the final video file to AWS S3, generates a link, and notifies the user.

[0707] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0708] The present invention relates to a system that automatically generates a chronological history video based on photos or video files uploaded by users. By combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[0709] 1. Uploading photo and video files

[0710] User: Log in to the system's web interface or dedicated application, then select the photo and video files and click the upload button.

[0711] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[0712] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[0713] 2. Analyzing files and obtaining emotional information

[0714] Server: Analyzes stored photos and video files based on image processing algorithms and an emotion engine. This analysis involves identifying people and estimating their ages using facial recognition technology, and extracting emotional information from their facial expressions using the emotion engine.

[0715] Server: Extracts the acquired age and emotion information, as well as metadata such as the date and time of the photo and location information.

[0716] 3. Scenario generation for history videos

[0717] Server: Generates time-series data for photos and video files based on age, emotion, and metadata. Based on this data, it creates a chronological event list.

[0718] Server: Automatically generates a scenario for a historical video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. Based on emotional information, it optimizes the transition effects and background music to emphasize emotionally appealing scenes.

[0719] 4. History video generation and notification

[0720] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video. The progress is displayed in real time during the generation process.

[0721] Server: Saves the completed history video to cloud storage and generates a download link.

[0722] Server: Notifies the user when generation is complete and provides a download link.

[0723] 5. Manual editing function (paid option)

[0724] User: If you are not satisfied with the generated history video, you can select the manual editing feature as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[0725] Server: Allows users to change the order of videos, edit text, change background music, etc. through editing tools.

[0726] User: Make any necessary corrections in a video editing tool.

[0727] On your device: Send your edits to the server.

[0728] 6. Edited video generation and notification

[0729] Server: Re-renders and generates the final history video based on the submitted edits.

[0730] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0731] Specific examples

[0732] For example, consider a case where a user wants to upload 10 photos and 2 videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in cloud storage and uses an emotion engine to recognize facial expressions and emotions. It also estimates ages and extracts metadata to generate time-series data.

[0733] Based on the analysis results, a scenario that takes emotional information into account is automatically generated and the history video is edited. The generated history video is then automatically edited and a download link is provided to the user. Furthermore, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0734] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos based on emotional information. Furthermore, a manual editing function can be provided as a paid option, improving both user convenience and profitability.

[0735] The processing flow will be explained below.

[0736] Step 1:

[0737] User: Logs in to the system's web interface or a dedicated application.

[0738] Step 2:

[0739] Server: Authenticates the user's login information and displays the upload screen.

[0740] Step 3:

[0741] User: Select the photo or video file you want to upload and click the upload button.

[0742] Step 4:

[0743] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[0744] Step 5:

[0745] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[0746] Step 6:

[0747] Server: Analyzes stored photo and video files using image processing algorithms and emotion engines. This analysis includes identifying people using facial recognition technology and obtaining their age and emotion information.

[0748] Step 7:

[0749] Server: Extracts the acquired age information, emotional information, and metadata such as the date and time of the photo and location information.

[0750] Step 8:

[0751] Server: Generates time series data for photos and video files based on age, emotion, and metadata, including creating an event list.

[0752] Step 9:

[0753] Server: Automatically generates a scenario for a history video based on the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. In particular, it optimizes the scenario by taking into account emotional information.

[0754] Step 10:

[0755] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[0756] Step 11:

[0757] Server: Displays the progress in real time during generation.

[0758] Step 12:

[0759] Server: Saves the completed history video to cloud storage and generates a download link.

[0760] Step 13:

[0761] Server: Notifies the user when generation is complete and provides a download link.

[0762] Step 14:

[0763] User: After viewing the generated history video, if they feel the need to edit it manually, they can select the manual editing feature as a paid option.

[0764] Step 15:

[0765] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[0766] Step 16:

[0767] User: Edit videos by changing the order, modifying text, changing background music, etc.

[0768] Step 17:

[0769] On your device: Send your edits to the server.

[0770] Step 18:

[0771] Server: Re-renders and generates the final history video based on the submitted edits.

[0772] Step 19:

[0773] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0774] Example 2

[0775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] It is technically difficult, time-consuming, and laborious to not only simply store photo and video files, but also to organize them chronologically and automatically generate history videos that take emotional information into account. Furthermore, there is a problem that when users want to manually edit the generated videos, there are limited ways to efficiently perform this editing process. Furthermore, there is a lack of systems that allow users to easily generate and quickly use the final version of a video that reflects their own edits.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0778] In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in a data management device, means for analyzing the saved photo or video files, recognizing people, and acquiring emotional and age information, means for extracting metadata such as shooting date and time and location information, means for generating a chronological event list based on the acquired age, emotional, and metadata information, means for automatically generating a scenario for the history video based on the event list, means for generating the history video using the automatically generated scenario, means for displaying progress in real time during generation, means for saving the generated history video in the data management device and generating a download link, and means for notifying the user of completion of generation and providing the download link. This enables users to automatically analyze their own photo or video files and quickly generate and edit high-quality history videos based on emotional information.

[0779] "User" refers to a person who uses the System to upload photos and video files and to create and edit history videos.

[0780] "Server" refers to the computer system that manages the overall processing of the system, receives, stores, analyzes photos and video files from users, and generates and notifies users of history videos.

[0781] "Photo or video files" refers to still images or video data uploaded by users to the system.

[0782] "Data management device" refers to a device that stores and manages data such as photos, video files, and generated history videos, including cloud storage and databases.

[0783] "Analysis" refers to the process by which the server examines and processes photo and video files to recognize people, obtain age and emotional information, and extract metadata.

[0784] "Emotional information" refers to emotional information obtained from the facial expressions of people appearing in photos and videos using emotion engines and algorithms.

[0785] "Age information" refers to information about the age of a person that the server recognizes from a photo or video file.

[0786] "Metadata" refers to supplementary information such as the date and time of the photo or video file, location information, etc.

[0787] "Time series data" refers to data that is ordered based on the date and time that a photo or video file was taken.

[0788] An "event list" refers to a list of events organized chronologically based on acquired information.

[0789] A "scenario" refers to a plan that includes elements such as the display order, inserted text, transition effects, and background music when generating a history video.

[0790] "History videos" refer to videos that are automatically generated based on photos and video files uploaded by users and are organized in chronological order.

[0791] The "manual editing function" refers to a function that allows a user to edit the automatically generated history video by changing the order, correcting the text, changing the background music, etc.

[0792] "Paid Option" refers to a service for which a user must pay a separate fee to use special features such as manual editing functionality.

[0793] "Download link" refers to the URL that allows a user to download the generated history video via the Internet.

[0794] "Notification" refers to the act of informing the user of important information, such as the completion of video generation or the provision of a download link.

[0795] "Real-time progress display" refers to a function that notifies the user of the progress of history video generation and editing in real time.

[0796] MODE FOR CARRYING OUT THE INVENTION

[0797] The present invention relates to a system that automatically generates a time-series history video based on photos or video files uploaded by users. Furthermore, by combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[0798] Uploading photo and video files

[0799] The user begins by logging in to the system's web interface or dedicated application. After logging in, the user selects a photo or video file and clicks the upload button. The device then sends the selected file to the server, displaying the progress in real time. The server then saves the received file to cloud storage (e.g., a data management device) and notifies the user when the file has been saved.

[0800] Analyzing files and extracting emotional information

[0801] The server analyzes the stored photos and video files using an image processing algorithm (e.g., an image processing library) to recognize people and obtain their age information. It also uses an emotion engine (e.g., an emotion recognition service) to obtain emotional information from people's facial expressions. It also extracts metadata such as the date and time of shooting and location information from the photos and video files.

[0802] History video scenario generation

[0803] The server generates time-series data for photos and video files based on the acquired age information, emotion information, and metadata. It then creates a chronological event list based on the generated time-series data. The server then automatically generates a scenario for the history video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, and more. In particular, it optimizes transition effects and background music based on the emotion information to emphasize emotionally appealing scenes.

[0804] History video generation and notification

[0805] The server applies a video editing algorithm (e.g., a video editing library) based on the automatically generated scenario to generate a history video. It also includes a function to display the progress in real time during generation. The completed history video is saved in cloud storage, and a download link is generated. The server notifies the user when generation is complete and provides the download link.

[0806] Manual editing function (paid option)

[0807] If users are dissatisfied with the generated history video, they can select the manual editing feature as a paid option. After completing the payment, the video editing tool operation screen will appear, allowing users to change the order, correct the text, change the background music, etc. The device will send the edited content to the server, and the server will re-render the final history video based on the content.

[0808] Edited video generation and notification

[0809] The server generates the final history video and saves it in cloud storage, and the user is notified and provided with a download link.

[0810] Specific examples

[0811] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in a data management device and uses an emotion recognition service to recognize facial expressions and emotions. Furthermore, it uses an image processing library to estimate age and extract metadata, generating time-series data.

[0812] Based on the analysis results, a scenario that takes emotional information into account is automatically generated, and the history video is edited using a video editing library. The generated history video is automatically edited, and a download link is provided to the user. If the user wishes to revise the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0813] Prompt Sentence Examples

[0814] Below are some examples of specific prompts that can be fed into a generative AI model:

[0815] "I uploaded 10 photos and 2 videos to record my family memories. I want to generate a history video that summarises the important moments based on their emotions."

[0816] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0817] Step 1: User Login

[0818] Input: User ID, Password

[0819] How it works: A user logs in to the system's web interface or a dedicated application. The user enters their user ID and password, which are then sent to the authentication server. The server then authenticates them by checking the information against an existing database.

[0820] Output: Authentication result (login success or failure)

[0821] Step 2: Select and upload files

[0822] Input: Photo or video file

[0823] Specific operation: After logging in, the user selects photo and video files on the upload screen and clicks the upload button. The device will send the selected files to the server and display the progress in real time.

[0824] Output: File saved in cloud storage

[0825] Step 3: Save the file

[0826] Input: Photo or video files sent from your device

[0827] Specific operation: The server saves the received photo and video files to cloud storage (data management device). Once the saving is complete, the server sends a notification to the user.

[0828] Output: Save completion notification

[0829] Step 4: Image and video analysis

[0830] Input: Photo and video files stored in cloud storage

[0831] Specific operation: The server analyzes these files using an image processing algorithm (image processing library) to recognize people. It then uses an emotion engine (emotion recognition service) to obtain emotional information from the people's facial expressions. In parallel, it extracts metadata such as the date and time of the photo and location information.

[0832] Output: Age information, emotion information, metadata

[0833] Step 5: Generate time series data

[0834] Input: Age information, Emotion information, Metadata

[0835] Specific operation: The server generates time-series data of photos and video files based on the acquired age information, emotion information, and metadata. The order is determined based on the shooting date and time of each file.

[0836] Output: Time series data

[0837] Step 6: Create an Event List

[0838] Input: Time series data

[0839] Specific operation: The server creates a chronological event list based on the generated time series data. Each event contains detailed information such as display order, inserted text, transition effects, and background music.

[0840] Output: Event List

[0841] Step 7: Automatic generation of historical video scenarios

[0842] Input: Event List

[0843] How it works: The server automatically generates a scenario for a history video based on the event list, including transition effects and background music optimized based on emotional information.

[0844] Output: History video scenario

[0845] Step 8: Generate History Video

[0846] Input: History video scenario

[0847] Specific operation: The server generates history videos using a video editing algorithm (video editing library), including a function to display the progress in real time during generation.

[0848] Output: Generated history video

[0849] Step 9: Save history video and notification

[0850] Input: Generated history video

[0851] Specific operation: The server saves the completed history video to cloud storage, generates a download link, and provides this download link to the user along with a notification that the video has been generated.

[0852] Output: Download link, notification

[0853] Step 10: Manual Editing (paid option)

[0854] Input: User edit request, payment information

[0855] Specific operation: If the user is dissatisfied with the generated history video, he / she can select the manual editing function as a paid option and complete the payment. The server provides an operation interface for the editing tool, allowing the user to change the video order, edit the text, change the background music, etc.

[0856] Output: Editing tool operation screen

[0857] Step 11: Submit your edits

[0858] Input:Edit

[0859] Specific operation: The user makes the necessary corrections through the editing tool and sends the content to the server. The terminal sends the edited content to the server, and the server receives it.

[0860] Output: Edited content

[0861] Step 12: Generate the final history video

[0862] Input:Edited content

[0863] Specific operation: The server re-renders and generates the final history video based on the submitted edits.

[0864] Output: Final history video

[0865] Step 13: Save and notify the final history video

[0866] Input: Final history video

[0867] Specific operation: The server saves the final history video in cloud storage and provides the user with a notification and a download link.

[0868] Output: Notification, download link

[0869] (Application example 2)

[0870] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0871] When creating history videos from photos and videos taken by users, manual editing and organization is cumbersome, making it difficult to efficiently generate high-quality videos that reflect emotions. Another issue is the lack of functionality to share the generated videos or check the progress.

[0872] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in cloud storage, means for analyzing the saved photo or video files to recognize people and acquire age information and emotional information, means for generating a chronological event list based on the acquired age information, emotional information, and metadata, means for automatically generating a scenario for a history video based on the event list and emotional information, means for generating an emotionally appealing history video using the automatically generated scenario, and means for notifying the user of the generated history video and providing a download link. This enables users to automatically and effectively generate history videos that take emotional information into consideration and easily share the videos.

[0873] A "photo or video file" is a media file that records a still image or sequence of images that a user provides to the system.

[0874] "Cloud storage" is an online storage service that stores data over the Internet.

[0875] "Emotional information" is data on psychological states and emotions extracted from the facial expressions and behavior of people included in photographs or video files.

[0876] "Metadata" is auxiliary information that accompanies a photo or video file, such as the date and time the photo was taken, location information, and camera settings.

[0877] An "event list" is a chronological list of events or activities that serves as the basis for generating history videos.

[0878] "History videos" are automatically generated video clips edited in chronological order based on photos and video files provided by users.

[0879] "Automatic scenario generation" is the process of automatically creating a storyline and structure for a video using captured data and emotional information.

[0880] A "download link" is a URL that allows users to download the completed history video on the Internet.

[0881] The "manual editing function" is a function that allows the user to customize the generated history video by changing the order, correcting text, changing background music, etc.

[0882] "Rendering" is the process of converting edited video and audio into the final video format.

[0883] "SNS" is an abbreviation for social networking service, an online platform for users to share information with each other.

[0884] "Real-time progress display" is a function that instantly notifies the user of the current processing status while generating or editing a history video.

[0885] A "transition effect" is a visual effect that provides a smooth transition between different scenes or images.

[0886] System Overview

[0887] This system automatically generates a history video based on emotion recognition and age estimation using photos or video files uploaded by users, generating a chronological event list, and then creating a history video based on that list. If users prefer to edit the list manually, a manual editing function is also available as a paid option. The main components of this system and their functions are described in detail below.

[0888] System Components

[0889] 1. Uploading photo and video files

[0890] Users use a device (smartphone or PC) to log in to the system's web interface or dedicated application, select and upload photo and video files, and the device sends the files selected by the user to the server, displaying the transfer progress in real time. The server then stores the received photo and video files in cloud storage.

[0891] 2. Analyzing files and obtaining emotional information

[0892] The server analyzes the stored photo and video files. The analysis algorithm uses facial recognition technology, age estimation models, and an emotion engine to recognize people in the photos and videos and obtain their age and emotion information.

[0893] 3. Scenario generation for history videos

[0894] The server uses the collected age information, emotional information, and metadata to generate a chronological event list. Based on this, it automatically generates a scenario for the history video. This scenario includes the file display order, inserted text, transition effects, background music, and more. It takes into account the emotional information in particular and optimizes the video to emphasize emotionally appealing scenes.

[0895] 4. History video generation and notification

[0896] The server uses the automatically generated scenario to apply a video editing algorithm to generate a history video. The progress is displayed in real time during the process, and once the video is complete, it is saved in cloud storage and a download link is provided to the user.

[0897] 5. Manual editing function (paid option)

[0898] If the user wishes to make edits to the generated history video, they can select the manual editing function as a paid option. After completing the payment, the server will provide the operation screen of the video editing tool, where the user can change the order of the videos, edit the text, change the background music, etc. The edited content will be sent to the server, which will then render the final version of the history video.

[0899] Specific examples

[0900] For example, suppose a user uploads ten photos and two videos documenting their child's growth to the system. The server stores these files and uses facial recognition technology and an emotion engine to recognize the person's facial expressions and emotions. It then extracts age information and metadata to generate time-series data. Based on the analysis results, it automatically generates a scenario that takes emotional information into account and edits the history video. The generated history video is automatically edited and a download link is provided to the user. In addition, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0901] Prompt Sentence Examples

[0902] "Analyze uploaded photos and videos, extract the emotional information from each media, and generate a moving chronological history video."

[0903] This allows users to automatically generate high-quality history videos that take emotional information into account, and also makes it easy to edit and share the videos.

[0904] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0905] Step 1:

[0906] Users use their devices to log in to the system's web interface or a dedicated application to select and upload photo and video files. As input, photo and video files are provided. These files are sent from the user's device to the server. As output, the progress of file selection and sending is displayed in real time.

[0907] Step 2:

[0908] The server stores the photo and video files received from the user in cloud storage. The received photo and video files are provided as input. These files are stored in cloud storage and the server notifies the user that the storage is complete. As output, a notification of the completion of the storage is sent to the user.

[0909] Step 3:

[0910] The server analyzes the photo and video files stored in the cloud storage. The stored photo and video files are provided as input. It uses face recognition technology to identify the person, and then uses an age estimation model and an emotion engine to obtain age and emotion information. This is done using an image processing algorithm and an emotion recognition model. The analyzed age and emotion information is obtained as output.

[0911] Step 4:

[0912] The server uses the acquired age information, emotion information, and metadata to generate a time-ordered event list. Age information, emotion information, and metadata are provided as input. Data analysis algorithms are used to integrate this information and generate a time-ordered event list. The output is a time-ordered event list.

[0913] Step 5:

[0914] The server automatically generates a scenario for the history video based on the generated event list and emotion information. The server receives the chronological event list and emotion information as input. It uses a scenario generation algorithm to create a scenario including elements such as display order, inserted text, transition effects, and background music. The server outputs the automatically generated scenario.

[0915] Step 6:

[0916] The server generates a history video using an automatically generated scenario. The scenario is provided as input. A video editing algorithm is used to generate a video based on the scenario. The progress is displayed in real time during the generation process. The generated history video is obtained as output.

[0917] Step 7:

[0918] The server saves the generated history video in cloud storage and generates a download link. The server receives the generated history video as input, saves the video file in cloud storage and provides the download link to the user, and provides the download link to the user as output.

[0919] Step 8:

[0920] If the user requests a manual editing function, the server provides the manual editing function as a paid option. The user's request is provided as input. After payment is completed, the server provides the operation screen of the video editing tool, allowing the user to change the video order, edit text, change background music, etc. The operation screen of the editing tool is provided to the user as output.

[0921] Step 9:

[0922] The user sends the edited content to the server. The edited content is provided as input. The server renders the final history video based on the edited content and saves it back to cloud storage. The final history video is obtained as output.

[0923] Step 10:

[0924] The server saves the edited final history video in cloud storage and notifies the user. The final history video is provided as input. After saving is complete, the server notifies the user and provides a download link again. As output, a download link for the final history video is provided to the user.

[0925] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0926] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0927] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0928] [Third embodiment]

[0929] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0930] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0931] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0932] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0933] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0934] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0935] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0936] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0937] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0938] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0939] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0940] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0941] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[0942] 1. Uploading photo and video files

[0943] User: The user registers as a member and logs into the system through the system's web interface or a dedicated application, then selects the photo and video files and clicks the upload button.

[0944] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[0945] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[0946] 2. Analyzing the file and generating time series data

[0947] Server: Analyzes the stored files using image processing algorithms, including facial recognition, age estimation, and extraction of metadata such as photo date and location.

[0948] Server: Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and shooting date and time.

[0949] 3. Automatic generation of history videos

[0950] Server: Automatically generate a historical video scenario based on time-series data, including the file display order, inserted text, transition effects, background music, etc.

[0951] Server: Edits the history video using the automatically generated scenario and generates the final video file.

[0952] 4. User Notification and Download

[0953] Server: Saves the generated history video to cloud storage and generates a download link.

[0954] Server: Notifies the user when the history video is complete and provides a download link.

[0955] 5. Manual editing function (paid option)

[0956] User: If you are not satisfied with the generated history video, you can select the manual editing function, which is available as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[0957] Server: Allows users to reorder videos, modify text, and change background music through editing tools.

[0958] User: Use the editing tools to make any necessary corrections.

[0959] On your device: Send your edits to the server.

[0960] 6. Edited video generation and notification

[0961] Server: Renders and generates the final history video based on the received edits.

[0962] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[0963] Specific examples

[0964] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server then stores these files in cloud storage and begins analysis. Based on age estimation and metadata extraction, time-series data is generated, and a history video scenario is automatically created based on this data.

[0965] The generated history video is automatically edited and a download link is provided to the user. Furthermore, if the user wishes to make further edits to the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[0966] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos. Furthermore, by providing a manual editing function as a paid option, it is possible to improve both user convenience and profitability.

[0967] The processing flow will be explained below.

[0968] Step 1:

[0969] User: Logs in to the system's web interface or a dedicated application.

[0970] Step 2:

[0971] Server: Authenticates the user's login information and displays the upload screen.

[0972] Step 3:

[0973] User: Select the photo or video file you want to upload and click the upload button.

[0974] Step 4:

[0975] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[0976] Step 5:

[0977] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[0978] Step 6:

[0979] Server: Analyzes stored photo and video files, using facial recognition technology to identify people and estimate their age, as well as extract metadata such as the date and time of the photo and location.

[0980] Step 7:

[0981] Server: Generates time series data for photos and video files based on the analysis results, including creating an event list based on age information and shooting date and time.

[0982] Step 8:

[0983] Server: Automatically generate a scenario for a historical video based on time-series data, including the display order of each file, inserted text, transition effects, and background music selection.

[0984] Step 9:

[0985] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[0986] Step 10:

[0987] Server: Stores the generated history video in cloud storage and provides users with a notification and a download link.

[0988] Step 11:

[0989] User: After viewing the generated history video, if they feel they need to edit it manually, they can select the manual editing feature as a paid option.

[0990] Step 12:

[0991] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[0992] Step 13:

[0993] User: Edit videos by changing the order, modifying text, changing background music, etc.

[0994] Step 14:

[0995] On your device: Send your edits to the server.

[0996] Step 15:

[0997] Server: Re-renders and generates the final history video based on the submitted edits.

[0998] Step 16:

[0999] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1000] Example 1

[1001] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1002] Conventional photo and video management systems simply store files and lack the ability to generate meaningful content along a timeline. This makes it difficult for users to effectively relive their memories. Furthermore, if users are dissatisfied with the automatically generated content, they often need specialized knowledge to manually edit it, leaving many users unsatisfied. The purpose of this invention is to solve these problems.

[1003] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1004] In this invention, the server includes means for receiving image or video data from a user, means for storing the received image or video data in a remote storage device, means for analyzing the stored image or video data and acquiring person and age information using facial recognition technology, means for generating a timeline-based event list based on the acquired age information and metadata, means for automatically generating a scenario based on the event list, means for generating a video file using the automatically generated scenario, and means for notifying the user of the generated video file and providing a download link. This allows the user to receive automatically generated high-quality history videos and also provides the option to manually edit them, thereby improving user satisfaction.

[1005] "User" refers to an individual or organization that uses the system to upload image and video data and generate history videos based on that data.

[1006] "Image or video data" refers to all photos and video files uploaded by users.

[1007] "Server" refers to the central computer device of this system, which receives, analyzes, stores, and processes data, and generates videos.

[1008] "Remote storage device" refers to a data storage device located in a physically separate location, such as cloud storage, that is used to store image and video data received from a server.

[1009] "Facial recognition technology" refers to technology that identifies a person's face from image or video data and identifies their age and individuality.

[1010] "Metadata" refers to additional information embedded in image and video data, such as the date and time of shooting, location information, and camera settings.

[1011] A "chronological event list" refers to a list of images and video data organized chronologically based on facial recognition technology and metadata.

[1012] A "scenario" refers to a plan for generating a chronological storyboard based on a user's image or video data, and setting the display order, including inserted text and transition effects.

[1013] "Video file" refers to video content that is generated based on a scenario and can be downloaded by users.

[1014] "Manual editing function" refers to the function that allows users to modify and edit the content of the history video they have created themselves.

[1015] "Progress display" refers to a feature that allows users to check the upload and video generation process in real time.

[1016] The "operation screen" refers to an interface that allows a user to use the manual editing function to change the order of videos, modify text, change background music, and so on.

[1017] MODE FOR CARRYING OUT THE INVENTION

[1018] The present invention is a system for automatically generating a history video along a time axis based on image or video data uploaded by a user. Specific embodiments of the present invention will be described below.

[1019] 1. System Overview

[1020] Users can use the system's web interface or a dedicated application to register and log in to upload images and video data. A file selection area and a progress bar are provided, allowing users to check the upload progress in real time.

[1021] Terminal: Sends the file selected by the user to the server. The file is temporarily stored and then transferred to cloud storage (e.g., AWS S3).

[1022] Server: Uses facial recognition and metadata analysis technologies to analyze received image and video data. Specifically, it uses OpenCV, TensorFlow, etc. to perform facial recognition and age estimation, and extracts metadata such as the shooting date and time and location information embedded in the image and video data. Based on this, it generates a timeline-based event list.

[1023] 2. Automatic generation of history videos

[1024] Server: Based on the event list, the server automatically generates a scenario for the history video. The scenario includes the display order of images and video data, the text to be inserted, transition effects, background music, etc. This forms a storyboard.

[1025] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg. The generated video file is then saved in cloud storage.

[1026] Server: Notifies the user of the download link for the generated history video. The user can download the video from the provided link.

[1027] 3. Manual editing function

[1028] Users: If you are not satisfied with the history video you have created, you can use the manual editing feature as a paid option. After successful payment, you will be provided with a video editing tool interface where you can change the order, edit the text, change the background music, etc.

[1029] Server: The final video is generated again based on the edits sent by the user. The generated final video is saved in cloud storage, and the download link is sent to the user again.

[1030] Specific examples

[1031] For example, consider the case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs in to the system's web interface, selects the files on the upload screen, and sends them. The device sends the files to the server while displaying the progress in real time. The server temporarily saves the files, then transfers them to cloud storage and begins analysis. The analysis involves estimating the age and extracting metadata, which is then used to generate time-series data. A history video scenario is automatically created based on this data, and the video is edited using FFmpeg. The final history video is saved on the server, and the user is notified of a download link.

[1032] Prompt Sentence Examples

[1033] For example, you can use the following prompt:

[1034] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[1035] This system allows users to easily create high-quality history videos, and manual editing functions can be used to improve satisfaction.

[1036] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1037] Program processing flow

[1038] Step 1: User registration and login

[1039] Input: Information entered by the user when registering as a member (email address, password, username).

[1040] Server: The user registers as a member using the system's web interface or a dedicated application. The registration information is saved in the database and an email is sent to the user confirming registration.

[1041] Output: Registration success message and login information.

[1042] Input: The information the user enters when logging in (email address, password).

[1043] Server: Authenticate the login information, and if authentication is successful, display the dashboard screen.

[1044] Output: A dashboard screen for the user.

[1045] Step 2: Upload your photo and video files

[1046] Input: Image or video data uploaded by the user.

[1047] Users: After logging in, click the "Upload" button on the dashboard and select the file.

[1048] Terminal: Prepares the selected file to be sent to the server. A progress bar is displayed to show the upload progress in real time.

[1049] Output: Progress of file submission.

[1050] Input: Image or video data sent from the device.

[1051] Server: Temporarily stores received files and eventually transfers them to cloud storage (e.g., AWS S3).

[1052] Output: Image or video data saved in cloud storage, upload completion notification.

[1053] Step 3: Parse the file and generate time series data

[1054] Input: Image or video data stored in cloud storage.

[1055] Server: Uses facial recognition and metadata analysis technologies (OpenCV, TensorFlow, etc.) to analyze the stored files. Extracts metadata such as facial recognition, age estimation, and photo date and location information.

[1056] Output: Face recognition results, age information, shooting date and location information.

[1057] Input: Analysis results (face recognition results, age information, photo date and time, and location information).

[1058] Server: Based on the analysis results, a timeline-based event list is generated for each file. Facial recognition and age information are used to organize the developmental process and events of people in chronological order.

[1059] Output: A chronological list of events.

[1060] Step 4: Automatic generation of history videos

[1061] Input: Event list.

[1062] Server: Automatically generates a scenario based on the event list, including the display order, inserted text, transition effects, background music, etc.

[1063] Output: The automatically generated scenario.

[1064] Input: An automatically generated scenario.

[1065] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg.

[1066] Output: The final history video file.

[1067] Step 5: Notify users and download

[1068] Input: The generated history video file.

[1069] Server: Saves the generated video file in cloud storage and generates a download link.

[1070] Output: Download link.

[1071] Input: Download link.

[1072] Server: Notify the user of the download link for the generated history video.

[1073] Output: A notification message with a download link.

[1074] Step 6: Manual Editing (paid option)

[1075] Input: User edit request.

[1076] User: If you are not satisfied with the history video, you can use the manual editing function as a paid option and make an online payment.

[1077] Server: After successful payment, the server provides the editing tool operation interface, allowing users to change the order, edit the text, change the background music, etc.

[1078] Output: Editing tool operation screen.

[1079] Input: The modifications made by the user in the editing tools.

[1080] On your device: Send your edits to the server.

[1081] Server: The final video is generated again based on the received edits. The generated final video is saved in cloud storage and the download link is sent to the user again.

[1082] Output: Final history video file, notification message with download link.

[1083] Specific examples

[1084] For example, if a user wants to upload 10 photos and 2 videos documenting their child's development, they would go through the following specific steps:

[1085] Step 1: The user logs in to the system and selects a file on the upload screen. The server displays the progress in real time and saves the file to cloud storage.

[1086] Step 2: The server analyzes the stored files, performs facial recognition and metadata analysis, and generates time series data based on this.

[1087] Step 3: The server automatically generates a scenario based on the generated time series data and generates the final history video using FFmpeg.

[1088] Step 4: The server saves the generated video to cloud storage and notifies the user of the download link.

[1089] Step 5: The user can manually edit the video as needed, generate the final version, and receive a notification again.

[1090] Prompt Sentence Examples

[1091] For example, you can use the following prompt:

[1092] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[1093] (Application example 1)

[1094] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] In modern society, the act of reminiscing about past events using photos and videos is becoming increasingly important. However, manually organizing these photos and videos and generating a chronological history video requires a great deal of time and effort. Furthermore, the means to enjoy the generated history video as a visually rich experience are limited, which can lead to a monotonous user experience. In particular, there is a need for a means to relive past events through an experience in a virtual reality (VR) environment. Therefore, there is a need for a system that can automatically analyze photo and video files, generate high-quality history videos, and simultaneously convert them into a format that can be played in 3D space, providing an experience in a VR environment.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1097] In this invention, the server includes means for receiving photo or video files from a user, means for storing the received photo or video files in cloud storage, means for analyzing the stored photo or video files to recognize people and acquire age information, means for generating a chronological event list based on the acquired age information and metadata, means for automatically generating a scenario for a history video based on the event list, means for generating the history video using the automatically generated scenario, means for notifying the user of the generated history video and providing a download link, and means for converting the generated history video into a format playable in 3D space so that the user can view it in a virtual reality environment. This allows users to not only easily view automatically generated high-quality history videos, but also to enjoy an immersive experience in a VR environment.

[1098] The "means for receiving photo or video files" is an interface for sending photos or video files uploaded by users to the server.

[1099] "Means for storing in cloud storage" refers to a system that provides server space and storage services for storing received photo and video files online.

[1100] "Means for analyzing photo or video files" refers to technology that analyzes the contents of stored photos and videos to recognize people and obtain age information.

[1101] "Means for recognizing people" are algorithms or technologies that identify faces in images or videos and identify who they are.

[1102] "Means for acquiring age information" refers to algorithms or technologies that estimate a person's age from a recognized facial image of the person.

[1103] "Metadata" is information that accompanies photos and videos, such as the date and time the photo was taken, location information, and camera settings.

[1104] The "means for generating a chronological event list" is a function that organizes related events in chronological order based on acquired age information and metadata.

[1105] "Means for automatically generating scenarios for history videos" is a technology that automatically determines the display order of videos, the text to be inserted, transition effects, etc. based on a chronological event list.

[1106] The "means for generating history videos" is a technology that uses an automatically generated scenario to create the final video file.

[1107] The "means for providing a download link" is a mechanism for generating and notifying a user of a link that allows the user to download the generated video.

[1108] "Means for converting into a format that can be played in 3D space" refers to a technology that converts the generated history video into a format that can be displayed as a 3D image.

[1109] A "virtual reality environment" is a computer-generated environment in which users can experience three-dimensional images and virtual spaces using devices such as VR headsets.

[1110] The "manual editing function" is a function that allows the user to tweak and customize the generated history video.

[1111] "Means of providing as a paid option" refers to a system that allows users to use specific functions or services by paying an additional fee.

[1112] "Means for displaying a preview in real time" is a display function that allows the user to instantly check the video being edited.

[1113] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[1114] 1. Uploading photo and video files

[1115] Users register and log in through the system's web interface or a dedicated application. They then select photo and video files and click the upload button. The device sends the files selected by the user to the server, displaying the progress in real time as they are sent. The server then saves the received photo and video files in cloud storage such as AWS S3 and notifies the user when the files have been saved.

[1116] 2. Analyzing the file and generating time series data

[1117] The server analyzes the stored files using image processing algorithms such as OpenCV and TensorFlow. This analysis includes facial recognition, age estimation, and the extraction of metadata such as the date and time of the photo and location. Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and photo date and time.

[1118] 3. Automatic generation of history videos

[1119] The server automatically generates a scenario for the history video based on the time-series data. This scenario includes the display order of the files, the text to be inserted, transition effects, background music, etc. The server then edits the history video using the automatically generated scenario to generate the final video file. These files are then converted into a format that can be played in 3D space, allowing users to view them in a virtual reality (VR) environment.

[1120] 4. User Notification and Download

[1121] The server stores the generated history video in cloud storage and generates a download link. It notifies the user that the history video has been generated and provides the download link. The user can then watch and experience the video using a smartphone or a virtual reality device.

[1122] 5. Manual editing function (paid option)

[1123] If the user is not satisfied with the generated history video, he or she can select the manual editing function, which is offered as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed. The server allows the user to change the order of the videos, edit the text, and change the background music through the editing tool. The user uses the editing tool to make the necessary corrections and sends the edited content to the server.

[1124] 6. Edited video generation and notification

[1125] The server renders and generates the final history video based on the received editing content. It also provides a function to preview the editing results in real time in the VR environment. The final history video is saved in cloud storage and a notification and download link are provided to the user. The user can then download the final video again and enjoy it in the VR environment.

[1126] Specific examples

[1127] For example, suppose a user uploads 50 photos and 10 videos documenting their child's growth from birth to the present. The system analyzes these and generates a history video that records the child's growth and milestones in chronological order. The user can then watch the video in a VR environment, experiencing it as if they were back in time.

[1128] Example prompt sentence:

[1129] Upload 50 photos and 10 videos to generate a history video showing your child's developmental history.

[1130] The above is a specific embodiment of the present invention, which is a system that not only allows users to easily watch automatically generated high-quality history videos, but also provides an immersive experience in a VR environment.

[1131] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1132] Step 1:

[1133] A user logs in to the system's web interface or a dedicated application, selects photo and video files, and clicks the upload button. The input is the photo and video files selected by the user, and the output is that these files are sent to the server via the device. Specifically, the user uses the file selection dialog on the device and presses the upload button, which causes the device to send the files to the server.

[1134] Step 2:

[1135] The device sends the photo and video files selected by the user to the server. In this case, the input is the photo and video files selected by the user, and the output is the files received by the server. The device also has the function of displaying the progress of the transmission in real time. Specifically, the device uses an HTTP request to upload the files to the server.

[1136] Step 3:

[1137] The server stores the received photo and video files in cloud storage such as AWS S3. The input is the photo and video files sent from the device, and the output is the files stored in the cloud storage. Specifically, the server receives the files and uploads them to the cloud storage using the AWS SDK. A notification is sent to the user when the files have been saved.

[1138] Step 4:

[1139] The server analyzes stored photo and video files, recognizes people, and obtains age information. The input is the photo and video files stored in cloud storage, and the output is age information and metadata as the analysis results. Specifically, the server uses OpenCV and TensorFlow to perform image analysis, recognize faces, and estimate ages. It also extracts the date and time of the photo and location information as metadata.

[1140] Step 5:

[1141] The server generates a chronological event list based on the acquired age information and metadata. The input is the analyzed age information and metadata, and the output is a chronological event list. Specifically, the server organizes the time-series data and builds the event list.

[1142] Step 6:

[1143] The server automatically generates a scenario for a historical video based on the event list. The input is a chronologically organized event list, and the output is a scenario for the historical video. Specifically, the server uses a scenario generation algorithm to automate the file display order, text insertion, transition effects, and background music application.

[1144] Step 7:

[1145] The server generates a history video using the automatically generated scenario. The input is the history video scenario, and the output is the completed history video file. Specifically, the server uses a video editing algorithm to combine the files to create the final video.

[1146] Step 8:

[1147] The server converts the generated history video into a format that can be played in 3D space. The input is the completed history video file, and the output is a video file converted into 3D format. Specifically, the server converts the video into a format that supports stereoscopic viewing.

[1148] Step 9:

[1149] The server saves the generated history video in cloud storage and generates a download link. The input is the video file converted to 3D format, and the output is a download link provided to the user. Specifically, the server uploads the video file to AWS S3, generates a link, and notifies the user.

[1150] Step 10:

[1151] Users can select the paid manual editing option as needed. The input is the history video the user wants to edit, and the output is the edited content created using the editing tool. Specifically, the user changes the order of the video, edits the text, and changes the background music, and the edited content is sent to the server.

[1152] Step 11:

[1153] The server renders the final history video based on the received edits. The input is the edits sent by the user, and the output is the final history video. Specifically, the server applies the video editing algorithm again to reflect the edits.

[1154] Step 12:

[1155] The server saves the edited final history video in cloud storage and notifies the user. The input is the final history video file, and the output is a download link provided to the user. Specifically, the server uploads the final video file to AWS S3, generates a link, and notifies the user.

[1156] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1157] The present invention relates to a system that automatically generates a chronological history video based on photos or video files uploaded by users. By combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[1158] 1. Uploading photo and video files

[1159] User: Log in to the system's web interface or dedicated application, then select the photo and video files and click the upload button.

[1160] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[1161] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[1162] 2. Analyzing files and obtaining emotional information

[1163] Server: Analyzes stored photos and video files based on image processing algorithms and an emotion engine. This analysis involves identifying people and estimating their ages using facial recognition technology, and extracting emotional information from their facial expressions using the emotion engine.

[1164] Server: Extracts the acquired age and emotion information, as well as metadata such as the date and time of the photo and location information.

[1165] 3. Scenario generation for history videos

[1166] Server: Generates time-series data for photos and video files based on age, emotion, and metadata. Based on this data, it creates a chronological event list.

[1167] Server: Automatically generates a scenario for a historical video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. Based on emotional information, it optimizes the transition effects and background music to emphasize emotionally appealing scenes.

[1168] 4. History video generation and notification

[1169] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video. The progress is displayed in real time during the generation process.

[1170] Server: Saves the completed history video to cloud storage and generates a download link.

[1171] Server: Notifies the user when generation is complete and provides a download link.

[1172] 5. Manual editing function (paid option)

[1173] User: If you are not satisfied with the generated history video, you can select the manual editing feature as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[1174] Server: Allows users to change the order of videos, edit text, change background music, etc. through editing tools.

[1175] User: Make any necessary corrections in a video editing tool.

[1176] On your device: Send your edits to the server.

[1177] 6. Edited video generation and notification

[1178] Server: Re-renders and generates the final history video based on the submitted edits.

[1179] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1180] Specific examples

[1181] For example, consider a case where a user wants to upload 10 photos and 2 videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in cloud storage and uses an emotion engine to recognize facial expressions and emotions. It also estimates ages and extracts metadata to generate time-series data.

[1182] Based on the analysis results, a scenario that takes emotional information into account is automatically generated and the history video is edited. The generated history video is then automatically edited and a download link is provided to the user. Furthermore, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1183] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos based on emotional information. Furthermore, a manual editing function can be provided as a paid option, improving both user convenience and profitability.

[1184] The processing flow will be explained below.

[1185] Step 1:

[1186] User: Logs in to the system's web interface or a dedicated application.

[1187] Step 2:

[1188] Server: Authenticates the user's login information and displays the upload screen.

[1189] Step 3:

[1190] User: Select the photo or video file you want to upload and click the upload button.

[1191] Step 4:

[1192] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[1193] Step 5:

[1194] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[1195] Step 6:

[1196] Server: Analyzes stored photo and video files using image processing algorithms and emotion engines. This analysis includes identifying people using facial recognition technology and obtaining their age and emotion information.

[1197] Step 7:

[1198] Server: Extracts the acquired age information, emotional information, and metadata such as the date and time of the photo and location information.

[1199] Step 8:

[1200] Server: Generates time series data for photos and video files based on age, emotion, and metadata, including creating an event list.

[1201] Step 9:

[1202] Server: Automatically generates a scenario for a history video based on the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. In particular, it optimizes the scenario by taking into account emotional information.

[1203] Step 10:

[1204] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[1205] Step 11:

[1206] Server: Displays the progress in real time during generation.

[1207] Step 12:

[1208] Server: Saves the completed history video to cloud storage and generates a download link.

[1209] Step 13:

[1210] Server: Notifies the user when generation is complete and provides a download link.

[1211] Step 14:

[1212] User: After viewing the generated history video, if they feel the need to edit it manually, they can select the manual editing feature as a paid option.

[1213] Step 15:

[1214] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[1215] Step 16:

[1216] User: Edit videos by changing the order, modifying text, changing background music, etc.

[1217] Step 17:

[1218] On your device: Send your edits to the server.

[1219] Step 18:

[1220] Server: Re-renders and generates the final history video based on the submitted edits.

[1221] Step 19:

[1222] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1223] Example 2

[1224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] It is technically difficult, time-consuming, and laborious to not only simply store photo and video files, but also to organize them chronologically and automatically generate history videos that take emotional information into account. Furthermore, there is a problem that when users want to manually edit the generated videos, there are limited ways to efficiently perform this editing process. Furthermore, there is a lack of systems that allow users to easily generate and quickly use the final version of a video that reflects their own edits.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1227] In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in a data management device, means for analyzing the saved photo or video files, recognizing people, and acquiring emotional and age information, means for extracting metadata such as shooting date and time and location information, means for generating a chronological event list based on the acquired age, emotional, and metadata information, means for automatically generating a scenario for the history video based on the event list, means for generating the history video using the automatically generated scenario, means for displaying progress in real time during generation, means for saving the generated history video in the data management device and generating a download link, and means for notifying the user of completion of generation and providing the download link. This enables users to automatically analyze their own photo or video files and quickly generate and edit high-quality history videos based on emotional information.

[1228] "User" refers to a person who uses the System to upload photos and video files and to create and edit history videos.

[1229] "Server" refers to the computer system that manages the overall processing of the system, receives, stores, analyzes photos and video files from users, and generates and notifies users of history videos.

[1230] "Photo or video files" refers to still images or video data uploaded by users to the system.

[1231] "Data management device" refers to a device that stores and manages data such as photos, video files, and generated history videos, including cloud storage and databases.

[1232] "Analysis" refers to the process by which the server examines and processes photo and video files to recognize people, obtain age and emotional information, and extract metadata.

[1233] "Emotional information" refers to emotional information obtained from the facial expressions of people appearing in photos and videos using emotion engines and algorithms.

[1234] "Age information" refers to information about the age of a person that the server recognizes from a photo or video file.

[1235] "Metadata" refers to supplementary information such as the date and time of the photo or video file, location information, etc.

[1236] "Time series data" refers to data that is ordered based on the date and time that a photo or video file was taken.

[1237] An "event list" refers to a list of events organized chronologically based on acquired information.

[1238] A "scenario" refers to a plan that includes elements such as the display order, inserted text, transition effects, and background music when generating a history video.

[1239] "History videos" refer to videos that are automatically generated based on photos and video files uploaded by users and are organized in chronological order.

[1240] The "manual editing function" refers to a function that allows a user to edit the automatically generated history video by changing the order, correcting the text, changing the background music, etc.

[1241] "Paid Option" refers to a service for which a user must pay a separate fee to use special features such as manual editing functionality.

[1242] "Download link" refers to the URL that allows a user to download the generated history video via the Internet.

[1243] "Notification" refers to the act of informing the user of important information, such as the completion of video generation or the provision of a download link.

[1244] "Real-time progress display" refers to a function that notifies the user of the progress of history video generation and editing in real time.

[1245] MODE FOR CARRYING OUT THE INVENTION

[1246] The present invention relates to a system that automatically generates a time-series history video based on photos or video files uploaded by users. Furthermore, by combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[1247] Uploading photo and video files

[1248] The user begins by logging in to the system's web interface or dedicated application. After logging in, the user selects a photo or video file and clicks the upload button. The device then sends the selected file to the server, displaying the progress in real time. The server then saves the received file to cloud storage (e.g., a data management device) and notifies the user when the file has been saved.

[1249] Analyzing files and extracting emotional information

[1250] The server analyzes the stored photos and video files using an image processing algorithm (e.g., an image processing library) to recognize people and obtain their age information. It also uses an emotion engine (e.g., an emotion recognition service) to obtain emotional information from people's facial expressions. It also extracts metadata such as the date and time of shooting and location information from the photos and video files.

[1251] History video scenario generation

[1252] The server generates time-series data for photos and video files based on the acquired age information, emotion information, and metadata. It then creates a chronological event list based on the generated time-series data. The server then automatically generates a scenario for the history video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, and more. In particular, it optimizes transition effects and background music based on the emotion information to emphasize emotionally appealing scenes.

[1253] History video generation and notification

[1254] The server applies a video editing algorithm (e.g., a video editing library) based on the automatically generated scenario to generate a history video. It also includes a function to display the progress in real time during generation. The completed history video is saved in cloud storage, and a download link is generated. The server notifies the user when generation is complete and provides the download link.

[1255] Manual editing function (paid option)

[1256] If users are dissatisfied with the generated history video, they can select the manual editing feature as a paid option. After completing the payment, the video editing tool operation screen will appear, allowing users to change the order, correct the text, change the background music, etc. The device will send the edited content to the server, and the server will re-render the final history video based on the content.

[1257] Edited video generation and notification

[1258] The server generates the final history video and saves it in cloud storage, and the user is notified and provided with a download link.

[1259] Specific examples

[1260] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in a data management device and uses an emotion recognition service to recognize facial expressions and emotions. Furthermore, it uses an image processing library to estimate age and extract metadata, generating time-series data.

[1261] Based on the analysis results, a scenario that takes emotional information into account is automatically generated, and the history video is edited using a video editing library. The generated history video is automatically edited, and a download link is provided to the user. If the user wishes to revise the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1262] Prompt Sentence Examples

[1263] Below are some examples of specific prompts that can be fed into a generative AI model:

[1264] "I uploaded 10 photos and 2 videos to record my family memories. I want to generate a history video that summarises the important moments based on their emotions."

[1265] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1266] Step 1: User Login

[1267] Input: User ID, Password

[1268] How it works: A user logs in to the system's web interface or a dedicated application. The user enters their user ID and password, which are then sent to the authentication server. The server then authenticates them by checking the information against an existing database.

[1269] Output: Authentication result (login success or failure)

[1270] Step 2: Select and upload files

[1271] Input: Photo or video file

[1272] Specific operation: After logging in, the user selects photo and video files on the upload screen and clicks the upload button. The device will send the selected files to the server and display the progress in real time.

[1273] Output: File saved in cloud storage

[1274] Step 3: Save the file

[1275] Input: Photo or video files sent from your device

[1276] Specific operation: The server saves the received photo and video files to cloud storage (data management device). Once the saving is complete, the server sends a notification to the user.

[1277] Output: Save completion notification

[1278] Step 4: Image and video analysis

[1279] Input: Photo and video files stored in cloud storage

[1280] Specific operation: The server analyzes these files using an image processing algorithm (image processing library) to recognize people. It then uses an emotion engine (emotion recognition service) to obtain emotional information from the people's facial expressions. In parallel, it extracts metadata such as the date and time of the photo and location information.

[1281] Output: Age information, emotion information, metadata

[1282] Step 5: Generate time series data

[1283] Input: Age information, Emotion information, Metadata

[1284] Specific operation: The server generates time-series data of photos and video files based on the acquired age information, emotion information, and metadata. The order is determined based on the shooting date and time of each file.

[1285] Output: Time series data

[1286] Step 6: Create an Event List

[1287] Input: Time series data

[1288] Specific operation: The server creates a chronological event list based on the generated time series data. Each event contains detailed information such as display order, inserted text, transition effects, and background music.

[1289] Output: Event List

[1290] Step 7: Automatic generation of historical video scenarios

[1291] Input: Event List

[1292] How it works: The server automatically generates a scenario for a history video based on the event list, including transition effects and background music optimized based on emotional information.

[1293] Output: History video scenario

[1294] Step 8: Generate History Video

[1295] Input: History video scenario

[1296] Specific operation: The server generates history videos using a video editing algorithm (video editing library), including a function to display the progress in real time during generation.

[1297] Output: Generated history video

[1298] Step 9: Save history video and notification

[1299] Input: Generated history video

[1300] Specific operation: The server saves the completed history video to cloud storage, generates a download link, and provides this download link to the user along with a notification that the video has been generated.

[1301] Output: Download link, notification

[1302] Step 10: Manual Editing (paid option)

[1303] Input: User edit request, payment information

[1304] Specific operation: If the user is dissatisfied with the generated history video, he / she can select the manual editing function as a paid option and complete the payment. The server provides an operation interface for the editing tool, allowing the user to change the video order, edit the text, change the background music, etc.

[1305] Output: Editing tool operation screen

[1306] Step 11: Submit your edits

[1307] Input:Edit

[1308] Specific operation: The user makes the necessary corrections through the editing tool and sends the content to the server. The terminal sends the edited content to the server, and the server receives it.

[1309] Output: Edited content

[1310] Step 12: Generate the final history video

[1311] Input:Edited content

[1312] Specific operation: The server re-renders and generates the final history video based on the submitted edits.

[1313] Output: Final history video

[1314] Step 13: Save and notify the final history video

[1315] Input: Final history video

[1316] Specific operation: The server saves the final history video in cloud storage and provides the user with a notification and a download link.

[1317] Output: Notification, download link

[1318] (Application example 2)

[1319] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1320] When creating history videos from photos and videos taken by users, manual editing and organization is cumbersome, making it difficult to efficiently generate high-quality videos that reflect emotions. Another issue is the lack of functionality to share the generated videos or check the progress.

[1321] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in cloud storage, means for analyzing the saved photo or video files to recognize people and acquire age information and emotional information, means for generating a chronological event list based on the acquired age information, emotional information, and metadata, means for automatically generating a scenario for a history video based on the event list and emotional information, means for generating an emotionally appealing history video using the automatically generated scenario, and means for notifying the user of the generated history video and providing a download link. This enables users to automatically and effectively generate history videos that take emotional information into consideration and easily share the videos.

[1322] A "photo or video file" is a media file that records a still image or sequence of images that a user provides to the system.

[1323] "Cloud storage" is an online storage service that stores data over the Internet.

[1324] "Emotional information" is data on psychological states and emotions extracted from the facial expressions and behavior of people included in photographs or video files.

[1325] "Metadata" is auxiliary information that accompanies a photo or video file, such as the date and time the photo was taken, location information, and camera settings.

[1326] An "event list" is a chronological list of events or activities that serves as the basis for generating history videos.

[1327] "History videos" are automatically generated video clips edited in chronological order based on photos and video files provided by users.

[1328] "Automatic scenario generation" is the process of automatically creating a storyline and structure for a video using captured data and emotional information.

[1329] A "download link" is a URL that allows users to download the completed history video on the Internet.

[1330] The "manual editing function" is a function that allows the user to customize the generated history video by changing the order, correcting text, changing background music, etc.

[1331] "Rendering" is the process of converting edited video and audio into the final video format.

[1332] "SNS" is an abbreviation for social networking service, an online platform for users to share information with each other.

[1333] "Real-time progress display" is a function that instantly notifies the user of the current processing status while generating or editing a history video.

[1334] A "transition effect" is a visual effect that provides a smooth transition between different scenes or images.

[1335] System Overview

[1336] This system automatically generates a history video based on emotion recognition and age estimation using photos or video files uploaded by users, generating a chronological event list, and then creating a history video based on that list. If users prefer to edit the list manually, a manual editing function is also available as a paid option. The main components of this system and their functions are described in detail below.

[1337] System Components

[1338] 1. Uploading photo and video files

[1339] Users use a device (smartphone or PC) to log in to the system's web interface or dedicated application, select and upload photo and video files, and the device sends the files selected by the user to the server, displaying the transfer progress in real time. The server then stores the received photo and video files in cloud storage.

[1340] 2. Analyzing files and obtaining emotional information

[1341] The server analyzes the stored photo and video files. The analysis algorithm uses facial recognition technology, age estimation models, and an emotion engine to recognize people in the photos and videos and obtain their age and emotion information.

[1342] 3. Scenario generation for history videos

[1343] The server uses the collected age information, emotional information, and metadata to generate a chronological event list. Based on this, it automatically generates a scenario for the history video. This scenario includes the file display order, inserted text, transition effects, background music, and more. It takes into account the emotional information in particular and optimizes the video to emphasize emotionally appealing scenes.

[1344] 4. History video generation and notification

[1345] The server uses the automatically generated scenario to apply a video editing algorithm to generate a history video. The progress is displayed in real time during the process, and once the video is complete, it is saved in cloud storage and a download link is provided to the user.

[1346] 5. Manual editing function (paid option)

[1347] If the user wishes to make edits to the generated history video, they can select the manual editing function as a paid option. After completing the payment, the server will provide the operation screen of the video editing tool, where the user can change the order of the videos, edit the text, change the background music, etc. The edited content will be sent to the server, which will then render the final version of the history video.

[1348] Specific examples

[1349] For example, suppose a user uploads ten photos and two videos documenting their child's growth to the system. The server stores these files and uses facial recognition technology and an emotion engine to recognize the person's facial expressions and emotions. It then extracts age information and metadata to generate time-series data. Based on the analysis results, it automatically generates a scenario that takes emotional information into account and edits the history video. The generated history video is automatically edited and a download link is provided to the user. In addition, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1350] Prompt Sentence Examples

[1351] "Analyze uploaded photos and videos, extract the emotional information from each media, and generate a moving chronological history video."

[1352] This allows users to automatically generate high-quality history videos that take emotional information into account, and also makes it easy to edit and share the videos.

[1353] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1354] Step 1:

[1355] Users use their devices to log in to the system's web interface or a dedicated application to select and upload photo and video files. As input, photo and video files are provided. These files are sent from the user's device to the server. As output, the progress of file selection and sending is displayed in real time.

[1356] Step 2:

[1357] The server stores the photo and video files received from the user in cloud storage. The received photo and video files are provided as input. These files are stored in cloud storage and the server notifies the user that the storage is complete. As output, a notification of the completion of the storage is sent to the user.

[1358] Step 3:

[1359] The server analyzes the photo and video files stored in the cloud storage. The stored photo and video files are provided as input. It uses face recognition technology to identify the person, and then uses an age estimation model and an emotion engine to obtain age and emotion information. This is done using an image processing algorithm and an emotion recognition model. The analyzed age and emotion information is obtained as output.

[1360] Step 4:

[1361] The server uses the acquired age information, emotion information, and metadata to generate a time-ordered event list. Age information, emotion information, and metadata are provided as input. Data analysis algorithms are used to integrate this information and generate a time-ordered event list. The output is a time-ordered event list.

[1362] Step 5:

[1363] The server automatically generates a scenario for the history video based on the generated event list and emotion information. The server receives the chronological event list and emotion information as input. It uses a scenario generation algorithm to create a scenario including elements such as display order, inserted text, transition effects, and background music. The server outputs the automatically generated scenario.

[1364] Step 6:

[1365] The server generates a history video using an automatically generated scenario. The scenario is provided as input. A video editing algorithm is used to generate a video based on the scenario. The progress is displayed in real time during the generation process. The generated history video is obtained as output.

[1366] Step 7:

[1367] The server saves the generated history video in cloud storage and generates a download link. The server receives the generated history video as input, saves the video file in cloud storage and provides the download link to the user, and provides the download link to the user as output.

[1368] Step 8:

[1369] If the user requests a manual editing function, the server provides the manual editing function as a paid option. The user's request is provided as input. After payment is completed, the server provides the operation screen of the video editing tool, allowing the user to change the video order, edit text, change background music, etc. The operation screen of the editing tool is provided to the user as output.

[1370] Step 9:

[1371] The user sends the edited content to the server. The edited content is provided as input. The server renders the final history video based on the edited content and saves it back to cloud storage. The final history video is obtained as output.

[1372] Step 10:

[1373] The server saves the edited final history video in cloud storage and notifies the user. The final history video is provided as input. After saving is complete, the server notifies the user and provides a download link again. As output, a download link for the final history video is provided to the user.

[1374] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1375] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1376] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1377] [Fourth embodiment]

[1378] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1379] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1380] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1381] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1382] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1384] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1385] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1386] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1387] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1388] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1389] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1390] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1391] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[1392] 1. Uploading photo and video files

[1393] User: The user registers as a member and logs into the system through the system's web interface or a dedicated application, then selects the photo and video files and clicks the upload button.

[1394] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[1395] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[1396] 2. Analyzing the file and generating time series data

[1397] Server: Analyzes the stored files using image processing algorithms, including facial recognition, age estimation, and extraction of metadata such as photo date and location.

[1398] Server: Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and shooting date and time.

[1399] 3. Automatic generation of history videos

[1400] Server: Automatically generate a historical video scenario based on time-series data, including the file display order, inserted text, transition effects, background music, etc.

[1401] Server: Edits the history video using the automatically generated scenario and generates the final video file.

[1402] 4. User Notification and Download

[1403] Server: Saves the generated history video to cloud storage and generates a download link.

[1404] Server: Notifies the user when the history video is complete and provides a download link.

[1405] 5. Manual editing function (paid option)

[1406] User: If you are not satisfied with the generated history video, you can select the manual editing function, which is available as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[1407] Server: Allows users to reorder videos, modify text, and change background music through editing tools.

[1408] User: Use the editing tools to make any necessary corrections.

[1409] On your device: Send your edits to the server.

[1410] 6. Edited video generation and notification

[1411] Server: Renders and generates the final history video based on the received edits.

[1412] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1413] Specific examples

[1414] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server then stores these files in cloud storage and begins analysis. Based on age estimation and metadata extraction, time-series data is generated, and a history video scenario is automatically created based on this data.

[1415] The generated history video is automatically edited and a download link is provided to the user. Furthermore, if the user wishes to make further edits to the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1416] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos. Furthermore, by providing a manual editing function as a paid option, it is possible to improve both user convenience and profitability.

[1417] The processing flow will be explained below.

[1418] Step 1:

[1419] User: Logs in to the system's web interface or a dedicated application.

[1420] Step 2:

[1421] Server: Authenticates the user's login information and displays the upload screen.

[1422] Step 3:

[1423] User: Select the photo or video file you want to upload and click the upload button.

[1424] Step 4:

[1425] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[1426] Step 5:

[1427] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[1428] Step 6:

[1429] Server: Analyzes stored photo and video files, using facial recognition technology to identify people and estimate their age, as well as extract metadata such as the date and time of the photo and location.

[1430] Step 7:

[1431] Server: Generates time series data for photos and video files based on the analysis results, including creating an event list based on age information and shooting date and time.

[1432] Step 8:

[1433] Server: Automatically generate a scenario for a historical video based on time-series data, including the display order of each file, inserted text, transition effects, and background music selection.

[1434] Step 9:

[1435] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[1436] Step 10:

[1437] Server: Stores the generated history video in cloud storage and provides users with a notification and a download link.

[1438] Step 11:

[1439] User: After viewing the generated history video, if they feel they need to edit it manually, they can select the manual editing feature as a paid option.

[1440] Step 12:

[1441] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[1442] Step 13:

[1443] User: Edit videos by changing the order, modifying text, changing background music, etc.

[1444] Step 14:

[1445] On your device: Send your edits to the server.

[1446] Step 15:

[1447] Server: Re-renders and generates the final history video based on the submitted edits.

[1448] Step 16:

[1449] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1450] Example 1

[1451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1452] Conventional photo and video management systems simply store files and lack the ability to generate meaningful content along a timeline. This makes it difficult for users to effectively relive their memories. Furthermore, if users are dissatisfied with the automatically generated content, they often need specialized knowledge to manually edit it, leaving many users unsatisfied. The purpose of this invention is to solve these problems.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1454] In this invention, the server includes means for receiving image or video data from a user, means for storing the received image or video data in a remote storage device, means for analyzing the stored image or video data and acquiring person and age information using facial recognition technology, means for generating a timeline-based event list based on the acquired age information and metadata, means for automatically generating a scenario based on the event list, means for generating a video file using the automatically generated scenario, and means for notifying the user of the generated video file and providing a download link. This allows the user to receive automatically generated high-quality history videos and also provides the option to manually edit them, thereby improving user satisfaction.

[1455] "User" refers to an individual or organization that uses the system to upload image and video data and generate history videos based on that data.

[1456] "Image or video data" refers to all photos and video files uploaded by users.

[1457] "Server" refers to the central computer device of this system, which receives, analyzes, stores, and processes data, and generates videos.

[1458] "Remote storage device" refers to a data storage device located in a physically separate location, such as cloud storage, that is used to store image and video data received from a server.

[1459] "Facial recognition technology" refers to technology that identifies a person's face from image or video data and identifies their age and individuality.

[1460] "Metadata" refers to additional information embedded in image and video data, such as the date and time of shooting, location information, and camera settings.

[1461] A "chronological event list" refers to a list of images and video data organized chronologically based on facial recognition technology and metadata.

[1462] A "scenario" refers to a plan for generating a chronological storyboard based on a user's image or video data, and setting the display order, including inserted text and transition effects.

[1463] "Video file" refers to video content that is generated based on a scenario and can be downloaded by users.

[1464] "Manual editing function" refers to the function that allows users to modify and edit the content of the history video they have created themselves.

[1465] "Progress display" refers to a feature that allows users to check the upload and video generation process in real time.

[1466] The "operation screen" refers to an interface that allows a user to use the manual editing function to change the order of videos, modify text, change background music, and so on.

[1467] MODE FOR CARRYING OUT THE INVENTION

[1468] The present invention is a system for automatically generating a history video along a time axis based on image or video data uploaded by a user. Specific embodiments of the present invention will be described below.

[1469] 1. System Overview

[1470] Users can use the system's web interface or a dedicated application to register and log in to upload images and video data. A file selection area and a progress bar are provided, allowing users to check the upload progress in real time.

[1471] Terminal: Sends the file selected by the user to the server. The file is temporarily stored and then transferred to cloud storage (e.g., AWS S3).

[1472] Server: Uses facial recognition and metadata analysis technologies to analyze received image and video data. Specifically, it uses OpenCV, TensorFlow, etc. to perform facial recognition and age estimation, and extracts metadata such as the shooting date and time and location information embedded in the image and video data. Based on this, it generates a timeline-based event list.

[1473] 2. Automatic generation of history videos

[1474] Server: Based on the event list, the server automatically generates a scenario for the history video. The scenario includes the display order of images and video data, the text to be inserted, transition effects, background music, etc. This forms a storyboard.

[1475] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg. The generated video file is then saved in cloud storage.

[1476] Server: Notifies the user of the download link for the generated history video. The user can download the video from the provided link.

[1477] 3. Manual editing function

[1478] Users: If you are not satisfied with the history video you have created, you can use the manual editing feature as a paid option. After successful payment, you will be provided with a video editing tool interface where you can change the order, edit the text, change the background music, etc.

[1479] Server: The final video is generated again based on the edits sent by the user. The generated final video is saved in cloud storage, and the download link is sent to the user again.

[1480] Specific examples

[1481] For example, consider the case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs in to the system's web interface, selects the files on the upload screen, and sends them. The device sends the files to the server while displaying the progress in real time. The server temporarily saves the files, then transfers them to cloud storage and begins analysis. The analysis involves estimating the age and extracting metadata, which is then used to generate time-series data. A history video scenario is automatically created based on this data, and the video is edited using FFmpeg. The final history video is saved on the server, and the user is notified of a download link.

[1482] Prompt Sentence Examples

[1483] For example, you can use the following prompt:

[1484] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[1485] This system allows users to easily create high-quality history videos, and manual editing functions can be used to improve satisfaction.

[1486] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1487] Program processing flow

[1488] Step 1: User registration and login

[1489] Input: Information entered by the user when registering as a member (email address, password, username).

[1490] Server: The user registers as a member using the system's web interface or a dedicated application. The registration information is saved in the database and an email is sent to the user confirming registration.

[1491] Output: Registration success message and login information.

[1492] Input: The information the user enters when logging in (email address, password).

[1493] Server: Authenticate the login information, and if authentication is successful, display the dashboard screen.

[1494] Output: A dashboard screen for the user.

[1495] Step 2: Upload your photo and video files

[1496] Input: Image or video data uploaded by the user.

[1497] Users: After logging in, click the "Upload" button on the dashboard and select the file.

[1498] Terminal: Prepares the selected file to be sent to the server. A progress bar is displayed to show the upload progress in real time.

[1499] Output: Progress of file submission.

[1500] Input: Image or video data sent from the device.

[1501] Server: Temporarily stores received files and eventually transfers them to cloud storage (e.g., AWS S3).

[1502] Output: Image or video data saved in cloud storage, upload completion notification.

[1503] Step 3: Parse the file and generate time series data

[1504] Input: Image or video data stored in cloud storage.

[1505] Server: Uses facial recognition and metadata analysis technologies (OpenCV, TensorFlow, etc.) to analyze the stored files. Extracts metadata such as facial recognition, age estimation, and photo date and location information.

[1506] Output: Face recognition results, age information, shooting date and location information.

[1507] Input: Analysis results (face recognition results, age information, photo date and time, and location information).

[1508] Server: Based on the analysis results, a timeline-based event list is generated for each file. Facial recognition and age information are used to organize the developmental process and events of people in chronological order.

[1509] Output: A chronological list of events.

[1510] Step 4: Automatic generation of history videos

[1511] Input: Event list.

[1512] Server: Automatically generates a scenario based on the event list, including the display order, inserted text, transition effects, background music, etc.

[1513] Output: The automatically generated scenario.

[1514] Input: An automatically generated scenario.

[1515] Server: Using the automatically generated scenario, the final video file is generated using video editing software such as FFmpeg.

[1516] Output: The final history video file.

[1517] Step 5: Notify users and download

[1518] Input: The generated history video file.

[1519] Server: Saves the generated video file in cloud storage and generates a download link.

[1520] Output: Download link.

[1521] Input: Download link.

[1522] Server: Notify the user of the download link for the generated history video.

[1523] Output: A notification message with a download link.

[1524] Step 6: Manual Editing (paid option)

[1525] Input: User edit request.

[1526] User: If you are not satisfied with the history video, you can use the manual editing function as a paid option and make an online payment.

[1527] Server: After successful payment, the server provides the editing tool operation interface, allowing users to change the order, edit the text, change the background music, etc.

[1528] Output: Editing tool operation screen.

[1529] Input: The modifications made by the user in the editing tools.

[1530] On your device: Send your edits to the server.

[1531] Server: The final video is generated again based on the received edits. The generated final video is saved in cloud storage and the download link is sent to the user again.

[1532] Output: Final history video file, notification message with download link.

[1533] Specific examples

[1534] For example, if a user wants to upload 10 photos and 2 videos documenting their child's development, they would go through the following specific steps:

[1535] Step 1: The user logs in to the system and selects a file on the upload screen. The server displays the progress in real time and saves the file to cloud storage.

[1536] Step 2: The server analyzes the stored files, performs facial recognition and metadata analysis, and generates time series data based on this.

[1537] Step 3: The server automatically generates a scenario based on the generated time series data and generates the final history video using FFmpeg.

[1538] Step 4: The server saves the generated video to cloud storage and notifies the user of the download link.

[1539] Step 5: The user can manually edit the video as needed, generate the final version, and receive a notification again.

[1540] Prompt Sentence Examples

[1541] For example, you can use the following prompt:

[1542] Please explain the specific process flow of a system that allows a user to upload 10 images and 2 videos documenting their child's growth and automatically convert them into a timeline history video. Please break down the process into detailed steps and clearly explain what happens at each step. Please also include the specific names of the hardware and software used.

[1543] (Application example 1)

[1544] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1545] In modern society, the act of reminiscing about past events using photos and videos is becoming increasingly important. However, manually organizing these photos and videos and generating a chronological history video requires a great deal of time and effort. Furthermore, the means to enjoy the generated history video as a visually rich experience are limited, which can lead to a monotonous user experience. In particular, there is a need for a means to relive past events through an experience in a virtual reality (VR) environment. Therefore, there is a need for a system that can automatically analyze photo and video files, generate high-quality history videos, and simultaneously convert them into a format that can be played in 3D space, providing an experience in a VR environment.

[1546] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1547] In this invention, the server includes means for receiving photo or video files from a user, means for storing the received photo or video files in cloud storage, means for analyzing the stored photo or video files to recognize people and acquire age information, means for generating a chronological event list based on the acquired age information and metadata, means for automatically generating a scenario for a history video based on the event list, means for generating the history video using the automatically generated scenario, means for notifying the user of the generated history video and providing a download link, and means for converting the generated history video into a format playable in 3D space so that the user can view it in a virtual reality environment. This allows users to not only easily view automatically generated high-quality history videos, but also to enjoy an immersive experience in a VR environment.

[1548] The "means for receiving photo or video files" is an interface for sending photos or video files uploaded by users to the server.

[1549] "Means for storing in cloud storage" refers to a system that provides server space and storage services for storing received photo and video files online.

[1550] "Means for analyzing photo or video files" refers to technology that analyzes the contents of stored photos and videos to recognize people and obtain age information.

[1551] "Means for recognizing people" are algorithms or technologies that identify faces in images or videos and identify who they are.

[1552] "Means for acquiring age information" refers to algorithms or technologies that estimate a person's age from a recognized facial image of the person.

[1553] "Metadata" is information that accompanies photos and videos, such as the date and time the photo was taken, location information, and camera settings.

[1554] The "means for generating a chronological event list" is a function that organizes related events in chronological order based on acquired age information and metadata.

[1555] "Means for automatically generating scenarios for history videos" is a technology that automatically determines the display order of videos, the text to be inserted, transition effects, etc. based on a chronological event list.

[1556] The "means for generating history videos" is a technology that uses an automatically generated scenario to create the final video file.

[1557] The "means for providing a download link" is a mechanism for generating and notifying a user of a link that allows the user to download the generated video.

[1558] "Means for converting into a format that can be played in 3D space" refers to a technology that converts the generated history video into a format that can be displayed as a 3D image.

[1559] A "virtual reality environment" is a computer-generated environment in which users can experience three-dimensional images and virtual spaces using devices such as VR headsets.

[1560] The "manual editing function" is a function that allows the user to tweak and customize the generated history video.

[1561] "Means of providing as a paid option" refers to a system that allows users to use specific functions or services by paying an additional fee.

[1562] "Means for displaying a preview in real time" is a display function that allows the user to instantly check the video being edited.

[1563] The present invention relates to a system for automatically generating a time-series history video based on photos or video files uploaded by a user. Specific embodiments of the present invention will be described below.

[1564] 1. Uploading photo and video files

[1565] Users register and log in through the system's web interface or a dedicated application. They then select photo and video files and click the upload button. The device sends the files selected by the user to the server, displaying the progress in real time as they are sent. The server then saves the received photo and video files in cloud storage such as AWS S3 and notifies the user when the files have been saved.

[1566] 2. Analyzing the file and generating time series data

[1567] The server analyzes the stored files using image processing algorithms such as OpenCV and TensorFlow. This analysis includes facial recognition, age estimation, and the extraction of metadata such as the date and time of the photo and location. Based on the analysis results, time series data for each file is generated. Specifically, the person's growth process and events are organized chronologically based on age and photo date and time.

[1568] 3. Automatic generation of history videos

[1569] The server automatically generates a scenario for the history video based on the time-series data. This scenario includes the display order of the files, the text to be inserted, transition effects, background music, etc. The server then edits the history video using the automatically generated scenario to generate the final video file. These files are then converted into a format that can be played in 3D space, allowing users to view them in a virtual reality (VR) environment.

[1570] 4. User Notification and Download

[1571] The server stores the generated history video in cloud storage and generates a download link. It notifies the user that the history video has been generated and provides the download link. The user can then watch and experience the video using a smartphone or a virtual reality device.

[1572] 5. Manual editing function (paid option)

[1573] If the user is not satisfied with the generated history video, he or she can select the manual editing function, which is offered as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed. The server allows the user to change the order of the videos, edit the text, and change the background music through the editing tool. The user uses the editing tool to make the necessary corrections and sends the edited content to the server.

[1574] 6. Edited video generation and notification

[1575] The server renders and generates the final history video based on the received editing content. It also provides a function to preview the editing results in real time in the VR environment. The final history video is saved in cloud storage and a notification and download link are provided to the user. The user can then download the final video again and enjoy it in the VR environment.

[1576] Specific examples

[1577] For example, suppose a user uploads 50 photos and 10 videos documenting their child's growth from birth to the present. The system analyzes these and generates a history video that records the child's growth and milestones in chronological order. The user can then watch the video in a VR environment, experiencing it as if they were back in time.

[1578] Example prompt sentence:

[1579] Upload 50 photos and 10 videos to generate a history video showing your child's developmental history.

[1580] The above is a specific embodiment of the present invention, which is a system that not only allows users to easily watch automatically generated high-quality history videos, but also provides an immersive experience in a VR environment.

[1581] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1582] Step 1:

[1583] A user logs in to the system's web interface or a dedicated application, selects photo and video files, and clicks the upload button. The input is the photo and video files selected by the user, and the output is that these files are sent to the server via the device. Specifically, the user uses the file selection dialog on the device and presses the upload button, which causes the device to send the files to the server.

[1584] Step 2:

[1585] The device sends the photo and video files selected by the user to the server. In this case, the input is the photo and video files selected by the user, and the output is the files received by the server. The device also has the function of displaying the progress of the transmission in real time. Specifically, the device uses an HTTP request to upload the files to the server.

[1586] Step 3:

[1587] The server stores the received photo and video files in cloud storage such as AWS S3. The input is the photo and video files sent from the device, and the output is the files stored in the cloud storage. Specifically, the server receives the files and uploads them to the cloud storage using the AWS SDK. A notification is sent to the user when the files have been saved.

[1588] Step 4:

[1589] The server analyzes stored photo and video files, recognizes people, and obtains age information. The input is the photo and video files stored in cloud storage, and the output is age information and metadata as the analysis results. Specifically, the server uses OpenCV and TensorFlow to perform image analysis, recognize faces, and estimate ages. It also extracts the date and time of the photo and location information as metadata.

[1590] Step 5:

[1591] The server generates a chronological event list based on the acquired age information and metadata. The input is the analyzed age information and metadata, and the output is a chronological event list. Specifically, the server organizes the time-series data and builds the event list.

[1592] Step 6:

[1593] The server automatically generates a scenario for a historical video based on the event list. The input is a chronologically organized event list, and the output is a scenario for the historical video. Specifically, the server uses a scenario generation algorithm to automate the file display order, text insertion, transition effects, and background music application.

[1594] Step 7:

[1595] The server generates a history video using the automatically generated scenario. The input is the history video scenario, and the output is the completed history video file. Specifically, the server uses a video editing algorithm to combine the files to create the final video.

[1596] Step 8:

[1597] The server converts the generated history video into a format that can be played in 3D space. The input is the completed history video file, and the output is a video file converted into 3D format. Specifically, the server converts the video into a format that supports stereoscopic viewing.

[1598] Step 9:

[1599] The server saves the generated history video in cloud storage and generates a download link. The input is the video file converted to 3D format, and the output is a download link provided to the user. Specifically, the server uploads the video file to AWS S3, generates a link, and notifies the user.

[1600] Step 10:

[1601] Users can select the paid manual editing option as needed. The input is the history video the user wants to edit, and the output is the edited content created using the editing tool. Specifically, the user changes the order of the video, edits the text, and changes the background music, and the edited content is sent to the server.

[1602] Step 11:

[1603] The server renders the final history video based on the received edits. The input is the edits sent by the user, and the output is the final history video. Specifically, the server applies the video editing algorithm again to reflect the edits.

[1604] Step 12:

[1605] The server saves the edited final history video in cloud storage and notifies the user. The input is the final history video file, and the output is a download link provided to the user. Specifically, the server uploads the final video file to AWS S3, generates a link, and notifies the user.

[1606] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1607] The present invention relates to a system that automatically generates a chronological history video based on photos or video files uploaded by users. By combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[1608] 1. Uploading photo and video files

[1609] User: Log in to the system's web interface or dedicated application, then select the photo and video files and click the upload button.

[1610] Terminal: Sends the user-selected file to the server, displaying the progress in real time.

[1611] Server: Saves the received photo and video files to cloud storage and notifies the user when the files have been saved.

[1612] 2. Analyzing files and obtaining emotional information

[1613] Server: Analyzes stored photos and video files based on image processing algorithms and an emotion engine. This analysis involves identifying people and estimating their ages using facial recognition technology, and extracting emotional information from their facial expressions using the emotion engine.

[1614] Server: Extracts the acquired age and emotion information, as well as metadata such as the date and time of the photo and location information.

[1615] 3. Scenario generation for history videos

[1616] Server: Generates time-series data for photos and video files based on age, emotion, and metadata. Based on this data, it creates a chronological event list.

[1617] Server: Automatically generates a scenario for a historical video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. Based on emotional information, it optimizes the transition effects and background music to emphasize emotionally appealing scenes.

[1618] 4. History video generation and notification

[1619] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video. The progress is displayed in real time during the generation process.

[1620] Server: Saves the completed history video to cloud storage and generates a download link.

[1621] Server: Notifies the user when generation is complete and provides a download link.

[1622] 5. Manual editing function (paid option)

[1623] User: If you are not satisfied with the generated history video, you can select the manual editing feature as a paid option. After completing the payment, the operation screen of the video editing tool will be displayed.

[1624] Server: Allows users to change the order of videos, edit text, change background music, etc. through editing tools.

[1625] User: Make any necessary corrections in a video editing tool.

[1626] On your device: Send your edits to the server.

[1627] 6. Edited video generation and notification

[1628] Server: Re-renders and generates the final history video based on the submitted edits.

[1629] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1630] Specific examples

[1631] For example, consider a case where a user wants to upload 10 photos and 2 videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in cloud storage and uses an emotion engine to recognize facial expressions and emotions. It also estimates ages and extracts metadata to generate time-series data.

[1632] Based on the analysis results, a scenario that takes emotional information into account is automatically generated and the history video is edited. The generated history video is then automatically edited and a download link is provided to the user. Furthermore, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1633] As described above, the present invention enables automatic analysis of photo and video files and the generation of high-quality history videos based on emotional information. Furthermore, a manual editing function can be provided as a paid option, improving both user convenience and profitability.

[1634] The processing flow will be explained below.

[1635] Step 1:

[1636] User: Logs in to the system's web interface or a dedicated application.

[1637] Step 2:

[1638] Server: Authenticates the user's login information and displays the upload screen.

[1639] Step 3:

[1640] User: Select the photo or video file you want to upload and click the upload button.

[1641] Step 4:

[1642] Terminal: Sends the selected file to the server, displaying real-time progress to the user during the sending process.

[1643] Step 5:

[1644] Server: Saves the received file to cloud storage and notifies the user when the save is complete.

[1645] Step 6:

[1646] Server: Analyzes stored photo and video files using image processing algorithms and emotion engines. This analysis includes identifying people using facial recognition technology and obtaining their age and emotion information.

[1647] Step 7:

[1648] Server: Extracts the acquired age information, emotional information, and metadata such as the date and time of the photo and location information.

[1649] Step 8:

[1650] Server: Generates time series data for photos and video files based on age, emotion, and metadata, including creating an event list.

[1651] Step 9:

[1652] Server: Automatically generates a scenario for a history video based on the event list. This scenario includes the file display order, inserted text, transition effects, background music, etc. In particular, it optimizes the scenario by taking into account emotional information.

[1653] Step 10:

[1654] Server: Based on the automatically generated scenario, a video editing algorithm is applied to generate a history video.

[1655] Step 11:

[1656] Server: Displays the progress in real time during generation.

[1657] Step 12:

[1658] Server: Saves the completed history video to cloud storage and generates a download link.

[1659] Step 13:

[1660] Server: Notifies the user when generation is complete and provides a download link.

[1661] Step 14:

[1662] User: After viewing the generated history video, if they feel the need to edit it manually, they can select the manual editing feature as a paid option.

[1663] Step 15:

[1664] Server: Confirms payment for paid options and displays the manual editing tool operation screen to the user.

[1665] Step 16:

[1666] User: Edit videos by changing the order, modifying text, changing background music, etc.

[1667] Step 17:

[1668] On your device: Send your edits to the server.

[1669] Step 18:

[1670] Server: Re-renders and generates the final history video based on the submitted edits.

[1671] Step 19:

[1672] Server: Stores the final history video in cloud storage and provides users with a notification and a download link.

[1673] Example 2

[1674] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1675] It is technically difficult, time-consuming, and laborious to not only simply store photo and video files, but also to organize them chronologically and automatically generate history videos that take emotional information into account. Furthermore, there is a problem that when users want to manually edit the generated videos, there are limited ways to efficiently perform this editing process. Furthermore, there is a lack of systems that allow users to easily generate and quickly use the final version of a video that reflects their own edits.

[1676] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1677] In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in a data management device, means for analyzing the saved photo or video files, recognizing people, and acquiring emotional and age information, means for extracting metadata such as shooting date and time and location information, means for generating a chronological event list based on the acquired age, emotional, and metadata information, means for automatically generating a scenario for the history video based on the event list, means for generating the history video using the automatically generated scenario, means for displaying progress in real time during generation, means for saving the generated history video in the data management device and generating a download link, and means for notifying the user of completion of generation and providing the download link. This enables users to automatically analyze their own photo or video files and quickly generate and edit high-quality history videos based on emotional information.

[1678] "User" refers to a person who uses the System to upload photos and video files and to create and edit history videos.

[1679] "Server" refers to the computer system that manages the overall processing of the system, receives, stores, analyzes photos and video files from users, and generates and notifies users of history videos.

[1680] "Photo or video files" refers to still images or video data uploaded by users to the system.

[1681] "Data management device" refers to a device that stores and manages data such as photos, video files, and generated history videos, including cloud storage and databases.

[1682] "Analysis" refers to the process by which the server examines and processes photo and video files to recognize people, obtain age and emotional information, and extract metadata.

[1683] "Emotional information" refers to emotional information obtained from the facial expressions of people appearing in photos and videos using emotion engines and algorithms.

[1684] "Age information" refers to information about the age of a person that the server recognizes from a photo or video file.

[1685] "Metadata" refers to supplementary information such as the date and time of the photo or video file, location information, etc.

[1686] "Time series data" refers to data that is ordered based on the date and time that a photo or video file was taken.

[1687] An "event list" refers to a list of events organized chronologically based on acquired information.

[1688] A "scenario" refers to a plan that includes elements such as the display order, inserted text, transition effects, and background music when generating a history video.

[1689] "History videos" refer to videos that are automatically generated based on photos and video files uploaded by users and are organized in chronological order.

[1690] The "manual editing function" refers to a function that allows a user to edit the automatically generated history video by changing the order, correcting the text, changing the background music, etc.

[1691] "Paid Option" refers to a service for which a user must pay a separate fee to use special features such as manual editing functionality.

[1692] "Download link" refers to the URL that allows a user to download the generated history video via the Internet.

[1693] "Notification" refers to the act of informing the user of important information, such as the completion of video generation or the provision of a download link.

[1694] "Real-time progress display" refers to a function that notifies the user of the progress of history video generation and editing in real time.

[1695] MODE FOR CARRYING OUT THE INVENTION

[1696] The present invention relates to a system that automatically generates a time-series history video based on photos or video files uploaded by users. Furthermore, by combining it with an emotion engine, the system provides a function to recognize the user's emotions and optimize the scenario of the history video. Specific embodiments of the present invention are described below.

[1697] Uploading photo and video files

[1698] The user begins by logging in to the system's web interface or dedicated application. After logging in, the user selects a photo or video file and clicks the upload button. The device then sends the selected file to the server, displaying the progress in real time. The server then saves the received file to cloud storage (e.g., a data management device) and notifies the user when the file has been saved.

[1699] Analyzing files and extracting emotional information

[1700] The server analyzes the stored photos and video files using an image processing algorithm (e.g., an image processing library) to recognize people and obtain their age information. It also uses an emotion engine (e.g., an emotion recognition service) to obtain emotional information from people's facial expressions. It also extracts metadata such as the date and time of shooting and location information from the photos and video files.

[1701] History video scenario generation

[1702] The server generates time-series data for photos and video files based on the acquired age information, emotion information, and metadata. It then creates a chronological event list based on the generated time-series data. The server then automatically generates a scenario for the history video using the event list. This scenario includes the file display order, inserted text, transition effects, background music, and more. In particular, it optimizes transition effects and background music based on the emotion information to emphasize emotionally appealing scenes.

[1703] History video generation and notification

[1704] The server applies a video editing algorithm (e.g., a video editing library) based on the automatically generated scenario to generate a history video. It also includes a function to display the progress in real time during generation. The completed history video is saved in cloud storage, and a download link is generated. The server notifies the user when generation is complete and provides the download link.

[1705] Manual editing function (paid option)

[1706] If users are dissatisfied with the generated history video, they can select the manual editing feature as a paid option. After completing the payment, the video editing tool operation screen will appear, allowing users to change the order, correct the text, change the background music, etc. The device will send the edited content to the server, and the server will re-render the final history video based on the content.

[1707] Edited video generation and notification

[1708] The server generates the final history video and saves it in cloud storage, and the user is notified and provided with a download link.

[1709] Specific examples

[1710] For example, consider a case where a user wants to upload 10 photos and two videos documenting their child's growth. The user logs into the system's web interface, selects the files on the upload screen, and submits them. The server stores these files in a data management device and uses an emotion recognition service to recognize facial expressions and emotions. Furthermore, it uses an image processing library to estimate age and extract metadata, generating time-series data.

[1711] Based on the analysis results, a scenario that takes emotional information into account is automatically generated, and the history video is edited using a video editing library. The generated history video is automatically edited, and a download link is provided to the user. If the user wishes to revise the video content, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1712] Prompt Sentence Examples

[1713] Below are some examples of specific prompts that can be fed into a generative AI model:

[1714] "I uploaded 10 photos and 2 videos to record my family memories. I want to generate a history video that summarises the important moments based on their emotions."

[1715] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1716] Step 1: User Login

[1717] Input: User ID, Password

[1718] How it works: A user logs in to the system's web interface or a dedicated application. The user enters their user ID and password, which are then sent to the authentication server. The server then authenticates them by checking the information against an existing database.

[1719] Output: Authentication result (login success or failure)

[1720] Step 2: Select and upload files

[1721] Input: Photo or video file

[1722] Specific operation: After logging in, the user selects photo and video files on the upload screen and clicks the upload button. The device will send the selected files to the server and display the progress in real time.

[1723] Output: File saved in cloud storage

[1724] Step 3: Save the file

[1725] Input: Photo or video files sent from your device

[1726] Specific operation: The server saves the received photo and video files to cloud storage (data management device). Once the saving is complete, the server sends a notification to the user.

[1727] Output: Save completion notification

[1728] Step 4: Image and video analysis

[1729] Input: Photo and video files stored in cloud storage

[1730] Specific operation: The server analyzes these files using an image processing algorithm (image processing library) to recognize people. It then uses an emotion engine (emotion recognition service) to obtain emotional information from the people's facial expressions. In parallel, it extracts metadata such as the date and time of the photo and location information.

[1731] Output: Age information, emotion information, metadata

[1732] Step 5: Generate time series data

[1733] Input: Age information, Emotion information, Metadata

[1734] Specific operation: The server generates time-series data of photos and video files based on the acquired age information, emotion information, and metadata. The order is determined based on the shooting date and time of each file.

[1735] Output: Time series data

[1736] Step 6: Create an Event List

[1737] Input: Time series data

[1738] Specific operation: The server creates a chronological event list based on the generated time series data. Each event contains detailed information such as display order, inserted text, transition effects, and background music.

[1739] Output: Event List

[1740] Step 7: Automatic generation of historical video scenarios

[1741] Input: Event List

[1742] How it works: The server automatically generates a scenario for a history video based on the event list, including transition effects and background music optimized based on emotional information.

[1743] Output: History video scenario

[1744] Step 8: Generate History Video

[1745] Input: History video scenario

[1746] Specific operation: The server generates history videos using a video editing algorithm (video editing library), including a function to display the progress in real time during generation.

[1747] Output: Generated history video

[1748] Step 9: Save history video and notification

[1749] Input: Generated history video

[1750] Specific operation: The server saves the completed history video to cloud storage, generates a download link, and provides this download link to the user along with a notification that the video has been generated.

[1751] Output: Download link, notification

[1752] Step 10: Manual Editing (paid option)

[1753] Input: User edit request, payment information

[1754] Specific operation: If the user is dissatisfied with the generated history video, he / she can select the manual editing function as a paid option and complete the payment. The server provides an operation interface for the editing tool, allowing the user to change the video order, edit the text, change the background music, etc.

[1755] Output: Editing tool operation screen

[1756] Step 11: Submit your edits

[1757] Input:Edit

[1758] Specific operation: The user makes the necessary corrections through the editing tool and sends the content to the server. The terminal sends the edited content to the server, and the server receives it.

[1759] Output: Edited content

[1760] Step 12: Generate the final history video

[1761] Input:Edited content

[1762] Specific operation: The server re-renders and generates the final history video based on the submitted edits.

[1763] Output: Final history video

[1764] Step 13: Save and notify the final history video

[1765] Input: Final history video

[1766] Specific operation: The server saves the final history video in cloud storage and provides the user with a notification and a download link.

[1767] Output: Notification, download link

[1768] (Application example 2)

[1769] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1770] When creating history videos from photos and videos taken by users, manual editing and organization is cumbersome, making it difficult to efficiently generate high-quality videos that reflect emotions. Another issue is the lack of functionality to share the generated videos or check the progress.

[1771] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo or video files from a user, means for saving the received photo or video files in cloud storage, means for analyzing the saved photo or video files to recognize people and acquire age information and emotional information, means for generating a chronological event list based on the acquired age information, emotional information, and metadata, means for automatically generating a scenario for a history video based on the event list and emotional information, means for generating an emotionally appealing history video using the automatically generated scenario, and means for notifying the user of the generated history video and providing a download link. This enables users to automatically and effectively generate history videos that take emotional information into consideration and easily share the videos.

[1772] A "photo or video file" is a media file that records a still image or sequence of images that a user provides to the system.

[1773] "Cloud storage" is an online storage service that stores data over the Internet.

[1774] "Emotional information" is data on psychological states and emotions extracted from the facial expressions and behavior of people included in photographs or video files.

[1775] "Metadata" is auxiliary information that accompanies a photo or video file, such as the date and time the photo was taken, location information, and camera settings.

[1776] An "event list" is a chronological list of events or activities that serves as the basis for generating history videos.

[1777] "History videos" are automatically generated video clips edited in chronological order based on photos and video files provided by users.

[1778] "Automatic scenario generation" is the process of automatically creating a storyline and structure for a video using captured data and emotional information.

[1779] A "download link" is a URL that allows users to download the completed history video on the Internet.

[1780] The "manual editing function" is a function that allows the user to customize the generated history video by changing the order, correcting text, changing background music, etc.

[1781] "Rendering" is the process of converting edited video and audio into the final video format.

[1782] "SNS" is an abbreviation for social networking service, an online platform for users to share information with each other.

[1783] "Real-time progress display" is a function that instantly notifies the user of the current processing status while generating or editing a history video.

[1784] A "transition effect" is a visual effect that provides a smooth transition between different scenes or images.

[1785] System Overview

[1786] This system automatically generates a history video based on emotion recognition and age estimation using photos or video files uploaded by users, generating a chronological event list, and then creating a history video based on that list. If users prefer to edit the list manually, a manual editing function is also available as a paid option. The main components of this system and their functions are described in detail below.

[1787] System Components

[1788] 1. Uploading photo and video files

[1789] Users use a device (smartphone or PC) to log in to the system's web interface or dedicated application, select and upload photo and video files, and the device sends the files selected by the user to the server, displaying the transfer progress in real time. The server then stores the received photo and video files in cloud storage.

[1790] 2. Analyzing files and obtaining emotional information

[1791] The server analyzes the stored photo and video files. The analysis algorithm uses facial recognition technology, age estimation models, and an emotion engine to recognize people in the photos and videos and obtain their age and emotion information.

[1792] 3. Scenario generation for history videos

[1793] The server uses the collected age information, emotional information, and metadata to generate a chronological event list. Based on this, it automatically generates a scenario for the history video. This scenario includes the file display order, inserted text, transition effects, background music, and more. It takes into account the emotional information in particular and optimizes the video to emphasize emotionally appealing scenes.

[1794] 4. History video generation and notification

[1795] The server uses the automatically generated scenario to apply a video editing algorithm to generate a history video. The progress is displayed in real time during the process, and once the video is complete, it is saved in cloud storage and a download link is provided to the user.

[1796] 5. Manual editing function (paid option)

[1797] If the user wishes to make edits to the generated history video, they can select the manual editing function as a paid option. After completing the payment, the server will provide the operation screen of the video editing tool, where the user can change the order of the videos, edit the text, change the background music, etc. The edited content will be sent to the server, which will then render the final version of the history video.

[1798] Specific examples

[1799] For example, suppose a user uploads ten photos and two videos documenting their child's growth to the system. The server stores these files and uses facial recognition technology and an emotion engine to recognize the person's facial expressions and emotions. It then extracts age information and metadata to generate time-series data. Based on the analysis results, it automatically generates a scenario that takes emotional information into account and edits the history video. The generated history video is automatically edited and a download link is provided to the user. In addition, if the user wishes to revise the content of the video, a manual editing function is available as a paid option. After editing is complete, the server generates the final version of the history video and notifies the user again.

[1800] Prompt Sentence Examples

[1801] "Analyze uploaded photos and videos, extract the emotional information from each media, and generate a moving chronological history video."

[1802] This allows users to automatically generate high-quality history videos that take emotional information into account, and also makes it easy to edit and share the videos.

[1803] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1804] Step 1:

[1805] Users use their devices to log in to the system's web interface or a dedicated application to select and upload photo and video files. As input, photo and video files are provided. These files are sent from the user's device to the server. As output, the progress of file selection and sending is displayed in real time.

[1806] Step 2:

[1807] The server stores the photo and video files received from the user in cloud storage. The received photo and video files are provided as input. These files are stored in cloud storage and the server notifies the user that the storage is complete. As output, a notification of the completion of the storage is sent to the user.

[1808] Step 3:

[1809] The server analyzes the photo and video files stored in the cloud storage. The stored photo and video files are provided as input. It uses face recognition technology to identify the person, and then uses an age estimation model and an emotion engine to obtain age and emotion information. This is done using an image processing algorithm and an emotion recognition model. The analyzed age and emotion information is obtained as output.

[1810] Step 4:

[1811] The server uses the acquired age information, emotion information, and metadata to generate a time-ordered event list. Age information, emotion information, and metadata are provided as input. Data analysis algorithms are used to integrate this information and generate a time-ordered event list. The output is a time-ordered event list.

[1812] Step 5:

[1813] The server automatically generates a scenario for the history video based on the generated event list and emotion information. The server receives the chronological event list and emotion information as input. It uses a scenario generation algorithm to create a scenario including elements such as display order, inserted text, transition effects, and background music. The server outputs the automatically generated scenario.

[1814] Step 6:

[1815] The server generates a history video using an automatically generated scenario. The scenario is provided as input. A video editing algorithm is used to generate a video based on the scenario. The progress is displayed in real time during the generation process. The generated history video is obtained as output.

[1816] Step 7:

[1817] The server saves the generated history video in cloud storage and generates a download link. The server receives the generated history video as input, saves the video file in cloud storage and provides the download link to the user, and provides the download link to the user as output.

[1818] Step 8:

[1819] If the user requests a manual editing function, the server provides the manual editing function as a paid option. The user's request is provided as input. After payment is completed, the server provides the operation screen of the video editing tool, allowing the user to change the video order, edit text, change background music, etc. The operation screen of the editing tool is provided to the user as output.

[1820] Step 9:

[1821] The user sends the edited content to the server. The edited content is provided as input. The server renders the final history video based on the edited content and saves it back to cloud storage. The final history video is obtained as output.

[1822] Step 10:

[1823] The server saves the edited final history video in cloud storage and notifies the user. The final history video is provided as input. After saving is complete, the server notifies the user and provides a download link again. As output, a download link for the final history video is provided to the user.

[1824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1825] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1827] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emot...

Claims

1. means for receiving a photo or video file from a user; A means for saving received photo or video files to cloud storage; A means for analyzing stored photo or video files to recognize people and obtain their age information; a means for generating a chronological event list based on the acquired age information and metadata; A means for automatically generating a scenario for a history video based on an event list; A means for generating a history video using an automatically generated scenario; A means for notifying a user of the generated history video and providing a download link; A system including:

2. A means for providing a manual editing function for the generated history video; A means to offer the use of manual editing functions as a paid option; A means of rendering the final history video based on the user's edits; The system of claim 1 further comprising:

3. a means for displaying the progress of the generated history video in real time; A means for providing a user with an operation screen of an editing tool, allowing the user to freely change the order of videos, modify text, and change background music; A method for saving the edited final history video in cloud storage and notifying the user; The system of claim 1 further comprising:

Images