System

Abstract

A system is provided.SOLUTION: A system comprising: means for taking pictures or videos with user-operable terminals and uploading the pictures or videos; means for receiving the uploaded pictures or videos on a server side and storing the pictures or videos in cloud storages; generation and AI means for analyzing the pictures or videos in the cloud storages and classifying and organizing the pictures or videos based on specific days or events; means for automatically generating storytelling type videos based on the pictures or videos organized by the generation and AI means; and means for providing the automatically generated videos to the terminals of users so that the videos can be played.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the widespread use of digital cameras and smartphones today, it has become easy to take and store large amounts of photos and videos. However, many of these photos and videos are unorganized and buried in vast amounts of data, making it difficult to easily relive memories. Furthermore, many existing storage services require fees, which many users find to be a financial burden. Furthermore, manually editing photos and videos is time-consuming and often requires specialized knowledge. There is a need for a way to solve these issues and allow users to visually enjoy their memories easily and for free. [Means for solving the problem]

[0005] This invention is a system that allows users to upload photos and videos they have taken to the cloud and automatically organize and edit them using generative AI. Specifically, these problems are solved by a system that includes the following means.

[0006] The system includes means for taking photos and videos using a device that can be operated by a user and uploading the photos and videos, means for receiving the uploaded photos and videos on the server side and saving them in cloud storage, generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific days or events, means for automatically generating storytelling-type videos based on the photos and videos organized by the generation AI means, and means for providing the automatically generated videos to the user's device and making them playable.

[0007] The system further includes a means for adding metadata (such as the date and time of shooting, location, and device information) to the user's photos and videos and sending them together when they are uploaded, a means for the generation AI means to perform analysis including face recognition, background analysis, and facial expression recognition and create a video scenario based on the analysis, a means for adding transition effects and background music to the storytelling-type video generated by the generation AI means, and a means for analyzing the contents of the photos and videos uploaded on the server side and suggesting the deletion of photos and videos that are deemed unnecessary.

[0008] "User" is a term that refers to any person or entity that uses the system to take and upload photos and videos.

[0009] A "terminal" is a device that is primarily operated by a user, and includes smartphones, tablets, and personal computers.

[0010] "Photo" refers to a still image file taken by the user using the camera on the device.

[0011] "Video" refers to a video file that a user has taken using the camera on their device.

[0012] "Upload" is a term that refers to the act of transferring photo or video data from a device to a server.

[0013] "Server" refers to a computer system that receives photo and video data via the Internet and stores it in cloud storage.

[0014] "Cloud storage" is online storage provided by a server, and refers to a virtual space where photos and videos are stored.

[0015] "Generative AI" refers to algorithms and programs that use artificial intelligence techniques to analyze, organize, and edit photos and videos.

[0016] "Storytelling videos" refer to visually appealing videos that are edited and processed by generative AI based on a specific day or event.

[0017] "Metadata" is additional information added to photos and videos, such as the date and time the photo was taken, the location, and device information.

[0018] "Facial recognition" refers to the technology that allows generative AI to identify people's faces in photos and videos.

[0019] "Contextual analysis" refers to the technology that allows generative AI to identify and analyze contextual information within photos and videos.

[0020] "Facial expression recognition" refers to the technology that allows generative AI to identify and analyze facial expressions of people in photos and videos.

[0021] "Transition effects" refers to the visual effects that generative AI uses to switch between scenes during the video editing process.

[0022] "Background music" refers to music inserted into videos by generative AI to complement the visual content. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] Hereinafter, embodiments of the present invention will be described in detail.

[0045] 1. User operations and photo / video uploads

[0046] User takes a photo with their smartphone:

[0047] Users use devices such as smartphones and tablets to take photos and videos of events such as their family's birthday parties.

[0048] User opens the app:

[0049] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set the title to "Birthday Party," for example.

[0050] User selects photos and videos:

[0051] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0052] 2. Transferring data from the device to the server

[0053] The device prepares the data:

[0054] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[0055] The device sends data:

[0056] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[0057] 3. Data reception and storage by the server

[0058] Server receives data:

[0059] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[0060] Server reconstructs data:

[0061] The photos and video files are reconstructed based on the received data and saved in a "Birthday Party" folder in cloud storage.

[0062] 4. Content analysis and video generation using generative AI

[0063] Generative AI analyzes content:

[0064] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of the event (e.g., cutting the cake or opening a gift).

[0065] Generative AI creates scenarios:

[0066] Based on the analysis results, it automatically creates a storytelling video scenario, placing key moments on a timeline and generating a visually engaging video.

[0067] Generative AI generates videos:

[0068] The generative AI then edits the video according to the scenario, adding transition effects and background music, and the finished video is saved as a new file in cloud storage.

[0069] 5. User Feedback and Video Playback

[0070] The server notifies the user:

[0071] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0072] User plays video:

[0073] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy their memories visually.

[0074] For example, users can take photos and videos of their child's birthday party and upload them through the app. The AI ​​then automatically analyzes the content and creates a storytelling video. Users can then play the video on their device and enjoy the memories of the party.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user takes a photo or video

[0078] A user uses a smartphone camera app to take photos and videos of a family birthday party.

[0079] Step 2:

[0080] The user opens the app

[0081] The user opens the storytelling app, selects "Create a new album" from the home screen, and enters "Birthday Party" as the album title.

[0082] Step 3:

[0083] User selects photos and videos and starts uploading

[0084] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0085] Step 4:

[0086] The device prepares the data

[0087] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[0088] Step 5:

[0089] The terminal divides and transmits the data

[0090] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[0091] Step 6:

[0092] The server receives the data

[0093] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[0094] Step 7:

[0095] The server reconstructs the data

[0096] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[0097] Step 8:

[0098] The server calls the generated AI

[0099] The server passes the path information of the photos and videos in cloud storage to the generation AI.

[0100] Step 9:

[0101] Generative AI analyzes content

[0102] The generative AI analyzes the metadata and content of each photo or video and uses techniques such as facial recognition, background analysis, and facial expression recognition.

[0103] Step 10:

[0104] Generative AI creates scenarios

[0105] Based on the analysis, key moments are selected and a storytelling video scenario is created.

[0106] Step 11:

[0107] Generative AI generates videos

[0108] The generative AI edits photos and videos according to a scenario, adds transition effects and background music, and generates a video that is then saved in cloud storage.

[0109] Step 12:

[0110] The server notifies the user

[0111] The server records the location of the generated video and sends a push notification to the user's smartphone.

[0112] Step 13:

[0113] The user plays the video

[0114] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] There is a need for an efficient method to manage and analyze the growing amount of photo and video data and reconstruct them into memorable formats. However, many users have to spend a lot of time and effort manually organizing and editing each photo and video. Therefore, a system is needed that allows users to easily upload their captured content and automatically analyzes, classifies, and reconstructs it into a visually enjoyable format.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes: means for taking photos and videos using a device operable by a user and uploading the photos and videos; means for adding metadata (such as the shooting date and time, location, and device information) to the uploaded photos and videos on the device side, storing them in a buffer, and then transmitting them to the server; means for receiving the transmitted data on the server side, verifying the sender's identification information, checking the data integrity, and then storing them in a cloud storage device; means for analyzing the photos and videos in the cloud storage device using a generative AI model, performing facial recognition, background analysis, facial expression recognition, etc., to identify important moments of the event; means for automatically generating storytelling-type videos based on the analysis results and adding transition effects and background music; and means for providing the automatically generated videos to the user's device and making them playable. This enables efficient management of photos and videos taken by users and automatically generating visually enjoyable videos.

[0120] "User" refers to an individual or organization that operates the system to take and upload photos and videos.

[0121] "Terminal" refers to a device that can be operated by a user, such as a smartphone, tablet, or PC.

[0122] "Photo" refers to a still image taken by a user on a device.

[0123] "Video" refers to a file format that contains moving images captured by a user on a device.

[0124] "Uploading" refers to the process of sending data such as photos or videos from a device to a server or cloud storage device.

[0125] "Metadata" refers to additional information associated with a photo or video, including the date and time it was taken, the location, and device information.

[0126] A "buffer" refers to a storage area that temporarily stores data.

[0127] "Server" means a computer that acts as a central processing unit and receives, processes, and stores uploaded data.

[0128] "Cloud storage" refers to a remote storage medium accessible over a network.

[0129] A "generative AI model" refers to an artificial intelligence algorithm that uses machine learning and deep learning technologies to analyze data and automatically generate and edit scenarios.

[0130] "Facial recognition" refers to the technology in which a generative AI model identifies the faces of people in photos and videos.

[0131] "Background analysis" refers to the technology in which a generative AI model analyzes the background or scenery in a photo or video.

[0132] "Facial expression recognition" refers to a technology in which a generative AI model distinguishes between facial expressions and analyzes emotions.

[0133] "Transition effects" refer to visual effects used when switching from one scene to another in video editing.

[0134] "Background music" refers to music or sound effects added to the background of a video.

[0135] "Storytelling video" refers to visual content that is edited around a specific scenario or narrative flow.

[0136] Hereinafter, embodiments of the present invention will be described in detail.

[0137] 1. User operations and photo / video uploads

[0138] A user uses a device such as a smartphone or tablet to take photos and videos of an event such as a family birthday party. For example, the user can take photos of a birthday cake or a video of a child opening a present. The user then opens a storytelling app. The app has a "Create a New Album" option, and the user creates a new album titled "Birthday Party." The user selects the photos and videos they want to upload from their device's gallery and taps the upload button to send the photos and videos.

[0139] 2. Transferring data from the device to the server

[0140] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos. Metadata adds important information to the uploaded content, making it useful for analysis and management. The device temporarily stores all data for upload in a buffer. The buffer ensures data consistency. The device then sends the photo and video data to the server. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer and makes retransmission easier if a communication error occurs.

[0141] 3. Data reception and storage by the server

[0142] The server receives the uploaded data and checks the session ID to identify the user who sent it. For example, the session ID can be used to identify who uploaded what data. The server uses a checksum to verify that the received data is correct, and requests a retransmission of any incomplete packets. This ensures the reliability of data transfer. The server reconstructs the photo and video files based on the received data. The reconstructed files are saved in the "Birthday Party" folder in the cloud storage.

[0143] 4. Content analysis and video generation using generative AI

[0144] The generation AI on the server analyzes the metadata and content of the stored photos and videos. Specifically, it uses technologies such as facial recognition, background analysis, and facial expression recognition to identify important moments (e.g., the moment a cake is cut or a gift is opened). Based on the analysis results, the generation AI automatically creates a storytelling video scenario. For example, the scenario may include scenes of cutting a cake or opening a gift on a timeline. The generation AI then edits the video according to the scenario, adding transition effects and background music to create a visually appealing video. The completed video is then saved in cloud storage.

[0145] 5. User Feedback and Video Playback

[0146] The server records the location where the generated video is saved and notifies the user that a new video has been generated. This is done via a push notification on the device or a message within the app. After checking the notification, the user opens the app and plays the generated video. This allows the user to visually enjoy the memories of the party.

[0147] Prompt Sentence Examples

[0148] Examples of prompts to input to a generative AI model include:

[0149] Users have uploaded photos and videos of their "birthday party." Use this content to generate a storytelling video that includes key moments like cutting the cake and opening gifts. Add transition effects and background music to create a visually appealing video.

[0150] The above is an embodiment of the invention.

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1:

[0153] Taking a photo and launching the app

[0154] A user takes photos or videos using a smartphone or tablet. For example, the user takes photos or videos of a birthday party, and then launches a storytelling app.

[0155] Input: Photos and videos taken by the user.

[0156] Output: Application launch.

[0157] Step 2:

[0158] Creating an album and selecting files

[0159] Users can select the "Create a new album" option within the app, set the title to "Birthday Party," then select the photos and videos they want to upload from their device's gallery and tap the upload button.

[0160] Input: User-specified new album name and selected photos and videos.

[0161] Output: Data ready for upload.

[0162] Step 3:

[0163] Adding metadata and saving it to the buffer

[0164] Your device will add metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos, giving them information about when and where they were taken. It then stores these files in a buffer and prepares them for transmission.

[0165] Input: Photo and video files selected by the user.

[0166] Output: Buffered photo and video files with metadata.

[0167] Step 4:

[0168] Splitting the data and sending it to the server

[0169] The device splits the large video file into chunks and sends the photos and split video data to the server, using an appropriate communication protocol to prevent data loss.

[0170] Input: Photo and video files with metadata stored in a buffer.

[0171] Output: The split data sent to the server.

[0172] Step 5:

[0173] Data reception and checksum verification by the server

[0174] The server validates the received data using checksums for integrity and requests retransmission of any incomplete packets, thus ensuring data consistency and quality.

[0175] Input: Split data sent from the terminal.

[0176] Output: Validated complete data.

[0177] Step 6:

[0178] Data reconstruction and cloud storage

[0179] The server reconstructs the received data to generate the original photo and video files, and then stores the reconstructed files in a "Birthday Party" folder in the cloud storage device.

[0180] Input: Validated data.

[0181] Output: Photo and video files saved in cloud storage.

[0182] Step 7:

[0183] Content analysis with generative AI

[0184] The server-based generative AI analyzes the metadata and content of photos and videos stored in cloud storage, using techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of an event.

[0185] Input: Photo and video files stored in cloud storage.

[0186] Output: Parsed content and identified key moments data.

[0187] Step 8:

[0188] Scenario creation and video generation

[0189] The generative AI creates a scenario based on the analysis results and automatically edits a storytelling video, adding transition effects and background music, and saving the generated video as a new file in cloud storage.

[0190] Input: Analysis results and data on key moments of identified events.

[0191] Output: Auto-generated storytelling videos.

[0192] Step 9:

[0193] User Notification

[0194] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0195] Input: An auto-generated video.

[0196] Output: Notification to the user.

[0197] Step 10:

[0198] Playing a video

[0199] The user confirms the notification, opens the app, and plays the generated video, allowing them to visually enjoy their birthday party memories.

[0200] Input: Notification from the server.

[0201] Output: The video played in the app.

[0202] In this way, each processing step works together and fulfills its role, allowing users to record and play back memories easily and quickly.

[0203] (Application example 1)

[0204] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] Previous technology lacked the means to promote products in stores or enhance the customer's purchasing experience. Furthermore, there was no system that allowed customers to take photos of products and effectively utilize those photos or videos. This made it difficult to effectively promote and introduce products to individual customers, limiting the ways to stimulate purchasing motivation.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0207] In this invention, the server includes a means for taking photos and videos using a user-operable device and uploading the photos and videos, a means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, a generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, a means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, a means for incorporating product introduction information and promotional information into the automatically generated videos, and a means for providing the automatically generated videos to the user's device and making them playable. This enables the automatic generation of effective promotional videos based on photos and videos of products and events taken by customers.

[0208] A "user-operable device" is an electronic device that can be operated by a user and used to take and upload photographs and videos.

[0209] "Means for uploading photos and videos" refers to a method or device for transmitting photos and video data taken by a user to a server.

[0210] "Means for receiving data on the server side and storing it in cloud storage" refers to a method or device for storing data received by the server in a cloud environment.

[0211] "Generative AI methods" refers to artificial intelligence technology that analyzes photos and videos uploaded and stored in cloud storage, categorizing and organizing them based on specific days or events.

[0212] A "means for automatically generating storytelling videos" refers to a method or device that automatically creates narrative videos using a sequence of photos and videos based on the analysis results of generative AI.

[0213] "Means for incorporating product introduction information and promotional information" refers to a method or device for adding product-related information and promotional content to the generated video.

[0214] "Means for providing the video to the user's terminal and enabling playback" refers to a method or device for delivering the generated video to the user's terminal and enabling playback on the terminal.

[0215] "Metadata" refers to additional information associated with a photo or video (e.g., date and time it was taken, location, device information, etc.).

[0216] "Facial recognition" is a technology that identifies people's faces in photos and videos and analyzes their features.

[0217] "Background analysis" refers to the technology of identifying background elements in photos and videos and analyzing their content.

[0218] "Facial expression recognition" is a technology that identifies facial expressions of people in photos and videos and analyzes their emotions and reactions.

[0219] "Creating a scenario" means designing a video with a narrative by determining the structure and order of the video based on the generated data.

[0220] MODE FOR CARRYING OUT THE INVENTION

[0221] Hereinafter, embodiments of the present invention will be described in detail.

[0222] 1. User operations and photo / video uploads

[0223] User takes a photo with their smartphone:

[0224] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in while in a physical store.

[0225] User opens the app:

[0226] Users open the app, access the photo and video upload screen, select the "Create a new promotional video" option, and set an appropriate title.

[0227] User selects photos and videos:

[0228] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0229] 2. Transferring data from the device to the server

[0230] The device prepares the data:

[0231] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[0232] The device sends data:

[0233] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[0234] 3. Data reception and storage by the server

[0235] Server receives data:

[0236] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[0237] Server reconstructs data:

[0238] The photo and video files are reconstructed based on the received data and saved in the appropriate folder in cloud storage.

[0239] 4. Content analysis and video generation using generative AI

[0240] Generative AI analyzes content:

[0241] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition to identify product features and promotional information.

[0242] Generative AI creates scenarios:

[0243] Based on the analysis results, the system automatically creates a storytelling video scenario, placing the key features of the product on a timeline and generating a visually appealing video.

[0244] Generative AI generates videos:

[0245] The AI ​​then edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The finished video is then saved as a new file in cloud storage.

[0246] 5. User Feedback and Video Playback

[0247] The server notifies the user:

[0248] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0249] User plays video:

[0250] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy product and promotional information visually.

[0251] Specific examples

[0252] Customers can take photos of products in the store through the app, and the photos are sent to a cloud server. The AI ​​analyzes the photos and automatically creates promotional videos for the products and stores. The created videos are then sent to customers via push notifications and can be played back within the app.

[0253] Example prompt sentence:

[0254] "Use a generative AI model to analyze this product photo and create a product video. The analysis should include the product name, price, and instructions for use."

[0255] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0256] Step 1:

[0257] User takes a photo with their smartphone:

[0258] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in in physical stores. At the same time, the date and time of the photo and location information are added to the images and videos as metadata. The input is photos and videos of the products, and the output is image and video files with the added metadata.

[0259] Step 2:

[0260] User opens the app:

[0261] The user opens the app and accesses the screen for uploading photos and videos they have taken. Here, they select the "Create a new promotional video" option and set an appropriate title. The input is the user's operation, and the output is the transition to the upload screen.

[0262] Step 3:

[0263] User selects photos and videos:

[0264] The user selects the photos or videos they want to upload from their device's gallery. Once selection is complete, they tap "Upload." The input is the selected media file, and the output is the start of the upload process.

[0265] Step 4:

[0266] The device prepares the data:

[0267] The device checks the metadata already attached to the selected photos and videos (such as the date and time of the photo, location, and device information) and stores these files in a buffer for transmission to the server. The input is the selected files, and the output is the data ready to be sent.

[0268] Step 5:

[0269] The device sends data:

[0270] The device sends photo and video data to the server in chunks. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer. The input is the data ready to be sent, and the output is the piece of data being sent.

[0271] Step 6:

[0272] Server receives data:

[0273] The server receives the uploaded data, checks the session ID to identify the sending user, verifies the received data is correct using checksums, and requests retransmission of any incomplete packets. The input is the piece of data being sent, and the output is the complete data file.

[0274] Step 7:

[0275] Server reconstructs data:

[0276] The server reconstructs the photo or video file from the received data fragments and saves it in the appropriate folder in the cloud storage. The input is the data fragments and the output is the reconstructed file.

[0277] Step 8:

[0278] Generative AI analyzes content:

[0279] The generative AI model in the server analyzes the metadata and content of stored photos and videos, identifying product features using techniques such as facial recognition, background analysis, and facial expression recognition. The input is the stored media files, and the output is the analyzed data.

[0280] Step 9:

[0281] Generative AI creates scenarios:

[0282] Based on the analysis results, the generative AI automatically creates a storytelling-style video scenario. It places the product's main features on a timeline and generates a visually appealing video. The input is analysis data, and the output is scenario data.

[0283] Step 10:

[0284] Generative AI generates videos:

[0285] The generative AI edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The completed video is saved as a new file in cloud storage. The input is the scenario data, and the output is the generated video file.

[0286] Step 11:

[0287] The server notifies the user:

[0288] The server records the location of the generated video and notifies the user that a new video has been generated. The notification is sent in the form of a push notification on the device or a message within the app. The input is the generated video information, and the output is a push notification.

[0289] Step 12:

[0290] User plays video:

[0291] The user confirms the notification and opens the app to play the generated video. This allows users to easily enjoy product and promotion information visually. The input is the push notification, and the output is the playing video.

[0292] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0293] Hereinafter, embodiments of the present invention will be described in detail.

[0294] 1. User operations and photo / video uploads

[0295] User takes a photo with their smartphone:

[0296] Users use devices such as smartphones and tablets to take photos and videos of events such as family birthday parties.

[0297] User opens the app:

[0298] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set a title, for example, "Birthday Party."

[0299] User selects photos and videos:

[0300] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0301] 2. Transferring data from the device to the server

[0302] The device prepares the data:

[0303] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and store the data in a temporary buffer.

[0304] The terminal splits and sends the data:

[0305] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[0306] 3. Data reception and storage by the server

[0307] Server receives data:

[0308] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[0309] Server reconstructs data:

[0310] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[0311] 4. Content analysis and video generation using generative AI and emotion engines

[0312] Generative AI analyzes content:

[0313] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition.

[0314] The emotion engine analyzes emotions:

[0315] The generative AI has a built-in emotion engine that analyzes the emotions (e.g., smiling, sad, surprised, etc.) of the people in each photo or video.

[0316] Generative AI creates scenarios:

[0317] Based on the analysis results, it automatically creates a storytelling video scenario, taking into account emotional information, selecting appropriate scenes, and arranging them along the timeline.

[0318] Generative AI generates videos:

[0319] The AI ​​then edits the photos and videos according to the scenario, adding transition effects and background music to create a video. The generated video is then saved as a new file in cloud storage.

[0320] 5. User Feedback and Video Playback

[0321] The server notifies the user:

[0322] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0323] User plays video:

[0324] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[0325] For example, a user can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and generate a storytelling video. The generated video includes key moments, such as cutting the birthday cake and opening presents, edited with emotional information, allowing the user to play the video on their device and enjoy memories of the party.

[0326] The processing flow will be explained below.

[0327] Step 1:

[0328] The user takes a photo with their smartphone

[0329] Users use devices such as smartphones and tablets to take photos and videos of family birthday parties.

[0330] Step 2:

[0331] The user opens the app

[0332] The user opens the Storytelling app and selects "Create a new album" from the home screen. They enter the album title as "Birthday Party."

[0333] Step 3:

[0334] User selects photos and videos and starts uploading

[0335] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0336] Step 4:

[0337] The device prepares the data

[0338] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[0339] Step 5:

[0340] The terminal divides and transmits the data

[0341] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[0342] Step 6:

[0343] The server receives the data

[0344] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[0345] Step 7:

[0346] The server reconstructs the data and stores it in cloud storage.

[0347] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[0348] Step 8:

[0349] The server calls the generative AI and emotion engine

[0350] The server passes the path information of the photos and videos in the cloud storage to the generation AI, which analyzes the metadata and content of the stored photos and videos, and the emotion engine analyzes the emotions (e.g., smile, sadness, surprise, etc.) of the people in each photo and video.

[0351] Step 9:

[0352] Generative AI and emotion engines analyze content

[0353] The generative AI analyzes photos and videos using technologies such as facial recognition, contextual analysis, and facial expression recognition. The emotion engine identifies changes in emotions and provides that information to the generative AI.

[0354] Step 10:

[0355] Generative AI creates scenarios

[0356] The generative AI uses information from the emotion engine to automatically create a storytelling video scenario, selecting key moments (e.g., cutting a birthday cake or opening a gift) and placing them on a timeline.

[0357] Step 11:

[0358] Generative AI generates videos

[0359] The AI ​​then edits the video based on the scenario, adding transition effects and background music to match the emotional changes, and the resulting video is saved as a new file in cloud storage.

[0360] Step 12:

[0361] The server notifies the user

[0362] The server records the location of the generated video and sends a push notification to the user's smartphone.

[0363] Step 13:

[0364] The user plays the video

[0365] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[0366] Example 2

[0367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] In recent years, the amount of photos and videos taken with digital devices has rapidly increased, creating a need for efficient classification and management of these images and videos, and for users to easily access them. Furthermore, to effectively preserve and share memories of special family events, simply storing photos and videos is insufficient; they must be reconstructed as storytelling content. Furthermore, there is a need for a method to automatically generate high-quality content with minimal user interaction.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for taking images and videos using a terminal operable by a user and uploading the images and videos, means for receiving the uploaded images and videos on the server side and saving them in shared storage, generation AI means for analyzing the images and videos in the shared storage and classifying and organizing them based on specific dates or events, means for automatically generating story-style videos based on the images and videos organized by the generation AI means, and means for providing the automatically generated videos to the user's terminal and making them playable. This allows users to efficiently manage their captured content and enjoy their special memories by automatically converting them into high-quality storytelling-style videos.

[0370] A "terminal" is a device that can be operated by a user, and includes, for example, a smartphone, a tablet, or a PC.

[0371] "Images and Videos" means still and moving images stored in digital format that are used to record events and moments in everyday life.

[0372] "Upload" refers to the operation or process of sending data from a device to a server.

[0373] A "server" is a computer system that receives, stores, analyzes, and generates data and communicates with terminals via a network.

[0374] "Shared storage" refers to a storage device connected to a server, used to centrally manage and store data from multiple users.

[0375] "Analysis" refers to the process of performing specific processing on digital content to extract and classify its content and characteristics.

[0376] "Generative AI" refers to a system that uses artificial intelligence techniques to process and analyze data and automatically create specific deliverables.

[0377] "Narrative video" refers to a video that is not simply a series of images, but is edited and reconstructed based on a specific theme or scenario.

[0378] "Delivery" refers to the process of making the generated deliverables available to users.

[0379] "Play" refers to the action initiated to watch or listen to a stored video or audio file.

[0380] "Metadata" refers to auxiliary information added to images and videos (e.g., shooting date and time, location, device information, etc.), and is used when organizing and analyzing data.

[0381] This invention is a system that uploads images and videos from users' devices to a server, analyzes and organizes them, and automatically generates story-style videos. This system allows users to efficiently manage their captured content and enjoy their special memories as high-quality storytelling videos.

[0382] Hardware and Software Configuration

[0383] Terminal

[0384] Users take pictures and videos using devices such as smartphones and tablets. These devices have a camera function and a dedicated storytelling application installed. The application provides an operation screen and supports the selection and uploading of images and videos.

[0385] server

[0386] The server is a cloud-connected computer system that receives and stores content uploaded by users. It is equipped with a generative AI and emotion analysis engine, which are used to analyze and organize images and videos, generate scenarios, and generate videos.

[0387] Generative AI and Sentiment Analysis Engine

[0388] The generative AI includes advanced image processing technology, enabling facial recognition, background analysis, and facial expression recognition. The emotion analysis engine analyzes the emotions of people in each image or video, and generates a scenario based on that information. Based on the generated scenario, appropriate effects and music are added to create a narrative video.

[0389] Data processing and calculation

[0390] Adding Metadata

[0391] The device automatically adds metadata to the images and videos it takes, including the date and time the image was taken, the location, and device information. When a user uploads an image or video, this metadata is also sent to the server.

[0392] Data division and transmission

[0393] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[0394] Receiving and Reconstructing Data

[0395] The server receives each chunk and checks it for errors using a checksum. If an error is found, the server requests that the chunk be retransmitted. Correctly received data chunks are reconstructed into the original file and saved to shared storage.

[0396] Analyzing and organizing content

[0397] Generative AI analyzes the metadata and content of images and videos stored on shared storage. Based on the analysis results, content is categorized and organized based on specific dates and events. Specific people and scenes are identified through facial recognition, context analysis, and facial expression recognition.

[0398] Scenario creation and video generation

[0399] The generative AI automatically generates a story-style video scenario based on the analysis results. The emotion analysis engine reflects the emotional information in each scene and selects and adds effects and music in line with the scenario. Finally, the automatically generated video is saved in cloud storage that users can access.

[0400] Specific examples

[0401] For example, users can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and create a storytelling video. The generated video includes key moments, such as cutting the birthday cake or opening presents, edited with emotional information, allowing users to play the video on their device and enjoy their memories of the party.

[0402] Example prompt sentence:

[0403] "Use photos and videos from a child's birthday party to generate a storytelling video incorporating emotional information."

[0404] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0405] Step 1:

[0406] The user takes a photo

[0407] Users use devices such as smartphones and tablets to take pictures and videos using the device's built-in camera app, and the content they take is automatically saved in the device's gallery.

[0408] Input: Images and videos taken with the camera app

[0409] Output: Images and videos stored in the device's gallery

[0410] Step 2:

[0411] The user opens the app

[0412] A user launches the storytelling application, authenticates by logging in, selects the "Create a new album" option within the app, and sets a title, for example, "Birthday Party."

[0413] Enter: storytelling application

[0414] Output: Create a new album (e.g. "Birthday Party")

[0415] Step 3:

[0416] Users can select and upload images and videos

[0417] Users can select the images or videos they want to upload from their device's gallery and tap the "Upload" button in the app. The selected content will be added with metadata such as the date and time it was taken, location, and device information.

[0418] Input: User-selected images, videos, and metadata

[0419] Output: Images, videos and metadata ready to upload

[0420] Step 4:

[0421] The terminal divides and transmits the data

[0422] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[0423] Input: Prepared images, videos and metadata

[0424] Output: Split data chunks, file ID, checksum

[0425] Step 5:

[0426] The server receives and checks the data

[0427] The server receives the data chunks sent by the device and checks for errors using the checksum. If an error is found, the server requests the device to retransmit the particular chunk to ensure error-free data.

[0428] Input: Data chunk, file ID, checksum

[0429] Output: Data chunk with errors corrected

[0430] Step 6:

[0431] The server reconstructs and saves the data

[0432] The server reconstructs correctly received data chunks into the original image or video file and stores it on shared storage.

[0433] Input: Data chunk with errors corrected

[0434] Output: Original image and video files saved in cloud storage

[0435] Step 7:

[0436] Generative AI analyzes content

[0437] The server-based generative AI analyzes the metadata and content of images and videos stored in cloud storage, identifying specific people and scenes through facial recognition, background analysis, and facial expression recognition.

[0438] Input: Images, videos and metadata from cloud storage

[0439] Output: Image and video analysis results (e.g., people, scenes, emotional information)

[0440] Step 8:

[0441] Generative AI creates scenarios

[0442] The generative AI automatically generates a story-style video scenario based on the analysis results, and uses an emotion analysis engine to select effects and music appropriate for each scene.

[0443] Input: Image and video analysis results, emotional information

[0444] Output: Narrative video scenario

[0445] Step 9:

[0446] Generative AI generates videos

[0447] The AI ​​then follows the user's scenario, edits images and videos, adds transition effects and background music, and generates a video, which is then saved in cloud storage.

[0448] Input: Narrative video scenario

[0449] Output: Generated story-style video file

[0450] Step 10:

[0451] The server notifies the user

[0452] The server records the location of the generated video and notifies the user that a new video is ready, either via a push notification on the device or an in-app message.

[0453] Input: Generated video file

[0454] Output: User notification

[0455] Step 11:

[0456] User plays video

[0457] The user will receive a notification and open the storytelling application to play the generated video, allowing them to enjoy their special memories as a storytelling video.

[0458] Input: Notification from the server

[0459] Output: Video played on the user's device

[0460] (Application example 2)

[0461] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0462] Previous applications only allowed users to individually store photos and videos, limiting their ability to visually enjoy specific events or experiences. Furthermore, there was a lack of effective ways to record and replay customer experiences in physical stores. As a result, users were unable to effectively reminisce about their memories and it was difficult to share their real-time experiences with others. Furthermore, the lack of an automatic generation function for storytelling videos that reflected the emotions and scenarios of each event required users to manually edit the content.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0464] In this invention, the server includes means for taking photos and videos using a device operable by a user and uploading the photos and videos, means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, means for providing the automatically generated videos to the user's device and making them playable, means for analyzing photos and videos taken of the user's experience in a physical store and generating scenarios of purchasing behavior and interactions, and means for editing the videos of the experience in the physical store based on the generated scenarios. This allows users to enjoy a specific event or experience in the store as an automatically edited video along with emotional information.

[0465] "Terminal" means an electronic device that can be operated by a user, such as a smartphone, tablet, or other computing device.

[0466] "Means for taking photos or videos" means having the ability to take images or videos using the device's built-in camera or a corresponding software application.

[0467] "Uploading means" refers to software or hardware capable of transmitting photos and videos to a server via the Internet.

[0468] "Server" refers to a remote computer system for data processing and management.

[0469] "Cloud storage" refers to a system for storing data on remote servers accessible via the Internet.

[0470] "Generative AI methods" refers to systems that use artificial intelligence techniques to analyze photos and videos and categorize and organize data based on specific days or events.

[0471] "Means for automatically generating storytelling videos" refers to a function that uses generative AI means to edit photos and videos and automatically create videos based on a scenario.

[0472] "Means for providing and making playable" refers to the functionality for transmitting the generated storytelling video to a user's device and enabling the user to watch the video.

[0473] "Means for analyzing photos and videos of experiences" refers to a system that has the ability to analyze images and videos taken by users in physical stores to understand their purchasing behavior and interactions.

[0474] "Means for generating scenarios of purchasing behavior and interactions" refers to a system that has the function of creating scenarios of users' behavior and experiences in stores based on the results of analysis.

[0475] "Means for editing recorded video" refers to a system that has the function of editing photos and videos according to the generated scenario and creating final video content.

[0476] MODE FOR CARRYING OUT THE INVENTION

[0477] System program for implementing application examples

[0478] User actions and photo / video uploads

[0479] Users take photos and videos of their in-store experience using a device such as a smartphone or tablet. After taking photos, users open the dedicated application and proceed to the upload screen. They create a new album, title it "Shopping Experience," select photos and videos from the gallery, and tap the upload button.

[0480] Transferring data from the device to the server

[0481] The device adds metadata to the selected photos and videos, such as the date and time of the photo, location, and device information, then splits the large video file into smaller chunks, attaches a file ID and checksum to each chunk, and sends these chunks to the server sequentially.

[0482] Data reception and storage by the server

[0483] The server receives the data chunks, checks them for errors using checksums, requests that the chunks be resent if an error is found, and reconstructs the original photo or video file from the received data chunks and stores it in cloud storage.

[0484] Content analysis and video generation with generative AI and emotion engines

[0485] The server-based generative AI analyzes the metadata and content of stored photos and videos. It uses advanced technologies such as facial recognition, background analysis, and facial expression recognition to analyze emotional information. Based on this information, the generative AI creates a scenario and generates a storytelling video that reflects specific events and experiences. Transition effects and background music are also automatically added to create the final video.

[0486] User feedback and video playback

[0487] The generated video is saved to cloud storage, the server records the location, and notifies the user that a video has been generated. The user can then confirm the notification and play the video within the app, allowing them to reminisce about their experience in the physical store.

[0488] Natural language explanation of program processing

[0489] Hardware and software usage

[0490] Hardware: Smartphones, cloud servers

[0491] Software: Dedicated applications, cloud storage, generative AI models (e.g., TENSORFLOW®), sentiment analysis engines (e.g., IBM Watson®)

[0492] Program processing flow

[0493] 1. User-taken and uploaded photos: When users upload photos or videos they have taken, metadata is automatically added within the app.

[0494] 2. Data transmission and reception: Data is sent from the device to the server in small chunks, which are then received, verified, and stored in cloud storage.

[0495] 3. Content Analysis and Video Generation: Generative AI analyzes each photo or video and automatically generates a scenario based on emotional information, then edits the video accordingly, adding background music and transition effects.

[0496] 4. User notification and playback: Once the generated video has been saved, the user will be notified and will be able to play the video within the app.

[0497] Specific examples

[0498] Users visit retail stores on launch days and film themselves tasting and trying on products in-store. They then upload this content through the app. Generative AI and emotion engines automatically create a "launch day experience video" and notify the user. Users can then play this video within the app to relive their experience in the retail store.

[0499] Prompt Sentence Examples

[0500] "Please enter a prompt for the generated AI:

[0501] To capture the product launch experience in a video, edit the video to focus on where the user is, their facial expressions, and their actions. Select key moments of tasting, trying on, and purchasing, and add effects and music to capture the emotional changes.

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] Users capture their in-store experiences

[0505] Users use their smartphones or tablets to take photos and videos of their in-store experiences.

[0506] Input: Photos and videos taken with your smartphone camera.

[0507] Output: The captured photos and videos will be saved in the device's gallery.

[0508] Specific actions: Launch the smartphone camera application and capture important moments in a physical store (e.g., trying on clothes, purchasing new items).

[0509] Step 2:

[0510] User opens the app, selects photos and videos to upload

[0511] Users open the dedicated application, create an album, select the photos and videos they have taken, and tap the upload button.

[0512] Input: Photos and videos taken, album title set by user (e.g. "Shopping Experience").

[0513] Output: The file ready to be uploaded.

[0514] What it does: Launch the app, go to the upload screen, set a title, select a photo or video from your gallery, then tap the "Upload" button.

[0515] Step 3:

[0516] The device adds metadata to the data, splits it into small chunks, and sends it to the server

[0517] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and will split large video files into smaller chunks for transmission.

[0518] Input: The photo or video file to upload.

[0519] Output: Data split into small chunks with metadata attached.

[0520] What happens: The application runs a routine to add metadata, splits the data into small chunks, adds a file ID and checksum to each chunk, and sends them to the server one by one.

[0521] Step 4:

[0522] The server receives and verifies the data chunks and stores them in cloud storage.

[0523] The server receives the transmitted data chunks, checks them for errors using checksums, and if an error is found, requests that the chunk be retransmitted. The error-free data chunks are reconstructed and stored in cloud storage.

[0524] Input: Data chunks.

[0525] Output: Complete photo and video files.

[0526] What happens: The server receives each data chunk, checks the checksum for errors, reconstructs the data and saves it to cloud storage if there are no errors, and requests a retransmission if there are errors.

[0527] Step 5:

[0528] The server analyzes the content using generative AI and emotion engine

[0529] The server-based generative AI analyzes the metadata and content of stored photos and videos to perform facial recognition, background analysis, and facial expression recognition. The emotion engine analyzes emotional information.

[0530] Input: Photos and videos in cloud storage.

[0531] Output: Analysis results for each content (face, background, facial expression, emotional information).

[0532] How it works: It activates a generative AI model to analyze the metadata and content of photos and videos, and then uses an emotion engine to analyze faces and expressions to extract emotional information.

[0533] Step 6:

[0534] Generative AI creates scenarios and generates storytelling videos

[0535] Based on the analysis, the generative AI creates a scenario and generates a video that reflects a specific event or experience, automatically adding transition effects and background music.

[0536] Input: Parsed content and sentiment information.

[0537] Output: The finished storytelling video file.

[0538] What it does: The generative AI creates a scenario, the editor engine edits photos and videos, adds transition effects and background music, and finally generates a video file.

[0539] Step 7:

[0540] The server notifies the user that the video has been generated

[0541] The generated video is stored in cloud storage, and the server records the location of the video and notifies the user that a new video has been generated.

[0542] Input: The finished video file.

[0543] Output: Notification to the user.

[0544] What happens: The server launches the notification system and sends a notification to the user in the form of a push notification or in-app message.

[0545] Step 8:

[0546] User plays a video within the app

[0547] Users can view the notification and play the video generated within the app, allowing them to relive their experience in the physical store in real time.

[0548] Input: Notification from the server.

[0549] Output: The video being played.

[0550] What happens: The user opens the notification and interacts with the in-app video player to play the video.

[0551] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0552] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0553] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0554] [Second embodiment]

[0555] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0556] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0557] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0558] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0559] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0560] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0561] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0562] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0563] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0564] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0565] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0566] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0567] Hereinafter, embodiments of the present invention will be described in detail.

[0568] 1. User operations and photo / video uploads

[0569] User takes a photo with their smartphone:

[0570] Users use devices such as smartphones and tablets to take photos and videos of events such as their family's birthday parties.

[0571] User opens the app:

[0572] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set the title to "Birthday Party," for example.

[0573] User selects photos and videos:

[0574] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0575] 2. Transferring data from the device to the server

[0576] The device prepares the data:

[0577] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[0578] The device sends data:

[0579] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[0580] 3. Data reception and storage by the server

[0581] Server receives data:

[0582] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[0583] Server reconstructs data:

[0584] The photos and video files are reconstructed based on the received data and saved in a "Birthday Party" folder in cloud storage.

[0585] 4. Content analysis and video generation using generative AI

[0586] Generative AI analyzes content:

[0587] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of the event (e.g., cutting the cake or opening a gift).

[0588] Generative AI creates scenarios:

[0589] Based on the analysis results, it automatically creates a storytelling video scenario, placing key moments on a timeline and generating a visually engaging video.

[0590] Generative AI generates videos:

[0591] The generative AI then edits the video according to the scenario, adding transition effects and background music, and the finished video is saved as a new file in cloud storage.

[0592] 5. User Feedback and Video Playback

[0593] The server notifies the user:

[0594] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0595] User plays video:

[0596] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy their memories visually.

[0597] For example, users can take photos and videos of their child's birthday party and upload them through the app. The AI ​​then automatically analyzes the content and creates a storytelling video. Users can then play the video on their device and enjoy the memories of the party.

[0598] The processing flow will be explained below.

[0599] Step 1:

[0600] The user takes a photo or video

[0601] A user uses a smartphone camera app to take photos and videos of a family birthday party.

[0602] Step 2:

[0603] The user opens the app

[0604] The user opens the storytelling app, selects "Create a new album" from the home screen, and enters "Birthday Party" as the album title.

[0605] Step 3:

[0606] User selects photos and videos and starts uploading

[0607] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0608] Step 4:

[0609] The device prepares the data

[0610] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[0611] Step 5:

[0612] The terminal divides and transmits the data

[0613] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[0614] Step 6:

[0615] The server receives the data

[0616] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[0617] Step 7:

[0618] The server reconstructs the data

[0619] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[0620] Step 8:

[0621] The server calls the generated AI

[0622] The server passes the path information of the photos and videos in cloud storage to the generation AI.

[0623] Step 9:

[0624] Generative AI analyzes content

[0625] The generative AI analyzes the metadata and content of each photo or video and uses techniques such as facial recognition, background analysis, and facial expression recognition.

[0626] Step 10:

[0627] Generative AI creates scenarios

[0628] Based on the analysis, key moments are selected and a storytelling video scenario is created.

[0629] Step 11:

[0630] Generative AI generates videos

[0631] The generative AI edits photos and videos according to a scenario, adds transition effects and background music, and generates a video that is then saved in cloud storage.

[0632] Step 12:

[0633] The server notifies the user

[0634] The server records the location of the generated video and sends a push notification to the user's smartphone.

[0635] Step 13:

[0636] The user plays the video

[0637] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[0638] Example 1

[0639] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] There is a need for an efficient method to manage and analyze the growing amount of photo and video data and reconstruct them into memorable formats. However, many users have to spend a lot of time and effort manually organizing and editing each photo and video. Therefore, a system is needed that allows users to easily upload their captured content and automatically analyzes, classifies, and reconstructs it into a visually enjoyable format.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0642] In this invention, the server includes: means for taking photos and videos using a device operable by a user and uploading the photos and videos; means for adding metadata (such as the shooting date and time, location, and device information) to the uploaded photos and videos on the device side, storing them in a buffer, and then transmitting them to the server; means for receiving the transmitted data on the server side, verifying the sender's identification information, checking the data integrity, and then storing them in a cloud storage device; means for analyzing the photos and videos in the cloud storage device using a generative AI model, performing facial recognition, background analysis, facial expression recognition, etc., to identify important moments of the event; means for automatically generating storytelling-type videos based on the analysis results and adding transition effects and background music; and means for providing the automatically generated videos to the user's device and making them playable. This enables efficient management of photos and videos taken by users and automatically generating visually enjoyable videos.

[0643] "User" refers to an individual or organization that operates the system to take and upload photos and videos.

[0644] "Terminal" refers to a device that can be operated by a user, such as a smartphone, tablet, or PC.

[0645] "Photo" refers to a still image taken by a user on a device.

[0646] "Video" refers to a file format that contains moving images captured by a user on a device.

[0647] "Uploading" refers to the process of sending data such as photos or videos from a device to a server or cloud storage device.

[0648] "Metadata" refers to additional information associated with a photo or video, including the date and time it was taken, the location, and device information.

[0649] A "buffer" refers to a storage area that temporarily stores data.

[0650] "Server" means a computer that acts as a central processing unit and receives, processes, and stores uploaded data.

[0651] "Cloud storage" refers to a remote storage medium accessible over a network.

[0652] A "generative AI model" refers to an artificial intelligence algorithm that uses machine learning and deep learning technologies to analyze data and automatically generate and edit scenarios.

[0653] "Facial recognition" refers to the technology in which a generative AI model identifies the faces of people in photos and videos.

[0654] "Background analysis" refers to the technology in which a generative AI model analyzes the background or scenery in a photo or video.

[0655] "Facial expression recognition" refers to a technology in which a generative AI model distinguishes between facial expressions and analyzes emotions.

[0656] "Transition effects" refer to visual effects used when switching from one scene to another in video editing.

[0657] "Background music" refers to music or sound effects added to the background of a video.

[0658] "Storytelling video" refers to visual content that is edited around a specific scenario or narrative flow.

[0659] Hereinafter, embodiments of the present invention will be described in detail.

[0660] 1. User operations and photo / video uploads

[0661] A user uses a device such as a smartphone or tablet to take photos and videos of an event such as a family birthday party. For example, the user can take photos of a birthday cake or a video of a child opening a present. The user then opens a storytelling app. The app has a "Create a New Album" option, and the user creates a new album titled "Birthday Party." The user selects the photos and videos they want to upload from their device's gallery and taps the upload button to send the photos and videos.

[0662] 2. Transferring data from the device to the server

[0663] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos. Metadata adds important information to the uploaded content, making it useful for analysis and management. The device temporarily stores all data for upload in a buffer. The buffer ensures data consistency. The device then sends the photo and video data to the server. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer and makes retransmission easier if a communication error occurs.

[0664] 3. Data reception and storage by the server

[0665] The server receives the uploaded data and checks the session ID to identify the user who sent it. For example, the session ID can be used to identify who uploaded what data. The server uses a checksum to verify that the received data is correct, and requests a retransmission of any incomplete packets. This ensures the reliability of data transfer. The server reconstructs the photo and video files based on the received data. The reconstructed files are saved in the "Birthday Party" folder in the cloud storage.

[0666] 4. Content analysis and video generation using generative AI

[0667] The generation AI on the server analyzes the metadata and content of the stored photos and videos. Specifically, it uses technologies such as facial recognition, background analysis, and facial expression recognition to identify important moments (e.g., the moment a cake is cut or a gift is opened). Based on the analysis results, the generation AI automatically creates a storytelling video scenario. For example, the scenario may include scenes of cutting a cake or opening a gift on a timeline. The generation AI then edits the video according to the scenario, adding transition effects and background music to create a visually appealing video. The completed video is then saved in cloud storage.

[0668] 5. User Feedback and Video Playback

[0669] The server records the location where the generated video is saved and notifies the user that a new video has been generated. This is done via a push notification on the device or a message within the app. After checking the notification, the user opens the app and plays the generated video. This allows the user to visually enjoy the memories of the party.

[0670] Prompt Sentence Examples

[0671] Examples of prompts to input to a generative AI model include:

[0672] Users have uploaded photos and videos of their "birthday party." Use this content to generate a storytelling video that includes key moments like cutting the cake and opening gifts. Add transition effects and background music to create a visually appealing video.

[0673] The above is an embodiment of the invention.

[0674] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0675] Step 1:

[0676] Taking a photo and launching the app

[0677] A user takes photos or videos using a smartphone or tablet. For example, the user takes photos or videos of a birthday party, and then launches a storytelling app.

[0678] Input: Photos and videos taken by the user.

[0679] Output: Application launch.

[0680] Step 2:

[0681] Creating an album and selecting files

[0682] Users can select the "Create a new album" option within the app, set the title to "Birthday Party," then select the photos and videos they want to upload from their device's gallery and tap the upload button.

[0683] Input: User-specified new album name and selected photos and videos.

[0684] Output: Data ready for upload.

[0685] Step 3:

[0686] Adding metadata and saving it to the buffer

[0687] Your device will add metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos, giving them information about when and where they were taken. It then stores these files in a buffer and prepares them for transmission.

[0688] Input: Photo and video files selected by the user.

[0689] Output: Buffered photo and video files with metadata.

[0690] Step 4:

[0691] Splitting the data and sending it to the server

[0692] The device splits the large video file into chunks and sends the photos and split video data to the server, using an appropriate communication protocol to prevent data loss.

[0693] Input: Photo and video files with metadata stored in a buffer.

[0694] Output: The split data sent to the server.

[0695] Step 5:

[0696] Data reception and checksum verification by the server

[0697] The server validates the received data using checksums for integrity and requests retransmission of any incomplete packets, thus ensuring data consistency and quality.

[0698] Input: Split data sent from the terminal.

[0699] Output: Validated complete data.

[0700] Step 6:

[0701] Data reconstruction and cloud storage

[0702] The server reconstructs the received data to generate the original photo and video files, and then stores the reconstructed files in a "Birthday Party" folder in the cloud storage device.

[0703] Input: Validated data.

[0704] Output: Photo and video files saved in cloud storage.

[0705] Step 7:

[0706] Content analysis with generative AI

[0707] The server-based generative AI analyzes the metadata and content of photos and videos stored in cloud storage, using techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of an event.

[0708] Input: Photo and video files stored in cloud storage.

[0709] Output: Parsed content and identified key moments data.

[0710] Step 8:

[0711] Scenario creation and video generation

[0712] The generative AI creates a scenario based on the analysis results and automatically edits a storytelling video, adding transition effects and background music, and saving the generated video as a new file in cloud storage.

[0713] Input: Analysis results and data on key moments of identified events.

[0714] Output: Auto-generated storytelling videos.

[0715] Step 9:

[0716] User Notification

[0717] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0718] Input: An auto-generated video.

[0719] Output: Notification to the user.

[0720] Step 10:

[0721] Playing a video

[0722] The user confirms the notification, opens the app, and plays the generated video, allowing them to visually enjoy their birthday party memories.

[0723] Input: Notification from the server.

[0724] Output: The video played in the app.

[0725] In this way, each processing step works together and fulfills its role, allowing users to record and play back memories easily and quickly.

[0726] (Application example 1)

[0727] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0728] Previous technology lacked the means to promote products in stores or enhance the customer's purchasing experience. Furthermore, there was no system that allowed customers to take photos of products and effectively utilize those photos or videos. This made it difficult to effectively promote and introduce products to individual customers, limiting the ways to stimulate purchasing motivation.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0730] In this invention, the server includes a means for taking photos and videos using a user-operable device and uploading the photos and videos, a means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, a generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, a means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, a means for incorporating product introduction information and promotional information into the automatically generated videos, and a means for providing the automatically generated videos to the user's device and making them playable. This enables the automatic generation of effective promotional videos based on photos and videos of products and events taken by customers.

[0731] A "user-operable device" is an electronic device that can be operated by a user and used to take and upload photographs and videos.

[0732] "Means for uploading photos and videos" refers to a method or device for transmitting photos and video data taken by a user to a server.

[0733] "Means for receiving data on the server side and storing it in cloud storage" refers to a method or device for storing data received by the server in a cloud environment.

[0734] "Generative AI methods" refers to artificial intelligence technology that analyzes photos and videos uploaded and stored in cloud storage, categorizing and organizing them based on specific days or events.

[0735] A "means for automatically generating storytelling videos" refers to a method or device that automatically creates narrative videos using a sequence of photos and videos based on the analysis results of generative AI.

[0736] "Means for incorporating product introduction information and promotional information" refers to a method or device for adding product-related information and promotional content to the generated video.

[0737] "Means for providing the video to the user's terminal and enabling playback" refers to a method or device for delivering the generated video to the user's terminal and enabling playback on the terminal.

[0738] "Metadata" refers to additional information associated with a photo or video (e.g., date and time it was taken, location, device information, etc.).

[0739] "Facial recognition" is a technology that identifies people's faces in photos and videos and analyzes their features.

[0740] "Background analysis" refers to the technology of identifying background elements in photos and videos and analyzing their content.

[0741] "Facial expression recognition" is a technology that identifies facial expressions of people in photos and videos and analyzes their emotions and reactions.

[0742] "Creating a scenario" means designing a video with a narrative by determining the structure and order of the video based on the generated data.

[0743] MODE FOR CARRYING OUT THE INVENTION

[0744] Hereinafter, embodiments of the present invention will be described in detail.

[0745] 1. User operations and photo / video uploads

[0746] User takes a photo with their smartphone:

[0747] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in while in a physical store.

[0748] User opens the app:

[0749] Users open the app, access the photo and video upload screen, select the "Create a new promotional video" option, and set an appropriate title.

[0750] User selects photos and videos:

[0751] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0752] 2. Transferring data from the device to the server

[0753] The device prepares the data:

[0754] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[0755] The device sends data:

[0756] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[0757] 3. Data reception and storage by the server

[0758] Server receives data:

[0759] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[0760] Server reconstructs data:

[0761] The photo and video files are reconstructed based on the received data and saved in the appropriate folder in cloud storage.

[0762] 4. Content analysis and video generation using generative AI

[0763] Generative AI analyzes content:

[0764] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition to identify product features and promotional information.

[0765] Generative AI creates scenarios:

[0766] Based on the analysis results, the system automatically creates a storytelling video scenario, placing the key features of the product on a timeline and generating a visually appealing video.

[0767] Generative AI generates videos:

[0768] The AI ​​then edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The finished video is then saved as a new file in cloud storage.

[0769] 5. User Feedback and Video Playback

[0770] The server notifies the user:

[0771] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0772] User plays video:

[0773] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy product and promotional information visually.

[0774] Specific examples

[0775] Customers can take photos of products in the store through the app, and the photos are sent to a cloud server. The AI ​​analyzes the photos and automatically creates promotional videos for the products and stores. The created videos are then sent to customers via push notifications and can be played back within the app.

[0776] Example prompt sentence:

[0777] "Use a generative AI model to analyze this product photo and create a product video. The analysis should include the product name, price, and instructions for use."

[0778] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0779] Step 1:

[0780] User takes a photo with their smartphone:

[0781] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in in physical stores. At the same time, the date and time of the photo and location information are added to the images and videos as metadata. The input is photos and videos of the products, and the output is image and video files with the added metadata.

[0782] Step 2:

[0783] User opens the app:

[0784] The user opens the app and accesses the screen for uploading photos and videos they have taken. Here, they select the "Create a new promotional video" option and set an appropriate title. The input is the user's operation, and the output is the transition to the upload screen.

[0785] Step 3:

[0786] User selects photos and videos:

[0787] The user selects the photos or videos they want to upload from their device's gallery. Once selection is complete, they tap "Upload." The input is the selected media file, and the output is the start of the upload process.

[0788] Step 4:

[0789] The device prepares the data:

[0790] The device checks the metadata already attached to the selected photos and videos (such as the date and time of the photo, location, and device information) and stores these files in a buffer for transmission to the server. The input is the selected files, and the output is the data ready to be sent.

[0791] Step 5:

[0792] The device sends data:

[0793] The device sends photo and video data to the server in chunks. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer. The input is the data ready to be sent, and the output is the piece of data being sent.

[0794] Step 6:

[0795] Server receives data:

[0796] The server receives the uploaded data, checks the session ID to identify the sending user, verifies the received data is correct using checksums, and requests retransmission of any incomplete packets. The input is the piece of data being sent, and the output is the complete data file.

[0797] Step 7:

[0798] Server reconstructs data:

[0799] The server reconstructs the photo or video file from the received data fragments and saves it in the appropriate folder in the cloud storage. The input is the data fragments and the output is the reconstructed file.

[0800] Step 8:

[0801] Generative AI analyzes content:

[0802] The generative AI model in the server analyzes the metadata and content of stored photos and videos, identifying product features using techniques such as facial recognition, background analysis, and facial expression recognition. The input is the stored media files, and the output is the analyzed data.

[0803] Step 9:

[0804] Generative AI creates scenarios:

[0805] Based on the analysis results, the generative AI automatically creates a storytelling-style video scenario. It places the product's main features on a timeline and generates a visually appealing video. The input is analysis data, and the output is scenario data.

[0806] Step 10:

[0807] Generative AI generates videos:

[0808] The generative AI edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The completed video is saved as a new file in cloud storage. The input is the scenario data, and the output is the generated video file.

[0809] Step 11:

[0810] The server notifies the user:

[0811] The server records the location of the generated video and notifies the user that a new video has been generated. The notification is sent in the form of a push notification on the device or a message within the app. The input is the generated video information, and the output is a push notification.

[0812] Step 12:

[0813] User plays video:

[0814] The user confirms the notification and opens the app to play the generated video. This allows users to easily enjoy product and promotion information visually. The input is the push notification, and the output is the playing video.

[0815] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0816] Hereinafter, embodiments of the present invention will be described in detail.

[0817] 1. User operations and photo / video uploads

[0818] User takes a photo with their smartphone:

[0819] Users use devices such as smartphones and tablets to take photos and videos of events such as family birthday parties.

[0820] User opens the app:

[0821] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set a title, for example, "Birthday Party."

[0822] User selects photos and videos:

[0823] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0824] 2. Transferring data from the device to the server

[0825] The device prepares the data:

[0826] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and store the data in a temporary buffer.

[0827] The terminal splits and sends the data:

[0828] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[0829] 3. Data reception and storage by the server

[0830] Server receives data:

[0831] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[0832] Server reconstructs data:

[0833] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[0834] 4. Content analysis and video generation using generative AI and emotion engines

[0835] Generative AI analyzes content:

[0836] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition.

[0837] The emotion engine analyzes emotions:

[0838] The generative AI has a built-in emotion engine that analyzes the emotions (e.g., smiling, sad, surprised, etc.) of the people in each photo or video.

[0839] Generative AI creates scenarios:

[0840] Based on the analysis results, it automatically creates a storytelling video scenario, taking into account emotional information, selecting appropriate scenes, and arranging them along the timeline.

[0841] Generative AI generates videos:

[0842] The AI ​​then edits the photos and videos according to the scenario, adding transition effects and background music to create a video. The generated video is then saved as a new file in cloud storage.

[0843] 5. User Feedback and Video Playback

[0844] The server notifies the user:

[0845] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[0846] User plays video:

[0847] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[0848] For example, a user can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and generate a storytelling video. The generated video includes key moments, such as cutting the birthday cake and opening presents, edited with emotional information, allowing the user to play the video on their device and enjoy memories of the party.

[0849] The processing flow will be explained below.

[0850] Step 1:

[0851] The user takes a photo with their smartphone

[0852] Users use devices such as smartphones and tablets to take photos and videos of family birthday parties.

[0853] Step 2:

[0854] The user opens the app

[0855] The user opens the Storytelling app and selects "Create a new album" from the home screen. They enter the album title as "Birthday Party."

[0856] Step 3:

[0857] User selects photos and videos and starts uploading

[0858] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[0859] Step 4:

[0860] The device prepares the data

[0861] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[0862] Step 5:

[0863] The terminal divides and transmits the data

[0864] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[0865] Step 6:

[0866] The server receives the data

[0867] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[0868] Step 7:

[0869] The server reconstructs the data and stores it in cloud storage.

[0870] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[0871] Step 8:

[0872] The server calls the generative AI and emotion engine

[0873] The server passes the path information of the photos and videos in the cloud storage to the generation AI, which analyzes the metadata and content of the stored photos and videos, and the emotion engine analyzes the emotions (e.g., smile, sadness, surprise, etc.) of the people in each photo and video.

[0874] Step 9:

[0875] Generative AI and emotion engines analyze content

[0876] The generative AI analyzes photos and videos using technologies such as facial recognition, contextual analysis, and facial expression recognition. The emotion engine identifies changes in emotions and provides that information to the generative AI.

[0877] Step 10:

[0878] Generative AI creates scenarios

[0879] The generative AI uses information from the emotion engine to automatically create a storytelling video scenario, selecting key moments (e.g., cutting a birthday cake or opening a gift) and placing them on a timeline.

[0880] Step 11:

[0881] Generative AI generates videos

[0882] The AI ​​then edits the video based on the scenario, adding transition effects and background music to match the emotional changes, and the resulting video is saved as a new file in cloud storage.

[0883] Step 12:

[0884] The server notifies the user

[0885] The server records the location of the generated video and sends a push notification to the user's smartphone.

[0886] Step 13:

[0887] The user plays the video

[0888] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[0889] Example 2

[0890] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0891] In recent years, the amount of photos and videos taken with digital devices has rapidly increased, creating a need for efficient classification and management of these images and videos, and for users to easily access them. Furthermore, to effectively preserve and share memories of special family events, simply storing photos and videos is insufficient; they must be reconstructed as storytelling content. Furthermore, there is a need for a method to automatically generate high-quality content with minimal user interaction.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for taking images and videos using a terminal operable by a user and uploading the images and videos, means for receiving the uploaded images and videos on the server side and saving them in shared storage, generation AI means for analyzing the images and videos in the shared storage and classifying and organizing them based on specific dates or events, means for automatically generating story-style videos based on the images and videos organized by the generation AI means, and means for providing the automatically generated videos to the user's terminal and making them playable. This allows users to efficiently manage their captured content and enjoy their special memories by automatically converting them into high-quality storytelling-style videos.

[0893] A "terminal" is a device that can be operated by a user, and includes, for example, a smartphone, a tablet, or a PC.

[0894] "Images and Videos" means still and moving images stored in digital format that are used to record events and moments in everyday life.

[0895] "Upload" refers to the operation or process of sending data from a device to a server.

[0896] A "server" is a computer system that receives, stores, analyzes, and generates data and communicates with terminals via a network.

[0897] "Shared storage" refers to a storage device connected to a server, used to centrally manage and store data from multiple users.

[0898] "Analysis" refers to the process of performing specific processing on digital content to extract and classify its content and characteristics.

[0899] "Generative AI" refers to a system that uses artificial intelligence techniques to process and analyze data and automatically create specific deliverables.

[0900] "Narrative video" refers to a video that is not simply a series of images, but is edited and reconstructed based on a specific theme or scenario.

[0901] "Delivery" refers to the process of making the generated deliverables available to users.

[0902] "Play" refers to the action initiated to watch or listen to a stored video or audio file.

[0903] "Metadata" refers to auxiliary information added to images and videos (e.g., shooting date and time, location, device information, etc.), and is used when organizing and analyzing data.

[0904] This invention is a system that uploads images and videos from users' devices to a server, analyzes and organizes them, and automatically generates story-style videos. This system allows users to efficiently manage their captured content and enjoy their special memories as high-quality storytelling videos.

[0905] Hardware and Software Configuration

[0906] Terminal

[0907] Users take pictures and videos using devices such as smartphones and tablets. These devices have a camera function and a dedicated storytelling application installed. The application provides an operation screen and supports the selection and uploading of images and videos.

[0908] server

[0909] The server is a cloud-connected computer system that receives and stores content uploaded by users. It is equipped with a generative AI and emotion analysis engine, which are used to analyze and organize images and videos, generate scenarios, and generate videos.

[0910] Generative AI and Sentiment Analysis Engine

[0911] The generative AI includes advanced image processing technology, enabling facial recognition, background analysis, and facial expression recognition. The emotion analysis engine analyzes the emotions of people in each image or video, and generates a scenario based on that information. Based on the generated scenario, appropriate effects and music are added to create a narrative video.

[0912] Data processing and calculation

[0913] Adding Metadata

[0914] The device automatically adds metadata to the images and videos it takes, including the date and time the image was taken, the location, and device information. When a user uploads an image or video, this metadata is also sent to the server.

[0915] Data division and transmission

[0916] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[0917] Receiving and Reconstructing Data

[0918] The server receives each chunk and checks it for errors using a checksum. If an error is found, the server requests that the chunk be retransmitted. Correctly received data chunks are reconstructed into the original file and saved to shared storage.

[0919] Analyzing and organizing content

[0920] Generative AI analyzes the metadata and content of images and videos stored on shared storage. Based on the analysis results, content is categorized and organized based on specific dates and events. Specific people and scenes are identified through facial recognition, context analysis, and facial expression recognition.

[0921] Scenario creation and video generation

[0922] The generative AI automatically generates a story-style video scenario based on the analysis results. The emotion analysis engine reflects the emotional information in each scene and selects and adds effects and music in line with the scenario. Finally, the automatically generated video is saved in cloud storage that users can access.

[0923] Specific examples

[0924] For example, users can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and create a storytelling video. The generated video includes key moments, such as cutting the birthday cake or opening presents, edited with emotional information, allowing users to play the video on their device and enjoy their memories of the party.

[0925] Example prompt sentence:

[0926] "Use photos and videos from a child's birthday party to generate a storytelling video incorporating emotional information."

[0927] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0928] Step 1:

[0929] The user takes a photo

[0930] Users use devices such as smartphones and tablets to take pictures and videos using the device's built-in camera app, and the content they take is automatically saved in the device's gallery.

[0931] Input: Images and videos taken with the camera app

[0932] Output: Images and videos stored in the device's gallery

[0933] Step 2:

[0934] The user opens the app

[0935] A user launches the storytelling application, authenticates by logging in, selects the "Create a new album" option within the app, and sets a title, for example, "Birthday Party."

[0936] Enter: storytelling application

[0937] Output: Create a new album (e.g. "Birthday Party")

[0938] Step 3:

[0939] Users can select and upload images and videos

[0940] Users can select the images or videos they want to upload from their device's gallery and tap the "Upload" button in the app. The selected content will be added with metadata such as the date and time it was taken, location, and device information.

[0941] Input: User-selected images, videos, and metadata

[0942] Output: Images, videos and metadata ready to upload

[0943] Step 4:

[0944] The terminal divides and transmits the data

[0945] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[0946] Input: Prepared images, videos and metadata

[0947] Output: Split data chunks, file ID, checksum

[0948] Step 5:

[0949] The server receives and checks the data

[0950] The server receives the data chunks sent by the device and checks for errors using the checksum. If an error is found, the server requests the device to retransmit the particular chunk to ensure error-free data.

[0951] Input: Data chunk, file ID, checksum

[0952] Output: Data chunk with errors corrected

[0953] Step 6:

[0954] The server reconstructs and saves the data

[0955] The server reconstructs correctly received data chunks into the original image or video file and stores it on shared storage.

[0956] Input: Data chunk with errors corrected

[0957] Output: Original image and video files saved in cloud storage

[0958] Step 7:

[0959] Generative AI analyzes content

[0960] The server-based generative AI analyzes the metadata and content of images and videos stored in cloud storage, identifying specific people and scenes through facial recognition, background analysis, and facial expression recognition.

[0961] Input: Images, videos and metadata from cloud storage

[0962] Output: Image and video analysis results (e.g., people, scenes, emotional information)

[0963] Step 8:

[0964] Generative AI creates scenarios

[0965] The generative AI automatically generates a story-style video scenario based on the analysis results, and uses an emotion analysis engine to select effects and music appropriate for each scene.

[0966] Input: Image and video analysis results, emotional information

[0967] Output: Narrative video scenario

[0968] Step 9:

[0969] Generative AI generates videos

[0970] The AI ​​then follows the user's scenario, edits images and videos, adds transition effects and background music, and generates a video, which is then saved in cloud storage.

[0971] Input: Narrative video scenario

[0972] Output: Generated story-style video file

[0973] Step 10:

[0974] The server notifies the user

[0975] The server records the location of the generated video and notifies the user that a new video is ready, either via a push notification on the device or an in-app message.

[0976] Input: Generated video file

[0977] Output: User notification

[0978] Step 11:

[0979] User plays video

[0980] The user will receive a notification and open the storytelling application to play the generated video, allowing them to enjoy their special memories as a storytelling video.

[0981] Input: Notification from the server

[0982] Output: Video played on the user's device

[0983] (Application example 2)

[0984] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0985] Previous applications only allowed users to individually store photos and videos, limiting their ability to visually enjoy specific events or experiences. Furthermore, there was a lack of effective ways to record and replay customer experiences in physical stores. As a result, users were unable to effectively reminisce about their memories and it was difficult to share their real-time experiences with others. Furthermore, the lack of an automatic generation function for storytelling videos that reflected the emotions and scenarios of each event required users to manually edit the content.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0987] In this invention, the server includes means for taking photos and videos using a device operable by a user and uploading the photos and videos, means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, means for providing the automatically generated videos to the user's device and making them playable, means for analyzing photos and videos taken of the user's experience in a physical store and generating scenarios of purchasing behavior and interactions, and means for editing the videos of the experience in the physical store based on the generated scenarios. This allows users to enjoy a specific event or experience in the store as an automatically edited video along with emotional information.

[0988] "Terminal" means an electronic device that can be operated by a user, such as a smartphone, tablet, or other computing device.

[0989] "Means for taking photos or videos" means having the ability to take images or videos using the device's built-in camera or a corresponding software application.

[0990] "Uploading means" refers to software or hardware capable of transmitting photos and videos to a server via the Internet.

[0991] "Server" refers to a remote computer system for data processing and management.

[0992] "Cloud storage" refers to a system for storing data on remote servers accessible via the Internet.

[0993] "Generative AI methods" refers to systems that use artificial intelligence techniques to analyze photos and videos and categorize and organize data based on specific days or events.

[0994] "Means for automatically generating storytelling videos" refers to a function that uses generative AI means to edit photos and videos and automatically create videos based on a scenario.

[0995] "Means for providing and making playable" refers to the functionality for transmitting the generated storytelling video to a user's device and enabling the user to watch the video.

[0996] "Means for analyzing photos and videos of experiences" refers to a system that has the ability to analyze images and videos taken by users in physical stores to understand their purchasing behavior and interactions.

[0997] "Means for generating scenarios of purchasing behavior and interactions" refers to a system that has the function of creating scenarios of users' behavior and experiences in stores based on the results of analysis.

[0998] "Means for editing recorded video" refers to a system that has the function of editing photos and videos according to the generated scenario and creating final video content.

[0999] MODE FOR CARRYING OUT THE INVENTION

[1000] System program for implementing application examples

[1001] User actions and photo / video uploads

[1002] Users take photos and videos of their in-store experience using a device such as a smartphone or tablet. After taking photos, users open the dedicated application and proceed to the upload screen. They create a new album, title it "Shopping Experience," select photos and videos from the gallery, and tap the upload button.

[1003] Transferring data from the device to the server

[1004] The device adds metadata to the selected photos and videos, such as the date and time of the photo, location, and device information, then splits the large video file into smaller chunks, attaches a file ID and checksum to each chunk, and sends these chunks to the server sequentially.

[1005] Data reception and storage by the server

[1006] The server receives the data chunks, checks them for errors using checksums, requests that the chunks be resent if an error is found, and reconstructs the original photo or video file from the received data chunks and stores it in cloud storage.

[1007] Content analysis and video generation with generative AI and emotion engines

[1008] The server-based generative AI analyzes the metadata and content of stored photos and videos. It uses advanced technologies such as facial recognition, background analysis, and facial expression recognition to analyze emotional information. Based on this information, the generative AI creates a scenario and generates a storytelling video that reflects specific events and experiences. Transition effects and background music are also automatically added to create the final video.

[1009] User feedback and video playback

[1010] The generated video is saved to cloud storage, the server records the location, and notifies the user that a video has been generated. The user can then confirm the notification and play the video within the app, allowing them to reminisce about their experience in the physical store.

[1011] Natural language explanation of program processing

[1012] Hardware and software usage

[1013] Hardware: Smartphones, cloud servers

[1014] Software: Dedicated applications, cloud storage, generative AI models (e.g., TensorFlow), sentiment analysis engines (e.g., IBM Watson)

[1015] Program processing flow

[1016] 1. User-taken and uploaded photos: When users upload photos or videos they have taken, metadata is automatically added within the app.

[1017] 2. Data transmission and reception: Data is sent from the device to the server in small chunks, which are then received, verified, and stored in cloud storage.

[1018] 3. Content Analysis and Video Generation: Generative AI analyzes each photo or video and automatically generates a scenario based on emotional information, then edits the video accordingly, adding background music and transition effects.

[1019] 4. User notification and playback: Once the generated video has been saved, the user will be notified and will be able to play the video within the app.

[1020] Specific examples

[1021] Users visit retail stores on launch days and film themselves tasting and trying on products in-store. They then upload this content through the app. Generative AI and emotion engines automatically create a "launch day experience video" and notify the user. Users can then play this video within the app to relive their experience in the retail store.

[1022] Prompt Sentence Examples

[1023] "Please enter a prompt for the generated AI:

[1024] To capture the product launch experience in a video, edit the video to focus on where the user is, their facial expressions, and their actions. Select key moments of tasting, trying on, and purchasing, and add effects and music to capture the emotional changes.

[1025] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1026] Step 1:

[1027] Users capture their in-store experiences

[1028] Users use their smartphones or tablets to take photos and videos of their in-store experiences.

[1029] Input: Photos and videos taken with your smartphone camera.

[1030] Output: The captured photos and videos will be saved in the device's gallery.

[1031] Specific actions: Launch the smartphone camera application and capture important moments in a physical store (e.g., trying on clothes, purchasing new items).

[1032] Step 2:

[1033] User opens the app, selects photos and videos to upload

[1034] Users open the dedicated application, create an album, select the photos and videos they have taken, and tap the upload button.

[1035] Input: Photos and videos taken, album title set by user (e.g. "Shopping Experience").

[1036] Output: The file ready to be uploaded.

[1037] What it does: Launch the app, go to the upload screen, set a title, select a photo or video from your gallery, then tap the "Upload" button.

[1038] Step 3:

[1039] The device adds metadata to the data, splits it into small chunks, and sends it to the server

[1040] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and will split large video files into smaller chunks for transmission.

[1041] Input: The photo or video file to upload.

[1042] Output: Data split into small chunks with metadata attached.

[1043] What happens: The application runs a routine to add metadata, splits the data into small chunks, adds a file ID and checksum to each chunk, and sends them to the server one by one.

[1044] Step 4:

[1045] The server receives and verifies the data chunks and stores them in cloud storage.

[1046] The server receives the transmitted data chunks, checks them for errors using checksums, and if an error is found, requests that the chunk be retransmitted. The error-free data chunks are reconstructed and stored in cloud storage.

[1047] Input: Data chunks.

[1048] Output: Complete photo and video files.

[1049] What happens: The server receives each data chunk, checks the checksum for errors, reconstructs the data and saves it to cloud storage if there are no errors, and requests a retransmission if there are errors.

[1050] Step 5:

[1051] The server analyzes the content using generative AI and emotion engine

[1052] The server-based generative AI analyzes the metadata and content of stored photos and videos to perform facial recognition, background analysis, and facial expression recognition. The emotion engine analyzes emotional information.

[1053] Input: Photos and videos in cloud storage.

[1054] Output: Analysis results for each content (face, background, facial expression, emotional information).

[1055] How it works: It activates a generative AI model to analyze the metadata and content of photos and videos, and then uses an emotion engine to analyze faces and expressions to extract emotional information.

[1056] Step 6:

[1057] Generative AI creates scenarios and generates storytelling videos

[1058] Based on the analysis, the generative AI creates a scenario and generates a video that reflects a specific event or experience, automatically adding transition effects and background music.

[1059] Input: Parsed content and sentiment information.

[1060] Output: The finished storytelling video file.

[1061] What it does: The generative AI creates a scenario, the editor engine edits photos and videos, adds transition effects and background music, and finally generates a video file.

[1062] Step 7:

[1063] The server notifies the user that the video has been generated

[1064] The generated video is stored in cloud storage, and the server records the location of the video and notifies the user that a new video has been generated.

[1065] Input: The finished video file.

[1066] Output: Notification to the user.

[1067] What happens: The server launches the notification system and sends a notification to the user in the form of a push notification or in-app message.

[1068] Step 8:

[1069] User plays a video within the app

[1070] Users can view the notification and play the video generated within the app, allowing them to relive their experience in the physical store in real time.

[1071] Input: Notification from the server.

[1072] Output: The video being played.

[1073] What happens: The user opens the notification and interacts with the in-app video player to play the video.

[1074] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1075] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1076] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1077] [Third embodiment]

[1078] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1079] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1080] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1081] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1082] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1083] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1084] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1085] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1086] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1087] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1088] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1089] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1090] Hereinafter, embodiments of the present invention will be described in detail.

[1091] 1. User operations and photo / video uploads

[1092] User takes a photo with their smartphone:

[1093] Users use devices such as smartphones and tablets to take photos and videos of events such as their family's birthday parties.

[1094] User opens the app:

[1095] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set the title to "Birthday Party," for example.

[1096] User selects photos and videos:

[1097] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1098] 2. Transferring data from the device to the server

[1099] The device prepares the data:

[1100] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[1101] The device sends data:

[1102] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[1103] 3. Data reception and storage by the server

[1104] Server receives data:

[1105] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[1106] Server reconstructs data:

[1107] The photos and video files are reconstructed based on the received data and saved in a "Birthday Party" folder in cloud storage.

[1108] 4. Content analysis and video generation using generative AI

[1109] Generative AI analyzes content:

[1110] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of the event (e.g., cutting the cake or opening a gift).

[1111] Generative AI creates scenarios:

[1112] Based on the analysis results, it automatically creates a storytelling video scenario, placing key moments on a timeline and generating a visually engaging video.

[1113] Generative AI generates videos:

[1114] The generative AI then edits the video according to the scenario, adding transition effects and background music, and the finished video is saved as a new file in cloud storage.

[1115] 5. User Feedback and Video Playback

[1116] The server notifies the user:

[1117] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1118] User plays video:

[1119] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy their memories visually.

[1120] For example, users can take photos and videos of their child's birthday party and upload them through the app. The AI ​​then automatically analyzes the content and creates a storytelling video. Users can then play the video on their device and enjoy the memories of the party.

[1121] The processing flow will be explained below.

[1122] Step 1:

[1123] The user takes a photo or video

[1124] A user uses a smartphone camera app to take photos and videos of a family birthday party.

[1125] Step 2:

[1126] The user opens the app

[1127] The user opens the storytelling app, selects "Create a new album" from the home screen, and enters "Birthday Party" as the album title.

[1128] Step 3:

[1129] User selects photos and videos and starts uploading

[1130] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1131] Step 4:

[1132] The device prepares the data

[1133] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[1134] Step 5:

[1135] The terminal divides and transmits the data

[1136] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[1137] Step 6:

[1138] The server receives the data

[1139] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[1140] Step 7:

[1141] The server reconstructs the data

[1142] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[1143] Step 8:

[1144] The server calls the generated AI

[1145] The server passes the path information of the photos and videos in cloud storage to the generation AI.

[1146] Step 9:

[1147] Generative AI analyzes content

[1148] The generative AI analyzes the metadata and content of each photo or video and uses techniques such as facial recognition, background analysis, and facial expression recognition.

[1149] Step 10:

[1150] Generative AI creates scenarios

[1151] Based on the analysis, key moments are selected and a storytelling video scenario is created.

[1152] Step 11:

[1153] Generative AI generates videos

[1154] The generative AI edits photos and videos according to a scenario, adds transition effects and background music, and generates a video that is then saved in cloud storage.

[1155] Step 12:

[1156] The server notifies the user

[1157] The server records the location of the generated video and sends a push notification to the user's smartphone.

[1158] Step 13:

[1159] The user plays the video

[1160] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[1161] Example 1

[1162] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1163] There is a need for an efficient method to manage and analyze the growing amount of photo and video data and reconstruct them into memorable formats. However, many users have to spend a lot of time and effort manually organizing and editing each photo and video. Therefore, a system is needed that allows users to easily upload their captured content and automatically analyzes, classifies, and reconstructs it into a visually enjoyable format.

[1164] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1165] In this invention, the server includes: means for taking photos and videos using a device operable by a user and uploading the photos and videos; means for adding metadata (such as the shooting date and time, location, and device information) to the uploaded photos and videos on the device side, storing them in a buffer, and then transmitting them to the server; means for receiving the transmitted data on the server side, verifying the sender's identification information, checking the data integrity, and then storing them in a cloud storage device; means for analyzing the photos and videos in the cloud storage device using a generative AI model, performing facial recognition, background analysis, facial expression recognition, etc., to identify important moments of the event; means for automatically generating storytelling-type videos based on the analysis results and adding transition effects and background music; and means for providing the automatically generated videos to the user's device and making them playable. This enables efficient management of photos and videos taken by users and automatically generating visually enjoyable videos.

[1166] "User" refers to an individual or organization that operates the system to take and upload photos and videos.

[1167] "Terminal" refers to a device that can be operated by a user, such as a smartphone, tablet, or PC.

[1168] "Photo" refers to a still image taken by a user on a device.

[1169] "Video" refers to a file format that contains moving images captured by a user on a device.

[1170] "Uploading" refers to the process of sending data such as photos or videos from a device to a server or cloud storage device.

[1171] "Metadata" refers to additional information associated with a photo or video, including the date and time it was taken, the location, and device information.

[1172] A "buffer" refers to a storage area that temporarily stores data.

[1173] "Server" means a computer that acts as a central processing unit and receives, processes, and stores uploaded data.

[1174] "Cloud storage" refers to a remote storage medium accessible over a network.

[1175] A "generative AI model" refers to an artificial intelligence algorithm that uses machine learning and deep learning technologies to analyze data and automatically generate and edit scenarios.

[1176] "Facial recognition" refers to the technology in which a generative AI model identifies the faces of people in photos and videos.

[1177] "Background analysis" refers to the technology in which a generative AI model analyzes the background or scenery in a photo or video.

[1178] "Facial expression recognition" refers to a technology in which a generative AI model distinguishes between facial expressions and analyzes emotions.

[1179] "Transition effects" refer to visual effects used when switching from one scene to another in video editing.

[1180] "Background music" refers to music or sound effects added to the background of a video.

[1181] "Storytelling video" refers to visual content that is edited around a specific scenario or narrative flow.

[1182] Hereinafter, embodiments of the present invention will be described in detail.

[1183] 1. User operations and photo / video uploads

[1184] A user uses a device such as a smartphone or tablet to take photos and videos of an event such as a family birthday party. For example, the user can take photos of a birthday cake or a video of a child opening a present. The user then opens a storytelling app. The app has a "Create a New Album" option, and the user creates a new album titled "Birthday Party." The user selects the photos and videos they want to upload from their device's gallery and taps the upload button to send the photos and videos.

[1185] 2. Transferring data from the device to the server

[1186] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos. Metadata adds important information to the uploaded content, making it useful for analysis and management. The device temporarily stores all data for upload in a buffer. The buffer ensures data consistency. The device then sends the photo and video data to the server. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer and makes retransmission easier if a communication error occurs.

[1187] 3. Data reception and storage by the server

[1188] The server receives the uploaded data and checks the session ID to identify the user who sent it. For example, the session ID can be used to identify who uploaded what data. The server uses a checksum to verify that the received data is correct, and requests a retransmission of any incomplete packets. This ensures the reliability of data transfer. The server reconstructs the photo and video files based on the received data. The reconstructed files are saved in the "Birthday Party" folder in the cloud storage.

[1189] 4. Content analysis and video generation using generative AI

[1190] The generation AI on the server analyzes the metadata and content of the stored photos and videos. Specifically, it uses technologies such as facial recognition, background analysis, and facial expression recognition to identify important moments (e.g., the moment a cake is cut or a gift is opened). Based on the analysis results, the generation AI automatically creates a storytelling video scenario. For example, the scenario may include scenes of cutting a cake or opening a gift on a timeline. The generation AI then edits the video according to the scenario, adding transition effects and background music to create a visually appealing video. The completed video is then saved in cloud storage.

[1191] 5. User Feedback and Video Playback

[1192] The server records the location where the generated video is saved and notifies the user that a new video has been generated. This is done via a push notification on the device or a message within the app. After checking the notification, the user opens the app and plays the generated video. This allows the user to visually enjoy the memories of the party.

[1193] Prompt Sentence Examples

[1194] Examples of prompts to input to a generative AI model include:

[1195] Users have uploaded photos and videos of their "birthday party." Use this content to generate a storytelling video that includes key moments like cutting the cake and opening gifts. Add transition effects and background music to create a visually appealing video.

[1196] The above is an embodiment of the invention.

[1197] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1198] Step 1:

[1199] Taking a photo and launching the app

[1200] A user takes photos or videos using a smartphone or tablet. For example, the user takes photos or videos of a birthday party, and then launches a storytelling app.

[1201] Input: Photos and videos taken by the user.

[1202] Output: Application launch.

[1203] Step 2:

[1204] Creating an album and selecting files

[1205] Users can select the "Create a new album" option within the app, set the title to "Birthday Party," then select the photos and videos they want to upload from their device's gallery and tap the upload button.

[1206] Input: User-specified new album name and selected photos and videos.

[1207] Output: Data ready for upload.

[1208] Step 3:

[1209] Adding metadata and saving it to the buffer

[1210] Your device will add metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos, giving them information about when and where they were taken. It then stores these files in a buffer and prepares them for transmission.

[1211] Input: Photo and video files selected by the user.

[1212] Output: Buffered photo and video files with metadata.

[1213] Step 4:

[1214] Splitting the data and sending it to the server

[1215] The device splits the large video file into chunks and sends the photos and split video data to the server, using an appropriate communication protocol to prevent data loss.

[1216] Input: Photo and video files with metadata stored in a buffer.

[1217] Output: The split data sent to the server.

[1218] Step 5:

[1219] Data reception and checksum verification by the server

[1220] The server validates the received data using checksums for integrity and requests retransmission of any incomplete packets, thus ensuring data consistency and quality.

[1221] Input: Split data sent from the terminal.

[1222] Output: Validated complete data.

[1223] Step 6:

[1224] Data reconstruction and cloud storage

[1225] The server reconstructs the received data to generate the original photo and video files, and then stores the reconstructed files in a "Birthday Party" folder in the cloud storage device.

[1226] Input: Validated data.

[1227] Output: Photo and video files saved in cloud storage.

[1228] Step 7:

[1229] Content analysis with generative AI

[1230] The server-based generative AI analyzes the metadata and content of photos and videos stored in cloud storage, using techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of an event.

[1231] Input: Photo and video files stored in cloud storage.

[1232] Output: Parsed content and identified key moments data.

[1233] Step 8:

[1234] Scenario creation and video generation

[1235] The generative AI creates a scenario based on the analysis results and automatically edits a storytelling video, adding transition effects and background music, and saving the generated video as a new file in cloud storage.

[1236] Input: Analysis results and data on key moments of identified events.

[1237] Output: Auto-generated storytelling videos.

[1238] Step 9:

[1239] User Notification

[1240] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1241] Input: An auto-generated video.

[1242] Output: Notification to the user.

[1243] Step 10:

[1244] Playing a video

[1245] The user confirms the notification, opens the app, and plays the generated video, allowing them to visually enjoy their birthday party memories.

[1246] Input: Notification from the server.

[1247] Output: The video played in the app.

[1248] In this way, each processing step works together and fulfills its role, allowing users to record and play back memories easily and quickly.

[1249] (Application example 1)

[1250] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1251] Previous technology lacked the means to promote products in stores or enhance the customer's purchasing experience. Furthermore, there was no system that allowed customers to take photos of products and effectively utilize those photos or videos. This made it difficult to effectively promote and introduce products to individual customers, limiting the ways to stimulate purchasing motivation.

[1252] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1253] In this invention, the server includes a means for taking photos and videos using a user-operable device and uploading the photos and videos, a means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, a generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, a means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, a means for incorporating product introduction information and promotional information into the automatically generated videos, and a means for providing the automatically generated videos to the user's device and making them playable. This enables the automatic generation of effective promotional videos based on photos and videos of products and events taken by customers.

[1254] A "user-operable device" is an electronic device that can be operated by a user and used to take and upload photographs and videos.

[1255] "Means for uploading photos and videos" refers to a method or device for transmitting photos and video data taken by a user to a server.

[1256] "Means for receiving data on the server side and storing it in cloud storage" refers to a method or device for storing data received by the server in a cloud environment.

[1257] "Generative AI methods" refers to artificial intelligence technology that analyzes photos and videos uploaded and stored in cloud storage, categorizing and organizing them based on specific days or events.

[1258] A "means for automatically generating storytelling videos" refers to a method or device that automatically creates narrative videos using a sequence of photos and videos based on the analysis results of generative AI.

[1259] "Means for incorporating product introduction information and promotional information" refers to a method or device for adding product-related information and promotional content to the generated video.

[1260] "Means for providing the video to the user's terminal and enabling playback" refers to a method or device for delivering the generated video to the user's terminal and enabling playback on the terminal.

[1261] "Metadata" refers to additional information associated with a photo or video (e.g., date and time it was taken, location, device information, etc.).

[1262] "Facial recognition" is a technology that identifies people's faces in photos and videos and analyzes their features.

[1263] "Background analysis" refers to the technology of identifying background elements in photos and videos and analyzing their content.

[1264] "Facial expression recognition" is a technology that identifies facial expressions of people in photos and videos and analyzes their emotions and reactions.

[1265] "Creating a scenario" means designing a video with a narrative by determining the structure and order of the video based on the generated data.

[1266] MODE FOR CARRYING OUT THE INVENTION

[1267] Hereinafter, embodiments of the present invention will be described in detail.

[1268] 1. User operations and photo / video uploads

[1269] User takes a photo with their smartphone:

[1270] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in while in a physical store.

[1271] User opens the app:

[1272] Users open the app, access the photo and video upload screen, select the "Create a new promotional video" option, and set an appropriate title.

[1273] User selects photos and videos:

[1274] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1275] 2. Transferring data from the device to the server

[1276] The device prepares the data:

[1277] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[1278] The device sends data:

[1279] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[1280] 3. Data reception and storage by the server

[1281] Server receives data:

[1282] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[1283] Server reconstructs data:

[1284] The photo and video files are reconstructed based on the received data and saved in the appropriate folder in cloud storage.

[1285] 4. Content analysis and video generation using generative AI

[1286] Generative AI analyzes content:

[1287] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition to identify product features and promotional information.

[1288] Generative AI creates scenarios:

[1289] Based on the analysis results, the system automatically creates a storytelling video scenario, placing the key features of the product on a timeline and generating a visually appealing video.

[1290] Generative AI generates videos:

[1291] The AI ​​then edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The finished video is then saved as a new file in cloud storage.

[1292] 5. User Feedback and Video Playback

[1293] The server notifies the user:

[1294] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1295] User plays video:

[1296] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy product and promotional information visually.

[1297] Specific examples

[1298] Customers can take photos of products in the store through the app, and the photos are sent to a cloud server. The AI ​​analyzes the photos and automatically creates promotional videos for the products and stores. The created videos are then sent to customers via push notifications and can be played back within the app.

[1299] Example prompt sentence:

[1300] "Use a generative AI model to analyze this product photo and create a product video. The analysis should include the product name, price, and instructions for use."

[1301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1302] Step 1:

[1303] User takes a photo with their smartphone:

[1304] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in in physical stores. At the same time, the date and time of the photo and location information are added to the images and videos as metadata. The input is photos and videos of the products, and the output is image and video files with the added metadata.

[1305] Step 2:

[1306] User opens the app:

[1307] The user opens the app and accesses the screen for uploading photos and videos they have taken. Here, they select the "Create a new promotional video" option and set an appropriate title. The input is the user's operation, and the output is the transition to the upload screen.

[1308] Step 3:

[1309] User selects photos and videos:

[1310] The user selects the photos or videos they want to upload from their device's gallery. Once selection is complete, they tap "Upload." The input is the selected media file, and the output is the start of the upload process.

[1311] Step 4:

[1312] The device prepares the data:

[1313] The device checks the metadata already attached to the selected photos and videos (such as the date and time of the photo, location, and device information) and stores these files in a buffer for transmission to the server. The input is the selected files, and the output is the data ready to be sent.

[1314] Step 5:

[1315] The device sends data:

[1316] The device sends photo and video data to the server in chunks. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer. The input is the data ready to be sent, and the output is the piece of data being sent.

[1317] Step 6:

[1318] Server receives data:

[1319] The server receives the uploaded data, checks the session ID to identify the sending user, verifies the received data is correct using checksums, and requests retransmission of any incomplete packets. The input is the piece of data being sent, and the output is the complete data file.

[1320] Step 7:

[1321] Server reconstructs data:

[1322] The server reconstructs the photo or video file from the received data fragments and saves it in the appropriate folder in the cloud storage. The input is the data fragments and the output is the reconstructed file.

[1323] Step 8:

[1324] Generative AI analyzes content:

[1325] The generative AI model in the server analyzes the metadata and content of stored photos and videos, identifying product features using techniques such as facial recognition, background analysis, and facial expression recognition. The input is the stored media files, and the output is the analyzed data.

[1326] Step 9:

[1327] Generative AI creates scenarios:

[1328] Based on the analysis results, the generative AI automatically creates a storytelling-style video scenario. It places the product's main features on a timeline and generates a visually appealing video. The input is analysis data, and the output is scenario data.

[1329] Step 10:

[1330] Generative AI generates videos:

[1331] The generative AI edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The completed video is saved as a new file in cloud storage. The input is the scenario data, and the output is the generated video file.

[1332] Step 11:

[1333] The server notifies the user:

[1334] The server records the location of the generated video and notifies the user that a new video has been generated. The notification is sent in the form of a push notification on the device or a message within the app. The input is the generated video information, and the output is a push notification.

[1335] Step 12:

[1336] User plays video:

[1337] The user confirms the notification and opens the app to play the generated video. This allows users to easily enjoy product and promotion information visually. The input is the push notification, and the output is the playing video.

[1338] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1339] Hereinafter, embodiments of the present invention will be described in detail.

[1340] 1. User operations and photo / video uploads

[1341] User takes a photo with their smartphone:

[1342] Users use devices such as smartphones and tablets to take photos and videos of events such as family birthday parties.

[1343] User opens the app:

[1344] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set a title, for example, "Birthday Party."

[1345] User selects photos and videos:

[1346] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1347] 2. Transferring data from the device to the server

[1348] The device prepares the data:

[1349] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and store the data in a temporary buffer.

[1350] The terminal splits and sends the data:

[1351] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[1352] 3. Data reception and storage by the server

[1353] Server receives data:

[1354] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[1355] Server reconstructs data:

[1356] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[1357] 4. Content analysis and video generation using generative AI and emotion engines

[1358] Generative AI analyzes content:

[1359] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition.

[1360] The emotion engine analyzes emotions:

[1361] The generative AI has a built-in emotion engine that analyzes the emotions (e.g., smiling, sad, surprised, etc.) of the people in each photo or video.

[1362] Generative AI creates scenarios:

[1363] Based on the analysis results, it automatically creates a storytelling video scenario, taking into account emotional information, selecting appropriate scenes, and arranging them along the timeline.

[1364] Generative AI generates videos:

[1365] The AI ​​then edits the photos and videos according to the scenario, adding transition effects and background music to create a video. The generated video is then saved as a new file in cloud storage.

[1366] 5. User Feedback and Video Playback

[1367] The server notifies the user:

[1368] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1369] User plays video:

[1370] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[1371] For example, a user can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and generate a storytelling video. The generated video includes key moments, such as cutting the birthday cake and opening presents, edited with emotional information, allowing the user to play the video on their device and enjoy memories of the party.

[1372] The processing flow will be explained below.

[1373] Step 1:

[1374] The user takes a photo with their smartphone

[1375] Users use devices such as smartphones and tablets to take photos and videos of family birthday parties.

[1376] Step 2:

[1377] The user opens the app

[1378] The user opens the Storytelling app and selects "Create a new album" from the home screen. They enter the album title as "Birthday Party."

[1379] Step 3:

[1380] User selects photos and videos and starts uploading

[1381] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1382] Step 4:

[1383] The device prepares the data

[1384] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[1385] Step 5:

[1386] The terminal divides and transmits the data

[1387] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[1388] Step 6:

[1389] The server receives the data

[1390] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[1391] Step 7:

[1392] The server reconstructs the data and stores it in cloud storage.

[1393] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[1394] Step 8:

[1395] The server calls the generative AI and emotion engine

[1396] The server passes the path information of the photos and videos in the cloud storage to the generation AI, which analyzes the metadata and content of the stored photos and videos, and the emotion engine analyzes the emotions (e.g., smile, sadness, surprise, etc.) of the people in each photo and video.

[1397] Step 9:

[1398] Generative AI and emotion engines analyze content

[1399] The generative AI analyzes photos and videos using technologies such as facial recognition, contextual analysis, and facial expression recognition. The emotion engine identifies changes in emotions and provides that information to the generative AI.

[1400] Step 10:

[1401] Generative AI creates scenarios

[1402] The generative AI uses information from the emotion engine to automatically create a storytelling video scenario, selecting key moments (e.g., cutting a birthday cake or opening a gift) and placing them on a timeline.

[1403] Step 11:

[1404] Generative AI generates videos

[1405] The AI ​​then edits the video based on the scenario, adding transition effects and background music to match the emotional changes, and the resulting video is saved as a new file in cloud storage.

[1406] Step 12:

[1407] The server notifies the user

[1408] The server records the location of the generated video and sends a push notification to the user's smartphone.

[1409] Step 13:

[1410] The user plays the video

[1411] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[1412] Example 2

[1413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1414] In recent years, the amount of photos and videos taken with digital devices has rapidly increased, creating a need for efficient classification and management of these images and videos, and for users to easily access them. Furthermore, to effectively preserve and share memories of special family events, simply storing photos and videos is insufficient; they must be reconstructed as storytelling content. Furthermore, there is a need for a method to automatically generate high-quality content with minimal user interaction.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for taking images and videos using a terminal operable by a user and uploading the images and videos, means for receiving the uploaded images and videos on the server side and saving them in shared storage, generation AI means for analyzing the images and videos in the shared storage and classifying and organizing them based on specific dates or events, means for automatically generating story-style videos based on the images and videos organized by the generation AI means, and means for providing the automatically generated videos to the user's terminal and making them playable. This allows users to efficiently manage their captured content and enjoy their special memories by automatically converting them into high-quality storytelling-style videos.

[1416] A "terminal" is a device that can be operated by a user, and includes, for example, a smartphone, a tablet, or a PC.

[1417] "Images and Videos" means still and moving images stored in digital format that are used to record events and moments in everyday life.

[1418] "Upload" refers to the operation or process of sending data from a device to a server.

[1419] A "server" is a computer system that receives, stores, analyzes, and generates data and communicates with terminals via a network.

[1420] "Shared storage" refers to a storage device connected to a server, used to centrally manage and store data from multiple users.

[1421] "Analysis" refers to the process of performing specific processing on digital content to extract and classify its content and characteristics.

[1422] "Generative AI" refers to a system that uses artificial intelligence techniques to process and analyze data and automatically create specific deliverables.

[1423] "Narrative video" refers to a video that is not simply a series of images, but is edited and reconstructed based on a specific theme or scenario.

[1424] "Delivery" refers to the process of making the generated deliverables available to users.

[1425] "Play" refers to the action initiated to watch or listen to a stored video or audio file.

[1426] "Metadata" refers to auxiliary information added to images and videos (e.g., shooting date and time, location, device information, etc.), and is used when organizing and analyzing data.

[1427] This invention is a system that uploads images and videos from users' devices to a server, analyzes and organizes them, and automatically generates story-style videos. This system allows users to efficiently manage their captured content and enjoy their special memories as high-quality storytelling videos.

[1428] Hardware and Software Configuration

[1429] Terminal

[1430] Users take pictures and videos using devices such as smartphones and tablets. These devices have a camera function and a dedicated storytelling application installed. The application provides an operation screen and supports the selection and uploading of images and videos.

[1431] server

[1432] The server is a cloud-connected computer system that receives and stores content uploaded by users. It is equipped with a generative AI and emotion analysis engine, which are used to analyze and organize images and videos, generate scenarios, and generate videos.

[1433] Generative AI and Sentiment Analysis Engine

[1434] The generative AI includes advanced image processing technology, enabling facial recognition, background analysis, and facial expression recognition. The emotion analysis engine analyzes the emotions of people in each image or video, and generates a scenario based on that information. Based on the generated scenario, appropriate effects and music are added to create a narrative video.

[1435] Data processing and calculation

[1436] Adding Metadata

[1437] The device automatically adds metadata to the images and videos it takes, including the date and time the image was taken, the location, and device information. When a user uploads an image or video, this metadata is also sent to the server.

[1438] Data division and transmission

[1439] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[1440] Receiving and Reconstructing Data

[1441] The server receives each chunk and checks it for errors using a checksum. If an error is found, the server requests that the chunk be retransmitted. Correctly received data chunks are reconstructed into the original file and saved to shared storage.

[1442] Analyzing and organizing content

[1443] Generative AI analyzes the metadata and content of images and videos stored on shared storage. Based on the analysis results, content is categorized and organized based on specific dates and events. Specific people and scenes are identified through facial recognition, context analysis, and facial expression recognition.

[1444] Scenario creation and video generation

[1445] The generative AI automatically generates a story-style video scenario based on the analysis results. The emotion analysis engine reflects the emotional information in each scene and selects and adds effects and music in line with the scenario. Finally, the automatically generated video is saved in cloud storage that users can access.

[1446] Specific examples

[1447] For example, users can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and create a storytelling video. The generated video includes key moments, such as cutting the birthday cake or opening presents, edited with emotional information, allowing users to play the video on their device and enjoy their memories of the party.

[1448] Example prompt sentence:

[1449] "Use photos and videos from a child's birthday party to generate a storytelling video incorporating emotional information."

[1450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1451] Step 1:

[1452] The user takes a photo

[1453] Users use devices such as smartphones and tablets to take pictures and videos using the device's built-in camera app, and the content they take is automatically saved in the device's gallery.

[1454] Input: Images and videos taken with the camera app

[1455] Output: Images and videos stored in the device's gallery

[1456] Step 2:

[1457] The user opens the app

[1458] A user launches the storytelling application, authenticates by logging in, selects the "Create a new album" option within the app, and sets a title, for example, "Birthday Party."

[1459] Enter: storytelling application

[1460] Output: Create a new album (e.g. "Birthday Party")

[1461] Step 3:

[1462] Users can select and upload images and videos

[1463] Users can select the images or videos they want to upload from their device's gallery and tap the "Upload" button in the app. The selected content will be added with metadata such as the date and time it was taken, location, and device information.

[1464] Input: User-selected images, videos, and metadata

[1465] Output: Images, videos and metadata ready to upload

[1466] Step 4:

[1467] The terminal divides and transmits the data

[1468] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[1469] Input: Prepared images, videos and metadata

[1470] Output: Split data chunks, file ID, checksum

[1471] Step 5:

[1472] The server receives and checks the data

[1473] The server receives the data chunks sent by the device and checks for errors using the checksum. If an error is found, the server requests the device to retransmit the particular chunk to ensure error-free data.

[1474] Input: Data chunk, file ID, checksum

[1475] Output: Data chunk with errors corrected

[1476] Step 6:

[1477] The server reconstructs and saves the data

[1478] The server reconstructs correctly received data chunks into the original image or video file and stores it on shared storage.

[1479] Input: Data chunk with errors corrected

[1480] Output: Original image and video files saved in cloud storage

[1481] Step 7:

[1482] Generative AI analyzes content

[1483] The server-based generative AI analyzes the metadata and content of images and videos stored in cloud storage, identifying specific people and scenes through facial recognition, background analysis, and facial expression recognition.

[1484] Input: Images, videos and metadata from cloud storage

[1485] Output: Image and video analysis results (e.g., people, scenes, emotional information)

[1486] Step 8:

[1487] Generative AI creates scenarios

[1488] The generative AI automatically generates a story-style video scenario based on the analysis results, and uses an emotion analysis engine to select effects and music appropriate for each scene.

[1489] Input: Image and video analysis results, emotional information

[1490] Output: Narrative video scenario

[1491] Step 9:

[1492] Generative AI generates videos

[1493] The AI ​​then follows the user's scenario, edits images and videos, adds transition effects and background music, and generates a video, which is then saved in cloud storage.

[1494] Input: Narrative video scenario

[1495] Output: Generated story-style video file

[1496] Step 10:

[1497] The server notifies the user

[1498] The server records the location of the generated video and notifies the user that a new video is ready, either via a push notification on the device or an in-app message.

[1499] Input: Generated video file

[1500] Output: User notification

[1501] Step 11:

[1502] User plays video

[1503] The user will receive a notification and open the storytelling application to play the generated video, allowing them to enjoy their special memories as a storytelling video.

[1504] Input: Notification from the server

[1505] Output: Video played on the user's device

[1506] (Application example 2)

[1507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1508] Previous applications only allowed users to individually store photos and videos, limiting their ability to visually enjoy specific events or experiences. Furthermore, there was a lack of effective ways to record and replay customer experiences in physical stores. As a result, users were unable to effectively reminisce about their memories and it was difficult to share their real-time experiences with others. Furthermore, the lack of an automatic generation function for storytelling videos that reflected the emotions and scenarios of each event required users to manually edit the content.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1510] In this invention, the server includes means for taking photos and videos using a device operable by a user and uploading the photos and videos, means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, means for providing the automatically generated videos to the user's device and making them playable, means for analyzing photos and videos taken of the user's experience in a physical store and generating scenarios of purchasing behavior and interactions, and means for editing the videos of the experience in the physical store based on the generated scenarios. This allows users to enjoy a specific event or experience in the store as an automatically edited video along with emotional information.

[1511] "Terminal" means an electronic device that can be operated by a user, such as a smartphone, tablet, or other computing device.

[1512] "Means for taking photos or videos" means having the ability to take images or videos using the device's built-in camera or a corresponding software application.

[1513] "Uploading means" refers to software or hardware capable of transmitting photos and videos to a server via the Internet.

[1514] "Server" refers to a remote computer system for data processing and management.

[1515] "Cloud storage" refers to a system for storing data on remote servers accessible via the Internet.

[1516] "Generative AI methods" refers to systems that use artificial intelligence techniques to analyze photos and videos and categorize and organize data based on specific days or events.

[1517] "Means for automatically generating storytelling videos" refers to a function that uses generative AI means to edit photos and videos and automatically create videos based on a scenario.

[1518] "Means for providing and making playable" refers to the functionality for transmitting the generated storytelling video to a user's device and enabling the user to watch the video.

[1519] "Means for analyzing photos and videos of experiences" refers to a system that has the ability to analyze images and videos taken by users in physical stores to understand their purchasing behavior and interactions.

[1520] "Means for generating scenarios of purchasing behavior and interactions" refers to a system that has the function of creating scenarios of users' behavior and experiences in stores based on the results of analysis.

[1521] "Means for editing recorded video" refers to a system that has the function of editing photos and videos according to the generated scenario and creating final video content.

[1522] MODE FOR CARRYING OUT THE INVENTION

[1523] System program for implementing application examples

[1524] User actions and photo / video uploads

[1525] Users take photos and videos of their in-store experience using a device such as a smartphone or tablet. After taking photos, users open the dedicated application and proceed to the upload screen. They create a new album, title it "Shopping Experience," select photos and videos from the gallery, and tap the upload button.

[1526] Transferring data from the device to the server

[1527] The device adds metadata to the selected photos and videos, such as the date and time of the photo, location, and device information, then splits the large video file into smaller chunks, attaches a file ID and checksum to each chunk, and sends these chunks to the server sequentially.

[1528] Data reception and storage by the server

[1529] The server receives the data chunks, checks them for errors using checksums, requests that the chunks be resent if an error is found, and reconstructs the original photo or video file from the received data chunks and stores it in cloud storage.

[1530] Content analysis and video generation with generative AI and emotion engines

[1531] The server-based generative AI analyzes the metadata and content of stored photos and videos. It uses advanced technologies such as facial recognition, background analysis, and facial expression recognition to analyze emotional information. Based on this information, the generative AI creates a scenario and generates a storytelling video that reflects specific events and experiences. Transition effects and background music are also automatically added to create the final video.

[1532] User feedback and video playback

[1533] The generated video is saved to cloud storage, the server records the location, and notifies the user that a video has been generated. The user can then confirm the notification and play the video within the app, allowing them to reminisce about their experience in the physical store.

[1534] Natural language explanation of program processing

[1535] Hardware and software usage

[1536] Hardware: Smartphones, cloud servers

[1537] Software: Dedicated applications, cloud storage, generative AI models (e.g., TensorFlow), sentiment analysis engines (e.g., IBM Watson)

[1538] Program processing flow

[1539] 1. User-taken and uploaded photos: When users upload photos or videos they have taken, metadata is automatically added within the app.

[1540] 2. Data transmission and reception: Data is sent from the device to the server in small chunks, which are then received, verified, and stored in cloud storage.

[1541] 3. Content Analysis and Video Generation: Generative AI analyzes each photo or video and automatically generates a scenario based on emotional information, then edits the video accordingly, adding background music and transition effects.

[1542] 4. User notification and playback: Once the generated video has been saved, the user will be notified and will be able to play the video within the app.

[1543] Specific examples

[1544] Users visit retail stores on launch days and film themselves tasting and trying on products in-store. They then upload this content through the app. Generative AI and emotion engines automatically create a "launch day experience video" and notify the user. Users can then play this video within the app to relive their experience in the retail store.

[1545] Prompt Sentence Examples

[1546] "Please enter a prompt for the generated AI:

[1547] To capture the product launch experience in a video, edit the video to focus on where the user is, their facial expressions, and their actions. Select key moments of tasting, trying on, and purchasing, and add effects and music to capture the emotional changes.

[1548] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1549] Step 1:

[1550] Users capture their in-store experiences

[1551] Users use their smartphones or tablets to take photos and videos of their in-store experiences.

[1552] Input: Photos and videos taken with your smartphone camera.

[1553] Output: The captured photos and videos will be saved in the device's gallery.

[1554] Specific actions: Launch the smartphone camera application and capture important moments in a physical store (e.g., trying on clothes, purchasing new items).

[1555] Step 2:

[1556] User opens the app, selects photos and videos to upload

[1557] Users open the dedicated application, create an album, select the photos and videos they have taken, and tap the upload button.

[1558] Input: Photos and videos taken, album title set by user (e.g. "Shopping Experience").

[1559] Output: The file ready to be uploaded.

[1560] What it does: Launch the app, go to the upload screen, set a title, select a photo or video from your gallery, then tap the "Upload" button.

[1561] Step 3:

[1562] The device adds metadata to the data, splits it into small chunks, and sends it to the server

[1563] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and will split large video files into smaller chunks for transmission.

[1564] Input: The photo or video file to upload.

[1565] Output: Data split into small chunks with metadata attached.

[1566] What happens: The application runs a routine to add metadata, splits the data into small chunks, adds a file ID and checksum to each chunk, and sends them to the server one by one.

[1567] Step 4:

[1568] The server receives and verifies the data chunks and stores them in cloud storage.

[1569] The server receives the transmitted data chunks, checks them for errors using checksums, and if an error is found, requests that the chunk be retransmitted. The error-free data chunks are reconstructed and stored in cloud storage.

[1570] Input: Data chunks.

[1571] Output: Complete photo and video files.

[1572] What happens: The server receives each data chunk, checks the checksum for errors, reconstructs the data and saves it to cloud storage if there are no errors, and requests a retransmission if there are errors.

[1573] Step 5:

[1574] The server analyzes the content using generative AI and emotion engine

[1575] The server-based generative AI analyzes the metadata and content of stored photos and videos to perform facial recognition, background analysis, and facial expression recognition. The emotion engine analyzes emotional information.

[1576] Input: Photos and videos in cloud storage.

[1577] Output: Analysis results for each content (face, background, facial expression, emotional information).

[1578] How it works: It activates a generative AI model to analyze the metadata and content of photos and videos, and then uses an emotion engine to analyze faces and expressions to extract emotional information.

[1579] Step 6:

[1580] Generative AI creates scenarios and generates storytelling videos

[1581] Based on the analysis, the generative AI creates a scenario and generates a video that reflects a specific event or experience, automatically adding transition effects and background music.

[1582] Input: Parsed content and sentiment information.

[1583] Output: The finished storytelling video file.

[1584] What it does: The generative AI creates a scenario, the editor engine edits photos and videos, adds transition effects and background music, and finally generates a video file.

[1585] Step 7:

[1586] The server notifies the user that the video has been generated

[1587] The generated video is stored in cloud storage, and the server records the location of the video and notifies the user that a new video has been generated.

[1588] Input: The finished video file.

[1589] Output: Notification to the user.

[1590] What happens: The server launches the notification system and sends a notification to the user in the form of a push notification or in-app message.

[1591] Step 8:

[1592] User plays a video within the app

[1593] Users can view the notification and play the video generated within the app, allowing them to relive their experience in the physical store in real time.

[1594] Input: Notification from the server.

[1595] Output: The video being played.

[1596] What happens: The user opens the notification and interacts with the in-app video player to play the video.

[1597] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1598] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1599] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1600] [Fourth embodiment]

[1601] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1602] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1603] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1604] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1605] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1606] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1607] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1608] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1609] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1610] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1611] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1612] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1613] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1614] Hereinafter, embodiments of the present invention will be described in detail.

[1615] 1. User operations and photo / video uploads

[1616] User takes a photo with their smartphone:

[1617] Users use devices such as smartphones and tablets to take photos and videos of events such as their family's birthday parties.

[1618] User opens the app:

[1619] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set the title to "Birthday Party," for example.

[1620] User selects photos and videos:

[1621] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1622] 2. Transferring data from the device to the server

[1623] The device prepares the data:

[1624] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[1625] The device sends data:

[1626] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[1627] 3. Data reception and storage by the server

[1628] Server receives data:

[1629] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[1630] Server reconstructs data:

[1631] The photos and video files are reconstructed based on the received data and saved in a "Birthday Party" folder in cloud storage.

[1632] 4. Content analysis and video generation using generative AI

[1633] Generative AI analyzes content:

[1634] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of the event (e.g., cutting the cake or opening a gift).

[1635] Generative AI creates scenarios:

[1636] Based on the analysis results, it automatically creates a storytelling video scenario, placing key moments on a timeline and generating a visually engaging video.

[1637] Generative AI generates videos:

[1638] The generative AI then edits the video according to the scenario, adding transition effects and background music, and the finished video is saved as a new file in cloud storage.

[1639] 5. User Feedback and Video Playback

[1640] The server notifies the user:

[1641] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1642] User plays video:

[1643] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy their memories visually.

[1644] For example, users can take photos and videos of their child's birthday party and upload them through the app. The AI ​​then automatically analyzes the content and creates a storytelling video. Users can then play the video on their device and enjoy the memories of the party.

[1645] The processing flow will be explained below.

[1646] Step 1:

[1647] The user takes a photo or video

[1648] A user uses a smartphone camera app to take photos and videos of a family birthday party.

[1649] Step 2:

[1650] The user opens the app

[1651] The user opens the storytelling app, selects "Create a new album" from the home screen, and enters "Birthday Party" as the album title.

[1652] Step 3:

[1653] User selects photos and videos and starts uploading

[1654] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1655] Step 4:

[1656] The device prepares the data

[1657] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[1658] Step 5:

[1659] The terminal divides and transmits the data

[1660] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[1661] Step 6:

[1662] The server receives the data

[1663] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[1664] Step 7:

[1665] The server reconstructs the data

[1666] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[1667] Step 8:

[1668] The server calls the generated AI

[1669] The server passes the path information of the photos and videos in cloud storage to the generation AI.

[1670] Step 9:

[1671] Generative AI analyzes content

[1672] The generative AI analyzes the metadata and content of each photo or video and uses techniques such as facial recognition, background analysis, and facial expression recognition.

[1673] Step 10:

[1674] Generative AI creates scenarios

[1675] Based on the analysis, key moments are selected and a storytelling video scenario is created.

[1676] Step 11:

[1677] Generative AI generates videos

[1678] The generative AI edits photos and videos according to a scenario, adds transition effects and background music, and generates a video that is then saved in cloud storage.

[1679] Step 12:

[1680] The server notifies the user

[1681] The server records the location of the generated video and sends a push notification to the user's smartphone.

[1682] Step 13:

[1683] The user plays the video

[1684] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[1685] Example 1

[1686] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1687] There is a need for an efficient method to manage and analyze the growing amount of photo and video data and reconstruct them into memorable formats. However, many users have to spend a lot of time and effort manually organizing and editing each photo and video. Therefore, a system is needed that allows users to easily upload their captured content and automatically analyzes, classifies, and reconstructs it into a visually enjoyable format.

[1688] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1689] In this invention, the server includes: means for taking photos and videos using a device operable by a user and uploading the photos and videos; means for adding metadata (such as the shooting date and time, location, and device information) to the uploaded photos and videos on the device side, storing them in a buffer, and then transmitting them to the server; means for receiving the transmitted data on the server side, verifying the sender's identification information, checking the data integrity, and then storing them in a cloud storage device; means for analyzing the photos and videos in the cloud storage device using a generative AI model, performing facial recognition, background analysis, facial expression recognition, etc., to identify important moments of the event; means for automatically generating storytelling-type videos based on the analysis results and adding transition effects and background music; and means for providing the automatically generated videos to the user's device and making them playable. This enables efficient management of photos and videos taken by users and automatically generating visually enjoyable videos.

[1690] "User" refers to an individual or organization that operates the system to take and upload photos and videos.

[1691] "Terminal" refers to a device that can be operated by a user, such as a smartphone, tablet, or PC.

[1692] "Photo" refers to a still image taken by a user on a device.

[1693] "Video" refers to a file format that contains moving images captured by a user on a device.

[1694] "Uploading" refers to the process of sending data such as photos or videos from a device to a server or cloud storage device.

[1695] "Metadata" refers to additional information associated with a photo or video, including the date and time it was taken, the location, and device information.

[1696] A "buffer" refers to a storage area that temporarily stores data.

[1697] "Server" means a computer that acts as a central processing unit and receives, processes, and stores uploaded data.

[1698] "Cloud storage" refers to a remote storage medium accessible over a network.

[1699] A "generative AI model" refers to an artificial intelligence algorithm that uses machine learning and deep learning technologies to analyze data and automatically generate and edit scenarios.

[1700] "Facial recognition" refers to the technology in which a generative AI model identifies the faces of people in photos and videos.

[1701] "Background analysis" refers to the technology in which a generative AI model analyzes the background or scenery in a photo or video.

[1702] "Facial expression recognition" refers to a technology in which a generative AI model distinguishes between facial expressions and analyzes emotions.

[1703] "Transition effects" refer to visual effects used when switching from one scene to another in video editing.

[1704] "Background music" refers to music or sound effects added to the background of a video.

[1705] "Storytelling video" refers to visual content that is edited around a specific scenario or narrative flow.

[1706] Hereinafter, embodiments of the present invention will be described in detail.

[1707] 1. User operations and photo / video uploads

[1708] A user uses a device such as a smartphone or tablet to take photos and videos of an event such as a family birthday party. For example, the user can take photos of a birthday cake or a video of a child opening a present. The user then opens a storytelling app. The app has a "Create a New Album" option, and the user creates a new album titled "Birthday Party." The user selects the photos and videos they want to upload from their device's gallery and taps the upload button to send the photos and videos.

[1709] 2. Transferring data from the device to the server

[1710] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos. Metadata adds important information to the uploaded content, making it useful for analysis and management. The device temporarily stores all data for upload in a buffer. The buffer ensures data consistency. The device then sends the photo and video data to the server. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer and makes retransmission easier if a communication error occurs.

[1711] 3. Data reception and storage by the server

[1712] The server receives the uploaded data and checks the session ID to identify the user who sent it. For example, the session ID can be used to identify who uploaded what data. The server uses a checksum to verify that the received data is correct, and requests a retransmission of any incomplete packets. This ensures the reliability of data transfer. The server reconstructs the photo and video files based on the received data. The reconstructed files are saved in the "Birthday Party" folder in the cloud storage.

[1713] 4. Content analysis and video generation using generative AI

[1714] The generation AI on the server analyzes the metadata and content of the stored photos and videos. Specifically, it uses technologies such as facial recognition, background analysis, and facial expression recognition to identify important moments (e.g., the moment a cake is cut or a gift is opened). Based on the analysis results, the generation AI automatically creates a storytelling video scenario. For example, the scenario may include scenes of cutting a cake or opening a gift on a timeline. The generation AI then edits the video according to the scenario, adding transition effects and background music to create a visually appealing video. The completed video is then saved in cloud storage.

[1715] 5. User Feedback and Video Playback

[1716] The server records the location where the generated video is saved and notifies the user that a new video has been generated. This is done via a push notification on the device or a message within the app. After checking the notification, the user opens the app and plays the generated video. This allows the user to visually enjoy the memories of the party.

[1717] Prompt Sentence Examples

[1718] Examples of prompts to input to a generative AI model include:

[1719] Users have uploaded photos and videos of their "birthday party." Use this content to generate a storytelling video that includes key moments like cutting the cake and opening gifts. Add transition effects and background music to create a visually appealing video.

[1720] The above is an embodiment of the invention.

[1721] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1722] Step 1:

[1723] Taking a photo and launching the app

[1724] A user takes photos or videos using a smartphone or tablet. For example, the user takes photos or videos of a birthday party, and then launches a storytelling app.

[1725] Input: Photos and videos taken by the user.

[1726] Output: Application launch.

[1727] Step 2:

[1728] Creating an album and selecting files

[1729] Users can select the "Create a new album" option within the app, set the title to "Birthday Party," then select the photos and videos they want to upload from their device's gallery and tap the upload button.

[1730] Input: User-specified new album name and selected photos and videos.

[1731] Output: Data ready for upload.

[1732] Step 3:

[1733] Adding metadata and saving it to the buffer

[1734] Your device will add metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos, giving them information about when and where they were taken. It then stores these files in a buffer and prepares them for transmission.

[1735] Input: Photo and video files selected by the user.

[1736] Output: Buffered photo and video files with metadata.

[1737] Step 4:

[1738] Splitting the data and sending it to the server

[1739] The device splits the large video file into chunks and sends the photos and split video data to the server, using an appropriate communication protocol to prevent data loss.

[1740] Input: Photo and video files with metadata stored in a buffer.

[1741] Output: The split data sent to the server.

[1742] Step 5:

[1743] Data reception and checksum verification by the server

[1744] The server validates the received data using checksums for integrity and requests retransmission of any incomplete packets, thus ensuring data consistency and quality.

[1745] Input: Split data sent from the terminal.

[1746] Output: Validated complete data.

[1747] Step 6:

[1748] Data reconstruction and cloud storage

[1749] The server reconstructs the received data to generate the original photo and video files, and then stores the reconstructed files in a "Birthday Party" folder in the cloud storage device.

[1750] Input: Validated data.

[1751] Output: Photo and video files saved in cloud storage.

[1752] Step 7:

[1753] Content analysis with generative AI

[1754] The server-based generative AI analyzes the metadata and content of photos and videos stored in cloud storage, using techniques such as facial recognition, background analysis, and facial expression recognition to identify key moments of an event.

[1755] Input: Photo and video files stored in cloud storage.

[1756] Output: Parsed content and identified key moments data.

[1757] Step 8:

[1758] Scenario creation and video generation

[1759] The generative AI creates a scenario based on the analysis results and automatically edits a storytelling video, adding transition effects and background music, and saving the generated video as a new file in cloud storage.

[1760] Input: Analysis results and data on key moments of identified events.

[1761] Output: Auto-generated storytelling videos.

[1762] Step 9:

[1763] User Notification

[1764] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1765] Input: An auto-generated video.

[1766] Output: Notification to the user.

[1767] Step 10:

[1768] Playing a video

[1769] The user confirms the notification, opens the app, and plays the generated video, allowing them to visually enjoy their birthday party memories.

[1770] Input: Notification from the server.

[1771] Output: The video played in the app.

[1772] In this way, each processing step works together and fulfills its role, allowing users to record and play back memories easily and quickly.

[1773] (Application example 1)

[1774] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1775] Previous technology lacked the means to promote products in stores or enhance the customer's purchasing experience. Furthermore, there was no system that allowed customers to take photos of products and effectively utilize those photos or videos. This made it difficult to effectively promote and introduce products to individual customers, limiting the ways to stimulate purchasing motivation.

[1776] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1777] In this invention, the server includes a means for taking photos and videos using a user-operable device and uploading the photos and videos, a means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, a generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, a means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, a means for incorporating product introduction information and promotional information into the automatically generated videos, and a means for providing the automatically generated videos to the user's device and making them playable. This enables the automatic generation of effective promotional videos based on photos and videos of products and events taken by customers.

[1778] A "user-operable device" is an electronic device that can be operated by a user and used to take and upload photographs and videos.

[1779] "Means for uploading photos and videos" refers to a method or device for transmitting photos and video data taken by a user to a server.

[1780] "Means for receiving data on the server side and storing it in cloud storage" refers to a method or device for storing data received by the server in a cloud environment.

[1781] "Generative AI methods" refers to artificial intelligence technology that analyzes photos and videos uploaded and stored in cloud storage, categorizing and organizing them based on specific days or events.

[1782] A "means for automatically generating storytelling videos" refers to a method or device that automatically creates narrative videos using a sequence of photos and videos based on the analysis results of generative AI.

[1783] "Means for incorporating product introduction information and promotional information" refers to a method or device for adding product-related information and promotional content to the generated video.

[1784] "Means for providing the video to the user's terminal and enabling playback" refers to a method or device for delivering the generated video to the user's terminal and enabling playback on the terminal.

[1785] "Metadata" refers to additional information associated with a photo or video (e.g., date and time it was taken, location, device information, etc.).

[1786] "Facial recognition" is a technology that identifies people's faces in photos and videos and analyzes their features.

[1787] "Background analysis" refers to the technology of identifying background elements in photos and videos and analyzing their content.

[1788] "Facial expression recognition" is a technology that identifies facial expressions of people in photos and videos and analyzes their emotions and reactions.

[1789] "Creating a scenario" means designing a video with a narrative by determining the structure and order of the video based on the generated data.

[1790] MODE FOR CARRYING OUT THE INVENTION

[1791] Hereinafter, embodiments of the present invention will be described in detail.

[1792] 1. User operations and photo / video uploads

[1793] User takes a photo with their smartphone:

[1794] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in while in a physical store.

[1795] User opens the app:

[1796] Users open the app, access the photo and video upload screen, select the "Create a new promotional video" option, and set an appropriate title.

[1797] User selects photos and videos:

[1798] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1799] 2. Transferring data from the device to the server

[1800] The device prepares the data:

[1801] The device adds metadata (such as the date and time of the photo, location, and device information) to the selected photos and videos and stores these files in a buffer for transmission to the server.

[1802] The device sends data:

[1803] Your device will send photo and video data in chunks, and large video files will be split into chunks (small pieces of data) for efficient data transfer.

[1804] 3. Data reception and storage by the server

[1805] Server receives data:

[1806] The server receives the uploaded data, checks the session ID to identify the user who sent it, verifies the received data using a checksum to ensure it is correct, and requests retransmission of any incomplete packets.

[1807] Server reconstructs data:

[1808] The photo and video files are reconstructed based on the received data and saved in the appropriate folder in cloud storage.

[1809] 4. Content analysis and video generation using generative AI

[1810] Generative AI analyzes content:

[1811] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition to identify product features and promotional information.

[1812] Generative AI creates scenarios:

[1813] Based on the analysis results, the system automatically creates a storytelling video scenario, placing the key features of the product on a timeline and generating a visually appealing video.

[1814] Generative AI generates videos:

[1815] The AI ​​then edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The finished video is then saved as a new file in cloud storage.

[1816] 5. User Feedback and Video Playback

[1817] The server notifies the user:

[1818] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1819] User plays video:

[1820] Users can check the notification, open the app and play the generated video, allowing them to easily enjoy product and promotional information visually.

[1821] Specific examples

[1822] Customers can take photos of products in the store through the app, and the photos are sent to a cloud server. The AI ​​analyzes the photos and automatically creates promotional videos for the products and stores. The created videos are then sent to customers via push notifications and can be played back within the app.

[1823] Example prompt sentence:

[1824] "Use a generative AI model to analyze this product photo and create a product video. The analysis should include the product name, price, and instructions for use."

[1825] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1826] Step 1:

[1827] User takes a photo with their smartphone:

[1828] Users use devices such as smartphones or tablets to take photos and videos of products they are interested in in physical stores. At the same time, the date and time of the photo and location information are added to the images and videos as metadata. The input is photos and videos of the products, and the output is image and video files with the added metadata.

[1829] Step 2:

[1830] User opens the app:

[1831] The user opens the app and accesses the screen for uploading photos and videos they have taken. Here, they select the "Create a new promotional video" option and set an appropriate title. The input is the user's operation, and the output is the transition to the upload screen.

[1832] Step 3:

[1833] User selects photos and videos:

[1834] The user selects the photos or videos they want to upload from their device's gallery. Once selection is complete, they tap "Upload." The input is the selected media file, and the output is the start of the upload process.

[1835] Step 4:

[1836] The device prepares the data:

[1837] The device checks the metadata already attached to the selected photos and videos (such as the date and time of the photo, location, and device information) and stores these files in a buffer for transmission to the server. The input is the selected files, and the output is the data ready to be sent.

[1838] Step 5:

[1839] The device sends data:

[1840] The device sends photo and video data to the server in chunks. Large video files are split into chunks (small pieces of data). This allows for efficient data transfer. The input is the data ready to be sent, and the output is the piece of data being sent.

[1841] Step 6:

[1842] Server receives data:

[1843] The server receives the uploaded data, checks the session ID to identify the sending user, verifies the received data is correct using checksums, and requests retransmission of any incomplete packets. The input is the piece of data being sent, and the output is the complete data file.

[1844] Step 7:

[1845] Server reconstructs data:

[1846] The server reconstructs the photo or video file from the received data fragments and saves it in the appropriate folder in the cloud storage. The input is the data fragments and the output is the reconstructed file.

[1847] Step 8:

[1848] Generative AI analyzes content:

[1849] The generative AI model in the server analyzes the metadata and content of stored photos and videos, identifying product features using techniques such as facial recognition, background analysis, and facial expression recognition. The input is the stored media files, and the output is the analyzed data.

[1850] Step 9:

[1851] Generative AI creates scenarios:

[1852] Based on the analysis results, the generative AI automatically creates a storytelling-style video scenario. It places the product's main features on a timeline and generates a visually appealing video. The input is analysis data, and the output is scenario data.

[1853] Step 10:

[1854] Generative AI generates videos:

[1855] The generative AI edits the video according to the scenario, adding transition effects and background music, and incorporating product information and promotional information. The completed video is saved as a new file in cloud storage. The input is the scenario data, and the output is the generated video file.

[1856] Step 11:

[1857] The server notifies the user:

[1858] The server records the location of the generated video and notifies the user that a new video has been generated. The notification is sent in the form of a push notification on the device or a message within the app. The input is the generated video information, and the output is a push notification.

[1859] Step 12:

[1860] User plays video:

[1861] The user confirms the notification and opens the app to play the generated video. This allows users to easily enjoy product and promotion information visually. The input is the push notification, and the output is the playing video.

[1862] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1863] Hereinafter, embodiments of the present invention will be described in detail.

[1864] 1. User operations and photo / video uploads

[1865] User takes a photo with their smartphone:

[1866] Users use devices such as smartphones and tablets to take photos and videos of events such as family birthday parties.

[1867] User opens the app:

[1868] A user opens a storytelling app and accesses a screen for uploading photos and videos. In the app, they select the "Create a new album" option and set a title, for example, "Birthday Party."

[1869] User selects photos and videos:

[1870] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1871] 2. Transferring data from the device to the server

[1872] The device prepares the data:

[1873] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and store the data in a temporary buffer.

[1874] The terminal splits and sends the data:

[1875] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[1876] 3. Data reception and storage by the server

[1877] Server receives data:

[1878] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[1879] Server reconstructs data:

[1880] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[1881] 4. Content analysis and video generation using generative AI and emotion engines

[1882] Generative AI analyzes content:

[1883] The server-based generative AI analyzes the metadata and content of stored photos and videos, using advanced technologies such as facial recognition, background analysis, and facial expression recognition.

[1884] The emotion engine analyzes emotions:

[1885] The generative AI has a built-in emotion engine that analyzes the emotions (e.g., smiling, sad, surprised, etc.) of the people in each photo or video.

[1886] Generative AI creates scenarios:

[1887] Based on the analysis results, it automatically creates a storytelling video scenario, taking into account emotional information, selecting appropriate scenes, and arranging them along the timeline.

[1888] Generative AI generates videos:

[1889] The AI ​​then edits the photos and videos according to the scenario, adding transition effects and background music to create a video. The generated video is then saved as a new file in cloud storage.

[1890] 5. User Feedback and Video Playback

[1891] The server notifies the user:

[1892] The server records the location of the generated video and notifies the user that a new video has been generated, either via a push notification on the device or a message within the app.

[1893] User plays video:

[1894] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[1895] For example, a user can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and generate a storytelling video. The generated video includes key moments, such as cutting the birthday cake and opening presents, edited with emotional information, allowing the user to play the video on their device and enjoy memories of the party.

[1896] The processing flow will be explained below.

[1897] Step 1:

[1898] The user takes a photo with their smartphone

[1899] Users use devices such as smartphones and tablets to take photos and videos of family birthday parties.

[1900] Step 2:

[1901] The user opens the app

[1902] The user opens the Storytelling app and selects "Create a new album" from the home screen. They enter the album title as "Birthday Party."

[1903] Step 3:

[1904] User selects photos and videos and starts uploading

[1905] Users can select the photos and videos they want to upload from their device's gallery, then tap "Upload."

[1906] Step 4:

[1907] The device prepares the data

[1908] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and the data will be temporarily stored in a buffer.

[1909] Step 5:

[1910] The terminal divides and transmits the data

[1911] A large video file is split into small chunks, each with a file ID and a checksum, and the device sends these chunks to the server sequentially.

[1912] Step 6:

[1913] The server receives the data

[1914] The server receives the transmitted data chunks, checks the checksums for errors, and if an error is found, requests that the chunk be resent.

[1915] Step 7:

[1916] The server reconstructs the data and stores it in cloud storage.

[1917] The received data chunks are reconstructed into the original photo and video files and saved in a "Birthday Party" folder in cloud storage.

[1918] Step 8:

[1919] The server calls the generative AI and emotion engine

[1920] The server passes the path information of the photos and videos in the cloud storage to the generation AI, which analyzes the metadata and content of the stored photos and videos, and the emotion engine analyzes the emotions (e.g., smile, sadness, surprise, etc.) of the people in each photo and video.

[1921] Step 9:

[1922] Generative AI and emotion engines analyze content

[1923] The generative AI analyzes photos and videos using technologies such as facial recognition, contextual analysis, and facial expression recognition. The emotion engine identifies changes in emotions and provides that information to the generative AI.

[1924] Step 10:

[1925] Generative AI creates scenarios

[1926] The generative AI uses information from the emotion engine to automatically create a storytelling video scenario, selecting key moments (e.g., cutting a birthday cake or opening a gift) and placing them on a timeline.

[1927] Step 11:

[1928] Generative AI generates videos

[1929] The AI ​​then edits the video based on the scenario, adding transition effects and background music to match the emotional changes, and the resulting video is saved as a new file in cloud storage.

[1930] Step 12:

[1931] The server notifies the user

[1932] The server records the location of the generated video and sends a push notification to the user's smartphone.

[1933] Step 13:

[1934] The user plays the video

[1935] Users can confirm the notification and play the generated video within the app, allowing them to easily enjoy their family memories.

[1936] Example 2

[1937] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1938] In recent years, the amount of photos and videos taken with digital devices has rapidly increased, creating a need for efficient classification and management of these images and videos, and for users to easily access them. Furthermore, to effectively preserve and share memories of special family events, simply storing photos and videos is insufficient; they must be reconstructed as storytelling content. Furthermore, there is a need for a method to automatically generate high-quality content with minimal user interaction.

[1939] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for taking images and videos using a terminal operable by a user and uploading the images and videos, means for receiving the uploaded images and videos on the server side and saving them in shared storage, generation AI means for analyzing the images and videos in the shared storage and classifying and organizing them based on specific dates or events, means for automatically generating story-style videos based on the images and videos organized by the generation AI means, and means for providing the automatically generated videos to the user's terminal and making them playable. This allows users to efficiently manage their captured content and enjoy their special memories by automatically converting them into high-quality storytelling-style videos.

[1940] A "terminal" is a device that can be operated by a user, and includes, for example, a smartphone, a tablet, or a PC.

[1941] "Images and Videos" means still and moving images stored in digital format that are used to record events and moments in everyday life.

[1942] "Upload" refers to the operation or process of sending data from a device to a server.

[1943] A "server" is a computer system that receives, stores, analyzes, and generates data and communicates with terminals via a network.

[1944] "Shared storage" refers to a storage device connected to a server, used to centrally manage and store data from multiple users.

[1945] "Analysis" refers to the process of performing specific processing on digital content to extract and classify its content and characteristics.

[1946] "Generative AI" refers to a system that uses artificial intelligence techniques to process and analyze data and automatically create specific deliverables.

[1947] "Narrative video" refers to a video that is not simply a series of images, but is edited and reconstructed based on a specific theme or scenario.

[1948] "Delivery" refers to the process of making the generated deliverables available to users.

[1949] "Play" refers to the action initiated to watch or listen to a stored video or audio file.

[1950] "Metadata" refers to auxiliary information added to images and videos (e.g., shooting date and time, location, device information, etc.), and is used when organizing and analyzing data.

[1951] This invention is a system that uploads images and videos from users' devices to a server, analyzes and organizes them, and automatically generates story-style videos. This system allows users to efficiently manage their captured content and enjoy their special memories as high-quality storytelling videos.

[1952] Hardware and Software Configuration

[1953] Terminal

[1954] Users take pictures and videos using devices such as smartphones and tablets. These devices have a camera function and a dedicated storytelling application installed. The application provides an operation screen and supports the selection and uploading of images and videos.

[1955] server

[1956] The server is a cloud-connected computer system that receives and stores content uploaded by users. It is equipped with a generative AI and emotion analysis engine, which are used to analyze and organize images and videos, generate scenarios, and generate videos.

[1957] Generative AI and Sentiment Analysis Engine

[1958] The generative AI includes advanced image processing technology, enabling facial recognition, background analysis, and facial expression recognition. The emotion analysis engine analyzes the emotions of people in each image or video, and generates a scenario based on that information. Based on the generated scenario, appropriate effects and music are added to create a narrative video.

[1959] Data processing and calculation

[1960] Adding Metadata

[1961] The device automatically adds metadata to the images and videos it takes, including the date and time the image was taken, the location, and device information. When a user uploads an image or video, this metadata is also sent to the server.

[1962] Data division and transmission

[1963] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[1964] Receiving and Reconstructing Data

[1965] The server receives each chunk and checks it for errors using a checksum. If an error is found, the server requests that the chunk be retransmitted. Correctly received data chunks are reconstructed into the original file and saved to shared storage.

[1966] Analyzing and organizing content

[1967] Generative AI analyzes the metadata and content of images and videos stored on shared storage. Based on the analysis results, content is categorized and organized based on specific dates and events. Specific people and scenes are identified through facial recognition, context analysis, and facial expression recognition.

[1968] Scenario creation and video generation

[1969] The generative AI automatically generates a story-style video scenario based on the analysis results. The emotion analysis engine reflects the emotional information in each scene and selects and adds effects and music in line with the scenario. Finally, the automatically generated video is saved in cloud storage that users can access.

[1970] Specific examples

[1971] For example, users can take photos and videos of their child's birthday party and upload them through the app. The generative AI and emotion engine then automatically analyze the content and create a storytelling video. The generated video includes key moments, such as cutting the birthday cake or opening presents, edited with emotional information, allowing users to play the video on their device and enjoy their memories of the party.

[1972] Example prompt sentence:

[1973] "Use photos and videos from a child's birthday party to generate a storytelling video incorporating emotional information."

[1974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1975] Step 1:

[1976] The user takes a photo

[1977] Users use devices such as smartphones and tablets to take pictures and videos using the device's built-in camera app, and the content they take is automatically saved in the device's gallery.

[1978] Input: Images and videos taken with the camera app

[1979] Output: Images and videos stored in the device's gallery

[1980] Step 2:

[1981] The user opens the app

[1982] A user launches the storytelling application, authenticates by logging in, selects the "Create a new album" option within the app, and sets a title, for example, "Birthday Party."

[1983] Enter: storytelling application

[1984] Output: Create a new album (e.g. "Birthday Party")

[1985] Step 3:

[1986] Users can select and upload images and videos

[1987] Users can select the images or videos they want to upload from their device's gallery and tap the "Upload" button in the app. The selected content will be added with metadata such as the date and time it was taken, location, and device information.

[1988] Input: User-selected images, videos, and metadata

[1989] Output: Images, videos and metadata ready to upload

[1990] Step 4:

[1991] The terminal divides and transmits the data

[1992] For large video files, the device splits the video into smaller chunks and attaches a file ID and checksum to each chunk, allowing for efficient and reliable data transfer over the network.

[1993] Input: Prepared images, videos and metadata

[1994] Output: Split data chunks, file ID, checksum

[1995] Step 5:

[1996] The server receives and checks the data

[1997] The server receives the data chunks sent by the device and checks for errors using the checksum. If an error is found, the server requests the device to retransmit the particular chunk to ensure error-free data.

[1998] Input: Data chunk, file ID, checksum

[1999] Output: Data chunk with errors corrected

[2000] Step 6:

[2001] The server reconstructs and saves the data

[2002] The server reconstructs correctly received data chunks into the original image or video file and stores it on shared storage.

[2003] Input: Data chunk with errors corrected

[2004] Output: Original image and video files saved in cloud storage

[2005] Step 7:

[2006] Generative AI analyzes content

[2007] The server-based generative AI analyzes the metadata and content of images and videos stored in cloud storage, identifying specific people and scenes through facial recognition, background analysis, and facial expression recognition.

[2008] Input: Images, videos and metadata from cloud storage

[2009] Output: Image and video analysis results (e.g., people, scenes, emotional information)

[2010] Step 8:

[2011] Generative AI creates scenarios

[2012] The generative AI automatically generates a story-style video scenario based on the analysis results, and uses an emotion analysis engine to select effects and music appropriate for each scene.

[2013] Input: Image and video analysis results, emotional information

[2014] Output: Narrative video scenario

[2015] Step 9:

[2016] Generative AI generates videos

[2017] The AI ​​then follows the user's scenario, edits images and videos, adds transition effects and background music, and generates a video, which is then saved in cloud storage.

[2018] Input: Narrative video scenario

[2019] Output: Generated story-style video file

[2020] Step 10:

[2021] The server notifies the user

[2022] The server records the location of the generated video and notifies the user that a new video is ready, either via a push notification on the device or an in-app message.

[2023] Input: Generated video file

[2024] Output: User notification

[2025] Step 11:

[2026] User plays video

[2027] The user will receive a notification and open the storytelling application to play the generated video, allowing them to enjoy their special memories as a storytelling video.

[2028] Input: Notification from the server

[2029] Output: Video played on the user's device

[2030] (Application example 2)

[2031] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2032] Previous applications only allowed users to individually store photos and videos, limiting their ability to visually enjoy specific events or experiences. Furthermore, there was a lack of effective ways to record and replay customer experiences in physical stores. As a result, users were unable to effectively reminisce about their memories and it was difficult to share their real-time experiences with others. Furthermore, the lack of an automatic generation function for storytelling videos that reflected the emotions and scenarios of each event required users to manually edit the content.

[2033] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2034] In this invention, the server includes means for taking photos and videos using a device operable by a user and uploading the photos and videos, means for receiving the uploaded photos and videos on the server side and storing them in cloud storage, generation AI means for analyzing the photos and videos in the cloud storage and classifying and organizing them based on specific dates or events, means for automatically generating storytelling videos based on the photos and videos organized by the generation AI means, means for providing the automatically generated videos to the user's device and making them playable, means for analyzing photos and videos taken of the user's experience in a physical store and generating scenarios of purchasing behavior and interactions, and means for editing the videos of the experience in the physical store based on the generated scenarios. This allows users to enjoy a specific event or experience in the store as an automatically edited video along with emotional information.

[2035] "Terminal" means an electronic device that can be operated by a user, such as a smartphone, tablet, or other computing device.

[2036] "Means for taking photos or videos" means having the ability to take images or videos using the device's built-in camera or a corresponding software application.

[2037] "Uploading means" refers to software or hardware capable of transmitting photos and videos to a server via the Internet.

[2038] "Server" refers to a remote computer system for data processing and management.

[2039] "Cloud storage" refers to a system for storing data on remote servers accessible via the Internet.

[2040] "Generative AI methods" refers to systems that use artificial intelligence techniques to analyze photos and videos and categorize and organize data based on specific days or events.

[2041] "Means for automatically generating storytelling videos" refers to a function that uses generative AI means to edit photos and videos and automatically create videos based on a scenario.

[2042] "Means for providing and making playable" refers to the functionality for transmitting the generated storytelling video to a user's device and enabling the user to watch the video.

[2043] "Means for analyzing photos and videos of experiences" refers to a system that has the ability to analyze images and videos taken by users in physical stores to understand their purchasing behavior and interactions.

[2044] "Means for generating scenarios of purchasing behavior and interactions" refers to a system that has the function of creating scenarios of users' behavior and experiences in stores based on the results of analysis.

[2045] "Means for editing recorded video" refers to a system that has the function of editing photos and videos according to the generated scenario and creating final video content.

[2046] MODE FOR CARRYING OUT THE INVENTION

[2047] System program for implementing application examples

[2048] User actions and photo / video uploads

[2049] Users take photos and videos of their in-store experience using a device such as a smartphone or tablet. After taking photos, users open the dedicated application and proceed to the upload screen. They create a new album, title it "Shopping Experience," select photos and videos from the gallery, and tap the upload button.

[2050] Transferring data from the device to the server

[2051] The device adds metadata to the selected photos and videos, such as the date and time of the photo, location, and device information, then splits the large video file into smaller chunks, attaches a file ID and checksum to each chunk, and sends these chunks to the server sequentially.

[2052] Data reception and storage by the server

[2053] The server receives the data chunks, checks them for errors using checksums, requests that the chunks be resent if an error is found, and reconstructs the original photo or video file from the received data chunks and stores it in cloud storage.

[2054] Content analysis and video generation with generative AI and emotion engines

[2055] The server-based generative AI analyzes the metadata and content of stored photos and videos. It uses advanced technologies such as facial recognition, background analysis, and facial expression recognition to analyze emotional information. Based on this information, the generative AI creates a scenario and generates a storytelling video that reflects specific events and experiences. Transition effects and background music are also automatically added to create the final video.

[2056] User feedback and video playback

[2057] The generated video is saved to cloud storage, the server records the location, and notifies the user that a video has been generated. The user can then confirm the notification and play the video within the app, allowing them to reminisce about their experience in the physical store.

[2058] Natural language explanation of program processing

[2059] Hardware and software usage

[2060] Hardware: Smartphones, cloud servers

[2061] Software: Dedicated applications, cloud storage, generative AI models (e.g., TensorFlow), sentiment analysis engines (e.g., IBM Watson)

[2062] Program processing flow

[2063] 1. User-taken and uploaded photos: When users upload photos or videos they have taken, metadata is automatically added within the app.

[2064] 2. Data transmission and reception: Data is sent from the device to the server in small chunks, which are then received, verified, and stored in cloud storage.

[2065] 3. Content Analysis and Video Generation: Generative AI analyzes each photo or video and automatically generates a scenario based on emotional information, then edits the video accordingly, adding background music and transition effects.

[2066] 4. User notification and playback: Once the generated video has been saved, the user will be notified and will be able to play the video within the app.

[2067] Specific examples

[2068] Users visit retail stores on launch days and film themselves tasting and trying on products in-store. They then upload this content through the app. Generative AI and emotion engines automatically create a "launch day experience video" and notify the user. Users can then play this video within the app to relive their experience in the retail store.

[2069] Prompt Sentence Examples

[2070] "Please enter a prompt for the generated AI:

[2071] To capture the product launch experience in a video, edit the video to focus on where the user is, their facial expressions, and their actions. Select key moments of tasting, trying on, and purchasing, and add effects and music to capture the emotional changes.

[2072] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2073] Step 1:

[2074] Users capture their in-store experiences

[2075] Users use their smartphones or tablets to take photos and videos of their in-store experiences.

[2076] Input: Photos and videos taken with your smartphone camera.

[2077] Output: The captured photos and videos will be saved in the device's gallery.

[2078] Specific actions: Launch the smartphone camera application and capture important moments in a physical store (e.g., trying on clothes, purchasing new items).

[2079] Step 2:

[2080] User opens the app, selects photos and videos to upload

[2081] Users open the dedicated application, create an album, select the photos and videos they have taken, and tap the upload button.

[2082] Input: Photos and videos taken, album title set by user (e.g. "Shopping Experience").

[2083] Output: The file ready to be uploaded.

[2084] What it does: Launch the app, go to the upload screen, set a title, select a photo or video from your gallery, then tap the "Upload" button.

[2085] Step 3:

[2086] The device adds metadata to the data, splits it into small chunks, and sends it to the server

[2087] Your device will add metadata to the selected photos and videos, such as the date and time they were taken, location, and device information, and will split large video files into smaller chunks for transmission.

[2088] Input: The photo or video file to upload.

[2089] Output: Data split into small chunks with metadata attached.

[2090] What happens: The application runs a routine to add metadata, splits the data into small chunks, adds a file ID and checksum to each chunk, and sends them to the server one by one.

[2091] Step 4:

[2092] The server receives and verifies the data chunks and stores them in cloud storage.

[2093] The server receives the transmitted data chunks, checks them for errors using checksums, and if an error is found, requests that the chunk be retransmitted. The error-free data chunks are reconstructed and stored in cloud storage.

[2094] Input: Data chunks.

[2095] Output: Complete photo and video files.

[2096] What happens: The server receives each data chunk, checks the checksum for errors, reconstructs the data and saves it to cloud storage if there are no errors, and requests a retransmission if there are errors.

[2097] Step 5:

[2098] The server analyzes the content using generative AI and emotion engine

[2099] The server-based generative AI analyzes the metadata and content of stored photos and videos to perform facial recognition, background analysis, and facial expression recognition. The emotion engine analyzes emotional information.

[2100] Input: Photos and videos in cloud storage.

[2101] Output: Analysis results for each content (face, background, facial expression, emotional information).

[2102] How it works: It activates a generative AI model to analyze the metadata and content of photos and videos, and then uses an emotion engine to analyze faces and expressions to extract emotional information.

[2103] Step 6:

[2104] Generative AI creates scenarios and generates storytelling videos

[2105] Based on the analysis, the generative AI creates a scenario and generates a video that reflects a specific event or experience, automatically adding transition effects and background music.

[2106] Input: Parsed content and sentiment information.

[2107] Output: The finished storytelling video file.

[2108] What it does: The generative AI creates a scenario, the editor engine edits photos and videos, adds transition effects and background music, and finally generates a video file.

[2109] Step 7:

[2110] The server notifies the user that the video has been generated

[2111] The generated video is stored in cloud storage, and the server records the location of the video and notifies the user that a new video has been generated.

[2112] Input: The finished video file.

[2113] Output: Notification to the user.

[2114] What happens: The server launches the notification system and sends a notification to the user in the form of a push notification or in-app message.

[2115] Step 8:

[2116] User plays a video within the app

[2117] Users can view the notification and play the video generated within the app, allowing them to relive their experience in the physical store in real time.

[2118] Input: Notification from the server.

[2119] Output: The video being played.

[2120] What happens: The user opens the notification and interacts with the in-app video player to play the video.

[2121] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2122] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2123] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2124] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2125] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2126] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2127] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2128] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2129] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2130] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2131] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2132] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2133] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2134] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2135] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2136] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2137] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2138] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2139] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2140] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2141] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2142] The following is further disclosed regarding the above embodiment.

[2143] (Claim 1)

[2144] a means for taking photos and videos using a user-operable terminal and uploading the photos and videos;

[2145] A means for receiving the uploaded photos and videos on a server side and storing them in a cloud storage;

[2146] a generating AI means for analyzing the photos and videos in the cloud storage and categorizing and organizing them based on specific days or events;

[2147] A means for automatically generating a storytelling video based on the photos and videos organized by the generating AI means;

[2148] means for providing the automatically generated video to a user's terminal for playback;

[2149] A system including:

[2150] (Claim 2)

[2151] 10. The system of claim 1, further comprising means for adding metadata (such as date and time of capture, location, device information, etc.) to a user's photos and videos and transmitting the metadata together with the photos and videos when they are uploaded.

[2152] (Claim 3)

[2153] The system of claim 1, wherein the generating AI means further comprises means for performing analysis including facial recognition, background analysis, and facial expression recognition, and creates a video scenario based thereon.

[2154] (Claim 4)

[2155] 10. The system of claim 1, further comprising: means for adding transition effects and background music to the storytelling type video generated by the generating AI means.

[2156] (Claim 5)

[2157] The system of claim 1, further comprising means for analyzing the content of the uploaded photos and videos on the server side and suggesting deletion of photos and videos deemed unnecessary.

[2158] "Example 1"

[2159] (Claim 1)

[2160] A means for taking photos and videos using a terminal operable by a user and uploading the photos and videos;

[2161] A means for adding metadata (such as the date and time of shooting, location, and device information) to the uploaded photos and videos on the terminal side, storing them in a buffer, and then transmitting them to a server;

[2162] A means for receiving the transmitted data on the server side, verifying the identity of the sender, checking the integrity of the data, and then storing the data in a cloud storage device;

[2163] A means for analyzing the photos and videos stored in the cloud storage device using a generative AI model to perform facial recognition, background analysis, facial expression recognition, etc., and to identify key moments of the event;

[2164] A means for automatically generating a storytelling video based on the analyzed results and adding transition effects and background music;

[2165] means for providing the automatically generated video to a user's terminal and making it playable;

[2166] A system including:

[2167] (Claim 2)

[2168] 10. The system of claim 1, further comprising means for adding metadata (such as date and time of taking, location, device information, etc.) to a user's photos and videos and transmitting the metadata together with the photos and videos when they are uploaded.

[2169] (Claim 3)

[2170] The system of claim 1, wherein the generative AI model further comprises means for performing analysis including facial recognition, background analysis, and facial expression recognition, and creates a video scenario based thereon.

[2171] "Application Example 1"

[2172] (Claim 1)

[2173] a means for taking photos and videos using a user-operable terminal and uploading the photos and videos;

[2174] A means for receiving the uploaded photos and videos on a server side and storing them in a cloud storage;

[2175] a generating AI means for analyzing the photos and videos in the cloud storage and categorizing and organizing them based on specific days or events;

[2176] A means for automatically generating a storytelling video based on the photos and videos organized by the generating AI means;

[2177] means for incorporating product introduction information and promotional information into the automatically generated video;

[2178] means for providing the automatically generated video to a user's terminal for playback;

[2179] A system including:

[2180] (Claim 2)

[2181] 10. The system of claim 1, further comprising means for adding metadata (such as date and time of capture, location, device information, etc.) to a user's photos and videos and transmitting the metadata together with the photos and videos when they are uploaded.

[2182] (Claim 3)

[2183] The system of claim 1, wherein the generating AI means further comprises means for performing analysis including facial recognition, background analysis, and facial expression recognition, and creates a video scenario based thereon.

[2184] "Example 2: Combining Emotion Engines"

[2185] (Claim 1)

[2186] A means for taking images and videos using a terminal operable by a user and uploading the images and videos;

[2187] A means for receiving the uploaded images and videos on the server side and storing them in a shared storage;

[2188] A generating AI means for analyzing images and videos in the shared storage and classifying and organizing them based on specific dates or events;

[2189] A means for automatically generating a story-style video based on the images and videos organized by the generating AI means;

[2190] means for providing the automatically generated video to a user's terminal and making it playable;

[2191] A system ...

Claims

1. a means for taking photos and videos using a user-operable terminal and uploading the photos and videos; A means for receiving the uploaded photos and videos on a server side and storing them in a cloud storage; a generating AI means for analyzing the photos and videos in the cloud storage and categorizing and organizing them based on specific days or events; A means for automatically generating a storytelling video based on the photos and videos organized by the generating AI means; means for providing the automatically generated video to a user's terminal for playback; A system including:

2. 10. The system of claim 1, further comprising means for adding metadata to a user's photos and videos and transmitting the metadata with the photos and videos when they are uploaded.

3. The system of claim 1 , wherein the generating AI means further comprises means for performing analysis including facial recognition, background analysis, and facial expression recognition, and creates a video scenario based thereon.

4. The system of claim 1 , further comprising: means for adding transition effects and background music to the storytelling type video generated by the generating AI means.

5. The system according to claim 1 , further comprising means for analyzing the contents of the uploaded photos and videos on the server side and suggesting the deletion of photos and videos deemed unnecessary.

Images