System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving and storing video data; means for analyzing the stored video data; means for detecting a particular risk based on the analysis; means for cutting or editing the detected risk location; means for generating edited video data; and means for providing the generated video data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional video posting process, it was common to manually edit videos after shooting them and then upload them. This editing process takes time and effort, and depending on the content, it often carries the risk of causing a backlash. Furthermore, checking and editing to avoid the risk of causing a backlash requires a high level of care and expertise, placing a heavy burden on ordinary video uploaders. There is a need for a system that can solve these issues and allow for efficient and safe video posting. [Means for solving the problem]

[0005] The present invention provides a video editing system that includes a means for receiving and saving video data, a means for analyzing the saved video data, a means for detecting specific risks based on the analysis, a means for cutting or editing the detected risk portions, a means for generating edited video data, and a means for providing the generated video data. In particular, the system includes a means for detecting specific risks from audio data using natural language processing technology and a means for detecting specific risks from video data using image recognition technology, enabling detailed content analysis and early detection of risks. The system also includes a means for inserting warning messages as captions into detected risk portions, a means for allowing users to preview the edited video data through a user interface, and a means for sharing or publishing the edited video data via a network selected by the user, providing an environment in which videos can be posted efficiently and safely.

[0006] "Video data" means video and audio data stored in digital format.

[0007] "Receiving" means receiving data through data transmission from a terminal to a server.

[0008] "Storage" means storing received data temporarily or permanently.

[0009] "Analysis" is the process of analyzing received and stored data to understand and interpret its content.

[0010] "Specific risks" refers to comments or videos that may cause outrage or inappropriate content.

[0011] "Detection" means discovering specific risks based on certain criteria based on the results of analysis.

[0012] "Cutting" means removing inappropriate parts and modifying the content without interrupting the flow of the video.

[0013] "Editing" means making changes to video data to create a format that suits your purpose.

[0014] "Generate" means to create a new video file after editing.

[0015] "Providing" means distributing the generated video data to users in a usable state.

[0016] "Natural language processing technology" is a technology that analyzes voice data as text and understands meaning and intent.

[0017] "Image recognition technology" is a technology that identifies and identifies objects and scenes in video data.

[0018] "Captions" are text messages that appear on top of videos and are used to convey information and warnings to viewers.

[0019] A "user interface" refers to the screen and operating means that allow a user to directly interact with and operate a system.

[0020] A "network" is an infrastructure that connects multiple devices and systems via communication and enables the exchange of data.

[0021] "Sharing" means using data jointly with others.

[0022] "Publishing" means making data accessible to the general public. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] This invention relates to a system that automatically analyzes and edits video data. Specifically, it is a system that uploads videos taken by users to a server, uses AI to detect comments or images that may pose a risk of causing an uproar, and performs appropriate editing. This system is a technology that significantly reduces the time users spend editing and reduces the risk of causing an uproar.

[0045] Overall system overview

[0046] 1. Upload your video

[0047] A user uploads a video taken using a terminal to a server.

[0048] The server stores the received video in storage.

[0049] 2. Video and audio analysis

[0050] The generative AI model on the server analyzes the video and audio of the uploaded video.

[0051] 3. Detecting the risk of online outrage

[0052] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[0053] 4. Cutting and Inserting Subtitles

[0054] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[0055] 5. Generate edited video

[0056] The server generates an edited video that reflects the cuts and inserted subtitles.

[0057] 6. View and download the video

[0058] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[0059] Explaining program processing in natural language

[0060] 1. Upload your video

[0061] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[0062] The server receives the video file and stores it in storage.

[0063] 2. Video and audio analysis

[0064] The server's generative AI model reads the video file and analyzes the video and audio data.

[0065] Voice data is converted into text using voice recognition technology within the server.

[0066] For video data, image recognition technology is used to recognize important objects and people in each scene.

[0067] 3. Detecting the risk of online outrage

[0068] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0069] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[0070] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[0071] 4. Cutting and Inserting Subtitles

[0072] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[0073] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[0074] 5. Generate edited video

[0075] The server applies the edits and generates a new video file.

[0076] Save the edited video file separately from the original video file.

[0077] 6. View and download the video

[0078] The user accesses the server on their device and previews the edited video.

[0079] The user checks the preview and, if there are no problems, downloads the edited video to their device.

[0080] Users can also post the edited video directly to social media or video sharing sites.

[0081] Specific examples

[0082] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Natural language processing technology is used to identify the insensitive remark. The remark is muted and a caption is inserted stating, "This remark has been deemed inappropriate."

[0083] This allows users to quickly create and publish videos with a low risk of causing controversy without editing. This system provides a safe and efficient video posting environment for both users and society.

[0084] The processing flow will be explained below.

[0085] Step 1: Select a video

[0086] The user selects a video file that was taken using the terminal.

[0087] Prepares to upload the video file selected by the user to the server.

[0088] Step 2: Upload your video

[0089] The video file selected by the user is sent to the server using the upload button.

[0090] The terminal transmits the data of the selected video file to the server.

[0091] The server stores the received video file in storage.

[0092] Step 3: Loading video data

[0093] The server loads the saved video file.

[0094] The server prepares to analyze the video data.

[0095] Step 4: Analyzing the audio data

[0096] The server's AI model separates the audio portion of the video.

[0097] The server uses speech recognition technology to convert the voice data into text.

[0098] Step 5: Analyze the video data

[0099] The server's AI model analyzes the video data.

[0100] The server uses image recognition technology to recognize important objects and people in each scene.

[0101] Step 6: Detecting the risk of a firestorm

[0102] The server's AI model analyzes the voice data that has been converted into text.

[0103] The server uses natural language processing technology to identify inappropriate comments and sensitive keywords.

[0104] The server's AI model analyzes the video data and uses image recognition technology to detect inappropriate footage.

[0105] Step 7: Marking risk areas

[0106] The server marks areas identified as at risk of a firestorm on the timeline.

[0107] Step 8: Placing cuts and silences

[0108] Silence or cut the appropriate time range for the server marked risk.

[0109] Step 9: Adding Text

[0110] The server inserts a warning message caption at the cut point.

[0111] The caption specifically includes a message such as "This comment has been deemed inappropriate."

[0112] Step 10: Apply edits and generate video

[0113] The server applies edits to the audio and video, including cuts and subtitles.

[0114] The server generates a new edited video file and stores it separately from the original video file.

[0115] Step 11: Preview your edited video

[0116] The user accesses the server on their device and previews the edited video.

[0117] The user checks the content of the video.

[0118] Step 12: Review and finalize your edits

[0119] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0120] The user can also request re-editing if necessary.

[0121] Step 13: Download and publish your edited video

[0122] The edited video that the user has confirmed is downloaded to the device.

[0123] Users can also upload edited videos directly to social media and video sharing sites.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] In recent years, opportunities for users to post videos they have taken to social networking sites and video sharing sites have increased, and this has also increased the risk of the video content causing outrage. Videos can contain inappropriate comments or sensitive content, which often leads to outrage and social criticism. Conventional methods require users to manually edit videos themselves, which is time-consuming and labor-intensive, and also carries the risk of oversight. Therefore, a system that automatically analyzes and edits video content is needed.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes means for receiving and saving video data, means for analyzing the video data, means for detecting specific risks based on the analysis, means for cutting or editing the detected risky parts and inserting warning messages, means for generating edited video data, and means for providing the generated video data. This enables users to automatically generate and edit videos with a low risk of causing controversy simply by uploading the videos.

[0129] "Video data" is a digital file containing video and audio captured by a user.

[0130] "Means for receiving and storing" refers to a mechanism by which the server receives video data sent by the user and stores it in a certain storage device.

[0131] "Means for analysis" refers to the technology that allows the server to analyze the video data stored therein and understand its content.

[0132] "Specific risks" are elements that could cause an outrage, such as inappropriate comments or sensitive footage contained in the video.

[0133] "Means of detection" are techniques that identify and detect specific risks from the analyzed data.

[0134] "Cutting or editing means" refers to techniques for muting or deleting detected risky parts or inserting warning messages.

[0135] A "warning message" is a warning message inserted at a specific risky point in a video.

[0136] The "means for generating edited video data" is a technology for generating a new video file in which the risk has been removed or corrected.

[0137] The "means for providing the generated video data" is a mechanism for providing the edited video data to the user.

[0138] "Natural language processing technology" is a technology for converting voice data into text format and analyzing its meaning.

[0139] "Image recognition technology" is a technology for identifying and analyzing important objects and people from each frame of video.

[0140] This invention relates to a system that automatically analyzes and edits video data. Specifically, this system uploads videos taken by users to a server, uses AI to detect comments or images that pose a risk of causing a firestorm, and performs appropriate editing. A specific embodiment of this system is described below.

[0141] Uploading videos

[0142] The user selects a video file taken on the device and uploads it to the server through the system's web interface. The server receives the video file and stores it in a specific directory (e.g., / uploads). At this time, the file's metadata (e.g., file name, upload time, user ID) is also stored in the database.

[0143] Video and audio analysis

[0144] The server detects a newly saved video file and calls a generative AI model (e.g., GPT-4, TensorFlow). The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it into text. It also uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. For example, it detects human faces and cars in each frame and records their coordinate information.

[0145] Detecting the risk of flame wars

[0146] The server's AI model receives audio and text data and video data as input and performs a flame war risk assessment. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords. The server also uses image recognition technology to identify images deemed inappropriate (e.g., violent scenes, discriminatory language). This process involves analyzing each frame and accumulating the results.

[0147] Cutting and Inserting Subtitles

[0148] The server marks identified risky sections on the timeline. Specific time information (e.g., the number of seconds the comment started and ended) is recorded. The server then mutes or cuts out the marked risky sections and inserts a warning message such as, "This comment has been deemed inappropriate."

[0149] Generate edited video

[0150] The server applies the edits to the timeline and generates a new video file. At this time, it determines the new file name and save location (e.g., / edited_videos directory) and saves it separately from the original video file. The server then updates the database to indicate that the edits are complete and prepares to notify the user.

[0151] View and download the video

[0152] The user accesses the system's web interface from their device and goes to the "Preview Edited Video" page. The server provides the user with a preview video link, allowing them to play and check the video. The user checks the edited video, and if there are no problems, clicks the download button to save the edited video to their device. They are also given the option to post the edited video directly to social media or video sharing sites.

[0153] Specific examples

[0154] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Using natural language processing technology, the insensitive remark is identified, muted, and a caption stating, "This remark has been deemed inappropriate" is inserted.

[0155] Example prompts to input to a generative AI model:

[0156] Prompt: Analyze the following text using natural language processing techniques to identify inappropriate statements.

[0157] Text: "I don't understand this culture at all and it doesn't interest me."

[0158] Output: Inappropriate remark detected. "I don't understand, and I'm not interested" is insensitive.

[0159] As a result, this system allows users to automatically generate and edit videos with a low risk of causing controversy simply by uploading them, enabling users to safely and efficiently publish videos without having to do any editing work.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] The user selects a video file taken using the device and uploads it to the server through the system's web interface. The input is the video file, and the output is the video file uploaded to the server. Specifically, the user clicks the "Select File" button on the browser, selects the video file, and then clicks the "Upload" button.

[0163] Step 2:

[0164] The server receives the video file and saves it in a specific directory (e.g., / uploads). The input for this operation is the video file from the user, and the output is the video file saved in storage. Specifically, this operation processes the HTTP POST request, receives the video file, saves it in storage on the server, and records the metadata in a database.

[0165] Step 3:

[0166] The server detects a newly saved video file and prepares to call a generative AI model (e.g., GPT-4, TensorFlow). The input is the path to the newly saved video file, and the output is the state where the analysis process is complete. Specifically, it looks at the path to the video file and adds it to the queue for analysis.

[0167] Step 4:

[0168] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it to text. The input is the video file, and the output is the extracted audio text data. Specifically, the process involves separating the audio track from the video file, sending it to the speech recognition API, and receiving the result as text.

[0169] Step 5:

[0170] The server uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. The input is the video file, and the output is data on the recognized objects and people. Specifically, it extracts each frame of the video as an image, inputs them into an image recognition model, and accumulates information on the detected objects.

[0171] Step 6:

[0172] The AI ​​model on the server receives voice, text, and video data as input and performs a flame war risk assessment. The input is text and video data, and the output is a list of specific risk areas. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords, and image recognition technology to identify videos deemed inappropriate.

[0173] Step 7:

[0174] The server marks identified risky areas on a timeline. The input is a list of identified risky areas, and the output is video data marked on the timeline. Specifically, it records the start and end times of each risky area and visualizes that information on the timeline.

[0175] Step 8:

[0176] The server mutes or cuts the marked risky parts and inserts a warning message such as "This comment has been deemed inappropriate." The input is the marked video data, and the output is the edited video data. Specifically, the server performs the muting or cutting process and sets the insertion position and text content of the caption.

[0177] Step 9:

[0178] The server applies the edits to the timeline and generates a new video file. The input is the edited timeline data, and the output is the new video file. Specifically, the server encodes and saves the new video file based on the timeline that reflects the edits.

[0179] Step 10:

[0180] The server updates the database to indicate that editing is complete and prepares to notify the user. The input is the creation information for the new video file, and the output is the status when notification preparation is complete. Specifically, this operation updates the database and prepares for email and system notifications to notify the user.

[0181] Step 11:

[0182] A user accesses the system's web interface from a terminal and navigates to the "Preview Edited Video" page. The input is a link to the edited video, and the output is a previewable state for the user. Specifically, the action is to click the preview link on the web page and play the video.

[0183] Step 12:

[0184] The user checks the edited video and, if there are no problems, clicks the download button to save the video to their device. They are also given the option to post it to a social networking site or video sharing site. The input is the user's confirmation action, and the output is the download of the video file or posting to the social networking site. Specifically, the user's click after confirmation downloads the video or posts it to the social networking site.

[0185] (Application example 1)

[0186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0187] Currently, when videos are distributed online, if they contain inappropriate comments or images, the task of manually identifying and editing them is extremely time-consuming and laborious. Furthermore, there are limits to human judgment, making it difficult to completely eliminate all risks. Posts on social media and video sharing platforms require a rapid response, but a delayed response can spark outrage, potentially having a major impact not only on individuals but also on companies and society as a whole. Therefore, a system that can efficiently and accurately eliminate risks is needed.

[0188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0189] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for inserting warning messages based on the analysis results, means for generating edited video data, and means for distributing the edited video data over a network. This allows users to automatically edit risky parts and insert warning messages, making it possible to safely and quickly release videos and reduce social risks.

[0190] "Means for receiving and storing video data" refers to a device or system that has the function of importing video files shot or created by users and storing them on a server or cloud storage.

[0191] "Means for analyzing stored video data" refers to a device or system that has the function of analyzing the contents of video files stored in storage and extracting or identifying data characteristics.

[0192] "Means for detecting specific risks based on analysis" refers to a device or system that has the functionality to identify high-risk elements, such as inappropriate comments or images, in a video based on the results of analysis.

[0193] "Means for cutting or editing detected risky parts" refers to a device or system that has the function of cutting identified risky parts, muting the audio, or inserting warning captions.

[0194] "Means for generating edited video data" refers to a device or system that has the function of generating a new video file that has been cut or edited, and outputting it with the editing results reflected.

[0195] "Means for providing generated video data" refers to a device or system that has the function of providing edited video files to users, making them available for download or uploading them directly to social networking sites or video sharing platforms.

[0196] "Means for inserting warning messages based on analysis results" refers to a device or system that has the function of inserting visual warning messages in the form of subtitles or the like into video for previously identified risk areas.

[0197] "Means for distributing edited video data over a network" refers to a device or system that has the function of distributing edited video files over the Internet or other networks and providing them to viewers.

[0198] This invention relates to a system that automatically analyzes, edits, and distributes video data. This system uses a generative AI model to detect inappropriate comments and images and then edits them appropriately to reduce risk.

[0199] System Overview and Configuration

[0200] A means of receiving and storing video data

[0201] Users upload videos they have taken using their devices to a server. This means has the function of receiving video files and saving them to cloud storage. Users can easily upload videos using a dedicated application.

[0202] A means of analyzing stored video data

[0203] The stored video data is analyzed by a generative AI model on the server, where audio data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and video data is analyzed using image recognition technology (e.g., Google Cloud Vision, OpenCV) to recognize important objects and people.

[0204] Analytics-based means of detecting specific risks

[0205] The generative AI model on the server identifies risks in the video based on the analysis results. At this stage, natural language processing techniques (e.g., BERT, GPT-4) are used for text analysis to detect inappropriate language and images. For example, if the video contains inappropriate language or violent scenes, those parts are identified.

[0206] A way to cut or edit detected risks

[0207] The generative AI model automatically mutes, cuts, or inserts warning messages for identified risky parts, making them clearly visible to users and preventing viewers from seeing inappropriate content.

[0208] A means of generating edited video data

[0209] Once editing is complete, the server generates a new video file, which is then saved with the edited results.

[0210] A means of providing generated video data

[0211] Users can access the server on their devices, preview the edited video, and if there are no problems, download the video or upload it directly to social media or video sharing platforms.

[0212] A means of inserting warning messages based on analysis results

[0213] When a risk is detected, the generative AI model inserts a visual warning message into the video as a caption, such as "This comment has been deemed inappropriate."

[0214] A means of distributing edited video data over a network

[0215] The edited video files are distributed via the Internet or other networks, allowing users to quickly distribute videos with risky parts automatically edited.

[0216] Specific examples

[0217] For example, consider a situation where a user shoots a video at a tourist spot and it contains inappropriate comments. The user uploads the video to a server using a dedicated application. A generative AI model on the server performs audio analysis and extracts the comments as text. Using natural language processing technology, the inappropriate comments are detected, muted, and a caption stating "This comment has been deemed inappropriate" is inserted. Furthermore, if inappropriate footage, such as a violent scene, is detected, the scene is cut.

[0218] Prompt Sentence Examples

[0219] "Detect inappropriate comments in the video and insert captions."

[0220] "Detect violent scenes and insert a warning message"

[0221] This allows users to safely and quickly publish videos without having to go through the trouble of editing out risky parts.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1:

[0224] The user uploads the video they have taken using their device to the server through a dedicated application. In this step, the video file selected on the device is sent to the server. The input is the video file, and the output is the video file saved on the server.

[0225] Step 2:

[0226] The server saves the uploaded video file in cloud storage. In this step, the video data is stored in storage and managed so that the user can access it again. The input is the video file sent in step 1, and the output is the video file saved in cloud storage.

[0227] Step 3:

[0228] The server reads the video and audio data using a generative AI model to analyze the stored video file. At this step, the video file is ready to be analyzed. The input is the video file in cloud storage, and the output is the data prepared for analysis.

[0229] Step 4:

[0230] The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The server generates text data from the voice data and prepares it for the next analysis step. The input is voice data, and the output is text data.

[0231] Step 5:

[0232] Image recognition technology (e.g., Google Cloud Vision, OpenCV) is used to recognize important objects and people in the video data. The server extracts specific features from the video data and saves them as analysis results. The input is video data, and the output is feature data.

[0233] Step 6:

[0234] The generative AI model detects specific risks based on text data and feature data. The server uses natural language processing technology (e.g., BERT, GPT-4) to identify inappropriate comments and videos. The input is text data and feature data, and the output is data with identified risks.

[0235] Step 7:

[0236] The server cuts or silences the identified risk sections and inserts warning captions as necessary. This process automates the editing process, eliminating the need for manual editing by the user. The input is the data with identified risks, and the output is the edited video data.

[0237] Step 8:

[0238] Edited video data is generated and saved as a new video file on the server. The server outputs the final editing results as a video file. The input is the edited data, and the output is the new video file.

[0239] Step 9:

[0240] The user accesses the server using a device to preview the edited video. This step allows the user to check the edited results. The input is the new video file, and the output is the preview screen for the user.

[0241] Step 10:

[0242] If the user is satisfied, they can download the edited video or upload it directly to social media or video sharing platforms. The user then finally publishes or saves the video. The input is the previewed video file, and the output is the downloaded file or uploaded content.

[0243] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0244] This invention relates to a system that automatically analyzes and edits video data, and in particular, it is a system that provides more advanced editing capabilities by combining an emotion engine that recognizes the user's emotions. This enables appropriate editing according to the user's emotions, further reducing the risk of online outrage and realizing the creation of high-quality videos.

[0245] Overall system overview

[0246] 1. Upload your video

[0247] A user uploads a video taken using a terminal to a server.

[0248] The server stores the received video in storage.

[0249] 2. Video and audio analysis

[0250] The generative AI model on the server analyzes the video and audio of the uploaded video.

[0251] 3. Detecting the risk of online outrage

[0252] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[0253] 4. User Emotion Recognition by Emotion Engine

[0254] The emotion engine in the server recognizes the user's emotions from the audio and video in the video.

[0255] It identifies emotions through voice analysis and highlights risk areas when certain emotions are detected.

[0256] Facial expression recognition technology is used to identify emotions in video, and appropriate subtitles are automatically inserted when a specific emotion is detected.

[0257] 5. Cutting and Inserting Subtitles

[0258] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[0259] 6. Generate edited video

[0260] The server generates an edited video that reflects the cuts and inserted subtitles.

[0261] 7. View and download videos

[0262] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[0263] Explaining program processing in natural language

[0264] 1. Upload your video

[0265] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[0266] The server receives the video file and stores it in storage.

[0267] 2. Video and audio analysis

[0268] The server's generative AI model reads the video file and analyzes the video and audio data.

[0269] Voice data is converted into text using voice recognition technology within the server.

[0270] For video data, image recognition technology is used to recognize important objects and people in each scene.

[0271] 3. Detecting the risk of online outrage

[0272] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0273] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[0274] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[0275] 4. User Emotion Recognition by Emotion Engine

[0276] The server's emotion engine identifies the user's emotion through voice analysis.

[0277] The server uses emotional information obtained from the voice data to highlight risk areas if a specific emotion is detected.

[0278] The server's emotion engine recognizes facial expressions in the video and identifies emotions.

[0279] Automatically insert appropriate captions based on emotion (e.g., "The user looked surprised here").

[0280] 5. Cutting and Inserting Subtitles

[0281] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[0282] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[0283] Appropriate subtitles are inserted based on the emotions identified by the emotion engine.

[0284] 6. Generate edited video

[0285] The server applies the edits and generates a new video file.

[0286] Save the edited video file separately from the original video file.

[0287] 7. View and download videos

[0288] The user accesses the server on their device and previews the edited video.

[0289] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0290] Once the user has finalized the edited video, they can download it to their device or upload it directly to social media or video sharing sites.

[0291] Specific examples

[0292] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[0293] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[0294] The processing flow will be explained below.

[0295] Step 1: Select a video

[0296] The user selects a video file that was taken using the terminal.

[0297] Prepares to upload the video file selected by the user to the server.

[0298] Step 2: Upload your video

[0299] The video file selected by the user is sent to the server using the upload button.

[0300] The terminal transmits the data of the selected video file to the server.

[0301] The server stores the received video file in storage.

[0302] Step 3: Loading video data

[0303] The server loads the saved video file.

[0304] The server prepares to analyze the video data.

[0305] Step 4: Analyzing the audio data

[0306] The server's AI model separates the audio portion of the video.

[0307] The server uses speech recognition technology to convert the voice data into text.

[0308] Step 5: Analyze the video data

[0309] The server's AI model analyzes the video data.

[0310] The server uses image recognition technology to recognize important objects and people in each scene.

[0311] Step 6: Recognizing user emotions with the emotion engine

[0312] The server's emotion engine identifies the user's emotion through voice analysis.

[0313] The server's emotion engine identifies the user's emotion using facial expression recognition technology from the video.

[0314] Step 7: Detecting the risk of a firestorm

[0315] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0316] The server uses natural language processing technology to analyze the text and identify inappropriate comments and sensitive keywords.

[0317] The server uses image recognition technology to analyze the video data and identify inappropriate content (e.g., violent scenes or discriminatory language).

[0318] Step 8: Marking risk and emotional areas

[0319] The server marks areas identified as at risk of a firestorm on the timeline.

[0320] The server also marks the emotional points identified by the emotion engine on the timeline.

[0321] Step 9: Place cuts and silences

[0322] The server will mute or cut the appropriate time range for the marked risk of a firestorm.

[0323] Step 10: Adding Text

[0324] The server will insert a warning message where the audio was cut or muted, for example, "This comment has been deemed inappropriate."

[0325] The server automatically inserts appropriate captions at the emotional points identified by the emotion engine. For example, it displays "The user is very excited here."

[0326] Step 11: Apply edits and generate new video file

[0327] The server applies edits to the audio and video, including cuts and subtitles.

[0328] The server generates a new edited video file and stores it separately from the original video file.

[0329] Step 12: Preview your edited video

[0330] The user accesses the server on their device and previews the edited video.

[0331] The user checks the video content in detail.

[0332] Step 13: Review and finalize your edits

[0333] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0334] The user can also request re-editing if necessary.

[0335] Step 14: Download and publish your edited video

[0336] The edited video that the user has confirmed is downloaded to the device.

[0337] Users can also upload edited videos directly to social media and video sharing sites.

[0338] Example 2

[0339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] In recent years, the spread of video sharing platforms has led to a rapid increase in video content. However, some of this content may contain inappropriate comments or images, posing a risk to viewers. Furthermore, while there is a demand for high-quality video editing that takes into account the emotions of viewers, conventional manual editing methods have the drawback of being too time-consuming and labor-intensive. The objective of this invention is to solve these problems and realize efficient and safe video editing.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0342] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for analyzing user emotions, means for cutting or editing detected risky parts and inserting appropriate captions based on the user emotions, means for generating edited video data, and means for providing the generated video data. This enables automatic detection and editing of risks of inappropriate content, and further enables appropriate editing according to the user emotions.

[0343] "Video data" refers to digital video and audio data, and is a media file that a user has filmed or collected.

[0344] "Means for receiving and storing" refers to devices or software for receiving data transmitted from outside and recording it on an internal or external recording medium.

[0345] "Means for analyzing" refers to devices or software that analyze received data and understand or classify the content of the data.

[0346] "Means for detecting specific risks" refers to devices or software that automatically identify inappropriate content or areas that pose a risk of causing an uproar based on the results of analysis.

[0347] "Means for analyzing user emotions" refers to devices or software that identify the user's emotions from the audio and video data and detect specific emotional states.

[0348] "Means for cutting or editing risky parts" refers to devices or software that remove or modify identified risky parts of data to make them safe.

[0349] "Means for inserting captions" refers to devices or software for displaying text information within edited video.

[0350] The "means for generating edited video data" refers to a device or software that generates a new file from the video after editing and stores or outputs it.

[0351] The "means for providing" refers to devices or software that make the generated video data accessible to users and enable downloading or streaming.

[0352] The present invention relates to a system for automatically analyzing and editing video data. Specific embodiments of the invention will be described below.

[0353] Uploading videos

[0354] The user uploads the video file taken on the device to the server. When the user clicks the upload button, the video file is sent to the server using the HTTP protocol. The server stores the received video file in cloud storage (for example, Amazon Web Services S3).

[0355] Video and audio analysis

[0356] The server analyzes the stored video files using a generative AI model (e.g., OpenAI's GPT-3). Audio data is converted to text using the Google Cloud Speech-to-Text API. For video data, OpenCV is used to analyze each frame and recognize important objects and people.

[0357] Detecting the risk of flame wars

[0358] The server uses the converted voice text and video data to detect the risk of a social media outrage using an AI model. The voice data is analyzed using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. The video data is also analyzed using machine learning models (e.g., TensorFlow) to detect inappropriate content such as violent scenes or discriminatory language.

[0359] Recognizing user emotions with an emotion engine

[0360] The server uses an emotion engine to analyze the user's emotions from audio and video data. Audio data is analyzed to identify specific emotions (e.g., anger, joy). For example, Microsoft Azure's Emotion API can be used to analyze facial expressions in the video to identify the emotion. When a specific emotion is detected, the relevant part is highlighted and an appropriate caption (e.g., "The user was surprised here") is automatically inserted.

[0361] Cutting and Inserting Subtitles

[0362] The server then edits the video based on the identified risky parts and emotions. Using tools such as FFmpeg, the server silences or cuts out risky parts. Furthermore, the server inserts warning messages and explanatory text based on the identified emotions.

[0363] Generate edited video

[0364] The server generates a new video file that reflects the edited content. The generated video file is saved separately from the original video file. This is done using tools such as FFmpeg.

[0365] View and download the video

[0366] The user can access the server to preview the edited video. If satisfied with the content, they can confirm the edited results. After confirming, the user can download the edited video to their device or upload it directly to a social networking site or video sharing site.

[0367] Specific examples

[0368] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[0369] Prompt Sentence Examples

[0370] "Please tell me about a system that analyzes video footage of an event, recognizes emotions, and edits the footage appropriately. For example, please explain how to insert appropriate captions in scenes where the user is excited, or cut out inappropriate scenes."

[0371] This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[0372] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0373] Step 1: Upload your video

[0374] Input: Video file taken by the user on the device

[0375] How it works: A user selects a video file using a dedicated application or web interface and clicks the "Upload" button. The device sends the video file to the server via an HTTP request.

[0376] Output: The server saves the received video files to cloud storage.

[0377] Step 2: Video and audio analysis

[0378] Input: Video file stored in cloud storage

[0379] How it works: The server reads the video file and extracts the video and audio data separately. The audio data is converted to text using the Google Cloud Speech-to-Text API. For the video data, OpenCV is used to analyze each frame and recognize important objects and people.

[0380] Output: Audio-text data and analyzed video data

[0381] Step 3: Detecting the risk of a firestorm

[0382] Input: Audio-text data and analyzed video data

[0383] How it works: The server's AI model analyzes the voice and text data using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. It also analyzes the video data using machine learning models (e.g., TensorFlow) to detect violent scenes and discriminatory language.

[0384] Output: List of inappropriate comments and inappropriate video scenes

[0385] Step 4: Recognizing user emotions with the emotion engine

[0386] Input: Audio and video data

[0387] How it works: The server's emotion engine analyzes the audio data and identifies specific emotions (e.g., anger, joy). It then uses Microsoft Azure's Emotion API to read facial expressions in the video and identify the emotion. If a specific emotion is detected, it is highlighted.

[0388] Output: Emotion recognition information and a list of highlighted emotion parts

[0389] Step 5: Cutting and adding captions

[0390] Input: List of inappropriate comments, list of inappropriate video scenes, emotion recognition information

[0391] What it does: The server uses an editing tool (e.g., FFmpeg) to mute or cut out identified risky parts. For inappropriate comments, it inserts a warning message caption (e.g., "This comment has been deemed inappropriate"). Based on the emotion identified by the emotion engine, it inserts an appropriate caption (e.g., "This is where the user was surprised").

[0392] Output: New video data reflecting the edited content

[0393] Step 6: Generate the edited video

[0394] Input: New video data reflecting edits

[0395] How it works: The server uses FFmpeg to apply the edits and generate a new video file, which is stored in cloud storage separately from the original video file.

[0396] Output: Final video data with full edits

[0397] Step 7: Check and download the video

[0398] Input: Final video data

[0399] How it works: The user accesses the server on their device and previews the edited video through a web interface. If they are satisfied with the edits, they confirm the edits by clicking the "Confirm" button. After confirming, the user can download the edited video to their device or upload it directly to social media or video sharing sites.

[0400] Output: Edited video ready for download or upload link

[0401] (Application example 2)

[0402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] The difficulty of automatically analyzing and editing video data lies in the difficulty of appropriately reflecting user emotions and the risk of a viral outbreak. In particular, there is still room for development in technology that recognizes user emotions and reflects them in editing. In today's world, where a large number of video data are posted, high-quality and safe video editing is required.

[0404] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for identifying the emotional state using an emotion engine that recognizes the user's emotions, means for cutting or editing based on the detected risky parts or emotional state, means for generating edited video data, and means for providing the generated video data. This enables high-quality and safe video editing that appropriately reflects the user's emotions and the risk of a controversy.

[0405] The "means for receiving and saving video data" is a mechanism for uploading videos taken by users to a server and saving them in storage within the server.

[0406] The "means for analyzing stored video data" is a mechanism for reading stored video data and analyzing the video and audio.

[0407] "Means for detecting specific risks based on analysis" refers to a mechanism for identifying inappropriate content or areas with a high risk of causing a backlash based on the results of analysis.

[0408] The "means for identifying an emotional state using an emotion engine that recognizes a user's emotion" is a mechanism for identifying a user's emotion from audio and video within a video using an emotion engine.

[0409] The "means for cutting or editing based on detected risky parts or emotional state" is a mechanism for cutting video or inserting subtitles based on detected risky parts or the user's emotional state.

[0410] The "means for generating edited video data" is a mechanism for generating a new video file that reflects cuts and edits.

[0411] The "means for providing the generated video data" is a mechanism for providing the edited video to users so that they can download or share it.

[0412] This invention provides an automatic analysis and editing system for video data. The system operates as follows.

[0413] First, the user uploads a video they have taken using their device to the server. The server then stores the received video in storage. Next, the server uses a generative AI model to analyze the video and audio of the uploaded video. The server then uses voice analysis technology to convert the audio data into text, and uses image recognition technology to recognize important objects and people in the video data.

[0414] The emotion engine in the server recognizes the user's emotions from voice and video. The server identifies emotions through voice analysis and highlights risky areas when a specific emotion is detected. It also uses facial expression recognition technology to identify emotions in video and automatically inserts appropriate subtitles.

[0415] Based on the analysis results, the server's AI model detects specific risks. It uses natural language processing technology to identify inappropriate comments and sensitive keywords from the audio data, and image recognition technology to detect inappropriate video. If a risky part is detected, the server cuts or silences the audio and inserts a warning message.

[0416] Finally, the server generates the edited video data. The generated video file is saved separately from the original video file, and the user can access the server on their device to check the edited video. If the user checks it and there are no problems, they can download the edited video or post it to social media, etc.

[0417] This system enables high-quality, safe video editing that appropriately reflects user emotions and the risk of online outrage.

[0418] As a specific example, consider a case where a user films an event and the video contains an exciting or moving scene. When the video is uploaded to the system, a generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. Furthermore, an emotion engine analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[0419] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[0420] The hardware used is a high-performance CPU or GPU, data storage, and the software used is Python, MoviePy, an emotion recognition module, and a flame risk detection module.

[0421] An example prompt for a generative AI model is:

[0422] Create a video editing program that recognizes the user's emotions and automatically detects and edits scenes that may pose a risk of causing controversy. Use Python and the MoviePy library, and also use the emotion recognition module and the controversy risk detection module. Finally, create a program that saves the edited video as a new file.

[0423] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0424] Step 1:

[0425] The user uses a device to shoot video data and uploads the video to the server. The input is a video file (e.g., .mp4 format), and the server receives this video file and saves it in storage. Specifically, the user selects a video on the file selection screen and clicks the upload button.

[0426] Step 2:

[0427] The server analyzes the stored video data using a generative AI model to analyze the video and audio data. The input is the stored video file, and the output is the text conversion result of the audio data and the analysis result of the video data. Specifically, it uses voice recognition technology to convert the audio data into text, and image recognition technology to identify important objects and people in the video.

[0428] Step 3:

[0429] The emotion engine in the server recognizes the user's emotions from the audio and video in the video. The input is the text conversion results and video analysis results obtained in step 2, and the output is the identification of the emotional state for each scene. Specifically, the emotion engine classifies the user's emotions based on the analysis results and identifies the emotions in each scene.

[0430] Step 4:

[0431] The server detects specific risks based on the analysis results. The input is the text conversion results of the audio data from step 2 and the video analysis results, and the output is the identification of risk areas. Specifically, it uses natural language processing technology and image recognition technology to identify inappropriate remarks and videos.

[0432] Step 5:

[0433] The server cuts or edits the video based on the detected risky parts or emotional state. The input is the emotional state identification result from step 3 and the risky part identification result from step 4, and the output is edited video data based on the editing instructions. Specifically, it mutes or cuts the identified risky parts, and inserts captions according to the detected emotions.

[0434] Step 6:

[0435] The server generates edited video data. The input is the edited video data based on the editing instructions in step 5, and the output is the final edited video file. Specifically, the server reconstructs the video file according to the editing instructions and generates a new video file.

[0436] Step 7:

[0437] The user uses their device to check the edited video that has been generated, and if necessary, downloads it or posts it to social media. The input is the final edited video file, and the output is the published or saved state of the video after the user has checked it. Specifically, the user accesses the server to preview the edited video, and after checking it, presses the download button or post button.

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0441] [Second embodiment]

[0442] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0443] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0448] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0449] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0450] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0453] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0454] This invention relates to a system that automatically analyzes and edits video data. Specifically, it is a system that uploads videos taken by users to a server, uses AI to detect comments or images that may pose a risk of causing an uproar, and performs appropriate editing. This system is a technology that significantly reduces the time users spend editing and reduces the risk of causing an uproar.

[0455] Overall system overview

[0456] 1. Upload your video

[0457] A user uploads a video taken using a terminal to a server.

[0458] The server stores the received video in storage.

[0459] 2. Video and audio analysis

[0460] The generative AI model on the server analyzes the video and audio of the uploaded video.

[0461] 3. Detecting the risk of online outrage

[0462] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[0463] 4. Cutting and Inserting Subtitles

[0464] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[0465] 5. Generate edited video

[0466] The server generates an edited video that reflects the cuts and inserted subtitles.

[0467] 6. View and download the video

[0468] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[0469] Explaining program processing in natural language

[0470] 1. Upload your video

[0471] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[0472] The server receives the video file and stores it in storage.

[0473] 2. Video and audio analysis

[0474] The server's generative AI model reads the video file and analyzes the video and audio data.

[0475] Voice data is converted into text using voice recognition technology within the server.

[0476] For video data, image recognition technology is used to recognize important objects and people in each scene.

[0477] 3. Detecting the risk of online outrage

[0478] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0479] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[0480] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[0481] 4. Cutting and Inserting Subtitles

[0482] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[0483] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[0484] 5. Generate edited video

[0485] The server applies the edits and generates a new video file.

[0486] Save the edited video file separately from the original video file.

[0487] 6. View and download the video

[0488] The user accesses the server on their device and previews the edited video.

[0489] The user checks the preview and, if there are no problems, downloads the edited video to their device.

[0490] Users can also post the edited video directly to social media or video sharing sites.

[0491] Specific examples

[0492] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Natural language processing technology is used to identify the insensitive remark. The remark is muted and a caption is inserted stating, "This remark has been deemed inappropriate."

[0493] This allows users to quickly create and publish videos with a low risk of causing controversy without editing. This system provides a safe and efficient video posting environment for both users and society.

[0494] The processing flow will be explained below.

[0495] Step 1: Select a video

[0496] The user selects a video file that was taken using the terminal.

[0497] Prepares to upload the video file selected by the user to the server.

[0498] Step 2: Upload your video

[0499] The video file selected by the user is sent to the server using the upload button.

[0500] The terminal transmits the data of the selected video file to the server.

[0501] The server stores the received video file in storage.

[0502] Step 3: Loading video data

[0503] The server loads the saved video file.

[0504] The server prepares to analyze the video data.

[0505] Step 4: Analyzing the audio data

[0506] The server's AI model separates the audio portion of the video.

[0507] The server uses speech recognition technology to convert the voice data into text.

[0508] Step 5: Analyze the video data

[0509] The server's AI model analyzes the video data.

[0510] The server uses image recognition technology to recognize important objects and people in each scene.

[0511] Step 6: Detecting the risk of a firestorm

[0512] The server's AI model analyzes the voice data that has been converted into text.

[0513] The server uses natural language processing technology to identify inappropriate comments and sensitive keywords.

[0514] The server's AI model analyzes the video data and uses image recognition technology to detect inappropriate footage.

[0515] Step 7: Marking risk areas

[0516] The server marks areas identified as at risk of a firestorm on the timeline.

[0517] Step 8: Placing cuts and silences

[0518] Silence or cut the appropriate time range for the server marked risk.

[0519] Step 9: Adding Text

[0520] The server inserts a warning message caption at the cut point.

[0521] The caption specifically includes a message such as "This comment has been deemed inappropriate."

[0522] Step 10: Apply edits and generate video

[0523] The server applies edits to the audio and video, including cuts and subtitles.

[0524] The server generates a new edited video file and stores it separately from the original video file.

[0525] Step 11: Preview your edited video

[0526] The user accesses the server on their device and previews the edited video.

[0527] The user checks the content of the video.

[0528] Step 12: Review and finalize your edits

[0529] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0530] The user can also request re-editing if necessary.

[0531] Step 13: Download and publish your edited video

[0532] The edited video that the user has confirmed is downloaded to the device.

[0533] Users can also upload edited videos directly to social media and video sharing sites.

[0534] Example 1

[0535] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0536] In recent years, opportunities for users to post videos they have taken to social networking sites and video sharing sites have increased, and this has also increased the risk of the video content causing outrage. Videos can contain inappropriate comments or sensitive content, which often leads to outrage and social criticism. Conventional methods require users to manually edit videos themselves, which is time-consuming and labor-intensive, and also carries the risk of oversight. Therefore, a system that automatically analyzes and edits video content is needed.

[0537] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0538] In this invention, the server includes means for receiving and saving video data, means for analyzing the video data, means for detecting specific risks based on the analysis, means for cutting or editing the detected risky parts and inserting warning messages, means for generating edited video data, and means for providing the generated video data. This enables users to automatically generate and edit videos with a low risk of causing controversy simply by uploading the videos.

[0539] "Video data" is a digital file containing video and audio captured by a user.

[0540] "Means for receiving and storing" refers to a mechanism by which the server receives video data sent by the user and stores it in a certain storage device.

[0541] "Means for analysis" refers to the technology that allows the server to analyze the video data stored therein and understand its content.

[0542] "Specific risks" are elements that could cause an outrage, such as inappropriate comments or sensitive footage contained in the video.

[0543] "Means of detection" are techniques that identify and detect specific risks from the analyzed data.

[0544] "Cutting or editing means" refers to techniques for muting or deleting detected risky parts or inserting warning messages.

[0545] A "warning message" is a warning message inserted at a specific risky point in a video.

[0546] The "means for generating edited video data" is a technology for generating a new video file in which the risk has been removed or corrected.

[0547] The "means for providing the generated video data" is a mechanism for providing the edited video data to the user.

[0548] "Natural language processing technology" is a technology for converting voice data into text format and analyzing its meaning.

[0549] "Image recognition technology" is a technology for identifying and analyzing important objects and people from each frame of video.

[0550] This invention relates to a system that automatically analyzes and edits video data. Specifically, this system uploads videos taken by users to a server, uses AI to detect comments or images that pose a risk of causing a firestorm, and performs appropriate editing. A specific embodiment of this system is described below.

[0551] Uploading videos

[0552] The user selects a video file taken on the device and uploads it to the server through the system's web interface. The server receives the video file and stores it in a specific directory (e.g., / uploads). At this time, the file's metadata (e.g., file name, upload time, user ID) is also stored in the database.

[0553] Video and audio analysis

[0554] The server detects a newly saved video file and calls a generative AI model (e.g., GPT-4, TensorFlow). The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it into text. It also uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. For example, it detects human faces and cars in each frame and records their coordinate information.

[0555] Detecting the risk of flame wars

[0556] The server's AI model receives audio and text data and video data as input and performs a flame war risk assessment. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords. The server also uses image recognition technology to identify images deemed inappropriate (e.g., violent scenes, discriminatory language). This process involves analyzing each frame and accumulating the results.

[0557] Cutting and Inserting Subtitles

[0558] The server marks identified risky sections on the timeline. Specific time information (e.g., the number of seconds the comment started and ended) is recorded. The server then mutes or cuts out the marked risky sections and inserts a warning message such as, "This comment has been deemed inappropriate."

[0559] Generate edited video

[0560] The server applies the edits to the timeline and generates a new video file. At this time, it determines the new file name and save location (e.g., / edited_videos directory) and saves it separately from the original video file. The server then updates the database to indicate that the edits are complete and prepares to notify the user.

[0561] View and download the video

[0562] The user accesses the system's web interface from their device and goes to the "Preview Edited Video" page. The server provides the user with a preview video link, allowing them to play and check the video. The user checks the edited video, and if there are no problems, clicks the download button to save the edited video to their device. They are also given the option to post the edited video directly to social media or video sharing sites.

[0563] Specific examples

[0564] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Using natural language processing technology, the insensitive remark is identified, muted, and a caption stating, "This remark has been deemed inappropriate" is inserted.

[0565] Example prompts to input to a generative AI model:

[0566] Prompt: Analyze the following text using natural language processing techniques to identify inappropriate statements.

[0567] Text: "I don't understand this culture at all and it doesn't interest me."

[0568] Output: Inappropriate remark detected. "I don't understand, and I'm not interested" is insensitive.

[0569] As a result, this system allows users to automatically generate and edit videos with a low risk of causing controversy simply by uploading them, enabling users to safely and efficiently publish videos without having to do any editing work.

[0570] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0571] Step 1:

[0572] The user selects a video file taken using the device and uploads it to the server through the system's web interface. The input is the video file, and the output is the video file uploaded to the server. Specifically, the user clicks the "Select File" button on the browser, selects the video file, and then clicks the "Upload" button.

[0573] Step 2:

[0574] The server receives the video file and saves it in a specific directory (e.g., / uploads). The input for this operation is the video file from the user, and the output is the video file saved in storage. Specifically, this operation processes the HTTP POST request, receives the video file, saves it in storage on the server, and records the metadata in a database.

[0575] Step 3:

[0576] The server detects a newly saved video file and prepares to call a generative AI model (e.g., GPT-4, TensorFlow). The input is the path to the newly saved video file, and the output is the state where the analysis process is complete. Specifically, it looks at the path to the video file and adds it to the queue for analysis.

[0577] Step 4:

[0578] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it to text. The input is the video file, and the output is the extracted audio text data. Specifically, the process involves separating the audio track from the video file, sending it to the speech recognition API, and receiving the result as text.

[0579] Step 5:

[0580] The server uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. The input is the video file, and the output is data on the recognized objects and people. Specifically, it extracts each frame of the video as an image, inputs them into an image recognition model, and accumulates information on the detected objects.

[0581] Step 6:

[0582] The AI ​​model on the server receives voice, text, and video data as input and performs a flame war risk assessment. The input is text and video data, and the output is a list of specific risk areas. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords, and image recognition technology to identify videos deemed inappropriate.

[0583] Step 7:

[0584] The server marks identified risky areas on a timeline. The input is a list of identified risky areas, and the output is video data marked on the timeline. Specifically, it records the start and end times of each risky area and visualizes that information on the timeline.

[0585] Step 8:

[0586] The server mutes or cuts the marked risky parts and inserts a warning message such as "This comment has been deemed inappropriate." The input is the marked video data, and the output is the edited video data. Specifically, the server performs the muting or cutting process and sets the insertion position and text content of the caption.

[0587] Step 9:

[0588] The server applies the edits to the timeline and generates a new video file. The input is the edited timeline data, and the output is the new video file. Specifically, the server encodes and saves the new video file based on the timeline that reflects the edits.

[0589] Step 10:

[0590] The server updates the database to indicate that editing is complete and prepares to notify the user. The input is the creation information for the new video file, and the output is the status when notification preparation is complete. Specifically, this operation updates the database and prepares for email and system notifications to notify the user.

[0591] Step 11:

[0592] A user accesses the system's web interface from a terminal and navigates to the "Preview Edited Video" page. The input is a link to the edited video, and the output is a previewable state for the user. Specifically, the action is to click the preview link on the web page and play the video.

[0593] Step 12:

[0594] The user checks the edited video and, if there are no problems, clicks the download button to save the video to their device. They are also given the option to post it to a social networking site or video sharing site. The input is the user's confirmation action, and the output is the download of the video file or posting to the social networking site. Specifically, the user's click after confirmation downloads the video or posts it to the social networking site.

[0595] (Application example 1)

[0596] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0597] Currently, when videos are distributed online, if they contain inappropriate comments or images, the task of manually identifying and editing them is extremely time-consuming and laborious. Furthermore, there are limits to human judgment, making it difficult to completely eliminate all risks. Posts on social media and video sharing platforms require a rapid response, but a delayed response can spark outrage, potentially having a major impact not only on individuals but also on companies and society as a whole. Therefore, a system that can efficiently and accurately eliminate risks is needed.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0599] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for inserting warning messages based on the analysis results, means for generating edited video data, and means for distributing the edited video data over a network. This allows users to automatically edit risky parts and insert warning messages, making it possible to safely and quickly release videos and reduce social risks.

[0600] "Means for receiving and storing video data" refers to a device or system that has the function of importing video files shot or created by users and storing them on a server or cloud storage.

[0601] "Means for analyzing stored video data" refers to a device or system that has the function of analyzing the contents of video files stored in storage and extracting or identifying data characteristics.

[0602] "Means for detecting specific risks based on analysis" refers to a device or system that has the functionality to identify high-risk elements, such as inappropriate comments or images, in a video based on the results of analysis.

[0603] "Means for cutting or editing detected risky parts" refers to a device or system that has the function of cutting identified risky parts, muting the audio, or inserting warning captions.

[0604] "Means for generating edited video data" refers to a device or system that has the function of generating a new video file that has been cut or edited, and outputting it with the editing results reflected.

[0605] "Means for providing generated video data" refers to a device or system that has the function of providing edited video files to users, making them available for download or uploading them directly to social networking sites or video sharing platforms.

[0606] "Means for inserting warning messages based on analysis results" refers to a device or system that has the function of inserting visual warning messages in the form of subtitles or the like into video for previously identified risk areas.

[0607] "Means for distributing edited video data over a network" refers to a device or system that has the function of distributing edited video files over the Internet or other networks and providing them to viewers.

[0608] This invention relates to a system that automatically analyzes, edits, and distributes video data. This system uses a generative AI model to detect inappropriate comments and images and then edits them appropriately to reduce risk.

[0609] System Overview and Configuration

[0610] A means of receiving and storing video data

[0611] Users upload videos they have taken using their devices to a server. This means has the function of receiving video files and saving them to cloud storage. Users can easily upload videos using a dedicated application.

[0612] A means of analyzing stored video data

[0613] The stored video data is analyzed by a generative AI model on the server, where audio data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and video data is analyzed using image recognition technology (e.g., Google Cloud Vision, OpenCV) to recognize important objects and people.

[0614] Analytics-based means of detecting specific risks

[0615] The generative AI model on the server identifies risks in the video based on the analysis results. At this stage, natural language processing techniques (e.g., BERT, GPT-4) are used for text analysis to detect inappropriate language and images. For example, if the video contains inappropriate language or violent scenes, those parts are identified.

[0616] A way to cut or edit detected risks

[0617] The generative AI model automatically mutes, cuts, or inserts warning messages for identified risky parts, making them clearly visible to users and preventing viewers from seeing inappropriate content.

[0618] A means of generating edited video data

[0619] Once editing is complete, the server generates a new video file, which is then saved with the edited results.

[0620] A means of providing generated video data

[0621] Users can access the server on their devices, preview the edited video, and if there are no problems, download the video or upload it directly to social media or video sharing platforms.

[0622] A means of inserting warning messages based on analysis results

[0623] When a risk is detected, the generative AI model inserts a visual warning message into the video as a caption, such as "This comment has been deemed inappropriate."

[0624] A means of distributing edited video data over a network

[0625] The edited video files are distributed via the Internet or other networks, allowing users to quickly distribute videos with risky parts automatically edited.

[0626] Specific examples

[0627] For example, consider a situation where a user shoots a video at a tourist spot and it contains inappropriate comments. The user uploads the video to a server using a dedicated application. A generative AI model on the server performs audio analysis and extracts the comments as text. Using natural language processing technology, the inappropriate comments are detected, muted, and a caption stating "This comment has been deemed inappropriate" is inserted. Furthermore, if inappropriate footage, such as a violent scene, is detected, the scene is cut.

[0628] Prompt Sentence Examples

[0629] "Detect inappropriate comments in the video and insert captions."

[0630] "Detect violent scenes and insert a warning message"

[0631] This allows users to safely and quickly publish videos without having to go through the trouble of editing out risky parts.

[0632] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0633] Step 1:

[0634] The user uploads the video they have taken using their device to the server through a dedicated application. In this step, the video file selected on the device is sent to the server. The input is the video file, and the output is the video file saved on the server.

[0635] Step 2:

[0636] The server saves the uploaded video file in cloud storage. In this step, the video data is stored in storage and managed so that the user can access it again. The input is the video file sent in step 1, and the output is the video file saved in cloud storage.

[0637] Step 3:

[0638] The server reads the video and audio data using a generative AI model to analyze the stored video file. At this step, the video file is ready to be analyzed. The input is the video file in cloud storage, and the output is the data prepared for analysis.

[0639] Step 4:

[0640] The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The server generates text data from the voice data and prepares it for the next analysis step. The input is voice data, and the output is text data.

[0641] Step 5:

[0642] Image recognition technology (e.g., Google Cloud Vision, OpenCV) is used to recognize important objects and people in the video data. The server extracts specific features from the video data and saves them as analysis results. The input is video data, and the output is feature data.

[0643] Step 6:

[0644] The generative AI model detects specific risks based on text data and feature data. The server uses natural language processing technology (e.g., BERT, GPT-4) to identify inappropriate comments and videos. The input is text data and feature data, and the output is data with identified risks.

[0645] Step 7:

[0646] The server cuts or silences the identified risk sections and inserts warning captions as necessary. This process automates the editing process, eliminating the need for manual editing by the user. The input is the data with identified risks, and the output is the edited video data.

[0647] Step 8:

[0648] Edited video data is generated and saved as a new video file on the server. The server outputs the final editing results as a video file. The input is the edited data, and the output is the new video file.

[0649] Step 9:

[0650] The user accesses the server using a device to preview the edited video. This step allows the user to check the edited results. The input is the new video file, and the output is the preview screen for the user.

[0651] Step 10:

[0652] If the user is satisfied, they can download the edited video or upload it directly to social media or video sharing platforms. The user then finally publishes or saves the video. The input is the previewed video file, and the output is the downloaded file or uploaded content.

[0653] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0654] This invention relates to a system that automatically analyzes and edits video data, and in particular, it is a system that provides more advanced editing capabilities by combining an emotion engine that recognizes the user's emotions. This enables appropriate editing according to the user's emotions, further reducing the risk of online outrage and realizing the creation of high-quality videos.

[0655] Overall system overview

[0656] 1. Upload your video

[0657] A user uploads a video taken using a terminal to a server.

[0658] The server stores the received video in storage.

[0659] 2. Video and audio analysis

[0660] The generative AI model on the server analyzes the video and audio of the uploaded video.

[0661] 3. Detecting the risk of online outrage

[0662] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[0663] 4. User Emotion Recognition by Emotion Engine

[0664] The emotion engine in the server recognizes the user's emotions from the audio and video in the video.

[0665] It identifies emotions through voice analysis and highlights risk areas when certain emotions are detected.

[0666] Facial expression recognition technology is used to identify emotions in video, and appropriate subtitles are automatically inserted when a specific emotion is detected.

[0667] 5. Cutting and Inserting Subtitles

[0668] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[0669] 6. Generate edited video

[0670] The server generates an edited video that reflects the cuts and inserted subtitles.

[0671] 7. View and download videos

[0672] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[0673] Explaining program processing in natural language

[0674] 1. Upload your video

[0675] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[0676] The server receives the video file and stores it in storage.

[0677] 2. Video and audio analysis

[0678] The server's generative AI model reads the video file and analyzes the video and audio data.

[0679] Voice data is converted into text using voice recognition technology within the server.

[0680] For video data, image recognition technology is used to recognize important objects and people in each scene.

[0681] 3. Detecting the risk of online outrage

[0682] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0683] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[0684] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[0685] 4. User Emotion Recognition by Emotion Engine

[0686] The server's emotion engine identifies the user's emotion through voice analysis.

[0687] The server uses emotional information obtained from the voice data to highlight risk areas if a specific emotion is detected.

[0688] The server's emotion engine recognizes facial expressions in the video and identifies emotions.

[0689] Automatically insert appropriate captions based on emotion (e.g., "The user looked surprised here").

[0690] 5. Cutting and Inserting Subtitles

[0691] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[0692] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[0693] Appropriate subtitles are inserted based on the emotions identified by the emotion engine.

[0694] 6. Generate edited video

[0695] The server applies the edits and generates a new video file.

[0696] Save the edited video file separately from the original video file.

[0697] 7. View and download videos

[0698] The user accesses the server on their device and previews the edited video.

[0699] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0700] Once the user has finalized the edited video, they can download it to their device or upload it directly to social media or video sharing sites.

[0701] Specific examples

[0702] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[0703] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[0704] The processing flow will be explained below.

[0705] Step 1: Select a video

[0706] The user selects a video file that was taken using the terminal.

[0707] Prepares to upload the video file selected by the user to the server.

[0708] Step 2: Upload your video

[0709] The video file selected by the user is sent to the server using the upload button.

[0710] The terminal transmits the data of the selected video file to the server.

[0711] The server stores the received video file in storage.

[0712] Step 3: Loading video data

[0713] The server loads the saved video file.

[0714] The server prepares to analyze the video data.

[0715] Step 4: Analyzing the audio data

[0716] The server's AI model separates the audio portion of the video.

[0717] The server uses speech recognition technology to convert the voice data into text.

[0718] Step 5: Analyze the video data

[0719] The server's AI model analyzes the video data.

[0720] The server uses image recognition technology to recognize important objects and people in each scene.

[0721] Step 6: Recognizing user emotions with the emotion engine

[0722] The server's emotion engine identifies the user's emotion through voice analysis.

[0723] The server's emotion engine identifies the user's emotion using facial expression recognition technology from the video.

[0724] Step 7: Detecting the risk of a firestorm

[0725] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0726] The server uses natural language processing technology to analyze the text and identify inappropriate comments and sensitive keywords.

[0727] The server uses image recognition technology to analyze the video data and identify inappropriate content (e.g., violent scenes or discriminatory language).

[0728] Step 8: Marking risk and emotional areas

[0729] The server marks areas identified as at risk of a firestorm on the timeline.

[0730] The server also marks the emotional points identified by the emotion engine on the timeline.

[0731] Step 9: Place cuts and silences

[0732] The server will mute or cut the appropriate time range for the marked risk of a firestorm.

[0733] Step 10: Adding Text

[0734] The server will insert a warning message where the audio was cut or muted, for example, "This comment has been deemed inappropriate."

[0735] The server automatically inserts appropriate captions at the emotional points identified by the emotion engine. For example, it displays "The user is very excited here."

[0736] Step 11: Apply edits and generate new video file

[0737] The server applies edits to the audio and video, including cuts and subtitles.

[0738] The server generates a new edited video file and stores it separately from the original video file.

[0739] Step 12: Preview your edited video

[0740] The user accesses the server on their device and previews the edited video.

[0741] The user checks the video content in detail.

[0742] Step 13: Review and finalize your edits

[0743] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0744] The user can also request re-editing if necessary.

[0745] Step 14: Download and publish your edited video

[0746] The edited video that the user has confirmed is downloaded to the device.

[0747] Users can also upload edited videos directly to social media and video sharing sites.

[0748] Example 2

[0749] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] In recent years, the spread of video sharing platforms has led to a rapid increase in video content. However, some of this content may contain inappropriate comments or images, posing a risk to viewers. Furthermore, while there is a demand for high-quality video editing that takes into account the emotions of viewers, conventional manual editing methods have the drawback of being too time-consuming and labor-intensive. The objective of this invention is to solve these problems and realize efficient and safe video editing.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0752] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for analyzing user emotions, means for cutting or editing detected risky parts and inserting appropriate captions based on the user emotions, means for generating edited video data, and means for providing the generated video data. This enables automatic detection and editing of risks of inappropriate content, and further enables appropriate editing according to the user emotions.

[0753] "Video data" refers to digital video and audio data, and is a media file that a user has filmed or collected.

[0754] "Means for receiving and storing" refers to devices or software for receiving data transmitted from outside and recording it on an internal or external recording medium.

[0755] "Means for analyzing" refers to devices or software that analyze received data and understand or classify the content of the data.

[0756] "Means for detecting specific risks" refers to devices or software that automatically identify inappropriate content or areas that pose a risk of causing an uproar based on the results of analysis.

[0757] "Means for analyzing user emotions" refers to devices or software that identify the user's emotions from the audio and video data and detect specific emotional states.

[0758] "Means for cutting or editing risky parts" refers to devices or software that remove or modify identified risky parts of data to make them safe.

[0759] "Means for inserting captions" refers to devices or software for displaying text information within edited video.

[0760] The "means for generating edited video data" refers to a device or software that generates a new file from the video after editing and stores or outputs it.

[0761] The "means for providing" refers to devices or software that make the generated video data accessible to users and enable downloading or streaming.

[0762] The present invention relates to a system for automatically analyzing and editing video data. Specific embodiments of the invention will be described below.

[0763] Uploading videos

[0764] The user uploads the video file taken on the device to the server. When the user clicks the upload button, the video file is sent to the server using the HTTP protocol. The server stores the received video file in cloud storage (for example, Amazon Web Services S3).

[0765] Video and audio analysis

[0766] The server analyzes the stored video files using a generative AI model (e.g., OpenAI's GPT-3). Audio data is converted to text using the Google Cloud Speech-to-Text API. For video data, OpenCV is used to analyze each frame and recognize important objects and people.

[0767] Detecting the risk of flame wars

[0768] The server uses the converted voice text and video data to detect the risk of a social media outrage using an AI model. The voice data is analyzed using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. The video data is also analyzed using machine learning models (e.g., TensorFlow) to detect inappropriate content such as violent scenes or discriminatory language.

[0769] Recognizing user emotions with an emotion engine

[0770] The server uses an emotion engine to analyze the user's emotions from audio and video data. Audio data is analyzed to identify specific emotions (e.g., anger, joy). For example, Microsoft Azure's Emotion API can be used to analyze facial expressions in the video to identify the emotion. When a specific emotion is detected, the relevant part is highlighted and an appropriate caption (e.g., "The user was surprised here") is automatically inserted.

[0771] Cutting and Inserting Subtitles

[0772] The server then edits the video based on the identified risky parts and emotions. Using tools such as FFmpeg, the server silences or cuts out risky parts. Furthermore, the server inserts warning messages and explanatory text based on the identified emotions.

[0773] Generate edited video

[0774] The server generates a new video file that reflects the edited content. The generated video file is saved separately from the original video file. This is done using tools such as FFmpeg.

[0775] View and download the video

[0776] The user can access the server to preview the edited video. If satisfied with the content, they can confirm the edited results. After confirming, the user can download the edited video to their device or upload it directly to a social networking site or video sharing site.

[0777] Specific examples

[0778] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[0779] Prompt Sentence Examples

[0780] "Please tell me about a system that analyzes video footage of an event, recognizes emotions, and edits the footage appropriately. For example, please explain how to insert appropriate captions in scenes where the user is excited, or cut out inappropriate scenes."

[0781] This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[0782] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0783] Step 1: Upload your video

[0784] Input: Video file taken by the user on the device

[0785] How it works: A user selects a video file using a dedicated application or web interface and clicks the "Upload" button. The device sends the video file to the server via an HTTP request.

[0786] Output: The server saves the received video files to cloud storage.

[0787] Step 2: Video and audio analysis

[0788] Input: Video file stored in cloud storage

[0789] How it works: The server reads the video file and extracts the video and audio data separately. The audio data is converted to text using the Google Cloud Speech-to-Text API. For the video data, OpenCV is used to analyze each frame and recognize important objects and people.

[0790] Output: Audio-text data and analyzed video data

[0791] Step 3: Detecting the risk of a firestorm

[0792] Input: Audio-text data and analyzed video data

[0793] How it works: The server's AI model analyzes the voice and text data using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. It also analyzes the video data using machine learning models (e.g., TensorFlow) to detect violent scenes and discriminatory language.

[0794] Output: List of inappropriate comments and inappropriate video scenes

[0795] Step 4: Recognizing user emotions with the emotion engine

[0796] Input: Audio and video data

[0797] How it works: The server's emotion engine analyzes the audio data and identifies specific emotions (e.g., anger, joy). It then uses Microsoft Azure's Emotion API to read facial expressions in the video and identify the emotion. If a specific emotion is detected, it is highlighted.

[0798] Output: Emotion recognition information and a list of highlighted emotion parts

[0799] Step 5: Cutting and adding captions

[0800] Input: List of inappropriate comments, list of inappropriate video scenes, emotion recognition information

[0801] What it does: The server uses an editing tool (e.g., FFmpeg) to mute or cut out identified risky parts. For inappropriate comments, it inserts a warning message caption (e.g., "This comment has been deemed inappropriate"). Based on the emotion identified by the emotion engine, it inserts an appropriate caption (e.g., "This is where the user was surprised").

[0802] Output: New video data reflecting the edited content

[0803] Step 6: Generate the edited video

[0804] Input: New video data reflecting edits

[0805] How it works: The server uses FFmpeg to apply the edits and generate a new video file, which is stored in cloud storage separately from the original video file.

[0806] Output: Final video data with full edits

[0807] Step 7: Check and download the video

[0808] Input: Final video data

[0809] How it works: The user accesses the server on their device and previews the edited video through a web interface. If they are satisfied with the edits, they confirm the edits by clicking the "Confirm" button. After confirming, the user can download the edited video to their device or upload it directly to social media or video sharing sites.

[0810] Output: Edited video ready for download or upload link

[0811] (Application example 2)

[0812] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0813] The difficulty of automatically analyzing and editing video data lies in the difficulty of appropriately reflecting user emotions and the risk of a viral outbreak. In particular, there is still room for development in technology that recognizes user emotions and reflects them in editing. In today's world, where a large number of video data are posted, high-quality and safe video editing is required.

[0814] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for identifying the emotional state using an emotion engine that recognizes the user's emotions, means for cutting or editing based on the detected risky parts or emotional state, means for generating edited video data, and means for providing the generated video data. This enables high-quality and safe video editing that appropriately reflects the user's emotions and the risk of a controversy.

[0815] The "means for receiving and saving video data" is a mechanism for uploading videos taken by users to a server and saving them in storage within the server.

[0816] The "means for analyzing stored video data" is a mechanism for reading stored video data and analyzing the video and audio.

[0817] "Means for detecting specific risks based on analysis" refers to a mechanism for identifying inappropriate content or areas with a high risk of causing a backlash based on the results of analysis.

[0818] The "means for identifying an emotional state using an emotion engine that recognizes a user's emotion" is a mechanism for identifying a user's emotion from audio and video within a video using an emotion engine.

[0819] The "means for cutting or editing based on detected risky parts or emotional state" is a mechanism for cutting video or inserting subtitles based on detected risky parts or the user's emotional state.

[0820] The "means for generating edited video data" is a mechanism for generating a new video file that reflects cuts and edits.

[0821] The "means for providing the generated video data" is a mechanism for providing the edited video to users so that they can download or share it.

[0822] This invention provides an automatic analysis and editing system for video data. The system operates as follows.

[0823] First, the user uploads a video they have taken using their device to the server. The server then stores the received video in storage. Next, the server uses a generative AI model to analyze the video and audio of the uploaded video. The server then uses voice analysis technology to convert the audio data into text, and uses image recognition technology to recognize important objects and people in the video data.

[0824] The emotion engine in the server recognizes the user's emotions from voice and video. The server identifies emotions through voice analysis and highlights risky areas when a specific emotion is detected. It also uses facial expression recognition technology to identify emotions in video and automatically inserts appropriate subtitles.

[0825] Based on the analysis results, the server's AI model detects specific risks. It uses natural language processing technology to identify inappropriate comments and sensitive keywords from the audio data, and image recognition technology to detect inappropriate video. If a risky part is detected, the server cuts or silences the audio and inserts a warning message.

[0826] Finally, the server generates the edited video data. The generated video file is saved separately from the original video file, and the user can access the server on their device to check the edited video. If the user checks it and there are no problems, they can download the edited video or post it to social media, etc.

[0827] This system enables high-quality, safe video editing that appropriately reflects user emotions and the risk of online outrage.

[0828] As a specific example, consider a case where a user films an event and the video contains an exciting or moving scene. When the video is uploaded to the system, a generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. Furthermore, an emotion engine analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[0829] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[0830] The hardware used is a high-performance CPU or GPU, data storage, and the software used is Python, MoviePy, an emotion recognition module, and a flame risk detection module.

[0831] An example prompt for a generative AI model is:

[0832] Create a video editing program that recognizes the user's emotions and automatically detects and edits scenes that may pose a risk of causing controversy. Use Python and the MoviePy library, and also use the emotion recognition module and the controversy risk detection module. Finally, create a program that saves the edited video as a new file.

[0833] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0834] Step 1:

[0835] The user uses a device to shoot video data and uploads the video to the server. The input is a video file (e.g., .mp4 format), and the server receives this video file and saves it in storage. Specifically, the user selects a video on the file selection screen and clicks the upload button.

[0836] Step 2:

[0837] The server analyzes the stored video data using a generative AI model to analyze the video and audio data. The input is the stored video file, and the output is the text conversion result of the audio data and the analysis result of the video data. Specifically, it uses voice recognition technology to convert the audio data into text, and image recognition technology to identify important objects and people in the video.

[0838] Step 3:

[0839] The emotion engine in the server recognizes the user's emotions from the audio and video in the video. The input is the text conversion results and video analysis results obtained in step 2, and the output is the identification of the emotional state for each scene. Specifically, the emotion engine classifies the user's emotions based on the analysis results and identifies the emotions in each scene.

[0840] Step 4:

[0841] The server detects specific risks based on the analysis results. The input is the text conversion results of the audio data from step 2 and the video analysis results, and the output is the identification of risk areas. Specifically, it uses natural language processing technology and image recognition technology to identify inappropriate remarks and videos.

[0842] Step 5:

[0843] The server cuts or edits the video based on the detected risky parts or emotional state. The input is the emotional state identification result from step 3 and the risky part identification result from step 4, and the output is edited video data based on the editing instructions. Specifically, it mutes or cuts the identified risky parts, and inserts captions according to the detected emotions.

[0844] Step 6:

[0845] The server generates edited video data. The input is the edited video data based on the editing instructions in step 5, and the output is the final edited video file. Specifically, the server reconstructs the video file according to the editing instructions and generates a new video file.

[0846] Step 7:

[0847] The user uses their device to check the edited video that has been generated, and if necessary, downloads it or posts it to social media. The input is the final edited video file, and the output is the published or saved state of the video after the user has checked it. Specifically, the user accesses the server to preview the edited video, and after checking it, presses the download button or post button.

[0848] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0849] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0850] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0851] [Third embodiment]

[0852] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0853] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0854] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0855] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0856] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0857] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0858] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0859] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0860] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0861] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0862] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0863] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0864] This invention relates to a system that automatically analyzes and edits video data. Specifically, it is a system that uploads videos taken by users to a server, uses AI to detect comments or images that may pose a risk of causing an uproar, and performs appropriate editing. This system is a technology that significantly reduces the time users spend editing and reduces the risk of causing an uproar.

[0865] Overall system overview

[0866] 1. Upload your video

[0867] A user uploads a video taken using a terminal to a server.

[0868] The server stores the received video in storage.

[0869] 2. Video and audio analysis

[0870] The generative AI model on the server analyzes the video and audio of the uploaded video.

[0871] 3. Detecting the risk of online outrage

[0872] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[0873] 4. Cutting and Inserting Subtitles

[0874] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[0875] 5. Generate edited video

[0876] The server generates an edited video that reflects the cuts and inserted subtitles.

[0877] 6. View and download the video

[0878] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[0879] Explaining program processing in natural language

[0880] 1. Upload your video

[0881] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[0882] The server receives the video file and stores it in storage.

[0883] 2. Video and audio analysis

[0884] The server's generative AI model reads the video file and analyzes the video and audio data.

[0885] Voice data is converted into text using voice recognition technology within the server.

[0886] For video data, image recognition technology is used to recognize important objects and people in each scene.

[0887] 3. Detecting the risk of online outrage

[0888] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[0889] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[0890] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[0891] 4. Cutting and Inserting Subtitles

[0892] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[0893] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[0894] 5. Generate edited video

[0895] The server applies the edits and generates a new video file.

[0896] Save the edited video file separately from the original video file.

[0897] 6. View and download the video

[0898] The user accesses the server on their device and previews the edited video.

[0899] The user checks the preview and, if there are no problems, downloads the edited video to their device.

[0900] Users can also post the edited video directly to social media or video sharing sites.

[0901] Specific examples

[0902] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Natural language processing technology is used to identify the insensitive remark. The remark is muted and a caption is inserted stating, "This remark has been deemed inappropriate."

[0903] This allows users to quickly create and publish videos with a low risk of causing controversy without editing. This system provides a safe and efficient video posting environment for both users and society.

[0904] The processing flow will be explained below.

[0905] Step 1: Select a video

[0906] The user selects a video file that was taken using the terminal.

[0907] Prepares to upload the video file selected by the user to the server.

[0908] Step 2: Upload your video

[0909] The video file selected by the user is sent to the server using the upload button.

[0910] The terminal transmits the data of the selected video file to the server.

[0911] The server stores the received video file in storage.

[0912] Step 3: Loading video data

[0913] The server loads the saved video file.

[0914] The server prepares to analyze the video data.

[0915] Step 4: Analyzing the audio data

[0916] The server's AI model separates the audio portion of the video.

[0917] The server uses speech recognition technology to convert the voice data into text.

[0918] Step 5: Analyze the video data

[0919] The server's AI model analyzes the video data.

[0920] The server uses image recognition technology to recognize important objects and people in each scene.

[0921] Step 6: Detecting the risk of a firestorm

[0922] The server's AI model analyzes the voice data that has been converted into text.

[0923] The server uses natural language processing technology to identify inappropriate comments and sensitive keywords.

[0924] The server's AI model analyzes the video data and uses image recognition technology to detect inappropriate footage.

[0925] Step 7: Marking risk areas

[0926] The server marks areas identified as at risk of a firestorm on the timeline.

[0927] Step 8: Placing cuts and silences

[0928] Silence or cut the appropriate time range for the server marked risk.

[0929] Step 9: Adding Text

[0930] The server inserts a warning message caption at the cut point.

[0931] The caption specifically includes a message such as "This comment has been deemed inappropriate."

[0932] Step 10: Apply edits and generate video

[0933] The server applies edits to the audio and video, including cuts and subtitles.

[0934] The server generates a new edited video file and stores it separately from the original video file.

[0935] Step 11: Preview your edited video

[0936] The user accesses the server on their device and previews the edited video.

[0937] The user checks the content of the video.

[0938] Step 12: Review and finalize your edits

[0939] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[0940] The user can also request re-editing if necessary.

[0941] Step 13: Download and publish your edited video

[0942] The edited video that the user has confirmed is downloaded to the device.

[0943] Users can also upload edited videos directly to social media and video sharing sites.

[0944] Example 1

[0945] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0946] In recent years, opportunities for users to post videos they have taken to social networking sites and video sharing sites have increased, and this has also increased the risk of the video content causing outrage. Videos can contain inappropriate comments or sensitive content, which often leads to outrage and social criticism. Conventional methods require users to manually edit videos themselves, which is time-consuming and labor-intensive, and also carries the risk of oversight. Therefore, a system that automatically analyzes and edits video content is needed.

[0947] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0948] In this invention, the server includes means for receiving and saving video data, means for analyzing the video data, means for detecting specific risks based on the analysis, means for cutting or editing the detected risky parts and inserting warning messages, means for generating edited video data, and means for providing the generated video data. This enables users to automatically generate and edit videos with a low risk of causing controversy simply by uploading the videos.

[0949] "Video data" is a digital file containing video and audio captured by a user.

[0950] "Means for receiving and storing" refers to a mechanism by which the server receives video data sent by the user and stores it in a certain storage device.

[0951] "Means for analysis" refers to the technology that allows the server to analyze the video data stored therein and understand its content.

[0952] "Specific risks" are elements that could cause an outrage, such as inappropriate comments or sensitive footage contained in the video.

[0953] "Means of detection" are techniques that identify and detect specific risks from the analyzed data.

[0954] "Cutting or editing means" refers to techniques for muting or deleting detected risky parts or inserting warning messages.

[0955] A "warning message" is a warning message inserted at a specific risky point in a video.

[0956] The "means for generating edited video data" is a technology for generating a new video file in which the risk has been removed or corrected.

[0957] The "means for providing the generated video data" is a mechanism for providing the edited video data to the user.

[0958] "Natural language processing technology" is a technology for converting voice data into text format and analyzing its meaning.

[0959] "Image recognition technology" is a technology for identifying and analyzing important objects and people from each frame of video.

[0960] This invention relates to a system that automatically analyzes and edits video data. Specifically, this system uploads videos taken by users to a server, uses AI to detect comments or images that pose a risk of causing a firestorm, and performs appropriate editing. A specific embodiment of this system is described below.

[0961] Uploading videos

[0962] The user selects a video file taken on the device and uploads it to the server through the system's web interface. The server receives the video file and stores it in a specific directory (e.g., / uploads). At this time, the file's metadata (e.g., file name, upload time, user ID) is also stored in the database.

[0963] Video and audio analysis

[0964] The server detects a newly saved video file and calls a generative AI model (e.g., GPT-4, TensorFlow). The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it into text. It also uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. For example, it detects human faces and cars in each frame and records their coordinate information.

[0965] Detecting the risk of flame wars

[0966] The server's AI model receives audio and text data and video data as input and performs a flame war risk assessment. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords. The server also uses image recognition technology to identify images deemed inappropriate (e.g., violent scenes, discriminatory language). This process involves analyzing each frame and accumulating the results.

[0967] Cutting and Inserting Subtitles

[0968] The server marks identified risky sections on the timeline. Specific time information (e.g., the number of seconds the comment started and ended) is recorded. The server then mutes or cuts out the marked risky sections and inserts a warning message such as, "This comment has been deemed inappropriate."

[0969] Generate edited video

[0970] The server applies the edits to the timeline and generates a new video file. At this time, it determines the new file name and save location (e.g., / edited_videos directory) and saves it separately from the original video file. The server then updates the database to indicate that the edits are complete and prepares to notify the user.

[0971] View and download the video

[0972] The user accesses the system's web interface from their device and goes to the "Preview Edited Video" page. The server provides the user with a preview video link, allowing them to play and check the video. The user checks the edited video, and if there are no problems, clicks the download button to save the edited video to their device. They are also given the option to post the edited video directly to social media or video sharing sites.

[0973] Specific examples

[0974] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Using natural language processing technology, the insensitive remark is identified, muted, and a caption stating, "This remark has been deemed inappropriate" is inserted.

[0975] Example prompts to input to a generative AI model:

[0976] Prompt: Analyze the following text using natural language processing techniques to identify inappropriate statements.

[0977] Text: "I don't understand this culture at all and it doesn't interest me."

[0978] Output: Inappropriate remark detected. "I don't understand, and I'm not interested" is insensitive.

[0979] As a result, this system allows users to automatically generate and edit videos with a low risk of causing controversy simply by uploading them, enabling users to safely and efficiently publish videos without having to do any editing work.

[0980] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0981] Step 1:

[0982] The user selects a video file taken using the device and uploads it to the server through the system's web interface. The input is the video file, and the output is the video file uploaded to the server. Specifically, the user clicks the "Select File" button on the browser, selects the video file, and then clicks the "Upload" button.

[0983] Step 2:

[0984] The server receives the video file and saves it in a specific directory (e.g., / uploads). The input for this operation is the video file from the user, and the output is the video file saved in storage. Specifically, this operation processes the HTTP POST request, receives the video file, saves it in storage on the server, and records the metadata in a database.

[0985] Step 3:

[0986] The server detects a newly saved video file and prepares to call a generative AI model (e.g., GPT-4, TensorFlow). The input is the path to the newly saved video file, and the output is the state where the analysis process is complete. Specifically, it looks at the path to the video file and adds it to the queue for analysis.

[0987] Step 4:

[0988] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it to text. The input is the video file, and the output is the extracted audio text data. Specifically, the process involves separating the audio track from the video file, sending it to the speech recognition API, and receiving the result as text.

[0989] Step 5:

[0990] The server uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. The input is the video file, and the output is data on the recognized objects and people. Specifically, it extracts each frame of the video as an image, inputs them into an image recognition model, and accumulates information on the detected objects.

[0991] Step 6:

[0992] The AI ​​model on the server receives voice, text, and video data as input and performs a flame war risk assessment. The input is text and video data, and the output is a list of specific risk areas. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords, and image recognition technology to identify videos deemed inappropriate.

[0993] Step 7:

[0994] The server marks identified risky areas on a timeline. The input is a list of identified risky areas, and the output is video data marked on the timeline. Specifically, it records the start and end times of each risky area and visualizes that information on the timeline.

[0995] Step 8:

[0996] The server mutes or cuts the marked risky parts and inserts a warning message such as "This comment has been deemed inappropriate." The input is the marked video data, and the output is the edited video data. Specifically, the server performs the muting or cutting process and sets the insertion position and text content of the caption.

[0997] Step 9:

[0998] The server applies the edits to the timeline and generates a new video file. The input is the edited timeline data, and the output is the new video file. Specifically, the server encodes and saves the new video file based on the timeline that reflects the edits.

[0999] Step 10:

[1000] The server updates the database to indicate that editing is complete and prepares to notify the user. The input is the creation information for the new video file, and the output is the status when notification preparation is complete. Specifically, this operation updates the database and prepares for email and system notifications to notify the user.

[1001] Step 11:

[1002] A user accesses the system's web interface from a terminal and navigates to the "Preview Edited Video" page. The input is a link to the edited video, and the output is a previewable state for the user. Specifically, the action is to click the preview link on the web page and play the video.

[1003] Step 12:

[1004] The user checks the edited video and, if there are no problems, clicks the download button to save the video to their device. They are also given the option to post it to a social networking site or video sharing site. The input is the user's confirmation action, and the output is the download of the video file or posting to the social networking site. Specifically, the user's click after confirmation downloads the video or posts it to the social networking site.

[1005] (Application example 1)

[1006] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1007] Currently, when videos are distributed online, if they contain inappropriate comments or images, the task of manually identifying and editing them is extremely time-consuming and laborious. Furthermore, there are limits to human judgment, making it difficult to completely eliminate all risks. Posts on social media and video sharing platforms require a rapid response, but a delayed response can spark outrage, potentially having a major impact not only on individuals but also on companies and society as a whole. Therefore, a system that can efficiently and accurately eliminate risks is needed.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1009] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for inserting warning messages based on the analysis results, means for generating edited video data, and means for distributing the edited video data over a network. This allows users to automatically edit risky parts and insert warning messages, making it possible to safely and quickly release videos and reduce social risks.

[1010] "Means for receiving and storing video data" refers to a device or system that has the function of importing video files shot or created by users and storing them on a server or cloud storage.

[1011] "Means for analyzing stored video data" refers to a device or system that has the function of analyzing the contents of video files stored in storage and extracting or identifying data characteristics.

[1012] "Means for detecting specific risks based on analysis" refers to a device or system that has the functionality to identify high-risk elements, such as inappropriate comments or images, in a video based on the results of analysis.

[1013] "Means for cutting or editing detected risky parts" refers to a device or system that has the function of cutting identified risky parts, muting the audio, or inserting warning captions.

[1014] "Means for generating edited video data" refers to a device or system that has the function of generating a new video file that has been cut or edited, and outputting it with the editing results reflected.

[1015] "Means for providing generated video data" refers to a device or system that has the function of providing edited video files to users, making them available for download or uploading them directly to social networking sites or video sharing platforms.

[1016] "Means for inserting warning messages based on analysis results" refers to a device or system that has the function of inserting visual warning messages in the form of subtitles or the like into video for previously identified risk areas.

[1017] "Means for distributing edited video data over a network" refers to a device or system that has the function of distributing edited video files over the Internet or other networks and providing them to viewers.

[1018] This invention relates to a system that automatically analyzes, edits, and distributes video data. This system uses a generative AI model to detect inappropriate comments and images and then edits them appropriately to reduce risk.

[1019] System Overview and Configuration

[1020] A means of receiving and storing video data

[1021] Users upload videos they have taken using their devices to a server. This means has the function of receiving video files and saving them to cloud storage. Users can easily upload videos using a dedicated application.

[1022] A means of analyzing stored video data

[1023] The stored video data is analyzed by a generative AI model on the server, where audio data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and video data is analyzed using image recognition technology (e.g., Google Cloud Vision, OpenCV) to recognize important objects and people.

[1024] Analytics-based means of detecting specific risks

[1025] The generative AI model on the server identifies risks in the video based on the analysis results. At this stage, natural language processing techniques (e.g., BERT, GPT-4) are used for text analysis to detect inappropriate language and images. For example, if the video contains inappropriate language or violent scenes, those parts are identified.

[1026] A way to cut or edit detected risks

[1027] The generative AI model automatically mutes, cuts, or inserts warning messages for identified risky parts, making them clearly visible to users and preventing viewers from seeing inappropriate content.

[1028] A means of generating edited video data

[1029] Once editing is complete, the server generates a new video file, which is then saved with the edited results.

[1030] A means of providing generated video data

[1031] Users can access the server on their devices, preview the edited video, and if there are no problems, download the video or upload it directly to social media or video sharing platforms.

[1032] A means of inserting warning messages based on analysis results

[1033] When a risk is detected, the generative AI model inserts a visual warning message into the video as a caption, such as "This comment has been deemed inappropriate."

[1034] A means of distributing edited video data over a network

[1035] The edited video files are distributed via the Internet or other networks, allowing users to quickly distribute videos with risky parts automatically edited.

[1036] Specific examples

[1037] For example, consider a situation where a user shoots a video at a tourist spot and it contains inappropriate comments. The user uploads the video to a server using a dedicated application. A generative AI model on the server performs audio analysis and extracts the comments as text. Using natural language processing technology, the inappropriate comments are detected, muted, and a caption stating "This comment has been deemed inappropriate" is inserted. Furthermore, if inappropriate footage, such as a violent scene, is detected, the scene is cut.

[1038] Prompt Sentence Examples

[1039] "Detect inappropriate comments in the video and insert captions."

[1040] "Detect violent scenes and insert a warning message"

[1041] This allows users to safely and quickly publish videos without having to go through the trouble of editing out risky parts.

[1042] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1043] Step 1:

[1044] The user uploads the video they have taken using their device to the server through a dedicated application. In this step, the video file selected on the device is sent to the server. The input is the video file, and the output is the video file saved on the server.

[1045] Step 2:

[1046] The server saves the uploaded video file in cloud storage. In this step, the video data is stored in storage and managed so that the user can access it again. The input is the video file sent in step 1, and the output is the video file saved in cloud storage.

[1047] Step 3:

[1048] The server reads the video and audio data using a generative AI model to analyze the stored video file. At this step, the video file is ready to be analyzed. The input is the video file in cloud storage, and the output is the data prepared for analysis.

[1049] Step 4:

[1050] The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The server generates text data from the voice data and prepares it for the next analysis step. The input is voice data, and the output is text data.

[1051] Step 5:

[1052] Image recognition technology (e.g., Google Cloud Vision, OpenCV) is used to recognize important objects and people in the video data. The server extracts specific features from the video data and saves them as analysis results. The input is video data, and the output is feature data.

[1053] Step 6:

[1054] The generative AI model detects specific risks based on text data and feature data. The server uses natural language processing technology (e.g., BERT, GPT-4) to identify inappropriate comments and videos. The input is text data and feature data, and the output is data with identified risks.

[1055] Step 7:

[1056] The server cuts or silences the identified risk sections and inserts warning captions as necessary. This process automates the editing process, eliminating the need for manual editing by the user. The input is the data with identified risks, and the output is the edited video data.

[1057] Step 8:

[1058] Edited video data is generated and saved as a new video file on the server. The server outputs the final editing results as a video file. The input is the edited data, and the output is the new video file.

[1059] Step 9:

[1060] The user accesses the server using a device to preview the edited video. This step allows the user to check the edited results. The input is the new video file, and the output is the preview screen for the user.

[1061] Step 10:

[1062] If the user is satisfied, they can download the edited video or upload it directly to social media or video sharing platforms. The user then finally publishes or saves the video. The input is the previewed video file, and the output is the downloaded file or uploaded content.

[1063] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1064] This invention relates to a system that automatically analyzes and edits video data, and in particular, it is a system that provides more advanced editing capabilities by combining an emotion engine that recognizes the user's emotions. This enables appropriate editing according to the user's emotions, further reducing the risk of online outrage and realizing the creation of high-quality videos.

[1065] Overall system overview

[1066] 1. Upload your video

[1067] A user uploads a video taken using a terminal to a server.

[1068] The server stores the received video in storage.

[1069] 2. Video and audio analysis

[1070] The generative AI model on the server analyzes the video and audio of the uploaded video.

[1071] 3. Detecting the risk of online outrage

[1072] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[1073] 4. User Emotion Recognition by Emotion Engine

[1074] The emotion engine in the server recognizes the user's emotions from the audio and video in the video.

[1075] It identifies emotions through voice analysis and highlights risk areas when certain emotions are detected.

[1076] Facial expression recognition technology is used to identify emotions in video, and appropriate subtitles are automatically inserted when a specific emotion is detected.

[1077] 5. Cutting and Inserting Subtitles

[1078] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[1079] 6. Generate edited video

[1080] The server generates an edited video that reflects the cuts and inserted subtitles.

[1081] 7. View and download videos

[1082] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[1083] Explaining program processing in natural language

[1084] 1. Upload your video

[1085] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[1086] The server receives the video file and stores it in storage.

[1087] 2. Video and audio analysis

[1088] The server's generative AI model reads the video file and analyzes the video and audio data.

[1089] Voice data is converted into text using voice recognition technology within the server.

[1090] For video data, image recognition technology is used to recognize important objects and people in each scene.

[1091] 3. Detecting the risk of online outrage

[1092] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[1093] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[1094] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[1095] 4. User Emotion Recognition by Emotion Engine

[1096] The server's emotion engine identifies the user's emotion through voice analysis.

[1097] The server uses emotional information obtained from the voice data to highlight risk areas if a specific emotion is detected.

[1098] The server's emotion engine recognizes facial expressions in the video and identifies emotions.

[1099] Automatically insert appropriate captions based on emotion (e.g., "The user looked surprised here").

[1100] 5. Cutting and Inserting Subtitles

[1101] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[1102] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[1103] Appropriate subtitles are inserted based on the emotions identified by the emotion engine.

[1104] 6. Generate edited video

[1105] The server applies the edits and generates a new video file.

[1106] Save the edited video file separately from the original video file.

[1107] 7. View and download videos

[1108] The user accesses the server on their device and previews the edited video.

[1109] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[1110] Once the user has finalized the edited video, they can download it to their device or upload it directly to social media or video sharing sites.

[1111] Specific examples

[1112] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[1113] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[1114] The processing flow will be explained below.

[1115] Step 1: Select a video

[1116] The user selects a video file that was taken using the terminal.

[1117] Prepares to upload the video file selected by the user to the server.

[1118] Step 2: Upload your video

[1119] The video file selected by the user is sent to the server using the upload button.

[1120] The terminal transmits the data of the selected video file to the server.

[1121] The server stores the received video file in storage.

[1122] Step 3: Loading video data

[1123] The server loads the saved video file.

[1124] The server prepares to analyze the video data.

[1125] Step 4: Analyzing the audio data

[1126] The server's AI model separates the audio portion of the video.

[1127] The server uses speech recognition technology to convert the voice data into text.

[1128] Step 5: Analyze the video data

[1129] The server's AI model analyzes the video data.

[1130] The server uses image recognition technology to recognize important objects and people in each scene.

[1131] Step 6: Recognizing user emotions with the emotion engine

[1132] The server's emotion engine identifies the user's emotion through voice analysis.

[1133] The server's emotion engine identifies the user's emotion using facial expression recognition technology from the video.

[1134] Step 7: Detecting the risk of a firestorm

[1135] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[1136] The server uses natural language processing technology to analyze the text and identify inappropriate comments and sensitive keywords.

[1137] The server uses image recognition technology to analyze the video data and identify inappropriate content (e.g., violent scenes or discriminatory language).

[1138] Step 8: Marking risk and emotional areas

[1139] The server marks areas identified as at risk of a firestorm on the timeline.

[1140] The server also marks the emotional points identified by the emotion engine on the timeline.

[1141] Step 9: Place cuts and silences

[1142] The server will mute or cut the appropriate time range for the marked risk of a firestorm.

[1143] Step 10: Adding Text

[1144] The server will insert a warning message where the audio was cut or muted, for example, "This comment has been deemed inappropriate."

[1145] The server automatically inserts appropriate captions at the emotional points identified by the emotion engine. For example, it displays "The user is very excited here."

[1146] Step 11: Apply edits and generate new video file

[1147] The server applies edits to the audio and video, including cuts and subtitles.

[1148] The server generates a new edited video file and stores it separately from the original video file.

[1149] Step 12: Preview your edited video

[1150] The user accesses the server on their device and previews the edited video.

[1151] The user checks the video content in detail.

[1152] Step 13: Review and finalize your edits

[1153] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[1154] The user can also request re-editing if necessary.

[1155] Step 14: Download and publish your edited video

[1156] The edited video that the user has confirmed is downloaded to the device.

[1157] Users can also upload edited videos directly to social media and video sharing sites.

[1158] Example 2

[1159] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1160] In recent years, the spread of video sharing platforms has led to a rapid increase in video content. However, some of this content may contain inappropriate comments or images, posing a risk to viewers. Furthermore, while there is a demand for high-quality video editing that takes into account the emotions of viewers, conventional manual editing methods have the drawback of being too time-consuming and labor-intensive. The objective of this invention is to solve these problems and realize efficient and safe video editing.

[1161] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1162] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for analyzing user emotions, means for cutting or editing detected risky parts and inserting appropriate captions based on the user emotions, means for generating edited video data, and means for providing the generated video data. This enables automatic detection and editing of risks of inappropriate content, and further enables appropriate editing according to the user emotions.

[1163] "Video data" refers to digital video and audio data, and is a media file that a user has filmed or collected.

[1164] "Means for receiving and storing" refers to devices or software for receiving data transmitted from outside and recording it on an internal or external recording medium.

[1165] "Means for analyzing" refers to devices or software that analyze received data and understand or classify the content of the data.

[1166] "Means for detecting specific risks" refers to devices or software that automatically identify inappropriate content or areas that pose a risk of causing an uproar based on the results of analysis.

[1167] "Means for analyzing user emotions" refers to devices or software that identify the user's emotions from the audio and video data and detect specific emotional states.

[1168] "Means for cutting or editing risky parts" refers to devices or software that remove or modify identified risky parts of data to make them safe.

[1169] "Means for inserting captions" refers to devices or software for displaying text information within edited video.

[1170] The "means for generating edited video data" refers to a device or software that generates a new file from the video after editing and stores or outputs it.

[1171] The "means for providing" refers to devices or software that make the generated video data accessible to users and enable downloading or streaming.

[1172] The present invention relates to a system for automatically analyzing and editing video data. Specific embodiments of the invention will be described below.

[1173] Uploading videos

[1174] The user uploads the video file taken on the device to the server. When the user clicks the upload button, the video file is sent to the server using the HTTP protocol. The server stores the received video file in cloud storage (for example, Amazon Web Services S3).

[1175] Video and audio analysis

[1176] The server analyzes the stored video files using a generative AI model (e.g., OpenAI's GPT-3). Audio data is converted to text using the Google Cloud Speech-to-Text API. For video data, OpenCV is used to analyze each frame and recognize important objects and people.

[1177] Detecting the risk of flame wars

[1178] The server uses the converted voice text and video data to detect the risk of a social media outrage using an AI model. The voice data is analyzed using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. The video data is also analyzed using machine learning models (e.g., TensorFlow) to detect inappropriate content such as violent scenes or discriminatory language.

[1179] Recognizing user emotions with an emotion engine

[1180] The server uses an emotion engine to analyze the user's emotions from audio and video data. Audio data is analyzed to identify specific emotions (e.g., anger, joy). For example, Microsoft Azure's Emotion API can be used to analyze facial expressions in the video to identify the emotion. When a specific emotion is detected, the relevant part is highlighted and an appropriate caption (e.g., "The user was surprised here") is automatically inserted.

[1181] Cutting and Inserting Subtitles

[1182] The server then edits the video based on the identified risky parts and emotions. Using tools such as FFmpeg, the server silences or cuts out risky parts. Furthermore, the server inserts warning messages and explanatory text based on the identified emotions.

[1183] Generate edited video

[1184] The server generates a new video file that reflects the edited content. The generated video file is saved separately from the original video file. This is done using tools such as FFmpeg.

[1185] View and download the video

[1186] The user can access the server to preview the edited video. If satisfied with the content, they can confirm the edited results. After confirming, the user can download the edited video to their device or upload it directly to a social networking site or video sharing site.

[1187] Specific examples

[1188] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[1189] Prompt Sentence Examples

[1190] "Please tell me about a system that analyzes video footage of an event, recognizes emotions, and edits the footage appropriately. For example, please explain how to insert appropriate captions in scenes where the user is excited, or cut out inappropriate scenes."

[1191] This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[1192] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1193] Step 1: Upload your video

[1194] Input: Video file taken by the user on the device

[1195] How it works: A user selects a video file using a dedicated application or web interface and clicks the "Upload" button. The device sends the video file to the server via an HTTP request.

[1196] Output: The server saves the received video files to cloud storage.

[1197] Step 2: Video and audio analysis

[1198] Input: Video file stored in cloud storage

[1199] How it works: The server reads the video file and extracts the video and audio data separately. The audio data is converted to text using the Google Cloud Speech-to-Text API. For the video data, OpenCV is used to analyze each frame and recognize important objects and people.

[1200] Output: Audio-text data and analyzed video data

[1201] Step 3: Detecting the risk of a firestorm

[1202] Input: Audio-text data and analyzed video data

[1203] How it works: The server's AI model analyzes the voice and text data using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. It also analyzes the video data using machine learning models (e.g., TensorFlow) to detect violent scenes and discriminatory language.

[1204] Output: List of inappropriate comments and inappropriate video scenes

[1205] Step 4: Recognizing user emotions with the emotion engine

[1206] Input: Audio and video data

[1207] How it works: The server's emotion engine analyzes the audio data and identifies specific emotions (e.g., anger, joy). It then uses Microsoft Azure's Emotion API to read facial expressions in the video and identify the emotion. If a specific emotion is detected, it is highlighted.

[1208] Output: Emotion recognition information and a list of highlighted emotion parts

[1209] Step 5: Cutting and adding captions

[1210] Input: List of inappropriate comments, list of inappropriate video scenes, emotion recognition information

[1211] What it does: The server uses an editing tool (e.g., FFmpeg) to mute or cut out identified risky parts. For inappropriate comments, it inserts a warning message caption (e.g., "This comment has been deemed inappropriate"). Based on the emotion identified by the emotion engine, it inserts an appropriate caption (e.g., "This is where the user was surprised").

[1212] Output: New video data reflecting the edited content

[1213] Step 6: Generate the edited video

[1214] Input: New video data reflecting edits

[1215] How it works: The server uses FFmpeg to apply the edits and generate a new video file, which is stored in cloud storage separately from the original video file.

[1216] Output: Final video data with full edits

[1217] Step 7: Check and download the video

[1218] Input: Final video data

[1219] How it works: The user accesses the server on their device and previews the edited video through a web interface. If they are satisfied with the edits, they confirm the edits by clicking the "Confirm" button. After confirming, the user can download the edited video to their device or upload it directly to social media or video sharing sites.

[1220] Output: Edited video ready for download or upload link

[1221] (Application example 2)

[1222] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1223] The difficulty of automatically analyzing and editing video data lies in the difficulty of appropriately reflecting user emotions and the risk of a viral outbreak. In particular, there is still room for development in technology that recognizes user emotions and reflects them in editing. In today's world, where a large number of video data are posted, high-quality and safe video editing is required.

[1224] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for identifying the emotional state using an emotion engine that recognizes the user's emotions, means for cutting or editing based on the detected risky parts or emotional state, means for generating edited video data, and means for providing the generated video data. This enables high-quality and safe video editing that appropriately reflects the user's emotions and the risk of a controversy.

[1225] The "means for receiving and saving video data" is a mechanism for uploading videos taken by users to a server and saving them in storage within the server.

[1226] The "means for analyzing stored video data" is a mechanism for reading stored video data and analyzing the video and audio.

[1227] "Means for detecting specific risks based on analysis" refers to a mechanism for identifying inappropriate content or areas with a high risk of causing a backlash based on the results of analysis.

[1228] The "means for identifying an emotional state using an emotion engine that recognizes a user's emotion" is a mechanism for identifying a user's emotion from audio and video within a video using an emotion engine.

[1229] The "means for cutting or editing based on detected risky parts or emotional state" is a mechanism for cutting video or inserting subtitles based on detected risky parts or the user's emotional state.

[1230] The "means for generating edited video data" is a mechanism for generating a new video file that reflects cuts and edits.

[1231] The "means for providing the generated video data" is a mechanism for providing the edited video to users so that they can download or share it.

[1232] This invention provides an automatic analysis and editing system for video data. The system operates as follows.

[1233] First, the user uploads a video they have taken using their device to the server. The server then stores the received video in storage. Next, the server uses a generative AI model to analyze the video and audio of the uploaded video. The server then uses voice analysis technology to convert the audio data into text, and uses image recognition technology to recognize important objects and people in the video data.

[1234] The emotion engine in the server recognizes the user's emotions from voice and video. The server identifies emotions through voice analysis and highlights risky areas when a specific emotion is detected. It also uses facial expression recognition technology to identify emotions in video and automatically inserts appropriate subtitles.

[1235] Based on the analysis results, the server's AI model detects specific risks. It uses natural language processing technology to identify inappropriate comments and sensitive keywords from the audio data, and image recognition technology to detect inappropriate video. If a risky part is detected, the server cuts or silences the audio and inserts a warning message.

[1236] Finally, the server generates the edited video data. The generated video file is saved separately from the original video file, and the user can access the server on their device to check the edited video. If the user checks it and there are no problems, they can download the edited video or post it to social media, etc.

[1237] This system enables high-quality, safe video editing that appropriately reflects user emotions and the risk of online outrage.

[1238] As a specific example, consider a case where a user films an event and the video contains an exciting or moving scene. When the video is uploaded to the system, a generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. Furthermore, an emotion engine analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[1239] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[1240] The hardware used is a high-performance CPU or GPU, data storage, and the software used is Python, MoviePy, an emotion recognition module, and a flame risk detection module.

[1241] An example prompt for a generative AI model is:

[1242] Create a video editing program that recognizes the user's emotions and automatically detects and edits scenes that may pose a risk of causing controversy. Use Python and the MoviePy library, and also use the emotion recognition module and the controversy risk detection module. Finally, create a program that saves the edited video as a new file.

[1243] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1244] Step 1:

[1245] The user uses a device to shoot video data and uploads the video to the server. The input is a video file (e.g., .mp4 format), and the server receives this video file and saves it in storage. Specifically, the user selects a video on the file selection screen and clicks the upload button.

[1246] Step 2:

[1247] The server analyzes the stored video data using a generative AI model to analyze the video and audio data. The input is the stored video file, and the output is the text conversion result of the audio data and the analysis result of the video data. Specifically, it uses voice recognition technology to convert the audio data into text, and image recognition technology to identify important objects and people in the video.

[1248] Step 3:

[1249] The emotion engine in the server recognizes the user's emotions from the audio and video in the video. The input is the text conversion results and video analysis results obtained in step 2, and the output is the identification of the emotional state for each scene. Specifically, the emotion engine classifies the user's emotions based on the analysis results and identifies the emotions in each scene.

[1250] Step 4:

[1251] The server detects specific risks based on the analysis results. The input is the text conversion results of the audio data from step 2 and the video analysis results, and the output is the identification of risk areas. Specifically, it uses natural language processing technology and image recognition technology to identify inappropriate remarks and videos.

[1252] Step 5:

[1253] The server cuts or edits the video based on the detected risky parts or emotional state. The input is the emotional state identification result from step 3 and the risky part identification result from step 4, and the output is edited video data based on the editing instructions. Specifically, it mutes or cuts the identified risky parts, and inserts captions according to the detected emotions.

[1254] Step 6:

[1255] The server generates edited video data. The input is the edited video data based on the editing instructions in step 5, and the output is the final edited video file. Specifically, the server reconstructs the video file according to the editing instructions and generates a new video file.

[1256] Step 7:

[1257] The user uses their device to check the edited video that has been generated, and if necessary, downloads it or posts it to social media. The input is the final edited video file, and the output is the published or saved state of the video after the user has checked it. Specifically, the user accesses the server to preview the edited video, and after checking it, presses the download button or post button.

[1258] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1259] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1260] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1261] [Fourth embodiment]

[1262] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1263] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1264] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1265] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1266] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1267] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1268] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1269] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1270] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1271] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1272] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1273] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1274] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] This invention relates to a system that automatically analyzes and edits video data. Specifically, it is a system that uploads videos taken by users to a server, uses AI to detect comments or images that may pose a risk of causing an uproar, and performs appropriate editing. This system is a technology that significantly reduces the time users spend editing and reduces the risk of causing an uproar.

[1276] Overall system overview

[1277] 1. Upload your video

[1278] A user uploads a video taken using a terminal to a server.

[1279] The server stores the received video in storage.

[1280] 2. Video and audio analysis

[1281] The generative AI model on the server analyzes the video and audio of the uploaded video.

[1282] 3. Detecting the risk of online outrage

[1283] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[1284] 4. Cutting and Inserting Subtitles

[1285] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[1286] 5. Generate edited video

[1287] The server generates an edited video that reflects the cuts and inserted subtitles.

[1288] 6. View and download the video

[1289] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[1290] Explaining program processing in natural language

[1291] 1. Upload your video

[1292] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[1293] The server receives the video file and stores it in storage.

[1294] 2. Video and audio analysis

[1295] The server's generative AI model reads the video file and analyzes the video and audio data.

[1296] Voice data is converted into text using voice recognition technology within the server.

[1297] For video data, image recognition technology is used to recognize important objects and people in each scene.

[1298] 3. Detecting the risk of online outrage

[1299] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[1300] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[1301] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[1302] 4. Cutting and Inserting Subtitles

[1303] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[1304] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[1305] 5. Generate edited video

[1306] The server applies the edits and generates a new video file.

[1307] Save the edited video file separately from the original video file.

[1308] 6. View and download the video

[1309] The user accesses the server on their device and previews the edited video.

[1310] The user checks the preview and, if there are no problems, downloads the edited video to their device.

[1311] Users can also post the edited video directly to social media or video sharing sites.

[1312] Specific examples

[1313] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Natural language processing technology is used to identify the insensitive remark. The remark is muted and a caption is inserted stating, "This remark has been deemed inappropriate."

[1314] This allows users to quickly create and publish videos with a low risk of causing controversy without editing. This system provides a safe and efficient video posting environment for both users and society.

[1315] The processing flow will be explained below.

[1316] Step 1: Select a video

[1317] The user selects a video file that was taken using the terminal.

[1318] Prepares to upload the video file selected by the user to the server.

[1319] Step 2: Upload your video

[1320] The video file selected by the user is sent to the server using the upload button.

[1321] The terminal transmits the data of the selected video file to the server.

[1322] The server stores the received video file in storage.

[1323] Step 3: Loading video data

[1324] The server loads the saved video file.

[1325] The server prepares to analyze the video data.

[1326] Step 4: Analyzing the audio data

[1327] The server's AI model separates the audio portion of the video.

[1328] The server uses speech recognition technology to convert the voice data into text.

[1329] Step 5: Analyze the video data

[1330] The server's AI model analyzes the video data.

[1331] The server uses image recognition technology to recognize important objects and people in each scene.

[1332] Step 6: Detecting the risk of a firestorm

[1333] The server's AI model analyzes the voice data that has been converted into text.

[1334] The server uses natural language processing technology to identify inappropriate comments and sensitive keywords.

[1335] The server's AI model analyzes the video data and uses image recognition technology to detect inappropriate footage.

[1336] Step 7: Marking risk areas

[1337] The server marks areas identified as at risk of a firestorm on the timeline.

[1338] Step 8: Placing cuts and silences

[1339] Silence or cut the appropriate time range for the server marked risk.

[1340] Step 9: Adding Text

[1341] The server inserts a warning message caption at the cut point.

[1342] The caption specifically includes a message such as "This comment has been deemed inappropriate."

[1343] Step 10: Apply edits and generate video

[1344] The server applies edits to the audio and video, including cuts and subtitles.

[1345] The server generates a new edited video file and stores it separately from the original video file.

[1346] Step 11: Preview your edited video

[1347] The user accesses the server on their device and previews the edited video.

[1348] The user checks the content of the video.

[1349] Step 12: Review and finalize your edits

[1350] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[1351] The user can also request re-editing if necessary.

[1352] Step 13: Download and publish your edited video

[1353] The edited video that the user has confirmed is downloaded to the device.

[1354] Users can also upload edited videos directly to social media and video sharing sites.

[1355] Example 1

[1356] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1357] In recent years, opportunities for users to post videos they have taken to social networking sites and video sharing sites have increased, and this has also increased the risk of the video content causing outrage. Videos can contain inappropriate comments or sensitive content, which often leads to outrage and social criticism. Conventional methods require users to manually edit videos themselves, which is time-consuming and labor-intensive, and also carries the risk of oversight. Therefore, a system that automatically analyzes and edits video content is needed.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1359] In this invention, the server includes means for receiving and saving video data, means for analyzing the video data, means for detecting specific risks based on the analysis, means for cutting or editing the detected risky parts and inserting warning messages, means for generating edited video data, and means for providing the generated video data. This enables users to automatically generate and edit videos with a low risk of causing controversy simply by uploading the videos.

[1360] "Video data" is a digital file containing video and audio captured by a user.

[1361] "Means for receiving and storing" refers to a mechanism by which the server receives video data sent by the user and stores it in a certain storage device.

[1362] "Means for analysis" refers to the technology that allows the server to analyze the video data stored therein and understand its content.

[1363] "Specific risks" are elements that could cause an outrage, such as inappropriate comments or sensitive footage contained in the video.

[1364] "Means of detection" are techniques that identify and detect specific risks from the analyzed data.

[1365] "Cutting or editing means" refers to techniques for muting or deleting detected risky parts or inserting warning messages.

[1366] A "warning message" is a warning message inserted at a specific risky point in a video.

[1367] The "means for generating edited video data" is a technology for generating a new video file in which the risk has been removed or corrected.

[1368] The "means for providing the generated video data" is a mechanism for providing the edited video data to the user.

[1369] "Natural language processing technology" is a technology for converting voice data into text format and analyzing its meaning.

[1370] "Image recognition technology" is a technology for identifying and analyzing important objects and people from each frame of video.

[1371] This invention relates to a system that automatically analyzes and edits video data. Specifically, this system uploads videos taken by users to a server, uses AI to detect comments or images that pose a risk of causing a firestorm, and performs appropriate editing. A specific embodiment of this system is described below.

[1372] Uploading videos

[1373] The user selects a video file taken on the device and uploads it to the server through the system's web interface. The server receives the video file and stores it in a specific directory (e.g., / uploads). At this time, the file's metadata (e.g., file name, upload time, user ID) is also stored in the database.

[1374] Video and audio analysis

[1375] The server detects a newly saved video file and calls a generative AI model (e.g., GPT-4, TensorFlow). The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it into text. It also uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. For example, it detects human faces and cars in each frame and records their coordinate information.

[1376] Detecting the risk of flame wars

[1377] The server's AI model receives audio and text data and video data as input and performs a flame war risk assessment. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords. The server also uses image recognition technology to identify images deemed inappropriate (e.g., violent scenes, discriminatory language). This process involves analyzing each frame and accumulating the results.

[1378] Cutting and Inserting Subtitles

[1379] The server marks identified risky sections on the timeline. Specific time information (e.g., the number of seconds the comment started and ended) is recorded. The server then mutes or cuts out the marked risky sections and inserts a warning message such as, "This comment has been deemed inappropriate."

[1380] Generate edited video

[1381] The server applies the edits to the timeline and generates a new video file. At this time, it determines the new file name and save location (e.g., / edited_videos directory) and saves it separately from the original video file. The server then updates the database to indicate that the edits are complete and prepares to notify the user.

[1382] View and download the video

[1383] The user accesses the system's web interface from their device and goes to the "Preview Edited Video" page. The server provides the user with a preview video link, allowing them to play and check the video. The user checks the edited video, and if there are no problems, clicks the download button to save the edited video to their device. They are also given the option to post the edited video directly to social media or video sharing sites.

[1384] Specific examples

[1385] For example, consider a case where a user shoots a video at a tourist spot and the narration contains a culturally insensitive remark. The user uploads the video to the system. A generative AI model on the server performs audio analysis and extracts the remark as text. Using natural language processing technology, the insensitive remark is identified, muted, and a caption stating, "This remark has been deemed inappropriate" is inserted.

[1386] Example prompts to input to a generative AI model:

[1387] Prompt: Analyze the following text using natural language processing techniques to identify inappropriate statements.

[1388] Text: "I don't understand this culture at all and it doesn't interest me."

[1389] Output: Inappropriate remark detected. "I don't understand, and I'm not interested" is insensitive.

[1390] As a result, this system allows users to automatically generate and edit videos with a low risk of causing controversy simply by uploading them, enabling users to safely and efficiently publish videos without having to do any editing work.

[1391] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1392] Step 1:

[1393] The user selects a video file taken using the device and uploads it to the server through the system's web interface. The input is the video file, and the output is the video file uploaded to the server. Specifically, the user clicks the "Select File" button on the browser, selects the video file, and then clicks the "Upload" button.

[1394] Step 2:

[1395] The server receives the video file and saves it in a specific directory (e.g., / uploads). The input for this operation is the video file from the user, and the output is the video file saved in storage. Specifically, this operation processes the HTTP POST request, receives the video file, saves it in storage on the server, and records the metadata in a database.

[1396] Step 3:

[1397] The server detects a newly saved video file and prepares to call a generative AI model (e.g., GPT-4, TensorFlow). The input is the path to the newly saved video file, and the output is the state where the analysis process is complete. Specifically, it looks at the path to the video file and adds it to the queue for analysis.

[1398] Step 4:

[1399] The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to extract audio data from the video and convert it to text. The input is the video file, and the output is the extracted audio text data. Specifically, the process involves separating the audio track from the video file, sending it to the speech recognition API, and receiving the result as text.

[1400] Step 5:

[1401] The server uses image recognition technology (e.g., TensorFlow Object Detection API) to recognize important objects and people in each video frame. The input is the video file, and the output is data on the recognized objects and people. Specifically, it extracts each frame of the video as an image, inputs them into an image recognition model, and accumulates information on the detected objects.

[1402] Step 6:

[1403] The AI ​​model on the server receives voice, text, and video data as input and performs a flame war risk assessment. The input is text and video data, and the output is a list of specific risk areas. Specifically, it uses natural language processing technology (e.g., GPT-4) to identify inappropriate comments and sensitive keywords, and image recognition technology to identify videos deemed inappropriate.

[1404] Step 7:

[1405] The server marks identified risky areas on a timeline. The input is a list of identified risky areas, and the output is video data marked on the timeline. Specifically, it records the start and end times of each risky area and visualizes that information on the timeline.

[1406] Step 8:

[1407] The server mutes or cuts the marked risky parts and inserts a warning message such as "This comment has been deemed inappropriate." The input is the marked video data, and the output is the edited video data. Specifically, the server performs the muting or cutting process and sets the insertion position and text content of the caption.

[1408] Step 9:

[1409] The server applies the edits to the timeline and generates a new video file. The input is the edited timeline data, and the output is the new video file. Specifically, the server encodes and saves the new video file based on the timeline that reflects the edits.

[1410] Step 10:

[1411] The server updates the database to indicate that editing is complete and prepares to notify the user. The input is the creation information for the new video file, and the output is the status when notification preparation is complete. Specifically, this operation updates the database and prepares for email and system notifications to notify the user.

[1412] Step 11:

[1413] A user accesses the system's web interface from a terminal and navigates to the "Preview Edited Video" page. The input is a link to the edited video, and the output is a previewable state for the user. Specifically, the action is to click the preview link on the web page and play the video.

[1414] Step 12:

[1415] The user checks the edited video and, if there are no problems, clicks the download button to save the video to their device. They are also given the option to post it to a social networking site or video sharing site. The input is the user's confirmation action, and the output is the download of the video file or posting to the social networking site. Specifically, the user's click after confirmation downloads the video or posts it to the social networking site.

[1416] (Application example 1)

[1417] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1418] Currently, when videos are distributed online, if they contain inappropriate comments or images, the task of manually identifying and editing them is extremely time-consuming and laborious. Furthermore, there are limits to human judgment, making it difficult to completely eliminate all risks. Posts on social media and video sharing platforms require a rapid response, but a delayed response can spark outrage, potentially having a major impact not only on individuals but also on companies and society as a whole. Therefore, a system that can efficiently and accurately eliminate risks is needed.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1420] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for inserting warning messages based on the analysis results, means for generating edited video data, and means for distributing the edited video data over a network. This allows users to automatically edit risky parts and insert warning messages, making it possible to safely and quickly release videos and reduce social risks.

[1421] "Means for receiving and storing video data" refers to a device or system that has the function of importing video files shot or created by users and storing them on a server or cloud storage.

[1422] "Means for analyzing stored video data" refers to a device or system that has the function of analyzing the contents of video files stored in storage and extracting or identifying data characteristics.

[1423] "Means for detecting specific risks based on analysis" refers to a device or system that has the functionality to identify high-risk elements, such as inappropriate comments or images, in a video based on the results of analysis.

[1424] "Means for cutting or editing detected risky parts" refers to a device or system that has the function of cutting identified risky parts, muting the audio, or inserting warning captions.

[1425] "Means for generating edited video data" refers to a device or system that has the function of generating a new video file that has been cut or edited, and outputting it with the editing results reflected.

[1426] "Means for providing generated video data" refers to a device or system that has the function of providing edited video files to users, making them available for download or uploading them directly to social networking sites or video sharing platforms.

[1427] "Means for inserting warning messages based on analysis results" refers to a device or system that has the function of inserting visual warning messages in the form of subtitles or the like into video for previously identified risk areas.

[1428] "Means for distributing edited video data over a network" refers to a device or system that has the function of distributing edited video files over the Internet or other networks and providing them to viewers.

[1429] This invention relates to a system that automatically analyzes, edits, and distributes video data. This system uses a generative AI model to detect inappropriate comments and images and then edits them appropriately to reduce risk.

[1430] System Overview and Configuration

[1431] A means of receiving and storing video data

[1432] Users upload videos they have taken using their devices to a server. This means has the function of receiving video files and saving them to cloud storage. Users can easily upload videos using a dedicated application.

[1433] A means of analyzing stored video data

[1434] The stored video data is analyzed by a generative AI model on the server, where audio data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and video data is analyzed using image recognition technology (e.g., Google Cloud Vision, OpenCV) to recognize important objects and people.

[1435] Analytics-based means of detecting specific risks

[1436] The generative AI model on the server identifies risks in the video based on the analysis results. At this stage, natural language processing techniques (e.g., BERT, GPT-4) are used for text analysis to detect inappropriate language and images. For example, if the video contains inappropriate language or violent scenes, those parts are identified.

[1437] A way to cut or edit detected risks

[1438] The generative AI model automatically mutes, cuts, or inserts warning messages for identified risky parts, making them clearly visible to users and preventing viewers from seeing inappropriate content.

[1439] A means of generating edited video data

[1440] Once editing is complete, the server generates a new video file, which is then saved with the edited results.

[1441] A means of providing generated video data

[1442] Users can access the server on their devices, preview the edited video, and if there are no problems, download the video or upload it directly to social media or video sharing platforms.

[1443] A means of inserting warning messages based on analysis results

[1444] When a risk is detected, the generative AI model inserts a visual warning message into the video as a caption, such as "This comment has been deemed inappropriate."

[1445] A means of distributing edited video data over a network

[1446] The edited video files are distributed via the Internet or other networks, allowing users to quickly distribute videos with risky parts automatically edited.

[1447] Specific examples

[1448] For example, consider a situation where a user shoots a video at a tourist spot and it contains inappropriate comments. The user uploads the video to a server using a dedicated application. A generative AI model on the server performs audio analysis and extracts the comments as text. Using natural language processing technology, the inappropriate comments are detected, muted, and a caption stating "This comment has been deemed inappropriate" is inserted. Furthermore, if inappropriate footage, such as a violent scene, is detected, the scene is cut.

[1449] Prompt Sentence Examples

[1450] "Detect inappropriate comments in the video and insert captions."

[1451] "Detect violent scenes and insert a warning message"

[1452] This allows users to safely and quickly publish videos without having to go through the trouble of editing out risky parts.

[1453] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1454] Step 1:

[1455] The user uploads the video they have taken using their device to the server through a dedicated application. In this step, the video file selected on the device is sent to the server. The input is the video file, and the output is the video file saved on the server.

[1456] Step 2:

[1457] The server saves the uploaded video file in cloud storage. In this step, the video data is stored in storage and managed so that the user can access it again. The input is the video file sent in step 1, and the output is the video file saved in cloud storage.

[1458] Step 3:

[1459] The server reads the video and audio data using a generative AI model to analyze the stored video file. At this step, the video file is ready to be analyzed. The input is the video file in cloud storage, and the output is the data prepared for analysis.

[1460] Step 4:

[1461] The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The server generates text data from the voice data and prepares it for the next analysis step. The input is voice data, and the output is text data.

[1462] Step 5:

[1463] Image recognition technology (e.g., Google Cloud Vision, OpenCV) is used to recognize important objects and people in the video data. The server extracts specific features from the video data and saves them as analysis results. The input is video data, and the output is feature data.

[1464] Step 6:

[1465] The generative AI model detects specific risks based on text data and feature data. The server uses natural language processing technology (e.g., BERT, GPT-4) to identify inappropriate comments and videos. The input is text data and feature data, and the output is data with identified risks.

[1466] Step 7:

[1467] The server cuts or silences the identified risk sections and inserts warning captions as necessary. This process automates the editing process, eliminating the need for manual editing by the user. The input is the data with identified risks, and the output is the edited video data.

[1468] Step 8:

[1469] Edited video data is generated and saved as a new video file on the server. The server outputs the final editing results as a video file. The input is the edited data, and the output is the new video file.

[1470] Step 9:

[1471] The user accesses the server using a device to preview the edited video. This step allows the user to check the edited results. The input is the new video file, and the output is the preview screen for the user.

[1472] Step 10:

[1473] If the user is satisfied, they can download the edited video or upload it directly to social media or video sharing platforms. The user then finally publishes or saves the video. The input is the previewed video file, and the output is the downloaded file or uploaded content.

[1474] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1475] This invention relates to a system that automatically analyzes and edits video data, and in particular, it is a system that provides more advanced editing capabilities by combining an emotion engine that recognizes the user's emotions. This enables appropriate editing according to the user's emotions, further reducing the risk of online outrage and realizing the creation of high-quality videos.

[1476] Overall system overview

[1477] 1. Upload your video

[1478] A user uploads a video taken using a terminal to a server.

[1479] The server stores the received video in storage.

[1480] 2. Video and audio analysis

[1481] The generative AI model on the server analyzes the video and audio of the uploaded video.

[1482] 3. Detecting the risk of online outrage

[1483] The AI ​​model evaluates the content of video and audio and automatically detects pre-defined statements and videos that pose a high risk of causing controversy.

[1484] 4. User Emotion Recognition by Emotion Engine

[1485] The emotion engine in the server recognizes the user's emotions from the audio and video in the video.

[1486] It identifies emotions through voice analysis and highlights risk areas when certain emotions are detected.

[1487] Facial expression recognition technology is used to identify emotions in video, and appropriate subtitles are automatically inserted when a specific emotion is detected.

[1488] 5. Cutting and Inserting Subtitles

[1489] When risky areas are detected, the AI ​​model edits them by making appropriate cuts or inserting subtitles.

[1490] 6. Generate edited video

[1491] The server generates an edited video that reflects the cuts and inserted subtitles.

[1492] 7. View and download videos

[1493] The user checks the edited video on their device, and if there are no problems, they can download it or post it directly to social media.

[1494] Explaining program processing in natural language

[1495] 1. Upload your video

[1496] The user selects the video file he or she has taken from the terminal and uploads it to the server.

[1497] The server receives the video file and stores it in storage.

[1498] 2. Video and audio analysis

[1499] The server's generative AI model reads the video file and analyzes the video and audio data.

[1500] Voice data is converted into text using voice recognition technology within the server.

[1501] For video data, image recognition technology is used to recognize important objects and people in each scene.

[1502] 3. Detecting the risk of online outrage

[1503] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[1504] The voice data is analyzed using natural language processing technology to identify inappropriate comments and sensitive keywords.

[1505] Image recognition technology is used to identify inappropriate content (e.g., violent scenes or discriminatory language) from video data.

[1506] 4. User Emotion Recognition by Emotion Engine

[1507] The server's emotion engine identifies the user's emotion through voice analysis.

[1508] The server uses emotional information obtained from the voice data to highlight risk areas if a specific emotion is detected.

[1509] The server's emotion engine recognizes facial expressions in the video and identifies emotions.

[1510] Automatically insert appropriate captions based on emotion (e.g., "The user looked surprised here").

[1511] 5. Cutting and Inserting Subtitles

[1512] The server marks areas identified as at risk of a controversy and displays them on a timeline.

[1513] The AI ​​model will mute or cut out the marked risky parts and insert a warning message, such as "This comment has been deemed inappropriate."

[1514] Appropriate subtitles are inserted based on the emotions identified by the emotion engine.

[1515] 6. Generate edited video

[1516] The server applies the edits and generates a new video file.

[1517] Save the edited video file separately from the original video file.

[1518] 7. View and download videos

[1519] The user accesses the server on their device and previews the edited video.

[1520] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[1521] Once the user has finalized the edited video, they can download it to their device or upload it directly to social media or video sharing sites.

[1522] Specific examples

[1523] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[1524] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[1525] The processing flow will be explained below.

[1526] Step 1: Select a video

[1527] The user selects a video file that was taken using the terminal.

[1528] Prepares to upload the video file selected by the user to the server.

[1529] Step 2: Upload your video

[1530] The video file selected by the user is sent to the server using the upload button.

[1531] The terminal transmits the data of the selected video file to the server.

[1532] The server stores the received video file in storage.

[1533] Step 3: Loading video data

[1534] The server loads the saved video file.

[1535] The server prepares to analyze the video data.

[1536] Step 4: Analyzing the audio data

[1537] The server's AI model separates the audio portion of the video.

[1538] The server uses speech recognition technology to convert the voice data into text.

[1539] Step 5: Analyze the video data

[1540] The server's AI model analyzes the video data.

[1541] The server uses image recognition technology to recognize important objects and people in each scene.

[1542] Step 6: Recognizing user emotions with the emotion engine

[1543] The server's emotion engine identifies the user's emotion through voice analysis.

[1544] The server's emotion engine identifies the user's emotion using facial expression recognition technology from the video.

[1545] Step 7: Detecting the risk of a firestorm

[1546] The server's AI model determines the risk of a controversy based on the converted audio text and video data.

[1547] The server uses natural language processing technology to analyze the text and identify inappropriate comments and sensitive keywords.

[1548] The server uses image recognition technology to analyze the video data and identify inappropriate content (e.g., violent scenes or discriminatory language).

[1549] Step 8: Marking risk and emotional areas

[1550] The server marks areas identified as at risk of a firestorm on the timeline.

[1551] The server also marks the emotional points identified by the emotion engine on the timeline.

[1552] Step 9: Place cuts and silences

[1553] The server will mute or cut the appropriate time range for the marked risk of a firestorm.

[1554] Step 10: Adding Text

[1555] The server will insert a warning message where the audio was cut or muted, for example, "This comment has been deemed inappropriate."

[1556] The server automatically inserts appropriate captions at the emotional points identified by the emotion engine. For example, it displays "The user is very excited here."

[1557] Step 11: Apply edits and generate new video file

[1558] The server applies edits to the audio and video, including cuts and subtitles.

[1559] The server generates a new edited video file and stores it separately from the original video file.

[1560] Step 12: Preview your edited video

[1561] The user accesses the server on their device and previews the edited video.

[1562] The user checks the video content in detail.

[1563] Step 13: Review and finalize your edits

[1564] The user checks the preview and, if there are no problems with the edited content, confirms the edited result.

[1565] The user can also request re-editing if necessary.

[1566] Step 14: Download and publish your edited video

[1567] The edited video that the user has confirmed is downloaded to the device.

[1568] Users can also upload edited videos directly to social media and video sharing sites.

[1569] Example 2

[1570] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1571] In recent years, the spread of video sharing platforms has led to a rapid increase in video content. However, some of this content may contain inappropriate comments or images, posing a risk to viewers. Furthermore, while there is a demand for high-quality video editing that takes into account the emotions of viewers, conventional manual editing methods have the drawback of being too time-consuming and labor-intensive. The objective of this invention is to solve these problems and realize efficient and safe video editing.

[1572] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1573] In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for detecting specific risks based on the analysis, means for analyzing user emotions, means for cutting or editing detected risky parts and inserting appropriate captions based on the user emotions, means for generating edited video data, and means for providing the generated video data. This enables automatic detection and editing of risks of inappropriate content, and further enables appropriate editing according to the user emotions.

[1574] "Video data" refers to digital video and audio data, and is a media file that a user has filmed or collected.

[1575] "Means for receiving and storing" refers to devices or software for receiving data transmitted from outside and recording it on an internal or external recording medium.

[1576] "Means for analyzing" refers to devices or software that analyze received data and understand or classify the content of the data.

[1577] "Means for detecting specific risks" refers to devices or software that automatically identify inappropriate content or areas that pose a risk of causing an uproar based on the results of analysis.

[1578] "Means for analyzing user emotions" refers to devices or software that identify the user's emotions from the audio and video data and detect specific emotional states.

[1579] "Means for cutting or editing risky parts" refers to devices or software that remove or modify identified risky parts of data to make them safe.

[1580] "Means for inserting captions" refers to devices or software for displaying text information within edited video.

[1581] The "means for generating edited video data" refers to a device or software that generates a new file from the video after editing and stores or outputs it.

[1582] The "means for providing" refers to devices or software that make the generated video data accessible to users and enable downloading or streaming.

[1583] The present invention relates to a system for automatically analyzing and editing video data. Specific embodiments of the invention will be described below.

[1584] Uploading videos

[1585] The user uploads the video file taken on the device to the server. When the user clicks the upload button, the video file is sent to the server using the HTTP protocol. The server stores the received video file in cloud storage (for example, Amazon Web Services S3).

[1586] Video and audio analysis

[1587] The server analyzes the stored video files using a generative AI model (e.g., OpenAI's GPT-3). Audio data is converted to text using the Google Cloud Speech-to-Text API. For video data, OpenCV is used to analyze each frame and recognize important objects and people.

[1588] Detecting the risk of flame wars

[1589] The server uses the converted voice text and video data to detect the risk of a social media outrage using an AI model. The voice data is analyzed using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. The video data is also analyzed using machine learning models (e.g., TensorFlow) to detect inappropriate content such as violent scenes or discriminatory language.

[1590] Recognizing user emotions with an emotion engine

[1591] The server uses an emotion engine to analyze the user's emotions from audio and video data. Audio data is analyzed to identify specific emotions (e.g., anger, joy). For example, Microsoft Azure's Emotion API can be used to analyze facial expressions in the video to identify the emotion. When a specific emotion is detected, the relevant part is highlighted and an appropriate caption (e.g., "The user was surprised here") is automatically inserted.

[1592] Cutting and Inserting Subtitles

[1593] The server then edits the video based on the identified risky parts and emotions. Using tools such as FFmpeg, the server silences or cuts out risky parts. Furthermore, the server inserts warning messages and explanatory text based on the identified emotions.

[1594] Generate edited video

[1595] The server generates a new video file that reflects the edited content. The generated video file is saved separately from the original video file. This is done using tools such as FFmpeg.

[1596] View and download the video

[1597] The user can access the server to preview the edited video. If satisfied with the content, they can confirm the edited results. After confirming, the user can download the edited video to their device or upload it directly to a social networking site or video sharing site.

[1598] Specific examples

[1599] For example, consider a situation where a user shoots a video of an event and it contains an exciting or moving scene. The user uploads the video to the system. A generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. An emotion engine then analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[1600] Prompt Sentence Examples

[1601] "Please tell me about a system that analyzes video footage of an event, recognizes emotions, and edits the footage appropriately. For example, please explain how to insert appropriate captions in scenes where the user is excited, or cut out inappropriate scenes."

[1602] This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[1603] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1604] Step 1: Upload your video

[1605] Input: Video file taken by the user on the device

[1606] How it works: A user selects a video file using a dedicated application or web interface and clicks the "Upload" button. The device sends the video file to the server via an HTTP request.

[1607] Output: The server saves the received video files to cloud storage.

[1608] Step 2: Video and audio analysis

[1609] Input: Video file stored in cloud storage

[1610] How it works: The server reads the video file and extracts the video and audio data separately. The audio data is converted to text using the Google Cloud Speech-to-Text API. For the video data, OpenCV is used to analyze each frame and recognize important objects and people.

[1611] Output: Audio-text data and analyzed video data

[1612] Step 3: Detecting the risk of a firestorm

[1613] Input: Audio-text data and analyzed video data

[1614] How it works: The server's AI model analyzes the voice and text data using natural language processing technology (e.g., spaCy) to identify inappropriate comments and sensitive keywords. It also analyzes the video data using machine learning models (e.g., TensorFlow) to detect violent scenes and discriminatory language.

[1615] Output: List of inappropriate comments and inappropriate video scenes

[1616] Step 4: Recognizing user emotions with the emotion engine

[1617] Input: Audio and video data

[1618] How it works: The server's emotion engine analyzes the audio data and identifies specific emotions (e.g., anger, joy). It then uses Microsoft Azure's Emotion API to read facial expressions in the video and identify the emotion. If a specific emotion is detected, it is highlighted.

[1619] Output: Emotion recognition information and a list of highlighted emotion parts

[1620] Step 5: Cutting and adding captions

[1621] Input: List of inappropriate comments, list of inappropriate video scenes, emotion recognition information

[1622] What it does: The server uses an editing tool (e.g., FFmpeg) to mute or cut out identified risky parts. For inappropriate comments, it inserts a warning message caption (e.g., "This comment has been deemed inappropriate"). Based on the emotion identified by the emotion engine, it inserts an appropriate caption (e.g., "This is where the user was surprised").

[1623] Output: New video data reflecting the edited content

[1624] Step 6: Generate the edited video

[1625] Input: New video data reflecting edits

[1626] How it works: The server uses FFmpeg to apply the edits and generate a new video file, which is stored in cloud storage separately from the original video file.

[1627] Output: Final video data with full edits

[1628] Step 7: Check and download the video

[1629] Input: Final video data

[1630] How it works: The user accesses the server on their device and previews the edited video through a web interface. If they are satisfied with the edits, they confirm the edits by clicking the "Confirm" button. After confirming, the user can download the edited video to their device or upload it directly to social media or video sharing sites.

[1631] Output: Edited video ready for download or upload link

[1632] (Application example 2)

[1633] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1634] The difficulty of automatically analyzing and editing video data lies in the difficulty of appropriately reflecting user emotions and the risk of a viral outbreak. In particular, there is still room for development in technology that recognizes user emotions and reflects them in editing. In today's world, where a large number of video data are posted, high-quality and safe video editing is required.

[1635] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving and saving video data, means for analyzing the saved video data, means for identifying the emotional state using an emotion engine that recognizes the user's emotions, means for cutting or editing based on the detected risky parts or emotional state, means for generating edited video data, and means for providing the generated video data. This enables high-quality and safe video editing that appropriately reflects the user's emotions and the risk of a controversy.

[1636] The "means for receiving and saving video data" is a mechanism for uploading videos taken by users to a server and saving them in storage within the server.

[1637] The "means for analyzing stored video data" is a mechanism for reading stored video data and analyzing the video and audio.

[1638] "Means for detecting specific risks based on analysis" refers to a mechanism for identifying inappropriate content or areas with a high risk of causing a backlash based on the results of analysis.

[1639] The "means for identifying an emotional state using an emotion engine that recognizes a user's emotion" is a mechanism for identifying a user's emotion from audio and video within a video using an emotion engine.

[1640] The "means for cutting or editing based on detected risky parts or emotional state" is a mechanism for cutting video or inserting subtitles based on detected risky parts or the user's emotional state.

[1641] The "means for generating edited video data" is a mechanism for generating a new video file that reflects cuts and edits.

[1642] The "means for providing the generated video data" is a mechanism for providing the edited video to users so that they can download or share it.

[1643] This invention provides an automatic analysis and editing system for video data. The system operates as follows.

[1644] First, the user uploads a video they have taken using their device to the server. The server then stores the received video in storage. Next, the server uses a generative AI model to analyze the video and audio of the uploaded video. The server then uses voice analysis technology to convert the audio data into text, and uses image recognition technology to recognize important objects and people in the video data.

[1645] The emotion engine in the server recognizes the user's emotions from voice and video. The server identifies emotions through voice analysis and highlights risky areas when a specific emotion is detected. It also uses facial expression recognition technology to identify emotions in video and automatically inserts appropriate subtitles.

[1646] Based on the analysis results, the server's AI model detects specific risks. It uses natural language processing technology to identify inappropriate comments and sensitive keywords from the audio data, and image recognition technology to detect inappropriate video. If a risky part is detected, the server cuts or silences the audio and inserts a warning message.

[1647] Finally, the server generates the edited video data. The generated video file is saved separately from the original video file, and the user can access the server on their device to check the edited video. If the user checks it and there are no problems, they can download the edited video or post it to social media, etc.

[1648] This system enables high-quality, safe video editing that appropriately reflects user emotions and the risk of online outrage.

[1649] As a specific example, consider a case where a user films an event and the video contains an exciting or moving scene. When the video is uploaded to the system, a generative AI model on the server performs audio analysis and converts the audio data into text. Next, natural language processing technology is used to identify inappropriate remarks and sensitive keywords. Furthermore, an emotion engine analyzes the voice and facial expressions in the video to identify the user's emotions. If a scene in which the user is excited is detected, for example, that part is highlighted and a caption such as "The user is very excited here" is automatically inserted.

[1650] Finally, users can preview the edited video and save or publish it as they see fit. This system allows users to create and publish safe videos with appropriate emotional content without spending time on editing.

[1651] The hardware used is a high-performance CPU or GPU, data storage, and the software used is Python, MoviePy, an emotion recognition module, and a flame risk detection module.

[1652] An example prompt for a generative AI model is:

[1653] Create a video editing program that recognizes the user's emotions and automatically detects and edits scenes that may pose a risk of causing controversy. Use Python and the MoviePy library, and also use the emotion recognition module and the controversy risk detection module. Finally, create a program that saves the edited video as a new file.

[1654] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1655] Step 1:

[1656] The user uses a device to shoot video data and uploads the video to the server. The input is a video file (e.g., .mp4 format), and the server receives this video file and saves it in storage. Specifically, the user selects a video on the file selection screen and clicks the upload button.

[1657] Step 2:

[1658] The server analyzes the stored video data using a generative AI model to analyze the video and audio data. The input is the stored video file, and the output is the text conversion result of the audio data and the analysis result of the video data. Specifically, it uses voice recognition technology to convert the audio data into text, and image recognition technology to identify important objects and people in the video.

[1659] Step 3:

[1660] The emotion engine in the server recognizes the user's emotions from the audio and video in the video. The input is the text conversion results and video analysis results obtained in step 2, and the output is the identification of the emotional state for each scene. Specifically, the emotion engine classifies the user's emotions based on the analysis results and identifies the emotions in each scene.

[1661] Step 4:

[1662] The server detects specific risks based on the analysis results. The input is the text conversion results of the audio data from step 2 and the video analysis results, and the output is the identification of risk areas. Specifically, it uses natural language processing technology and image recognition technology to identify inappropriate remarks and videos.

[1663] Step 5:

[1664] The server cuts or edits the video based on the detected risky parts or emotional state. The input is the emotional state identification result from step 3 and the risky part identification result from step 4, and the output is edited video data based on the editing instructions. Specifically, it mutes or cuts the identified risky parts, and inserts captions according to the detected emotions.

[1665] Step 6:

[1666] The server generates edited video data. The input is the edited video data based on the editing instructions in step 5, and the output is the final edited video file. Specifically, the server reconstructs the video file according to the editing instructions and generates a new video file.

[1667] Step 7:

[1668] The user uses their device to check the edited video that has been generated, and if necessary, downloads it or posts it to social media. The input is the final edited video file, and the output is the published or saved state of the video after the user has checked it. Specifically, the user accesses the server to preview the edited video, and after checking it, presses the download button or post button.

[1669] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1670] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1671] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1672] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1673] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1674] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1675] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1676] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1677] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1678] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1679] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1680] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1681] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1682] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1683] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1684] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1685] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1686] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1687] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1688] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1689] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1690] The following is further disclosed regarding the above embodiment.

[1691] (Claim 1)

[1692] means for receiving and storing video data;

[1693] means for analyzing the stored video data;

[1694] A means of detecting specific risks based on the analysis;

[1695] A means for cutting or editing the detected risk portion;

[1696] means for generating edited video data;

[1697] means for providing the generated video data;

[1698] A system including:

[1699] (Claim 2)

[1700] 2. The system according to claim 1, further comprising means for detecting the specific risk from voice data using natural language processing technology.

[1701] (Claim 3)

[1702] 2. The system according to claim 1, further comprising means for detecting the specific risk from video data using image recognition technology.

[1703] (Claim 4)

[1704] 2. The system according to claim 1, further comprising means for inserting a warning message as a caption at the detected risk location.

[1705] (Claim 5)

[1706] 2. The system according to claim 1, further comprising means for enabling the edited video data to be previewed through a user interface.

[1707] (Claim 6)

[1708] 2. The system according to claim 1, further comprising means for sharing or publishing the edited video data via a network selected by the user.

[1709] "Example 1"

[1710] (Claim 1)

[1711] means for receiving and storing video data;

[1712] means for analyzing the stored video data;

[1713] A means of detecting specific risks based on the analysis;

[1714] A means for cutting or editing the detected risk portion and inserting a warning message;

[1715] means for generating edited video data;

[1716] means for providing the generated video data;

[1717] A system including:

[1718] (Claim 2)

[1719] 2. The system according to claim 1, further comprising means for detecting the specific risk from voice data using natural language processing technology.

[1720] (Claim 3)

[1721] 2. The system according to claim 1, further comprising means for detecting the specific risk from video data using image recognition technology.

[1722] "Application Example 1"

[1723] (Claim 1)

[1724] means for receiving and storing video data;

[1725] means for analyzing the stored video data;

[1726] A means of detecting specific risks based on the analysis;

[1727] A means for cutting or editing the detected risk portion;

[1728] means for generating edited video data;

[1729] means for providing the generated video data;

[1730] means for inserting a warning message based on the analysis results;

[1731] a means for distributing the edited video data via a network;

[1732] A system including:

[1733] (Claim 2)

[1734] 2. The system according to claim 1, further comprising means for detecting the specific risk from voice data using natural language processing technology.

[1735] (Claim 3)

[1736] 2. The system according to claim 1, further comprising means for detecting the specific risk from video data using image recognition technology.

[1737] "Example 2: Combining Emotion Engines"

[1738] (Claim 1)

[1739] means for receiving and storing video data;

[1740] means for analyzing the stored video data;

[1741] A means of detecting specific risks based on the analysis;

[1742] means for analyzing user emotions;

[1743] a means for cutting or editing the detected risky portion and inserting an appropriate caption based on the user's emotion;

[1744] means for generating edited video data;

[1745] means for providing the generated video data;

[1746] A system including:

[1747] (Claim 2)

[1748] 2. The system according to claim 1, further comprising means for detecting the specific risk from voice data using natural language processing technology.

[1749] (Claim 3)

[1750] 2. The system according to claim 1, further comprising means for detecting the specific risk from video data using image recognition technology.

[1751] "Application example 2 when combining emotion engines"

[1752] (Claim 1)

[1753] means for receiving and storing video data;

[1754] means for analyzing the stored video data;

[1755] A means of detecting specific risks based on the analysis;

[1756] means for identifying an emotional state using an emotion engine that recognizes the user's emotions;

[1757] means for cutting or editing based on the detected risk points or emotional state;

[1758] means for generating edited video data;

[1759] means for providing the generated video data;

[1760] A system including:

[1761] (Claim 2)

[1762] 2. The system according to claim 1, further comprising means for detecting the specific risk from voice data using natural language processing technology.

[1763] (Claim 3)

[1764] 2. The system according to claim 1, further comprising means for detecting the specific risk from video data using image recognition technology. [Explanation of symbols]

[1765] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving and storing video data; means for analyzing the stored video data; A means of detecting specific risks based on the analysis; A means for cutting or editing the detected risk portion; means for generating edited video data; means for providing the generated video data; A system including:

2. 2. The system according to claim 1, further comprising means for detecting the specific risk from voice data using natural language processing techniques.

3. The system according to claim 1, further comprising means for detecting the specific risk from video data using image recognition technology.

4. 2. The system according to claim 1, further comprising means for inserting a warning message as a caption at the detected risk portion.

5. 2. The system according to claim 1, further comprising means for enabling previewing of the edited video data through a user interface.

6. 2. The system according to claim 1, further comprising means for sharing or publishing the edited video data via a network selected by the user.

Images