Video editing systems and video editing methods

Abstract

Edit videos with simple operations, without requiring high-spec hardware. [Solution] The video editing system 1 inputs the target video to be edited on the video editing server 2, and uses a machine learning model that distinguishes between voiced and silent sections, segment sections, and detects scene transition points in the video to divide the audio data of the target video into voiced and silent sections, transcribes the voiced sections into segment sections, detects scene transition points, and provides the user with an editing screen 100 in which segment section objects 103, silent section objects 104, and scene transition point objects 105 are arranged as a video audio document 102. On the editing screen 100, the user edits the video audio document 102 according to the operation of the segment section objects 103, silent section objects 104, and scene transition point objects 105, and creates an edited video by editing the target video according to the editing result of the video audio document 102.

Description

Technical Field

[0001] The present invention relates to a video editing system and a video editing method for editing videos.

Background Art

[0002] Conventionally, an information processing device such as a personal computer functions as a video editing device by installing software (program) for editing videos.

[0003] For example, the video editing device disclosed in Patent Document 1 includes an edited video generation unit that generates edited video data having a length desired by the user and including the content of the portion that the user wants to view, in response to the user's request.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, conventional video editing devices require the installation of dedicated video editing software (video editing software) on an information processing device such as a personal computer. This dedicated video editing software is a versatile, general-purpose software with numerous features, resulting in slow processing and poor responsiveness when editing videos using the device. Furthermore, the user interface becomes complex to provide these features, and editing videos requires many steps and considerable time. Additionally, video editing devices require high-spec hardware on the information processing device (such as a personal computer) running this video editing software.

[0006] Therefore, taking the above circumstances into consideration, the present invention aims to provide a video editing system and video editing method that allow video editing with simple operations without requiring high-spec hardware. [Means for solving the problem]

[0007] To solve the above problems, the video editing system of the present invention takes a target video to be edited as input, uses a machine learning model that divides the audio data of the target video into voiced and silent sections, divides the voiced sections into phrase sections, and detects scene transition points in the video, transcribes the voiced sections of the audio data into phrase sections, detects scene transition points from the target video, provides the user with an editing screen in which phrase section objects indicating the phrase sections of the data, silent section objects indicating the silent sections of the audio data, and scene transition point objects indicating the scene transition points of the target video are arranged in chronological order as a series of video audio documents, edits the video audio document in accordance with the operation of the phrase section objects, silent section objects and / or scene transition point objects on the editing screen, and creates an edited video by editing the target video according to the editing result of the video audio document.

[0008] Furthermore, in order to solve the above problems, the present invention provides a video editing method in which a target video to be edited is input, and a learning model that has been trained on machine learning to distinguish between voiced and silent sections in the video, distinguish between phrase sections within the voiced sections, and detect scene transition points in the video is used to divide the audio data of the target video into voiced and silent sections, transcribe the voiced sections of the audio data and divide them into phrase sections, detect scene transition points from the target video, and provides the user with an editing screen in which phrase section objects indicating the phrase sections of the data, silent section objects indicating the silent sections of the audio data, and scene transition point objects indicating the scene transition points of the target video are arranged in chronological order as a series of video audio documents, and the video audio document is edited in accordance with the operation of the phrase section objects, silent section objects and / or scene transition point objects on the editing screen, and an edited video is created by editing the target video according to the editing result of the video audio document. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a video editing system and video editing method that allow video editing with simple operations without requiring high-spec hardware. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram illustrating a video editing system according to one embodiment of the present invention. [Figure 2] This is a schematic diagram showing an example of an editing screen provided by a video editing server in a video editing system according to one embodiment of the present invention. [Modes for carrying out the invention]

[0011] First, the overall configuration of the video editing system 1 according to an embodiment of the present invention will be described with reference to Figure 1. As shown in Figure 1, the video editing system 1 comprises a video editing server 2 that performs video editing and a terminal device 3 for performing video editing operations, and the video editing server 2 and the terminal device 3 are connected to each other via a network 5 such as the Internet. The video editing system 1 is a system in which a video containing video data and audio data is treated as the target video for video editing, and the video editing server 2 edits the target video and provides it to the terminal device 3 in response to editing operations on the target video on the terminal device 3.

[0012] Although Figure 1 illustrates an example where the video editing system 1 has one terminal device 3 connected to one video editing server 2, it is possible to have multiple terminal devices 3 connected to one video editing server 2.

[0013] The video editing server 2 is configured to include, for example, a control unit 10, a storage unit 11, a communication unit 12, a display unit 13, and an operation unit 14.

[0014] The control unit 10 provides overall control over all parts and functions of the video editing server 2. It is composed of a computer such as a CPU (Central Processing Unit) and is connected to the storage unit 11 and the communication unit 12. The storage unit 11 is composed of memory such as ROM (Read Only Memory) and RAM (Random Access Memory), and recording media such as a hard disk, and stores programs and data for controlling all parts and functions of the video editing server 2. The communication unit 12 is an interface for the video editing server 2 to connect to the network 5, that is, it connects the video editing server 2 to the terminal device 3 via the network 5.

[0015] The control unit 10 controls various components and functions of the video editing server 2 by executing calculations based on programs and data stored in the storage unit 11. For example, the control unit 10 operates as a video input unit 20, a section detection unit 21, a translation unit 22, an editing screen provision unit 23, and a video editing unit 24 by executing a program stored in the storage unit 11. The video input unit 20, section detection unit 21, translation unit 22, editing screen provision unit 23, and video editing unit 24 realize the video input process, section detection process, translation process, editing screen provision process, and video editing process of the video editing method according to the present invention.

[0016] The video input unit 20 inputs the target video to be edited, which is transmitted or transferred (uploaded) from the terminal device 3, and stores it in the storage unit 11. For example, the video input unit 20 provides a website through the network 5 that allows the uploading of target videos, and inputs the target video uploaded via this website displayed in the browser installed on the terminal device 3. The video input unit 20 may accept input (upload) of the target video through a website that displays an editing screen 100 (see Figure 2) for editing the target video, which will be described later, or it may accept input (upload) of the target video through a website that displays a dedicated input screen (not shown) for inputting the target video.

[0017] The section detection unit 21 divides the audio data of the target video input by the video input unit 20 into voiced sections consisting of consecutive voiced sounds and silent sections consisting of consecutive voiceless sounds, and stores the time information of the voiced and silent sections in the target video (playback start time and playback end time in the target video) in the storage unit 11, associating them with the voiced and silent sections, respectively.

[0018] Furthermore, the section detection unit 21 transcribes the voiced segments of the audio data to extract words, divides the smallest set of words that form a translatable sentence into a phrase segment, and stores the time information of the phrase segment in the target video (playback start time and playback end time in the target video) in the storage unit 11 in association with the phrase segment. For example, the section detection unit 21 divides the target video into phrase segments and silent segments in units of WebVTT.

[0019] Furthermore, the section detection unit 21 detects scene transition points in the target video where scenes change, and stores the time information of the scene transition point in the target video (scene transition time in the target video) in the storage unit 11 in association with the scene transition point.

[0020] Furthermore, regardless of whether the input video or a video obtainable via network 5 is a target video, video editing server 2 can utilize a machine learning model that uses artificial intelligence to distinguish between voiced and silent sections in a video, distinguish between phrase sections within the voiced sections, and detect scene transition points in the video. It can also utilize artificial intelligence such as a Large Language Model (LLM) that uses this learning model. Video editing server 2 may function as an artificial intelligence server that creates, stores, and uses this learning model itself, or it may have an external artificial intelligence server create, store, and use this learning model.

[0021] Then, the section detection unit 21 uses artificial intelligence based on this learning model to classify the audio data of the target video into a voiced section and a voiceless section. Also, the section detection unit 21 uses artificial intelligence based on this learning model to perform speech recognition on the voiced section of the audio data and classify the voiced section into clause sections. Further, the section detection unit 21 uses artificial intelligence based on this learning model to detect scene change points from the target video.

[0022] The translation unit 22 translates the clause sections of the voiced section of the audio data of the target video detected by the section detection unit 21, and stores the translation results in the storage unit 11 in association with the clause sections.

[0023] For example, the translation unit 22 may use an external translation server to translate each clause section of the target video. At this time, regardless of the input target video or videos that can be obtained via the network 5, the translation unit 22 causes the external translation server to translate the clause sections of the voiced section in the video, and accumulates the translation results in the external translation server by machine learning using artificial intelligence based on a learning model. Then, the translation unit 22 causes the artificial intelligence of the external translation server using this learning model to translate each clause section of the target video.

[0024] Alternatively, the translation unit 22 may translate each clause section of the target video by internal processing of the video editing server 2. At this time, as internal processing of the video editing server 2, regardless of the input target video or videos that can be obtained via the network 5, the translation unit 22 generates a learning model obtained by machine learning of the translation results of the clause sections of the voiced section in the video using artificial intelligence, and accumulates it in the storage unit 11 or a database (not shown). Then, the translation unit 22 uses the artificial intelligence based on this learning model to translate each clause section of the target video.

[0025] The editing screen provider 23 provides the terminal device 3 with an editing screen 100, as shown in Figure 2, for editing the target video input by the video input unit 20. For example, the editing screen provider 23 provides a website via the network 5 that displays the editing screen 100 on which the target video can be edited, and accepts editing operations for the target video through this website displayed in the browser installed in the terminal device 3. The editing screen 100 provided by the editing screen provider 23 has an edit confirmation button 101, and in response to the operation of the edit confirmation button 101, the editing screen provider 23 creates editing information for the target video corresponding to the editing operation for the target video and stores it in the storage unit 11.

[0026] Specifically, the editing screen providing unit 23 arranges the following objects in chronological order on the editing screen 100: a phrase section object 103 indicating a phrase section of the audio data detected by the section detection unit 21, a silent section object 104 indicating a silent section of the audio data, and a scene transition point object 105 indicating a scene transition point of the target video. These objects are then displayed as a series of video audio documents 102 (text) that document (convert to text) the audio data of the target video, and each object can be selected individually.

[0027] The editing screen provider 23 accepts editing of the video audio document 102 (transcript), which consists of a phrase section object 103, a silent section object 104, and a scene transition point object 105, on the editing screen 100, and the editing of the video audio document 102 on the editing screen 100 becomes a preliminary edit of the target video. The editing screen provider 23 confirms the editing of the video audio document 102 in response to the operation of the edit confirmation button 101, and creates editing information for the target video corresponding to the editing of the video audio document 102.

[0028] The phrase segment object 103, the silent segment object 104, and the scene transition point object 105 displayed on the editing screen 100 are each individual GUI buttons, and the editing screen provider unit 23 accepts drag-and-drop operations for the phrase segment object 103, the silent segment object 104, and the scene transition point object 105 on the editing screen 100.

[0029] The phrase section object 103 displays the characters of the corresponding phrase section. The silent section object 104 and the scene transition point object 105 are displayed with marks that can distinguish them from other objects. For example, the silent section object 104 displays a green circular mark (shown as a white circle in Figure 2), and the scene transition point object 105 displays a red circular mark (shown as a black circle in Figure 2).

[0030] For example, the editing screen provider 23 accepts a delete operation specifying a phrase section object 103, a silent section object 104, and a scene transition point object 105 in order to delete phrase sections, silent sections, and scene transition points from the target video. The editing screen provider 23 accepts a position change operation specifying a phrase section object 103, a silent section object 104, a scene transition point object 105, and the position to be changed in order to change the position of phrase sections, silent sections, and scene transition points in the target video. The editing screen provider 23 accepts a split operation specifying a phrase section object 103 and the split position in order to split one phrase section into two or more phrase sections. The editing screen provider 23 accepts a merge operation specifying two or more phrase section objects 103 in order to merge two or more phrase sections into one phrase section.

[0031] Furthermore, the editing screen provider 23 accepts event addition operations on the segment section object 103, the silent section object 104, and / or the scene switching point object 105 in order to add predetermined events to the segment sections of the audio data, the silent sections of the audio data, and / or the positions of the target video corresponding to the scene switching points of the target video, and edits the video audio document 102 according to the event addition operations. For example, the editing screen provider 23 displays GUI buttons for event objects 106 in the editing screen 100 to accept various event addition operations, and accepts drag-and-drop operations for event objects 106.

[0032] At this time, the editing screen provider 23 may accept operations on the editing screen 100 to specify the objects to which events are to be added (phrase section object 103, silent section object 104, scene transition point object 105) and the period for which events are to be added (i.e., the playback period during which events are played). Furthermore, when accepting operations to add events to the phrase section object 103 and / or silent section object 104, the editing screen provider 23 may accept operations on the editing screen 100 to specify the addition time in the phrase section and / or silent section (the start time and / or end time of events in the phrase section and / or silent section).

[0033] For example, the editing screen provider 23 accepts event addition operations in the editing screen 100 to specify and insert events such as titles (eye-catching videos that show the target video or each scene), subtitles (Japanese subtitles, English subtitles, etc.), captions (text that is overlaid on the video, etc.), audio (narration, etc.), music, images, and scene transitions (fade in, fade out, etc.). The editing screen provider 23 may also accept event addition operations in the editing screen 100 to insert other events such as double speed or normalize into the segment section object 103 or the silent section object 104.

[0034] The video editing server 2 (section detection unit 21) may analyze the video of each scene of the target video, which is divided by scene switching points, and predict candidate events (titles, subtitles, captions, audio, music, images, scene transitions) to be inserted into each scene using artificial intelligence. The editing screen provision unit 23 may provide the predicted event candidates on the editing screen 100.

[0035] Furthermore, the editing screen providing unit 23 may display scene screens 110, such as thumbnails and previews, corresponding to the video data of the target images in the selected object (phrase section object 103, silent section object 104, scene transition point object 105) in the video audio document 102, on the editing screen 100.

[0036] When the editing of the video audio document 102 is confirmed in response to the operation of the editing confirmation button 101 on the editing screen 100 provided to the editing screen provisioning unit 23, the video editing unit 24 performs transcoding of the target video, creates an edited video by editing the target video according to the editing information of the target video, and stores it in the storage unit 11. The video editing unit 24 stores the created edited video in the storage unit 11 and also sends or transfers (downloads) it to the terminal device 3 for output.

[0037] For example, if the video editing unit 24 confirms the deletion of a phrase section object 103, a silent section object 104, and a scene transition point object 105 on the editing screen 100, it will delete the corresponding phrase section, silent section, and scene transition point and edit the target video.

[0038] When the video editing unit 24 confirms the position change operation of the phrase section object 103, the silent section object 104, and the scene transition point object 105 on the editing screen 100, it edits the target video by changing the positions of the corresponding phrase sections, silent sections, and scene transition points to the specified changed positions.

[0039] When the splitting or merging operation of the segment section object 103 is confirmed on the editing screen 100, the video editing unit 24 edits the target video by splitting or merging the corresponding segment sections to form WebVTT units.

[0040] When the video editing unit 24 confirms the addition of events to the phrase section object 103, silent section object 104, and scene transition point object 105 on the editing screen 100, it adds events to the corresponding phrase sections, silent sections, and scene transition points and edits the target video.

[0041] The video editing unit 24 may automatically send or transfer the edited video to the terminal device 3 upon completion of the creation of the edited video, or it may transfer the edited video to the terminal device 3 in response to a download operation on the terminal device 3.

[0042] For example, the video editing unit 24 provides a website via the network 5 that allows users to download the edited video they have created, and allows users to download the edited video through this website displayed in the browser installed on the terminal device 3. The video editing unit 24 may accept the output (download) of the edited video through a website displaying the editing screen 100, or it may accept the output (download) of the edited video through a website displaying a dedicated output screen for editing video output.

[0043] Terminal device 3 is a device for the user to perform video editing operations, and is composed of, for example, a personal computer, smartphone, or tablet terminal. Terminal device 3 is composed of, for example, a control unit 30, a storage unit 31, a communication unit 32, a display unit 33, and an operation unit 34.

[0044] The control unit 30 provides overall control over all parts and functions of the terminal device 3. It is composed of a computer such as a CPU and is connected to the storage unit 31 and the communication unit 32. The storage unit 31 is composed of memory such as ROM or RAM and recording media such as a hard disk, and stores programs and data for controlling all parts and functions of the terminal device 3. The storage unit 31 stores target videos for video editing and edited videos edited by the video editing server 2. The communication unit 32 is an interface for the terminal device 3 to connect to the network 5, that is, it connects the terminal device 3 to the video editing server 2 via the network 5.

[0045] In this embodiment, the terminal device 3 is equipped with a browser and displays the editing screen 100 provided by the video editing server 2, which is connected via the network 5, on the display unit 33 using the browser. The terminal device 3 can perform operations such as uploading the target video, editing the video audio document 102 of the target video, and downloading the edited video using the operation unit 34 via the editing screen 100 displayed on the display unit 33 by the browser.

[0046] In this embodiment, as described above, the video editing system 1 receives the target video to be edited via the video input unit 20 in the video editing server 2, and the section detection unit 21 uses a machine learning model to divide the audio data of the target video into voiced and silent sections, divide the voiced sections into phrase sections, and detect scene transition points in the video, and transcribes the voiced sections of the audio data into phrase sections, detects scene transition points from the target video, and the editing screen providing unit 23 provides the text of the audio data of the target video. The user is provided with an editing screen 100 in which a segment segment object 103 indicating a segment, a silent segment object 104 indicating a silent segment of the audio data of the target video, and a scene transition point object 105 indicating a scene transition point of the target video are arranged in chronological order as a series of video-audio documents 102. On the editing screen 100, the user edits the video-audio documents 102 according to operations on the segment segment object 103, the silent segment object 104, and / or the scene transition point object 105, and the video editing unit 24 creates an edited video by editing the target video according to the editing results of the video-audio documents 102.

[0047] In other words, the video editing method of the present invention takes a target video to be edited as input, uses a machine learning model that divides the audio data of the target video into voiced and silent sections, divides the voiced sections into phrase sections, and detects scene transition points in the video, divides the audio data of the target video into voiced and silent sections, transcribes the voiced sections of the audio data of the target video into phrase sections, detects scene transition points from the target video, and provides the user with an editing screen 100 in which phrase section objects 103 indicating the phrase sections of the audio data of the target video, silent section objects 104 indicating the silent sections of the audio data of the target video, and scene transition point objects 105 indicating the scene transition points of the target video are arranged in chronological order as a series of video audio documents 102, and on the editing screen 100 the video audio documents 102 are edited according to the operation of the phrase section objects 103, silent section objects 104 and / or scene transition point objects 105, and an edited video is created by editing the target video according to the editing result of the video audio documents 102.

[0048] With this configuration, according to this embodiment, in the terminal device 3 that performs editing operations on the target video, the editing results of the video audio document 102, which consists of a segment section object 103 indicating a segment section of the audio data of the target video, a silent section object 104 indicating a silent section of the audio data of the target video, and a scene transition point object 105 indicating a scene transition point of the target video, are directly reflected in the editing of the target video, and an edited video is created. Therefore, in video editing, the terminal device 3 can easily identify editing points and utilize a user interface (UI) / user experience (UX) that is easy to use both visually and operationally. Thus, according to the video editing system 1 of the present invention, videos can be edited with simple operations without requiring high-spec hardware.

[0049] Furthermore, in this embodiment, the video editing system 1, in the video editing server 2, receives event addition operations on the segment section object 103, the silent section object 104, and / or the scene transition point object 105 in order to add predetermined events to the segment sections of the audio data, the silent sections of the audio data, and / or the scene transition points of the target video on the editing screen 100, and the video editing unit 24 creates an edited video by editing the target video in accordance with the event addition operations.

[0050] With this configuration, according to this embodiment, in the editing screen 100, simply by performing an event addition operation on the phrase section object 103, silent section object 104, and scene transition point object 105 of the video audio document 102, the target video can be edited to add events to the phrase sections, silent sections, and scene transition points.

[0051] In the embodiment described above, the editing screen providing unit 23 confirms the editing of the video audio document 102 in response to the operation of the editing confirmation button 101 on the editing screen 100, creates editing information for the target video corresponding to the editing of the video audio document 102, and the video editing unit 24 creates an edited video by editing the target video according to the editing information for the target video. However, the present invention is not limited to this example.

[0052] In another example, in response to the operation of the edit confirmation button 101 on the editing screen 100, the editing of the video audio document 102 is confirmed on the editing screen 100 of the terminal device 3, editing information for the target video corresponding to the editing of the video audio document 102 is created, the editing information for the target video is sent from the terminal device 3 to the video editing server 2, and the video editing unit 24 creates an edited video by editing the target video according to the editing information for the target video received from the terminal device 3.

[0053] The present invention may be modified as appropriate, provided that it does not contradict the gist or idea of ​​the invention as can be read from the claims and the specification as a whole, and video editing systems and video editing methods that involve such modifications are also included in the technical concept of the present invention. [Explanation of Symbols]

[0054] 1. Video editing system 2. Video editing server 3 Terminal devices 5 Network 10 Control Unit 11 Storage section 12 Communications Department 13 Display section 14 Control section 20 Video Input Section 21 Section detection unit 22 Translation Department 23 Editing Screen Provision Department 24 Video Editing Department 30 Control Unit 31 Storage section 32 Communications Department 33 Display section 34 Control section 100 Editing screen 101 Edit Confirmation Button 102 Video Audio Document 103 Sentence segment object 104 Silent interval object 105 Scene Transition Point Object 106 Event Objects 110 Scene Screen

Claims

1. Enter the video to be edited, Using a machine learning model that distinguishes between voiced and silent sections in a video, distinguishes between phrase sections within the voiced sections, and detects scene transitions in the video, the audio data of the target video is divided into voiced and silent sections, the voiced sections of the audio data are transcribed and divided into phrase sections, and scene transitions are detected from the target video. The system provides the user with an editing screen in which a series of video-audio documents are arranged chronologically, with a segment segment object indicating a segment segment of the audio data, a silent segment object indicating a silent segment of the audio data, and a scene transition point object indicating a scene transition point of the target video. A video editing system characterized in that, in the editing screen, the system edits the video audio document in accordance with the operation of the phrase section object, the silent section object, and / or the scene switching point object, and creates an edited video by editing the target video in accordance with the editing result of the video audio document.

2. The video editing system according to claim 1, characterized in that, in the editing screen, it accepts an event addition operation on the phrase section object, the silent section object and / or the scene switching point object in order to add a predetermined event to the position of the target video corresponding to the phrase section of the audio data, the silent section of the audio data and / or the scene switching point of the target video, and creates the edited video by editing the target video in accordance with the event addition operation.

3. Enter the video to be edited, Using a machine learning model that distinguishes between voiced and silent sections in a video, distinguishes between phrase sections within the voiced sections, and detects scene transitions in the video, the audio data of the target video is divided into voiced and silent sections, the voiced sections of the audio data are transcribed and divided into phrase sections, and scene transitions are detected from the target video. The system provides the user with an editing screen in which a series of video-audio documents are arranged chronologically, with a segment segment object indicating a segment segment of the audio data, a silent segment object indicating a silent segment of the audio data, and a scene transition point object indicating a scene transition point of the target video. A video editing method characterized by editing the video audio document in accordance with operations on the phrase section object, the silent section object, and / or the scene switching point object in the editing screen, and creating an edited video by editing the target video according to the editing result of the video audio document.

Images