Video Editing System Having Precise Time Synchronization and Clip Editing Functions

Abstract

A video editing system having precise time synchronization and clip editing functions according to the present invention includes a client that processes editing of a clip on a timeline of video editing software based on editing point information received from a server, and the client may include a time conversion unit that converts time information of an editing target section included in the editing point information into an integer value in tick units, which is an internal time unit of the video editing software, and calculates the timeline position of the editing target section based on integer operations, and an editing execution unit that processes editing of the clip by selecting one of a plurality of editing processing methods based on the timeline position and the position occupied by the editing target section within the corresponding clip.

Description

Technology Field

[0001] The present invention relates to a video editing system equipped with precise time synchronization and clip editing functions, and more specifically, to a video editing system that converts time information of a section to be edited into integer-based tick units to eliminate accumulated error and automatically selects an editing processing method according to a position within the clip to edit the clip. Background Technology

[0002] In the production of speech-centered video content such as interviews, lectures, and podcasts, raw footage typically contains unintentional speech elements, including speaker interjections, stuttering, unnecessary silences, and word repetition. Since editors must manually search for and remove these elements by listening to them one by one, the time and effort required for editing increase significantly as the video length grows. Furthermore, determining which speech is unnecessary relies on the editor's personal subjectivity, making it difficult to ensure consistency in the editing results.

[0003] To address these inconveniences, text-based editing methods utilizing speech recognition technology have been proposed. However, conventional text-based editing services are primarily designed for English-speaking languages, which has resulted in insufficient accuracy in detecting unnecessary utterances for languages ​​with unique phonological systems, such as Korean.

[0004] Furthermore, conventional methods for detecting unnecessary utterances tended to process the same words uniformly regardless of context, leaving editors with the inconvenience of having to re-examine and correct the results during actual editing. Although editing standards vary depending on the user's editing habits or content types, there was no structure in place for the system to learn and reflect these variations.

[0005] Meanwhile, even in attempts to automate editing through the extension features of video editing software, there was a difficulty in precisely splitting clips on the timeline or maintaining synchronization due to the limitations of the interface provided by the software.

[0006] Therefore, a method to resolve these problems is required. Prior art literature

[0007] Korean Patent Publication No. 10-2018-0129265 The problem to be solved

[0008] The present invention is an invention devised to solve the problems of the aforementioned prior art, and has the purpose of providing a video editing system capable of eliminating floating-point accumulation errors that occur when processing a section to be edited on the timeline of video editing software, and non-destructively performing clip splitting and restoration.

[0009] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0010] A video editing system having precise time synchronization and clip editing functions according to the present invention for achieving the above-mentioned purpose includes a client that processes editing of a clip on a timeline of video editing software based on editing point information received from a server, and the client may include a time conversion unit that converts time information of a section to be edited included in the editing point information into an integer value in tick units, which is an internal time unit of the video editing software, and calculates the timeline position of the section to be edited based on integer operations, and an editing execution unit that processes editing of the clip by selecting one of a plurality of editing processing methods according to the position occupied by the section to be edited within the corresponding clip based on the timeline position.

[0011] Additionally, the time conversion unit may perform a time conversion algorithm comprising the steps of: (a-1) acquiring time information in milliseconds included in the edit point information; (a-2) converting the time information in milliseconds into an integer value in ticks; and (a-3) calculating the current timeline position of the edit target section by subtracting the sum of ticks of sections deleted prior to the edit target section from the integer value.

[0012] At this time, in step (a-2) above, the integer value of the tick unit can be calculated using the number of ticks per second defined based on the least common multiple of a plurality of frame rates.

[0013] And, the above editing execution unit may perform a clip editing algorithm comprising: a step (b-1) of parsing the above editing point information to calculate a time interval for each editing target; a step (b-2) of determining a position within the corresponding clip for each of the above time intervals; a step (b-3) of selecting one of the above multiple editing processing methods according to the position; and a step (b-4) of editing the clip according to the selected editing processing method.

[0014] In addition, the above-mentioned multiple editing processing methods may include a total deletion method that ripples the entire clip when the section to be edited corresponds to the entire clip, a start point adjustment method that adjusts the in point of the clip when the section to be edited is located at the start point of the clip, an end point adjustment method that adjusts the out point of the clip when the section to be edited is located at the end point of the clip, and an intermediate splitting method that splits the clip and removes the corresponding section when the section to be edited is located in the middle of the clip.

[0015] In addition, if the intermediate splitting method is selected in step (b-3), step (b-4) may include: step (b-4-1) of creating a first duplicate clip and a second duplicate clip by duplicating the clip twice at the end of the sequence of the timeline; step (b-4-2) of changing the out point of the second duplicate clip to the start point of the section to be edited; step (b-4-3) of rippling the second duplicate clip to the original position of the clip; step (b-4-4) of deleting the second duplicate clip remaining at the end of the sequence; step (b-4-5) of changing the in point of the first duplicate clip to the end point of the section to be edited; step (b-4-6) of rippling the first duplicate clip to the back of the second duplicate clip; and step (b-4-7) of rippling the original of the clip.

[0016] Meanwhile, if the above-mentioned editing execution unit is deleted, it can restore the above-mentioned restoration target section by selecting one of a plurality of restoration processing methods depending on the existence and location of a clip adjacent to the restoration target section.

[0017] At this time, the plurality of restoration processing methods may include a bidirectional merging method in which, when adjacent clips exist on the left and right sides of the section to be restored, the left adjacent clip is duplicated and the out point is expanded, and then the left adjacent clip and the right adjacent clip are deleted; a left expansion method in which, when adjacent clips exist only on the left side of the section to be restored, the left adjacent clip is duplicated and the out point is expanded; a right expansion method in which, when adjacent clips exist only on the right side of the section to be restored, the right adjacent clip is duplicated and the in point is expanded; and a new creation method in which, when there are no adjacent clips in the section to be restored, an adjacent clip is duplicated to create a new clip corresponding to the section to be restored.

[0018] In addition, the present invention may further include an audio extraction unit that extracts audio from the video editing software and transmits it to the server.

[0019] And, the audio extraction unit may perform an audio extraction algorithm comprising the steps of: (c-1) identifying a plurality of audio tracks of the video editing software; (c-2) muting the remaining tracks excluding a designated track among the plurality of audio tracks; (c-3) rendering and extracting the audio of the designated track; (c-4) transmitting the extracted audio to the server; and (c-5) restoring the audio track muted in step (c-2) to its original state. Effects of the invention

[0020] The video editing system of the present invention, equipped with precise time synchronization and clip editing functions to solve the above-mentioned problem, has the advantage of being able to eliminate floating-point accumulation errors that may occur during the process of sequentially processing multiple editing target sections by converting time information included in editing point information into an integer value in tick units, which is an internal time unit of video editing software, and calculating the timeline position based on integer operations.

[0021] In addition, the present invention has the advantage that the removal of the section to be edited can be performed regardless of its position within the clip, as the editing execution unit automatically selects one of a plurality of editing processing methods based on the position occupied by the section to be edited within the corresponding clip.

[0022] Furthermore, the present invention has the advantage that the user can complete the editing work within the video editing software without going through a separate external tool, as the client directly edits the clip on the timeline of the video editing software based on edit point information received from the server.

[0023] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0024] FIG. 1 is a diagram showing the overall configuration of a video editing system according to one embodiment of the present invention. FIG. 2 is a diagram showing a time conversion algorithm performed by a time conversion unit in a video editing system according to an embodiment of the present invention. FIG. 3 is a diagram showing a clip editing algorithm performed by an editing execution unit in a video editing system according to an embodiment of the present invention. FIG. 4 is a diagram showing the detailed process of step (b-4) performed when an intermediate splitting method is selected among clip editing algorithms in a video editing system according to one embodiment of the present invention. FIG. 5 is a diagram showing an audio extraction algorithm performed by an audio extraction unit in a video editing system according to an embodiment of the present invention. Specific details for implementing the invention

[0025] In this specification, where a component (or region, layer, part, etc.) is described as being "on," "connected," or "combined" with another component, it means that it may be directly placed / connected / combined with the other component, or that a third component may be placed between them.

[0026] Identical reference numerals denote identical components. Additionally, in the drawings, the thicknesses, proportions, and dimensions of the components are exaggerated for the effective illustration of the technical content.

[0027] "And / or" includes all one or more combinations that the associated configurations can define.

[0028] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0029] Additionally, terms such as "below," "lower side," "above," and "upper side" are used to describe the relationships between the components depicted in the drawings. These terms are relative concepts and are described based on the directions indicated in the drawings.

[0030] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Additionally, terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and are explicitly defined herein unless interpreted in an ideal or overly formal sense.

[0031] Terms such as "include" or "have" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] Furthermore, when it is stated in this specification that a first component operates or is executed on (ON) a second component, it should be understood that the first component operates or is executed in an environment where the second component operates or is executed, or operates or is executed through direct or indirect interaction with the second component.

[0033] Where any component, device, or system is described as including a component consisting of a program or software, it should be understood that, even without explicit mention, that component, device, or system includes hardware (e.g., memory, CPU, etc.) or other programs or software (e.g., an operating system or drivers required to run the hardware) necessary for the execution or operation of that program or software.

[0034] Furthermore, unless otherwise specified regarding the implementation of a component, it should be understood that the component may be implemented in software, hardware, or both software and hardware.

[0035] Furthermore, the terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components to the mentioned components.

[0036] Additionally, in this specification, terms such as 'part', 'device', etc., may be intended to refer to hardware and the functional and structural combination of software driven by said hardware or for driving said hardware. For example, the hardware here may be a data processing device including a CPU or other processor. Furthermore, the software driven by the hardware may refer to a running process, object, executable, thread of execution, program, etc.

[0037] Furthermore, it can be easily inferred by an average expert in the art of the present invention that the above terms may refer to a specific code and a logical unit of hardware resources for executing the said specific code, and do not necessarily refer to physically connected code or a single type of hardware.

[0038] FIG. 1 is a diagram showing the overall configuration of a video editing system according to one embodiment of the present invention.

[0039] As illustrated in FIG. 1, the video editing system in this embodiment may include a client (100) and a server (200). The client (100) and the server (200) can transmit and receive data to and from each other via a network, and the client (100) can operate in conjunction with video editing software.

[0040] In this embodiment, the client (100) can edit a clip on the timeline of the video editing software based on edit point information received from the server (200). The client (100) is a computing device on which the video editing software is installed and executed, and can be implemented as a desktop computer, laptop, workstation, etc. However, the client (100) can be implemented in various forms and is not limited to such forms. The client (100) may include a processor, memory, and a network interface, and can drive the time conversion unit (110), the editing execution unit (120), and the audio extraction unit (130) in the form of a plugin or extension program of the video editing software.

[0041] In this embodiment, the client (100) may include a time conversion unit (110) and an editing execution unit (120). Additionally, the client (100) may further include an audio extraction unit (130).

[0042] The time conversion unit (110) converts the time information of the target section to be edited included in the edit point information into an integer value in tick units, which is an internal time unit of the video editing software, and can calculate the timeline position of the target section to be edited based on integer operations. The video editing software can internally use a time unit called a tick to represent a position on the timeline, and such a tick may correspond to a unit in which 1 second is divided into very large integers. By converting the time information in millisecond units included in the edit point information into an integer value in tick units, the time conversion unit (110) can eliminate errors associated with floating-point operations. Additionally, the time conversion unit (110) can calculate the current timeline position of the target section to be edited by subtracting the sum of tick values ​​of sections deleted prior to the target section to be edited, and since this subtraction process is performed as an integer operation, cumulative errors that may occur during the process of sequentially processing multiple target sections to be edited can be prevented.

[0043] The editing execution unit (120) can edit a clip by selecting one of a plurality of editing processing methods based on the timeline position calculated by the time conversion unit (110) and the position that the section to be edited occupies within the corresponding clip. That is, the editing execution unit (120) can automatically select different editing processing methods depending on whether the section to be edited corresponds to the entire clip, is located at the beginning of the clip, is located at the end of the clip, or is located in the middle of the clip. Additionally, if the section to be edited is deleted, the editing execution unit (120) can restore the section to be restored by selecting one of a plurality of restoration processing methods based on the existence and position of a clip adjacent to the section to be restored.

[0044] The audio extraction unit (130) can extract audio from video editing software and transmit it to the server (200). The audio extraction unit (130) can separate audio data from a video placed on the timeline of the video editing software and transmit the separated audio data to the server (200). At this time, the audio extraction unit (130) can select and extract a track containing the speaker's voice among a plurality of audio tracks of the video editing software, thereby ensuring that background music or sound effects are not included in the audio data.

[0045] The server (200) can analyze voice data received from the client (100) to generate edit point information and transmit the edit point information to the client (100). In this embodiment, the server (200) may include a voice recognition processing unit (210), an edit point analysis unit (220), an artificial intelligence verification unit (230), an edit point transmission unit (240), and a feedback learning unit (250). The server (200) may be implemented as a physical server, a cloud-based virtual server, or a distributed processing system composed of multiple servers. However, the server (200) may be implemented in various forms and is not limited to such forms. The server (200) may include a processor, memory, a storage device, and a network interface, and may operate the voice recognition processing unit (210), the edit point analysis unit (220), the artificial intelligence verification unit (230), the edit point transmission unit (240), and the feedback learning unit (250) in software.

[0046] The client (100) and the server (200) may be connected via a wired or wireless network and may transmit and receive data using communication protocols such as HTTP, HTTPS, and WebSocket. However, the communication method between the client (100) and the server (200) may be implemented in various forms and is not limited to such forms.

[0047] The speech recognition processing unit (210) can perform speech recognition on speech data received from the client (100) to generate text conversion data including time information per word. Such text conversion data may include time information regarding the start and end times of speech of each word included in the speech data, and the speech recognition processing unit (210) can generate such text conversion data by utilizing speech-to-text (STT) technology.

[0048] The edit point analysis unit (220) can detect edit point candidates from text conversion data and determine the edit target based on the confidence level calculated for each edit point candidate. Such edit point candidates may include sections corresponding to unintentional utterances, such as interjections, stuttering, unnecessary silent intervals, and word repetitions, and the edit point analysis unit (220) can calculate a confidence level, which is a numerical value indicating the need for editing, for each edit point candidate. Based on the calculated confidence level, the edit point analysis unit (220) can determine the edit point candidates that are judged to require editing as the edit target.

[0049] The artificial intelligence verification unit (230) may request verification from an external artificial intelligence model only for edit point candidates whose reliability calculated by the edit point analysis unit (220) falls within an intermediate range between a predetermined upper threshold and a predetermined lower threshold. Additionally, the artificial intelligence verification unit (230) may determine edit point candidates whose reliability is above the upper threshold as targets for editing, and determine edit point candidates whose reliability is below the lower threshold as unnecessary for editing. Therefore, since the request for verification to the external artificial intelligence model is limited to edit point candidates in the intermediate range, the number of calls to the artificial intelligence model can be reduced.

[0050] The edit point transmission unit (240) can transmit the edit target determined through the edit point analysis unit (220) and the artificial intelligence verification unit (230) to the client (100) as edit point information. Such edit point information may include word-specific time information, edit point type, and reliability corresponding to each edit target. The edit point transmission unit (240) can serialize the edit point information into a predetermined data format such as JSON or XML, and transmit the serialized edit point information to the client (100) via a network. However, the data format used for serialization can be implemented in various forms and is not limited to such forms.

[0051] The feedback learning unit (250) can receive acceptance or rejection feedback regarding an editing target input through the client (100). Based on the received feedback, the feedback learning unit (250) calculates the acceptance rate of editing point candidates by pattern and can adjust the reliability calculated by the editing point analysis unit (220) based on the calculated acceptance rate. Accordingly, as the user repeatedly accepts or rejects an editing target, the reliability calculation of the editing point analysis unit (220) can be adjusted to match the user's editing tendency.

[0052] Below, the algorithms performed in each of the time conversion unit (110), the editing execution unit (120), and the audio extraction unit (130) will be described in detail.

[0053] FIG. 2 is a diagram showing a time conversion algorithm performed by a time conversion unit (110) in a video editing system according to one embodiment of the present invention.

[0054] As illustrated in FIG. 2, in this embodiment, the time conversion unit (110) can perform a time conversion algorithm. Such a time conversion algorithm may include steps (a-1), (a-2), and (a-3).

[0055] Step (a-1) is a process of obtaining time information in milliseconds included in the edit point information. The time conversion unit (110) can parse the edit point information received from the edit point transmission unit (240) of the server (200) to obtain time information in milliseconds corresponding to the start and end times of each edit target section. Such time information in milliseconds can correspond to the start and end times of each word-by-word utterance included in the text conversion data generated by the speech recognition processing unit (210) of the server (200).

[0056] Step (a-2) is a process of converting time information in millisecond units into an integer value in tick units. The time conversion unit (110) can convert the time information in millisecond units obtained in step (a-1) into an integer value in tick units, which is an internal time unit of video editing software. At this time, the integer value in tick units can be calculated using the number of ticks per second defined based on the least common multiple of multiple frame rates. For example, the number of ticks per second can be defined as a value based on the least common multiple of these frame rates so that integer frame boundaries can be expressed for multiple frame rates such as 24fps, 25fps, 29.97fps, 30fps, and 60fps. The time conversion unit (110) can calculate the integer value in tick units through an operation of multiplying the time information in millisecond units by the number of ticks per second and dividing by 1000, and such an operation can be performed using an integer data type such as BigInt. Accordingly, errors associated with floating-point operations may not occur.

[0057] Step (a-3) is a process of calculating the current timeline position of the section to be edited by subtracting the sum of ticks of sections deleted prior to the section to be edited from an integer value. The time conversion unit (110) can sum the tick values ​​of sections that were already deleted chronologically prior to the section to be edited currently being processed while processing the section to be edited sequentially. The time conversion unit (110) can calculate the current timeline position of the section to be edited by subtracting the summed tick value from the tick unit integer value of the section to be edited converted in step (a-2). Since this subtraction process is performed as an integer operation, floating-point accumulation error may not occur even when processing multiple sections to be edited sequentially. Whenever the deletion of a section to be edited is completed, the time conversion unit (110) can add the tick length of the corresponding section to the sum value and reflect it in the calculation of the timeline position of the next section to be edited.

[0058] FIG. 3 is a diagram showing a clip editing algorithm performed by an editing execution unit (120) in a video editing system according to one embodiment of the present invention.

[0059] As illustrated in FIG. 3, in this embodiment, the editing execution unit (120) can perform a clip editing algorithm. Such a clip editing algorithm may include steps (b-1), (b-2), (b-3), and (b-4).

[0060] Step (b-1) is a process of parsing edit point information to calculate time intervals for each edit target. The edit execution unit (120) can parse edit point information received from the edit point transmission unit (240) to extract the start and end times of speech corresponding to each edit target. The edit execution unit (120) can calculate time intervals for each edit target based on the extracted start and end times. Such time intervals may correspond to intervals that are subject to removal or editing processing on the timeline of the video editing software.

[0061] Step (b-2) is a process of determining the position within the corresponding clip for each time interval. For each time interval calculated in Step (b-1), the editing execution unit (120) can search to see which clip on the timeline the time interval belongs to and determine the position within the corresponding clip. For example, it can be determined whether the time interval corresponds to the entire clip, is located at the beginning of the clip, is located at the end of the clip, or is located in the middle of the clip. Such position determination can be referenced to select an editing processing method in the subsequent Step (b-3).

[0062] Step (b-3) is a process of selecting one of a plurality of editing processing methods based on the location. The editing execution unit (120) can select an editing processing method for the corresponding clip based on the location determined in Step (b-2). In this embodiment, the plurality of editing processing methods may include a total deletion method, a start point adjustment method, an end point adjustment method, and an intermediate splitting method. The total deletion method is a method of ripple-deleting the entire clip when the section to be edited corresponds to the entire clip. The start point adjustment method is a method of adjusting the in point of the clip when the section to be edited is located at the start point of the clip. The end point adjustment method is a method of adjusting the out point of the clip when the section to be edited is located at the end point of the clip. The intermediate splitting method is a method of splitting the clip and removing the corresponding section when the section to be edited is located in the middle of the clip. In this way, the editing execution unit (120) can automatically select different editing processing methods based on the location of the section to be edited within the clip.

[0063] Step (b-4) is a process of editing a clip according to the selected editing processing method. The editing execution unit (120) can perform editing processing on the clip on the timeline of the video editing software according to the editing processing method selected in Step (b-3). Such editing processing may include deleting ripples from the clip, adjusting the in or out points of the clip, splitting the clip, and removing the split sections. After the editing processing is completed, the editing execution unit (120) can perform ripple processing so that the subsequent clip fills the empty sections created by the editing processing on the timeline. Accordingly, the continuity of the timeline can be maintained. The detailed process of Step (b-4) when an intermediate splitting method is selected in Step (b-3) will be described later with reference to FIG. 4.

[0064] Additionally, the editing execution unit (120) can restore the section to be restored by selecting one of a plurality of restoration processing methods based on the existence and location of clips adjacent to the section to be restored when the section to be edited is deleted. In this embodiment, the plurality of restoration processing methods may include a bidirectional merging method, a left expansion method, a right expansion method, and a new creation method. The bidirectional merging method is a method in which, when adjacent clips exist on the left and right sides of the section to be restored, the left adjacent clip is duplicated, the out point is expanded, and then the left adjacent clip and the right adjacent clip are deleted. The left expansion method is a method in which, when adjacent clips exist only on the left side of the section to be restored, the left adjacent clip is duplicated and the out point is expanded. The right expansion method is a method in which, when adjacent clips exist only on the right side of the section to be restored, the right adjacent clip is duplicated and the in point is expanded. The new creation method is a method in which, when there are no adjacent clips in the section to be restored, adjacent clips are duplicated to create a new clip corresponding to the section to be restored. In this way, the editing execution unit (120) can automatically select different restoration processing methods depending on the surrounding clip conditions of the section to be restored, so that word-unit selective restoration of the deleted section can be performed.

[0065] FIG. 4 is a diagram showing the detailed process of step (b-4) performed when an intermediate splitting method is selected among clip editing algorithms in a video editing system according to one embodiment of the present invention.

[0066] As illustrated in FIG. 4, in this embodiment, if an intermediate splitting method is selected in step (b-3), step (b-4) may include steps (b-4-1), (b-4-2), (b-4-3), (b-4-4), (b-4-5), (b-4-6), and (b-4-7) in detail.

[0067] Step (b-4-1) is a process of creating a first duplicate clip and a second duplicate clip by duplicating the clip twice at the end of the timeline sequence. The editing execution unit (120) can duplicate the original clip where the section to be edited is located twice at the end of the timeline sequence. By utilizing the end of the sequence as a temporary workspace in this way, operations on the duplicate clip can be performed without interference with other clips placed on the existing timeline. A first duplicate clip and a second duplicate clip can be created by duplicating twice.

[0068] Step (b-4-2) is a process of changing the out point of the second duplicate clip to the starting point of the section to be edited. The editing execution unit (120) can change the out point of the second duplicate clip to a position corresponding to the starting point of the section to be edited. Accordingly, the second duplicate clip can be formed to include only the section from the starting point of the original clip to the starting point of the section to be edited.

[0069] Step (b-4-3) is a process of ripple-moving the second duplicate clip to the original position of the clip. The editing execution unit (120) can ripple-move the second duplicate clip, whose out point was changed in step (b-4-2), to the original position on the timeline where the original clip is located. Such ripple-moving can be performed in the form of insertion editing and can be configured so that the second duplicate clip is inserted in front of the original clip.

[0070] Step (b-4-4) is a process of deleting the second duplicate clip remaining at the end of the sequence. The editing execution unit (120) can delete the second duplicate clip remaining at the end of the sequence after the ripple movement is completed in step (b-4-3). Accordingly, the temporary workspace at the end of the sequence can be cleaned up.

[0071] Step (b-4-5) is a process of changing the in point of the first duplicate clip to the end point of the section to be edited. The editing execution unit (120) can change the in point of the first duplicate clip remaining at the end of the sequence to a position corresponding to the end point of the section to be edited. Accordingly, the first duplicate clip can be formed to include only the section from the end point of the section to be edited to the end point of the original clip.

[0072] Step (b-4-6) is a process of moving the first duplicate clip behind the second duplicate clip. The editing execution unit (120) can move the first duplicate clip, whose point was changed in step (b-4-5), to be positioned behind the second duplicate clip moved in step (b-4-3) on the timeline. Accordingly, the second duplicate clip and the first duplicate clip can be placed consecutively on the timeline, and the portion corresponding to the editing target section between the two clips can be excluded.

[0073] Step (b-4-7) is a process of ripple-deleting the original clip. The editing execution unit (120) can ripple-delete the original clip remaining on the timeline. Accordingly, the section occupied by the original clip is removed, and the second duplicate clip and the first duplicate clip can be positioned to replace the position of the original clip. Through steps (b-4-1) through (b-4-7) such as this, the editing target section located in the middle of the clip can be divided and removed even if the video editing software does not provide a direct razor or cut command.

[0074] FIG. 5 is a diagram showing an audio extraction algorithm performed by an audio extraction unit (130) in a video editing system according to one embodiment of the present invention.

[0075] As illustrated in FIG. 5, in this embodiment, the audio extraction unit (130) can perform an audio extraction algorithm. Such an audio extraction algorithm may include steps (c-1), (c-2), (c-3), (c-4), and (c-5).

[0076] Step (c-1) is a process of identifying multiple audio tracks of video editing software. In addition to voice tracks, multiple audio tracks containing background music, sound effects, etc., may exist on the timeline of the video editing software. The audio extraction unit (130) can identify multiple audio tracks included in the timeline in step (c-1) and obtain index and attribute information of each audio track.

[0077] Step (c-2) is a process of muting the remaining tracks among multiple audio tracks, excluding the designated track. The audio extraction unit (130) can set the track containing the voice of the speaker that is the subject of speech recognition as the designated track and mute the remaining audio tracks. For example, if the speaker's voice in an interview video is placed in the first audio track, the audio extraction unit (130) can mute the remaining audio tracks excluding the first audio track. Through such muting, background music or sound effects can be prevented from being included in the audio extraction result.

[0078] Step (c-3) is a process of rendering and extracting the audio of a designated track. The audio extraction unit (130) can render the audio of the designated track into a predetermined audio format and extract it as a file while the remaining tracks are muted in step (c-2). Through this rendering process, an audio file containing only the speaker's voice can be generated, thereby improving the accuracy of subsequent speech recognition processing.

[0079] Step (c-4) is the process of transmitting the extracted audio to the server (200). The audio extraction unit (130) can transmit the audio file extracted in step (c-3) to the server (200). The voice recognition processing unit (210) of the server (200) can receive the transmitted audio file and perform voice recognition.

[0080] Step (c-5) is a process of restoring the audio track that was muted in Step (c-2) to its original state. The audio extraction unit (130) can restore the remaining audio track that was muted in Step (c-2) to its original state after the transmission of the audio file is completed in Step (c-4). Accordingly, the muted state of the audio track changed during the audio extraction process can be prevented from affecting the timeline of the video editing software.

[0081] For the above, a video editing system equipped with precise time synchronization and clip editing functions according to one embodiment of the present invention has been described in detail. Below, algorithms that can be additionally applied to the present invention will be described.

[0082] First, in this embodiment, the time conversion unit (110) can perform a frame snap correction algorithm that corrects the start and end points of the edit target section converted in tick units to the nearest frame boundary. This frame snap correction algorithm can be performed between steps (a-2) and (a-3) of the time conversion algorithm.

[0083] The time conversion unit (110) can calculate the number of ticks per frame corresponding to the frame rate set in the current timeline for the integer value of the tick unit converted in step (a-2). For example, if the number of ticks per second is defined as a predetermined value and the frame rate is 30fps, the number of ticks per frame can be calculated as an integer value obtained by dividing the number of ticks per second by 30. The time conversion unit (110) calculates the remainder obtained by dividing the integer value of the tick corresponding to the starting point of the section to be edited by the number of ticks per frame, and if the remainder is not 0, it can determine that the starting point is not aligned with the frame boundary.

[0084] The time conversion unit (110) can correct the starting point to the closer of the previous frame boundary or the next frame boundary if the starting point is not aligned with the frame boundary. In this case, the time conversion unit (110) can correct to the previous frame boundary if the remaining value is less than half of the number of ticks per frame, and correct to the next frame boundary if the remaining value is more than half of the number of ticks per frame. The time conversion unit (110) can also perform snap correction on the ending point in the same way.

[0085] However, the time conversion unit (110) may selectively apply a conservative snap mode that corrects the starting point in a direction that reduces the editing target section and corrects the ending point in a direction that reduces the editing target section. When such a conservative snap mode is applied, the starting point may be corrected to the boundary of the immediately following frame, and the ending point may be corrected to the boundary of the immediately following frame. Accordingly, it is prevented that valid utterances outside the editing target section are unintentionally deleted.

[0086] Additionally, the time conversion unit (110) may exclude the section to be edited from the editing target if the tick interval between the start point and the end point of the section to be edited after snap correction is less than the number of ticks per frame. Due to this exclusion process, if the section to be edited is reduced to less than one frame by snap correction, the editing process may not be performed.

[0087] With this frame snap correction algorithm, the start and end points of the section to be edited are aligned with the frame boundaries, so it is possible to prevent unintended remaining frames from being left on the timeline due to a cut occurring in the middle of the frame.

[0088] Next, in this embodiment, the editing execution unit (120) can perform an editing history stack-based multiple restoration algorithm that stores the editing history in a stack structure whenever editing processing is performed, and sequentially pops the editing history from the stack according to the user's restoration request to perform restoration in reverse order.

[0089] The editing execution unit (120) may generate an editing history for the corresponding editing process and push it onto a stack whenever the editing process is completed in step (b-4) of the clip editing algorithm. Such an editing history may include the tick-unit start and end points of the section to be edited, the selected editing processing method, clip state information before the editing process, and clip state information after the editing process. The clip state information may include the in point, out point, position on the timeline, and clip identifier of the corresponding clip. Additionally, if the editing process is performed by an intermediate splitting method, the editing history may additionally include the identifiers of the first duplicate clip and the second duplicate clip generated in each of steps (b-4-1) through (b-4-7), and the identifier of the original clip.

[0090] When the editing execution unit (120) receives a restoration request from a user, it can pop an editing history from the top of the stack and perform the editing processing recorded in the editing history in reverse order. For example, if an editing history deleted by the total deletion method is popped, the editing execution unit (120) can refer to the clip status information included in the editing history and restore the deleted clip to its original timeline position. Additionally, if an editing history divided by the intermediate splitting method is popped, the editing execution unit (120) can delete the first duplicate clip and the second duplicate clip and restore the original clip to its original timeline position.

[0091] The edit execution unit (120) can repeat reverse restoration by sequentially popping edit history from the stack when the user continuously inputs restoration requests. Accordingly, the user can restore sequentially starting from the most recently performed edit processing. Additionally, if the user inputs a restoration request while the stack is empty, the edit execution unit (120) can notify the user through the client (100) that there is no edit history to restore.

[0092] Such an editing history stack-based multiple restoration algorithm can operate independently of the undo function of the video editing software itself and can provide an independent restoration path for the editing process of the present invention. However, the maximum storage depth of the editing history stack may be limited to a predetermined value, and when the storage depth of the stack reaches the maximum value, the oldest editing history may be removed from the bottom of the stack.

[0093] Next, in this embodiment, the editing execution unit (120) may perform a clip boundary collision verification algorithm to verify whether the editing target section spans multiple clips before determining the location in step (b-2) of the clip editing algorithm.

[0094] The editing execution unit (120) can determine whether the clip where the starting point of the time interval is located and the clip where the ending point is located are the same for each time interval for each editing target calculated in step (b-1). If the starting point and the ending point are located within the same clip, the editing execution unit (120) determines the editing target interval as an editing target within a single clip and can perform the process after step (b-2).

[0095] On the other hand, if the clip where the starting point is located and the clip where the ending point is located are different, the editing execution unit (120) can determine that the section to be edited spans across multiple clips. In this case, the editing execution unit (120) can divide the section to be edited into multiple sub-sections based on the clip boundaries. For example, if the section to be edited spans a first clip and a second clip, the editing execution unit (120) can divide the section from the starting point of the section to be edited to the ending point of the first clip into a first sub-section, and the section from the starting point of the second clip to the ending point of the section to be edited into a second sub-section.

[0096] Additionally, if the section to be edited spans three or more clips, the editing execution unit (120) can identify an intermediate clip existing between the clip where the starting point is located and the clip where the ending point is located. The editing execution unit (120) can classify the entire intermediate clip into a single sub-section and apply a total deletion method to the sub-section.

[0097] The editing execution unit (120) can perform steps (b-2) through (b-4) individually for each divided sub-section. That is, for the first sub-section, an editing processing method can be selected according to the position within the first clip, and for the second sub-section, an editing processing method can be selected according to the position within the second clip. At this time, since the first sub-section is located at the end point of the first clip, an end point adjustment method can be selected, and since the second sub-section is located at the start point of the second clip, a start point adjustment method can be selected.

[0098] Through this clip boundary collision verification algorithm, even if the time interval of the speech recognition result does not match the clip boundary on the timeline, the section to be edited can be divided into clip units, and an appropriate editing processing method can be applied to each clip.

[0099] Preferred embodiments according to the present invention have been described above, and it is obvious to those skilled in the art that, in addition to the embodiments described above, the present invention may be embodied in other specific forms without departing from the spirit or scope thereof. Therefore, the embodiments described above should be regarded as illustrative rather than restrictive, and accordingly, the present invention is not limited to the description above but may be modified within the scope of the appended claims and their equivalents. Explanation of the symbols

[0100] 100: Client 110: Time conversion unit 120: Edit execution section 130: Audio Extraction Section 200: Server 210: Speech recognition processing unit 220: Edit Point Analysis Section 230: Artificial Intelligence Verification Department 240: Edit point transmission section 250: Feedback Learning Department

Claims

Claim 1 A video editing system comprising a client that processes editing of a clip on a timeline of video editing software based on editing point information received from a server, wherein the client comprises: a time conversion unit that converts time information of an editing target section included in the editing point information into an integer value in tick units, which is an internal time unit of the video editing software, and calculates the timeline position of the editing target section based on integer operations; and an editing execution unit that processes editing of the clip by selecting one of a plurality of editing processing methods according to the position occupied by the editing target section within the corresponding clip based on the timeline position; wherein the time conversion unit performs a time conversion algorithm comprising: (a-1) a step of acquiring time information in millisecond units included in the editing point information; (a-2) a step of converting the time information in millisecond units into an integer value in tick units; and (a-3) a step of calculating the current timeline position of the editing target section by subtracting the sum of ticks of sections deleted prior to the editing target section from the integer value. Claim 2 A video editing system according to claim 1, wherein in step (a-2), the integer value of the tick unit is calculated using the number of ticks per second defined based on the least common multiple of a plurality of frame rates. Claim 3 A video editing system comprising a client that processes editing on a clip on the timeline of video editing software based on edit point information received from a server, wherein the client comprises: a time conversion unit that converts time information of an editing target section included in the edit point information into an integer value in tick units, which is an internal time unit of the video editing software, and calculates the timeline position of the editing target section based on integer operations; and an editing execution unit that processes editing on the clip by selecting one of a plurality of editing processing methods according to the position occupied by the editing target section within a corresponding clip based on the timeline position; wherein the editing execution unit comprises: a step (b-1) of parsing the edit point information to calculate a time section for each editing target; a step (b-2) of determining the position within a corresponding clip for each of the time sections; and a step (b-3) of selecting one of the plurality of editing processing methods according to the position. A video editing system comprising: a clip editing algorithm including step (b-4) of editing the clip according to the selected editing processing method; wherein the plurality of editing processing methods include: a total deletion method that ripples and deletes the entire clip when the editing target section corresponds to the entire clip; a start point adjustment method that adjusts the in point of the clip when the editing target section is located at the start point of the clip; an end point adjustment method that adjusts the out point of the clip when the editing target section is located at the end point of the clip; and an intermediate splitting method that splits the clip and removes the corresponding section when the editing target section is located in the middle of the clip. Claim 4 In claim 3, if the intermediate splitting method is selected in step (b-3), step (b-4) comprises: (b-4-1) a step of creating a first duplicate clip and a second duplicate clip by duplicating the clip twice at the end of the sequence of the timeline; (b-4-2) a step of changing the out point of the second duplicate clip to the start point of the section to be edited; (b-4-3) a step of rippling the second duplicate clip to the original position of the clip; (b-4-4) a step of deleting the second duplicate clip remaining at the end of the sequence; (b-4-5) a step of changing the in point of the first duplicate clip to the end point of the section to be edited; (b-4-6) a step of rippling the first duplicate clip to the back of the second duplicate clip; and (b-4-7) a step of rippling the original of the clip. Claim 5 A video editing system comprising a client that processes editing of a clip on a timeline of video editing software based on edit point information received from a server, wherein the client comprises a time conversion unit that converts time information of an editing target section included in the edit point information into an integer value in tick units, which is an internal time unit of the video editing software, and calculates the timeline position of the editing target section based on integer operations; A video editing system comprising: an editing execution unit that, based on the timeline position, selects one of a plurality of editing processing methods to edit the clip according to the position the editing target section occupies within the corresponding clip; wherein, when the editing target section is deleted, the editing execution unit selects one of a plurality of restoration processing methods to restore the editing target section according to the existence and position of a clip adjacent to the restoration target section, and the plurality of restoration processing methods include: a bidirectional merging method in which, when adjacent clips exist on the left and right sides of the restoration target section respectively, the left adjacent clip is duplicated and the out point is expanded, and then the left adjacent clip and the right adjacent clip are deleted; a left expansion method in which, when an adjacent clip exists only on the left side of the restoration target section, the left adjacent clip is duplicated and the out point is expanded; a right expansion method in which, when an adjacent clip exists only on the right side of the restoration target section, the right adjacent clip is duplicated and the in point is expanded; and a new creation method in which, when no adjacent clip exists in the restoration target section, an adjacent clip is duplicated to create a new clip corresponding to the restoration target section. Claim 6 A video editing system comprising a client that processes editing of a clip on the timeline of video editing software based on edit point information received from a server, wherein the client comprises: a time conversion unit that converts time information of an editing target section included in the edit point information into an integer value in tick units, which is an internal time unit of the video editing software, and calculates the timeline position of the editing target section based on integer operations; an editing execution unit that processes editing of the clip by selecting one of a plurality of editing processing methods according to the position occupied by the editing target section within the corresponding clip based on the timeline position; and an audio extraction unit that extracts audio from the video editing software and transmits it to the server; wherein the audio extraction unit comprises: a step (c-1) of identifying a plurality of audio tracks of the video editing software; a step (c-2) of muting the remaining tracks excluding a designated track among the plurality of audio tracks; a step (c-3) of rendering and extracting the audio of the designated track; and a step (c-4) of transmitting the extracted audio to the server. A video editing system that performs an audio extraction algorithm comprising: a step (c-5) of restoring the audio track muted in step (c-2) to its original state. Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 delete

Images