Video Editing System for Speech Recognition-Based Edit Point Detection and Clip Editing Based on Server-Client Interworking

Abstract

A video editing system for detecting edit points and editing clips based on server-client linkage according to the present invention comprises a server and a client, wherein the client includes an audio extraction unit that extracts audio from video editing software and transmits it to the server, and an editing execution unit that receives edit point information from the server and processes the editing of a clip of the video editing software based on the edit point information, and the server may include a voice recognition processing unit that performs voice recognition on the audio received from the client and generates text conversion data including time information per word, an edit point analysis unit that detects edit point candidates from the text conversion data and determines an edit target based on the reliability calculated for each edit point candidate, and an edit point transmission unit that transmits the edit target to the client as the edit point information.

Description

Technology Field

[0001] The present invention relates to a video editing system, and more specifically, to a video editing system that performs voice recognition-based edit point detection and clip editing through the interaction of a server and a client. Background Technology

[0002] In the production of speech-centered video content such as interviews, lectures, and podcasts, raw footage typically contains unintentional speech elements, including speaker interjections, stuttering, unnecessary silences, and word repetition. Since editors must manually search for and remove these elements by listening to them one by one, the time and effort required for editing increase significantly as the video length grows. Furthermore, determining which speech is unnecessary relies on the editor's personal subjectivity, making it difficult to ensure consistency in the editing results.

[0003] To address these inconveniences, text-based editing methods utilizing speech recognition technology have been proposed. However, conventional text-based editing services are primarily designed for English-speaking languages, which has resulted in insufficient accuracy in detecting unnecessary utterances for languages ​​with unique phonological systems, such as Korean.

[0004] Furthermore, conventional methods for detecting unnecessary utterances tended to process the same words uniformly regardless of context, leaving editors with the inconvenience of having to re-examine and correct the results during actual editing. Although editing standards vary depending on the user's editing habits or content types, there was no structure in place for the system to learn and reflect these variations.

[0005] Meanwhile, even in attempts to automate editing through the extended features of video editing software, there was a difficulty in precisely splitting clips on the timeline or maintaining synchronization due to the limitations of the interface provided by the software.

[0006] Therefore, a method to resolve these problems is required. Prior art literature

[0007] Korean Patent Publication No. 10-2018-0129265 The problem to be solved

[0008] The present invention is devised to solve the problems of the aforementioned prior art and aims to provide a video editing system capable of automatically detecting unnecessary speech elements within a video, determining editing targets based on context, and precisely reflecting them on the timeline of video editing software.

[0009] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0010] A video editing system for detecting edit points and editing clips based on server-client interoperability according to the present invention for achieving the above-mentioned purpose comprises a server and a client, wherein the client includes an audio extraction unit that extracts audio from video editing software and transmits it to the server, and an editing execution unit that receives edit point information from the server and processes the editing of a clip of the video editing software based on the edit point information, and the server may include a speech recognition processing unit that performs speech recognition on the audio received from the client and generates text conversion data including time information per word, an edit point analysis unit that detects edit point candidates from the text conversion data and determines an edit target based on the reliability calculated for each edit point candidate, and an edit point transmission unit that transmits the edit target to the client as the edit point information.

[0011] Additionally, the audio extraction unit may perform an audio extraction algorithm comprising the steps of: (a-1) identifying a plurality of audio tracks of the video editing software; (a-2) muting the remaining tracks excluding a designated track among the plurality of audio tracks; (a-3) rendering and extracting the audio of the designated track; and (a-4) transmitting the extracted audio to the server.

[0012] In addition, the above edit point analysis unit may perform a context-based reliability calculation algorithm comprising the steps of (b-1) obtaining the context before and after each word from the text conversion data, (b-2) determining the context type of each word based on the context before and after, and (b-3) calculating different reliability for the same word according to the context type.

[0013] In addition, the edit point analysis unit may perform an edit point detection algorithm comprising: a step (c-1) of detecting silent intervals from the text conversion data and classifying silent intervals exceeding a dynamic threshold calculated based on speaker-specific utterance patterns as edit point candidates; a step (c-2) of detecting words corresponding to interjections from the text conversion data and classifying them into a predetermined category; a step (c-3) of detecting utterance repetition by analyzing the similarity between consecutive words from the text conversion data; and a step (c-4) of synthesizing the results detected in steps (c-1) through (c-3) as edit point candidates.

[0014] Meanwhile, the present invention may further include an artificial intelligence verification unit that requests verification from an external artificial intelligence model only for edit point candidates whose reliability falls within an intermediate interval between a predetermined upper threshold and a predetermined lower threshold, confirms edit point candidates above the upper threshold as targets for editing, and confirms edit point candidates below the lower threshold as unnecessary for editing.

[0015] In addition, the artificial intelligence verification unit may perform an artificial intelligence verification algorithm comprising: a step (d-1) of determining the domain type of the edit point candidate; a step (d-2) of selecting an artificial intelligence prompt corresponding to the domain type; a step (d-3) of grouping the edit point candidates corresponding to the intermediate section into a predetermined unit; and a step (d-4) of requesting verification from the external artificial intelligence model together with the artificial intelligence prompt for each of the grouped units.

[0016] Additionally, the above edit point transmission unit may perform an edit point transmission algorithm comprising the steps of: (e-1) generating edit point information including word-specific time information, edit point type, and reliability corresponding to each of the edit targets; (e-2) serializing the edit point information into a predetermined data format; and (e-3) transmitting the serialized edit point information to the client.

[0017] And, the above editing execution unit may perform a clip editing algorithm comprising: a step (f-1) of parsing the editing point information received from the server to calculate a time interval for each editing target; a step (f-2) of determining a position within the clip corresponding to each of the time intervals; a step (f-3) of determining an editing processing method for the clip according to the position; and a step (f-4) of editing the clip according to the determined editing processing method.

[0018] At this time, the present invention may further include a feedback learning unit that receives acceptance or rejection feedback regarding the editing target input through the client, calculates an acceptance rate for each pattern of editing point candidates based on the feedback, and adjusts the reliability calculated by the editing point analysis unit based on the acceptance rate.

[0019] In addition, the feedback learning unit may perform a feedback learning algorithm comprising: (g-1) a step of storing learning data including text patterns of the edit point candidates, edit point types, preceding and succeeding contexts, domains, and user identification information; (g-2) a step of calculating an acceptance rate for each text pattern from the learning data; (g-3) a step of classifying patterns with an acceptance rate above a predetermined upper limit criterion as edit-recommended patterns and patterns with an acceptance rate below a predetermined lower limit criterion as retain-recommended patterns; and (g-4) a step of increasing the confidence for the edit-recommended patterns and decreasing the confidence for the retain-recommended patterns. Effects of the invention

[0020] The video editing system for voice recognition edit point detection and clip editing based on the server-client linkage of the present invention, which solves the above-mentioned problem, has the advantage of reducing the time and effort required for an editor to manually search for and remove unnecessary speech elements by automatically detecting edit point candidates from text conversion data generated through voice recognition on the server and determining the editing target based on the calculated reliability.

[0021] In addition, the present invention has the advantage of concentrating the computational load of speech recognition and edit point analysis on the server and reducing the processing burden on the client side by adopting a server-client interlocking structure in which the client extracts audio from video editing software and transmits it to the server, and receives edit point information from the server to process the clip editing.

[0022] Furthermore, the present invention has the advantage of being able to select editing targets based on consistent criteria without relying on the editor's subjective judgment, as the editing point analysis unit calculates the reliability of each editing point candidate and determines the editing target based thereon.

[0023] In addition, the present invention transmits an editing target determined by the server to the client as editing point information, and the client's editing execution unit directly processes the editing of the clip in the video editing software based on this, so the user has the advantage of being able to complete the editing work within the video editing software without going through a separate external tool.

[0024] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0025] FIG. 1 is a diagram showing the overall configuration of a video editing system according to one embodiment of the present invention. FIG. 2 is a diagram showing an audio extraction algorithm performed by an audio extraction unit in a video editing system according to an embodiment of the present invention. FIG. 3 is a diagram showing a context-based reliability calculation algorithm performed by an edit point analysis unit in a video editing system according to an embodiment of the present invention. FIG. 4 is a diagram showing an edit point detection algorithm performed by an edit point analysis unit in an image editing system according to an embodiment of the present invention. FIG. 5 is a diagram showing an artificial intelligence verification algorithm performed by an artificial intelligence verification unit in an image editing system according to an embodiment of the present invention. FIG. 6 is a diagram showing an edit point transmission algorithm performed by an edit point transmission unit in an image editing system according to an embodiment of the present invention. FIG. 7 is a diagram showing a clip editing algorithm performed by an editing execution unit in a video editing system according to an embodiment of the present invention. FIG. 8 is a diagram showing a feedback learning algorithm performed by a feedback learning unit in an image editing system according to an embodiment of the present invention. Specific details for implementing the invention

[0026] In this specification, where a component (or region, layer, part, etc.) is described as being "on," "connected," or "combined" with another component, it means that it may be directly placed / connected / combined with the other component, or that a third component may be placed between them.

[0027] Identical reference numerals denote identical components. Additionally, in the drawings, the thicknesses, proportions, and dimensions of the components are exaggerated for the effective illustration of the technical content.

[0028] "And / or" includes all one or more combinations that the associated configurations can define.

[0029] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0030] Additionally, terms such as "below," "lower side," "above," and "upper side" are used to describe the relationships between the components depicted in the drawings. These terms are relative concepts and are described based on the directions indicated in the drawings.

[0031] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Additionally, terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and are explicitly defined herein unless interpreted in an ideal or overly formal sense.

[0032] Terms such as "include" or "have" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0033] Furthermore, when it is stated in this specification that a first component operates or is executed on (ON) a second component, it should be understood that the first component operates or is executed in an environment where the second component operates or is executed, or operates or is executed through direct or indirect interaction with the second component.

[0034] Where any component, device, or system is described as including a component consisting of a program or software, it should be understood that, even without explicit mention, that component, device, or system includes hardware (e.g., memory, CPU, etc.) or other programs or software (e.g., an operating system or drivers required to run the hardware) necessary for the execution or operation of that program or software.

[0035] Furthermore, unless otherwise specified regarding the implementation of a component, it should be understood that the component may be implemented in software, hardware, or both software and hardware.

[0036] Furthermore, the terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components to the mentioned components.

[0037] Additionally, in this specification, terms such as 'part', 'device', etc., may be intended to refer to hardware and the functional and structural combination of software driven by said hardware or for driving said hardware. For example, the hardware here may be a data processing device including a CPU or other processor. Furthermore, the software driven by the hardware may refer to a running process, object, executable, thread of execution, program, etc.

[0038] Furthermore, it can be easily inferred by an average expert in the art of the present invention that the above terms may refer to a specific code and a logical unit of hardware resources for executing the said specific code, and do not necessarily refer to physically connected code or a single type of hardware.

[0039] FIG. 1 is a diagram showing the overall configuration of a video editing system according to one embodiment of the present invention.

[0040] As illustrated in FIG. 1, the video editing system in this embodiment may include a client (100) and a server (200). The client (100) and the server (200) can transmit and receive data to and from each other via a network, and the client (100) can operate in conjunction with video editing software.

[0041] In this embodiment, the client (100) may include an audio extraction unit (110) and an editing execution unit (120).

[0042] In addition, in this embodiment, the server (200) may include a voice recognition processing unit (210), an edit point analysis unit (220), and an edit point transmission unit (240). Furthermore, the server (200) may further include an artificial intelligence verification unit (230) and a feedback learning unit (250).

[0043] The client (100) is a computing device on which video editing software is installed and executed, and can be implemented as a desktop computer, laptop, workstation, etc. However, the client (100) can be implemented in various forms and is not limited to such forms. The client (100) may include a processor, memory, and a network interface, and can drive the audio extraction unit (110) and the editing execution unit (120) in the form of a plugin or extension program of the video editing software.

[0044] The server (200) is a computing device that processes audio received from the client (100) and generates edit point information, and can be implemented as a physical server, a cloud-based virtual server, or a distributed processing system composed of multiple servers. However, the server (200) can be implemented in various forms and is not limited to such forms. The server (200) may include a processor, memory, a storage device, and a network interface, and can software-operate a voice recognition processing unit (210), an edit point analysis unit (220), an artificial intelligence verification unit (230), an edit point transmission unit (240), and a feedback learning unit (250).

[0045] The client (100) and the server (200) may be connected via a wired or wireless network and may transmit and receive data using communication protocols such as HTTP, HTTPS, and WebSocket. However, the communication method between the client (100) and the server (200) may be implemented in various forms and is not limited to such forms.

[0046] The audio extraction unit (110) can extract audio from video editing software and transmit it to the server (200). That is, the audio extraction unit (110) can perform the role of separating audio data from a video placed on the timeline of the video editing software and transmitting the separated audio data to the server (200).

[0047] The speech recognition processing unit (210) can perform speech recognition on audio received from the client (100) to generate text conversion data including time information per word. Such text conversion data may include time information regarding the start and end times of speech of each word included in the audio, and the speech recognition processing unit (210) can generate such text conversion data by utilizing Speech-to-Text (STT) technology.

[0048] The edit point analysis unit (220) can detect edit point candidates from text conversion data and determine the edit target based on the confidence level calculated for each edit point candidate. Such edit point candidates may include sections corresponding to unintentional utterances, such as interjections, stuttering, unnecessary silent intervals, and word repetitions, and the edit point analysis unit (220) can calculate a confidence level, which is a numerical value indicating the need for editing, for each edit point candidate. Based on the calculated confidence level, the edit point analysis unit (220) can determine the edit point candidates that are judged to require editing as the edit target.

[0049] The artificial intelligence verification unit (230) may request verification from an external artificial intelligence model only for edit point candidates whose reliability calculated by the edit point analysis unit (220) falls within an intermediate range between a predetermined upper threshold and a predetermined lower threshold. Additionally, the artificial intelligence verification unit (230) may determine edit point candidates whose reliability is above the upper threshold as targets for editing, and determine edit point candidates whose reliability is below the lower threshold as unnecessary for editing. Therefore, since the request for verification to the external artificial intelligence model is limited to edit point candidates in the intermediate range, the number of calls to the artificial intelligence model can be reduced.

[0050] The edit point transmission unit (240) can transmit the edit target determined through the edit point analysis unit (220) and the artificial intelligence verification unit (230) to the client (100) as edit point information. Such edit point information may include word-specific time information, edit point type, and reliability corresponding to each edit target.

[0051] The feedback learning unit (250) can receive acceptance or rejection feedback regarding an editing target input through the client (100). Based on the received feedback, the feedback learning unit (250) calculates the acceptance rate of editing point candidates by pattern and can adjust the reliability calculated by the editing point analysis unit (220) based on the calculated acceptance rate. Accordingly, as the user repeatedly accepts or rejects an editing target, the reliability calculation of the editing point analysis unit (220) can be adjusted to match the user's editing tendency.

[0052] The editing execution unit (120) receives editing point information from the server (200) and can edit the clip of the video editing software based on the editing point information. That is, the editing execution unit (120) can refer to the word-by-word time information included in the editing point information and perform editing processing such as splitting, deleting, or moving the clip corresponding to the time interval on the timeline of the video editing software.

[0053] Below, the algorithms performed in each of these components will be explained in detail.

[0054] FIG. 2 is a diagram showing an audio extraction algorithm performed by an audio extraction unit (110) in a video editing system according to one embodiment of the present invention.

[0055] As illustrated in FIG. 2, in this embodiment, the audio extraction unit (110) can perform an audio extraction algorithm. Such an audio extraction algorithm may include steps (a-1), (a-2), (a-3), and (a-4).

[0056] Step (a-1) is a process of identifying multiple audio tracks of video editing software. In addition to voice tracks, multiple audio tracks containing background music, sound effects, etc., may exist on the timeline of the video editing software. The audio extraction unit (110) can identify multiple audio tracks included in the timeline in step (a-1) and obtain index and attribute information of each audio track.

[0057] Step (a-2) is a process of muting the remaining tracks among multiple audio tracks, excluding the designated track. The audio extraction unit (110) can set the track containing the voice of the speaker that is the subject of speech recognition as the designated track and mute the remaining audio tracks. For example, if the speaker's voice in an interview video is placed in the first audio track, the audio extraction unit (110) can mute the remaining audio tracks excluding the first audio track. Through such muting, background music or sound effects can be prevented from being included in the audio extraction result.

[0058] Step (a-3) is a process of rendering and extracting audio of a designated track. The audio extraction unit (110) can render the audio of the designated track into a predetermined audio format and extract it as a file while the remaining tracks are muted in step (a-2). Through this rendering process, an audio file containing only the speaker's voice can be generated, thereby improving the accuracy of subsequent speech recognition processing.

[0059] Step (a-4) is the process of transmitting the extracted audio to the server (200). The audio extraction unit (110) can transmit the audio file extracted in step (a-3) to the server (200). The voice recognition processing unit (210) of the server (200) can receive the transmitted audio file and perform voice recognition. Additionally, after the transmission of the audio file is completed, the audio extraction unit (110) can restore the audio track that was muted in step (a-2) to its original state.

[0060] FIG. 3 is a diagram showing a context-based reliability calculation algorithm performed by an edit point analysis unit (220) in an image editing system according to one embodiment of the present invention.

[0061] As illustrated in FIG. 3, in this embodiment, the edit point analysis unit (220) can perform a context-based reliability calculation algorithm. Such a context-based reliability calculation algorithm may include steps (b-1), (b-2), and (b-3).

[0062] Step (b-1) is a process of obtaining the context before and after each word from the text conversion data. The edit point analysis unit (220) can obtain a sequence of words within a predetermined range located before and after the word as the context before and after for each word included in the text conversion data. For example, if the utterance "Yes, I" follows the utterance "The weather is nice, isn't it?", the context before the word "Yes" can be "The weather is nice, isn't it?" and the context after it can be "I." In this way, in Step (b-1), individual words are not treated independently, and context information before and after the utterance of the word can be obtained together.

[0063] Step (b-2) is a process of determining the context type of each word based on the preceding and succeeding contexts. The edit point analysis unit (220) can determine which context type the word corresponds to by analyzing the preceding and succeeding contexts obtained in Step (b-1). For example, in the case of the word "yes," if the preceding context is a question-type utterance, it can be determined as a context type corresponding to the start of an answer, and if the preceding context is in the middle of a sentence that is not a question-type utterance, it can be determined as a context type corresponding to a habitual interjection. In addition, even if "yes" is uttered alone for a short period of time, it can be determined as a context type corresponding to a habitual interjection. Thus, in Step (b-2), even the same word can be assigned different context types depending on the preceding and succeeding contexts.

[0064] Step (b-3) is a process of calculating different reliability for the same word depending on the context type. The edit point analysis unit (220) can calculate the reliability of each word based on the context type determined in step (b-2). For example, if the word "yes" is located immediately after a question and is determined to be the start of an answer, a low reliability may be calculated, and in this case, the word may not be classified as an edit target. On the other hand, if the word "yes" is located in the middle of a sentence and is determined to be a habitual interjection, a high reliability may be calculated, and in this case, the word may be more likely to be classified as an edit target. Therefore, according to the context-based reliability calculation algorithm, different reliabilitys are calculated for the same word depending on the utterance context, so the determination of the edit target can be subdivided compared to a method that processes uniformly without considering the context.

[0065] FIG. 4 is a diagram showing an edit point detection algorithm performed by an edit point analysis unit (220) in an image editing system according to one embodiment of the present invention.

[0066] As illustrated in FIG. 4, in this embodiment, the edit point analysis unit (220) can perform an edit point detection algorithm. Such an edit point detection algorithm may include steps (c-1), (c-2), (c-3), and (c-4).

[0067] Step (c-1) is a process of detecting silent intervals from text conversion data and classifying silent intervals that exceed a dynamic threshold calculated based on speaker-specific speech patterns as edit point candidates. The edit point analysis unit (220) can calculate the time interval between consecutive words by referring to word-specific time information included in the text conversion data, and can detect intervals where such time interval is greater than a predetermined length as silent intervals. At this time, the edit point analysis unit (220) can calculate a dynamic threshold based on speaker-specific speech patterns instead of a fixed threshold. Such a dynamic threshold can be calculated based on the speaker's average time interval between words and its distribution, and different dynamic thresholds can be applied to speakers with fast speech speeds and speakers with slow speech speeds, respectively. The edit point analysis unit (220) can classify silent intervals that exceed the dynamic threshold as edit point candidates.

[0068] Step (c-2) is a process of detecting words corresponding to interjections from text conversion data and classifying them into a predetermined category. The edit point analysis unit (220) can detect words corresponding to interjections by analyzing each word included in the text conversion data. Such interjections may include filling expressions such as "um," "uh," and "ah"; softening expressions such as "uh," "what," and "somewhat"; response expressions such as "yes" and "yes"; restart expressions such as "no, no, no"; and thought expressions such as "well" and "there." The edit point analysis unit (220) can classify the detected interjections into a predetermined category, and the degree of editing recommendation may be set differently depending on each category. For example, the degree of editing recommendation may be set high for interjections corresponding to filling expressions or restart expressions, and the editing status for interjections corresponding to response expressions or thought expressions may vary depending on the context.

[0069] Step (c-3) is a process of detecting utterance repetition by analyzing the similarity between consecutive words in text conversion data. The edit point analysis unit (220) can analyze the similarity between words that appear consecutively in the text conversion data. For example, a pattern in which a complete word follows the repetition of the first syllable, such as when "ga" is followed by "price," or a pattern in which a complete word follows a partial utterance, such as when "geureo" is followed by "geureo," can be detected as utterance repetition. Additionally, utterance repetition can be detected when the same word is repeated consecutively. The edit point analysis unit (220) can classify the detected utterance repetitions as edit point candidates.

[0070] Step (c-4) is a process of synthesizing the results detected in steps (c-1) through (c-3) as edit point candidates. The edit point analysis unit (220) can synthesize the silent segments classified in step (c-1), the interjections classified in step (c-2), and the utterance repetitions detected in step (c-3) as edit point candidates. For each of the edit point candidates synthesized in this way, the edit point analysis unit (220) can assign an edit point type and reliability, and the final edit target can be determined through additional verification by a context-based reliability calculation algorithm or an artificial intelligence verification unit (230).

[0071] FIG. 5 is a diagram showing an artificial intelligence verification algorithm performed by an artificial intelligence verification unit (230) in an image editing system according to one embodiment of the present invention.

[0072] As illustrated in FIG. 5, in this embodiment, the artificial intelligence verification unit (230) can perform an artificial intelligence verification algorithm. Such an artificial intelligence verification algorithm may include steps (d-1), (d-2), (d-3), and (d-4).

[0073] Step (d-1) is a process of determining the domain type of the edit point candidate. The artificial intelligence verification unit (230) can determine the domain type based on the content characteristics of the video or audio currently being processed. Such domain types may include interviews, lectures, podcasts, etc., and the criteria for unnecessary utterances may differ depending on the domain type. For example, in the interview domain, "yes" as a response to a question may be retained, but in the lecture domain, the same "yes" may be classified as a habitual interjection.

[0074] Step (d-2) is a process of selecting an AI prompt corresponding to a domain type. The AI ​​verification unit (230) can select an AI prompt corresponding to the domain type identified in step (d-1). Such AI prompts can be predefined for each domain and may include guidelines for utterance types recommended for editing and utterance types not recommended for editing in the corresponding domain. For example, in an AI prompt corresponding to an interview domain, interjections such as habitual “um” and “uh” or stuttering may be designated as targets for recommended editing, while responses after a question such as “yes” or emotional expressions such as “wa” and “ah” may be designated as targets not recommended for editing.

[0075] Step (d-3) is a process of grouping edit point candidates corresponding to the intermediate interval into a predetermined number of units. The artificial intelligence verification unit (230) can group edit point candidates corresponding to the intermediate interval between the upper threshold and the lower threshold of confidence calculated by the edit point analysis unit (220) into a predetermined number of units. For example, if there are multiple edit point candidates corresponding to the intermediate interval, the artificial intelligence verification unit (230) can group them into a predetermined number of units (e.g., 15). Through such grouping, the number of requests to the external artificial intelligence model can be reduced, and verification results for multiple edit point candidates can be received with a single request.

[0076] Step (d-4) is a process of requesting verification from an external AI model along with an AI prompt on a batch group basis. The AI ​​verification unit (230) may request verification from an external AI model along with an AI prompt selected in Step (d-2) for each batch group grouped in Step (d-3). The external AI model may determine whether to edit each edit point candidate by referring to the instructions included in the AI ​​prompt and the text and contextual information of each edit point candidate. Based on the verification results received from the external AI model, the AI ​​verification unit (230) may determine whether the edit point candidate corresponding to the intermediate section is an edit target or does not require editing.

[0077] FIG. 6 is a diagram showing an edit point transmission algorithm performed by an edit point transmission unit (240) in an image editing system according to one embodiment of the present invention.

[0078] As illustrated in FIG. 6, in this embodiment, the edit point transmission unit (240) can perform an edit point transmission algorithm. Such an edit point transmission algorithm may include steps (e-1), (e-2), and (e-3).

[0079] (e-1) Step is a process of generating edit point information including word-specific time information, edit point type, and reliability corresponding to each edit target. The edit point transmission unit (240) can generate edit point information for each edit target determined by the edit point analysis unit (220) and the artificial intelligence verification unit (230), including word-specific time information indicating the start and end times of the utterance of the word corresponding to the edit target, edit point types such as silent intervals, interjections, and utterance repetitions, and reliability calculated by the edit point analysis unit (220). Such edit point information can be referenced by the edit execution unit (120) of the client (100) to identify the corresponding section on the timeline and perform editing processing.

[0080] Step (e-2) is a process of serializing edit point information into a predetermined data format. The edit point transmission unit (240) can serialize the edit point information generated in Step (e-1) into a predetermined data format such as JSON or XML. However, the data format used for serialization can be implemented in various forms and is not limited to such forms. Through such serialization, the edit point information can be converted into a form suitable for transmission over a network.

[0081] Step (e-3) is a process of transmitting serialized edit point information to the client (100). The edit point transmission unit (240) can transmit the serialized edit point information from Step (e-2) to the client (100) via a network. The edit execution unit (120) of the client (100) can receive the transmitted edit point information and utilize it in a subsequent clip editing algorithm.

[0082] FIG. 7 is a diagram showing a clip editing algorithm performed by an editing execution unit (120) in a video editing system according to one embodiment of the present invention.

[0083] As illustrated in FIG. 7, in this embodiment, the editing execution unit (120) can perform a clip editing algorithm. Such a clip editing algorithm may include steps (f-1), (f-2), (f-3), and (f-4).

[0084] Step (f-1) is a process of parsing edit point information received from the server (200) to calculate a time interval for each edit target. The edit execution unit (120) can parse edit point information received from the edit point transmission unit (240) to extract the start and end times of speech corresponding to each edit target. The edit execution unit (120) can calculate a time interval for each edit target based on the extracted start and end times. Such a time interval may correspond to a section that is subject to removal or editing processing on the timeline of the video editing software.

[0085] Step (f-2) is a process of determining the position within the corresponding clip for each time interval. For each time interval calculated in Step (f-1), the editing execution unit (120) can search for which clip on the timeline the time interval belongs to and determine the position within the corresponding clip. For example, it can be determined whether the time interval is located at the beginning of the clip, at the end of the clip, at the middle of the clip, or in the entire clip. Such position determination can be referenced to determine the editing processing method in the subsequent Step (f-3).

[0086] Step (f-3) is a process of determining the editing processing method for the clip based on the location. The editing execution unit (120) can determine the editing processing method for the clip based on the location determined in step (f-2). For example, if the time interval corresponds to the entire clip, a method of deleting the entire clip may be determined, and if the time interval is located at the beginning of the clip, a method of adjusting the beginning of the clip may be determined. Additionally, if the time interval is located at the end of the clip, a method of adjusting the end of the clip may be determined, and if the time interval is located at the middle of the clip, a method of splitting the clip and removing the corresponding section may be determined. In this way, the editing execution unit (120) can automatically determine different editing processing methods depending on the location of the time interval.

[0087] Step (f-4) is a process of editing a clip according to a determined editing method. The editing execution unit (120) can perform editing processing on the clip on the timeline of the video editing software according to the editing method determined in step (f-3). Such editing processing may include deleting the clip, adjusting the start or end point of the clip, splitting the clip, and removing the split section. After the editing processing is completed, the editing execution unit (120) can perform ripple processing so that a subsequent clip fills the empty section created by the editing processing on the timeline. Accordingly, the continuity of the timeline can be maintained.

[0088] FIG. 8 is a diagram showing a feedback learning algorithm performed by a feedback learning unit (250) in a video editing system according to one embodiment of the present invention.

[0089] As illustrated in FIG. 8, in this embodiment, the feedback learning unit (250) can perform a feedback learning algorithm. Such a feedback learning algorithm may include steps (g-1), (g-2), (g-3), and (g-4).

[0090] Step (g-1) is a process of storing training data that includes text patterns of edit point candidates, edit point types, context before and after, domain, and user identification information. When a user inputs feedback of acceptance or rejection for each edit target through the client (100), the feedback learning unit (250) can store the text patterns, edit point types, context before and after, domain, and user identification information corresponding to the edit target as training data. For example, if a user inputs rejection feedback for an edit target that is "Yes," the feedback learning unit (250) can store training data that includes "Yes" as a text pattern, an interjection as an edit point type, utterance content before and after the word as context before and after, an interview as a domain, and user identification information of the user. Such training data can be stored cumulatively whenever the user inputs feedback.

[0091] Step (g-2) is a process of calculating the acceptance rate for each text pattern from the training data. The feedback learning unit (250) can refer to the training data stored in step (g-1), aggregate the number of acceptance feedbacks for each text pattern and the total number of feedbacks, and calculate the acceptance rate for each text pattern. For example, if 3 out of 10 total feedbacks for the text pattern "Yes" are acceptances and 7 are rejections, the acceptance rate for that text pattern can be calculated as 30%. Since the calculation of such acceptance rates can be performed individually for user identification information and domains, different acceptance rates may be calculated depending on the user and domain, even for the same text pattern.

[0092] Step (g-3) is a process of classifying patterns with an acceptance rate above a predetermined upper limit as patterns recommended for editing, and patterns with an acceptance rate below a predetermined lower limit as patterns recommended for retention. Based on the acceptance rates for each text pattern calculated in Step (g-2), the feedback learning unit (250) can classify text patterns with an acceptance rate above a predetermined upper limit as patterns recommended for editing. Such patterns recommended for editing are patterns that are frequently accepted by the user, and thus have a higher probability of being classified as a target for editing when the same pattern is detected later. Additionally, the feedback learning unit (250) can classify text patterns with an acceptance rate below a predetermined lower limit as patterns recommended for retention. Such patterns recommended for retention are patterns that are frequently rejected by the user, and thus have a higher probability of being excluded from the target for editing when the same pattern is detected later.

[0093] Step (g-4) is a process of increasing the reliability for editing recommended patterns and decreasing the reliability for maintenance recommended patterns. The feedback learning unit (250) can increase the reliability of an editing point candidate corresponding to an editing recommended pattern classified in Step (g-3) when the editing point analysis unit (220) detects the candidate. Additionally, the feedback learning unit (250) can decrease the reliability of an editing point candidate corresponding to a maintenance recommended pattern when the candidate is detected. Accordingly, as the user repeatedly inputs feedback, the reliability calculated by the editing point analysis unit (220) can be gradually adjusted to match the user's editing tendencies. For example, if a specific user repeatedly inputs rejection feedback for a pattern of "Yes," the reliability calculated by the editing point analysis unit (220) may be decreased when "Yes" is subsequently detected in the user's video, so that it may not be classified as an editing target.

[0094] For the above, a video editing system for voice recognition editing point detection and clip editing based on server-client integration according to one embodiment of the present invention has been described in detail. Below, algorithms that can be additionally applied to the present invention will be described.

[0095] First, in this embodiment, the feedback learning unit (250) can perform a threshold automatic adjustment algorithm that automatically adjusts the upper threshold and lower threshold of the artificial intelligence verification unit (230). Based on the learning data accumulated in the feedback learning unit (250), such a threshold automatic adjustment algorithm can individually adjust the upper threshold and lower threshold of the artificial intelligence verification unit (230) for each user.

[0096] The feedback learning unit (250) can determine whether the total number of feedback accumulated for a specific user is greater than or equal to a predetermined number of feedback accumulation criteria. If the total number of feedback is greater than or equal to the number of feedback accumulation criteria, the feedback learning unit (250) can determine that the user is in a state where threshold adjustment is possible. On the other hand, if the total number of feedback is less than the number of feedback accumulation criteria, the feedback learning unit (250) can maintain the upper threshold and lower threshold of the artificial intelligence verification unit (230) as initial values.

[0097] The feedback learning unit (250) can calculate a feedback match rate from the learning data for a user determined to be in a state where threshold adjustment is possible. This feedback match rate can be calculated as the ratio of the user's actual feedback to the determination of whether to edit based on the reliability calculated by the edit point analysis unit (220). That is, cases where the user inputs acceptance feedback for an edit point candidate determined to be an edit target, and cases where the user inputs rejection feedback for an edit point candidate determined to be unnecessary to edit, are considered as matches, and the ratio of the number of matches to the total number of feedbacks can be calculated as the feedback match rate.

[0098] The feedback learning unit (250) can compare the calculated feedback match rate with a predetermined upper limit standard for the match rate and a predetermined lower limit standard for the match rate. If the feedback match rate is greater than or equal to the upper limit standard for the match rate, the feedback learning unit (250) can determine that the reliability calculation of the edit point analysis unit (220) is sufficiently consistent with the user's editing tendency. In this case, the feedback learning unit (250) can lower the upper threshold of the artificial intelligence verification unit (230) by a predetermined downward adjustment range and raise the lower threshold by a predetermined upward adjustment range. Accordingly, the intermediate interval between the upper threshold and the lower threshold is reduced, and the proportion of edit point candidates for which the artificial intelligence verification unit (230) requests verification from an external artificial intelligence model can be reduced.

[0099] On the other hand, if the feedback match rate is below the lower limit of the match rate standard, the feedback learning unit (250) can determine that the reliability calculation of the edit point analysis unit (220) does not sufficiently match the editing tendency of the user. In this case, the feedback learning unit (250) can raise the upper threshold of the artificial intelligence verification unit (230) by a predetermined upward adjustment range and lower the lower threshold by a predetermined downward adjustment range. Accordingly, the intermediate interval between the upper threshold and the lower threshold is expanded, and the proportion of edit point candidates for which the artificial intelligence verification unit (230) requests verification from an external artificial intelligence model can increase.

[0100] Additionally, if the feedback matching rate is below the upper matching rate threshold and above the lower matching rate threshold, the feedback learning unit (250) can maintain the upper threshold and lower threshold of the artificial intelligence verification unit (230) at their current values.

[0101] The feedback learning unit (250) can restrict the upper threshold value from exceeding a predetermined upper allowable range and the lower threshold value from falling below a predetermined lower allowable range when adjusting the upper threshold value and the lower threshold value. Additionally, the feedback learning unit (250) can restrict the upper threshold value and the lower threshold value to maintain a predetermined minimum interval width between the upper threshold value and the lower threshold value. By such restrictions, the intermediate interval may be prevented from being excessively reduced or expanded.

[0102] With this threshold automatic adjustment algorithm, for users with sufficient accumulated feedback, the rate at which the editing target is determined solely by the reliability adjustment by the editing point analysis unit (220) and the feedback learning unit (250) can be increased, and for users with insufficient accumulated feedback, the verification rate by the artificial intelligence verification unit (230) can be maintained or increased.

[0103] Next, in this embodiment, the edit point analysis unit (220) can perform an edit point clustering algorithm that merges multiple temporally adjacent edit point candidates into a single continuous edit section. Such an edit point clustering algorithm can be performed on the edit point candidates aggregated in step (c-4) of the edit point detection algorithm.

[0104] The edit point analysis unit (220) can sort the aggregated edit point candidates in chronological order and calculate the time interval between adjacent edit point candidates. If the calculated time interval is less than or equal to a predetermined merging standard interval, the edit point analysis unit (220) can merge the corresponding adjacent edit point candidates into a single continuous edit section. For example, if the time interval between the end point of an edit point candidate corresponding to an interjection and the start point of an edit point candidate corresponding to a silent section is less than or equal to the merging standard interval, the edit point analysis unit (220) can merge the time from the start point of the interjection to the end point of the silent section into a single continuous edit section.

[0105] Additionally, even after being merged into a continuous editing section, if the time interval between the end point of the merged continuous editing section and the start point of the next editing point candidate is less than or equal to the merging standard interval, the editing point analysis unit (220) may additionally merge the corresponding editing point candidate into the continuous editing section. This merging process may be performed repeatedly until the time interval between adjacent editing point candidates exceeds the merging standard interval.

[0106] The edit point analysis unit (220) can calculate a single reliability for a merged continuous editing section by combining the reliability of multiple edit point candidates included in the continuous editing section. For example, the edit point analysis unit (220) can calculate the reliability of the continuous editing section by taking the maximum value among the reliability of each edit point candidate included in the continuous editing section, or by taking the average value of the reliability of each edit point candidate as the reliability of the continuous editing section. However, the method of calculating the reliability of the continuous editing section can be implemented in various forms and is not limited to only such forms.

[0107] With such an editing point clustering algorithm, the editing execution unit (120) can process the merged continuous editing section as a single editing target, so the number of times the clip editing algorithm is executed can be reduced compared to the case where individual editing point candidates are processed separately.

[0108] Next, in this embodiment, the edit point transmission unit (240) may perform an edit intensity filtering algorithm that filters the edit target according to the edit intensity level set by the user. Such an edit intensity filtering algorithm may be performed prior to step (e-1) of the edit point transmission algorithm.

[0109] The client (100) can receive input of an editing intensity level from the user. The editing intensity level may be divided into multiple levels, and in this embodiment, may include a first level, a second level, and a third level. The client (100) can transmit the editing intensity level selected by the user to the server (200).

[0110] The edit point transmission unit (240) can determine a filtering criterion reliability corresponding to an edit intensity level received from the client (100). In this embodiment, a predetermined first filtering criterion reliability may correspond to the first level, a predetermined second filtering criterion reliability may correspond to the second level, and a predetermined third filtering criterion reliability may correspond to the third level. At this time, the first filtering criterion reliability may be a value higher than the second filtering criterion reliability, and the second filtering criterion reliability may be a value higher than the third filtering criterion reliability. Therefore, when the first level is selected, only edit targets with a reliability greater than or equal to the first filtering criterion reliability are included in the edit point information, and when the third level is selected, edit targets with a reliability greater than or equal to the third filtering criterion reliability are included in the edit point information, so more edit targets may be included in the third level.

[0111] The edit point transmission unit (240) can select edit targets among the edit targets determined through the edit point analysis unit (220) and the artificial intelligence verification unit (230) that have a reliability level greater than or equal to the filtering criterion reliability level determined. The edit point transmission unit (240) can generate edit point information and transmit it to the client (100) by performing steps (e-1) through (e-3) of the edit point transmission algorithm only on the selected edit targets.

[0112] Through such an editing intensity filtering algorithm, users can select an editing intensity level based on the nature of the content or the editing intent, and the scope of the editing target can be adjusted according to the selected editing intensity level. However, the number of editing intensity levels and the confidence values ​​of the filtering criteria corresponding to each level can be implemented in various forms and are not limited to this form.

[0113] Preferred embodiments according to the present invention have been described above, and it is obvious to those skilled in the art that, in addition to the embodiments described above, the present invention may be embodied in other specific forms without departing from the spirit or scope thereof. Therefore, the embodiments described above should be regarded as illustrative rather than restrictive, and accordingly, the present invention is not limited to the description above but may be modified within the scope of the appended claims and their equivalents. Explanation of the symbols

[0114] 100: Client 110: Audio extraction section 120: Edit execution section 200: Server 210: Speech recognition processing unit 220: Edit Point Analysis Section 230: Artificial Intelligence Verification Department 240: Edit point transmission section 250: Feedback Learning Department

Claims

Claim 1 In a video editing system comprising a server and a client, the client comprises: an audio extraction unit that extracts audio from video editing software and transmits it to the server; and an editing execution unit that receives edit point information from the server and processes the editing of a clip of the video editing software based on the edit point information; and the server comprises: a speech recognition processing unit that performs speech recognition on the audio received from the client and generates text conversion data including time information per word; and an edit point analysis unit that detects edit point candidates from the text conversion data and determines an editing target based on the confidence level calculated for each edit point candidate. A video editing system comprising: an edit point transmission unit that transmits the edit target to the client as edit point information; wherein the edit point analysis unit performs an edit point detection algorithm comprising: (c-1) a step of detecting silent intervals from the text conversion data and classifying silent intervals exceeding a dynamic threshold calculated based on a speaker-specific utterance pattern as edit point candidates; (c-2) a step of detecting words corresponding to interjections from the text conversion data and classifying them into a predetermined category; (c-3) a step of detecting utterance repetition by analyzing the similarity between consecutive words from the text conversion data; and (c-4) a step of synthesizing the results detected in steps (c-1) to (c-3) as edit point candidates. Claim 2 A video editing system comprising a server and a client, wherein the client comprises: an audio extraction unit that extracts audio from video editing software and transmits it to the server; and an editing execution unit that receives edit point information from the server and processes the editing of a clip of the video editing software based on the edit point information; and the server comprises: a speech recognition processing unit that performs speech recognition on the audio received from the client and generates text conversion data including time information per word; an edit point analysis unit that detects edit point candidates from the text conversion data and determines an edit target based on the confidence calculated for each edit point candidate; an edit point transmission unit that transmits the edit target to the client as the edit point information; and an artificial intelligence verification unit that requests verification from an external artificial intelligence model only for edit point candidates whose confidence corresponds to an intermediate interval between a predetermined upper threshold and a predetermined lower threshold, confirms edit point candidates above the upper threshold as edit targets, and confirms edit point candidates below the lower threshold as unnecessary for editing. Claim 3 In paragraph 2, the artificial intelligence verification unit performs an artificial intelligence verification algorithm comprising: (d-1) a step of determining the domain type of the edit point candidate; (d-2) a step of selecting an artificial intelligence prompt corresponding to the domain type; (d-3) a step of grouping the edit point candidates corresponding to the intermediate section into a predetermined unit; and (d-4) a step of requesting verification from the external artificial intelligence model together with the artificial intelligence prompt in the unit of the grouped units. Claim 4 A video editing system comprising a server and a client, wherein the client comprises: an audio extraction unit that extracts audio from video editing software and transmits it to the server; and an editing execution unit that receives edit point information from the server and processes the editing of a clip of the video editing software based on the edit point information; and the server comprises: a speech recognition processing unit that performs speech recognition on the audio received from the client and generates text conversion data including time information per word; an edit point analysis unit that detects edit point candidates from the text conversion data and determines an edit target based on the reliability calculated for each edit point candidate; an edit point transmission unit that transmits the edit target to the client as the edit point information; and a feedback learning unit that receives acceptance or rejection feedback regarding the edit target input through the client, calculates a pattern-specific acceptance rate of the edit point candidates based on the feedback, and adjusts the reliability calculated by the edit point analysis unit based on the acceptance rate. Claim 5 In claim 4, the feedback learning unit performs a feedback learning algorithm comprising: (g-1) a step of storing learning data including text pattern of the edit point candidate, edit point type, preceding and succeeding context, domain and user identification information; (g-2) a step of calculating an acceptance rate for the text pattern from the learning data; (g-3) a step of classifying a pattern with an acceptance rate above a predetermined upper limit criterion as an edit recommended pattern and a pattern with an acceptance rate below a predetermined lower limit criterion as a retention recommended pattern; and (g-4) a step of increasing the confidence for the edit recommended pattern and decreasing the confidence for the retention recommended pattern; an image editing system. Claim 6 A video editing system according to any one of claims 1, 2, or 4, wherein the audio extraction unit performs an audio extraction algorithm comprising: (a-1) a step of identifying a plurality of audio tracks of the video editing software; (a-2) a step of muting the remaining tracks excluding a designated track among the plurality of audio tracks; (a-3) a step of rendering and extracting the audio of the designated track; and (a-4) a step of transmitting the extracted audio to the server. Claim 7 A video editing system that performs a context-based reliability calculation algorithm, wherein, in any one of claims 1, 2, or 4, the edit point analysis unit comprises: (b-1) a step of obtaining the context before and after each word from the text conversion data; (b-2) a step of determining the context type of each word based on the context before and after; and (b-3) a step of calculating different reliability for the same word according to the context type. Claim 8 A video editing system that performs an edit point transmission algorithm, wherein, in any one of claims 1, 2, or 4, the edit point transmission unit comprises: (e-1) a step of generating edit point information including word-by-word time information, an edit point type, and a reliability corresponding to each of the edit targets; (e-2) a step of serializing the edit point information into a predetermined data format; and (e-3) a step of transmitting the serialized edit point information to the client. Claim 9 A video editing system that performs a clip editing algorithm, wherein, in any one of claims 1, 2, or 4, the editing execution unit comprises: (f-1) a step of parsing the editing point information received from the server to calculate a time interval for each editing target; (f-2) a step of determining a position within the clip corresponding to each of the time intervals; (f-3) a step of determining an editing processing method for the clip according to the position; and (f-4) a step of editing the clip according to the determined editing processing method. Claim 10 delete

Images