Video Editing System Having Automatic Edit Point Detection Function Based on Context-Based Confidence

Abstract

A video editing system having an automatic edit point detection function based on context-based reliability according to the present invention includes a server that analyzes audio data of a video received from a client to generate edit point information and transmits the edit point information to the client. The server may include a voice recognition processing unit that performs voice recognition on the audio data to generate text conversion data including time information per word, and an edit point analysis unit that detects edit point candidates from the text conversion data and determines an edit target based on reliability calculated for each edit point candidate.

Description

Technology Field

[0001] The present invention relates to a video editing system having an automatic edit point detection function based on context-based reliability, and more specifically, to a video editing system that analyzes the context of text conversion data generated from voice data to differentially calculate reliability for each edit point candidate and automatically determines an editing target based thereon. Background Technology

[0002] In the production of speech-centered video content such as interviews, lectures, and podcasts, raw footage typically contains unintentional speech elements, including speaker interjections, stuttering, unnecessary silences, and word repetition. Since editors must manually search for and remove these elements by listening to them one by one, the time and effort required for editing increase significantly as the video length grows. Furthermore, determining which speech is unnecessary relies on the editor's personal subjectivity, making it difficult to ensure consistency in the editing results.

[0003] To address these inconveniences, text-based editing methods utilizing speech recognition technology have been proposed. However, conventional text-based editing services are primarily designed for English-speaking languages, which has resulted in insufficient accuracy in detecting unnecessary utterances for languages ​​with unique phonological systems, such as Korean.

[0004] Furthermore, conventional methods for detecting unnecessary utterances tended to process the same words uniformly regardless of context, leaving editors with the inconvenience of having to re-examine and correct the results during actual editing. Although editing standards vary depending on the user's editing habits or content types, there was no structure in place for the system to learn and reflect these variations.

[0005] Meanwhile, even in attempts to automate editing through the extended features of video editing software, there was a difficulty in precisely splitting clips on the timeline or maintaining synchronization due to the limitations of the interface provided by the software.

[0006] Therefore, a method to resolve these problems is required. Prior art literature

[0007] Korean Patent Publication No. 10-2018-0129265 The problem to be solved

[0008] The present invention is an invention devised to solve the problems of the aforementioned prior art, and has the purpose of providing a video editing system that automatically detects unnecessary speech elements within a video and can differentially determine whether to edit even the same word depending on the speech context.

[0009] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0010] A video editing system having an automatic edit point detection function based on context-based reliability according to the present invention for achieving the above-mentioned purpose includes a server that analyzes audio data of a video received from a client to generate edit point information and transmits the edit point information to the client. The server may include a voice recognition processing unit that performs voice recognition on the audio data to generate text conversion data including time information per word, and an edit point analysis unit that detects edit point candidates from the text conversion data and determines an edit target based on reliability calculated for each edit point candidate.

[0011] Additionally, the edit point analysis unit may perform a context-based reliability calculation algorithm comprising the steps of: (a-1) obtaining the context before and after each word from the text conversion data; (a-2) determining the context type of each word based on the context before and after; and (a-3) calculating different reliability for the same word according to the context type.

[0012] And, the above edit point analysis unit may perform an edit point detection algorithm comprising: a step (b-1) of detecting silent intervals from the text conversion data and classifying silent intervals exceeding a dynamic threshold calculated based on speaker-specific utterance patterns as edit point candidates; a step (b-2) of detecting words corresponding to interjections from the text conversion data and classifying them into a predetermined category; a step (b-3) of detecting utterance repetition by analyzing the similarity between consecutive words from the text conversion data; and a step (b-4) of synthesizing the results detected in steps (b-1) through (b-3) as edit point candidates.

[0013] At this time, the above step (b-1) may include a step (b-1-1) of calculating the time interval between consecutive words from the word-by-word time information included in the text conversion data, a step (b-1-2) of calculating the average value and standard deviation of the time interval for each speaker, a step (b-1-3) of calculating a dynamic threshold for each speaker based on the average value and standard deviation, and a step (b-1-4) of classifying silent intervals exceeding the dynamic threshold as edit point candidates.

[0014] In addition, the above step (b-2) may include step (b-2-1) of determining whether each word included in the text conversion data corresponds to an interjection by comparing it with a predefined list of interjections, step (b-2-2) of classifying the determined interjections into one of the categories of filler expressions, relief expressions, response expressions, restart expressions, or thought expressions, and step (b-2-3) of assigning different initial confidence levels to each interjection according to the category.

[0015] In addition, the above step (b-3) may include a step (b-3-1) of extracting an initial consonant from each of the consecutive words in the text conversion data, a step (b-3-2) of determining a type of utterance repetition among a syllable restart, a word restart, or a similar sound repetition based on whether the extracted initial consonant matches, and a step (b-3-3) of classifying a preceding incomplete utterance among the words corresponding to the utterance repetition type as an edit point candidate.

[0016] Meanwhile, the present invention may further include an artificial intelligence verification unit that requests verification from an external artificial intelligence model only for edit point candidates whose reliability falls within an intermediate interval between a predetermined upper threshold and a predetermined lower threshold, confirms edit point candidates above the upper threshold as targets for editing, and confirms edit point candidates below the lower threshold as unnecessary for editing.

[0017] At this time, the artificial intelligence verification unit may perform an artificial intelligence verification algorithm comprising the steps of: (c-1) determining the domain type of the edit point candidate; (c-2) selecting an artificial intelligence prompt corresponding to the domain type; (c-3) grouping the edit point candidates corresponding to the intermediate section into a predetermined unit; and (c-4) requesting verification from the external artificial intelligence model together with the artificial intelligence prompt in the unit of the group.

[0018] In addition, the present invention may further include a feedback learning unit that receives acceptance or rejection feedback regarding the editing target input through the client, calculates an acceptance rate by pattern of editing point candidates based on the feedback, and adjusts the reliability calculated by the editing point analysis unit based on the acceptance rate.

[0019] And, the feedback learning unit may perform a feedback learning algorithm comprising: (d-1) a step of storing learning data including text patterns of edit point candidates, edit point types, preceding and succeeding contexts, domains, and user identification information; (d-2) a step of individually calculating acceptance rates for each text pattern according to the user identification information and domains from the learning data; (d-3) a step of classifying patterns with acceptance rates above a predetermined upper limit criterion as edit recommended patterns and patterns with acceptance rates below a predetermined lower limit criterion as retained recommended patterns; and (d-4) a step of increasing the confidence for the edit recommended patterns and decreasing the confidence for the retained recommended patterns. Effects of the invention

[0020] The video editing system of the present invention, which has an automatic edit point detection function based on context-based reliability to solve the above-mentioned problem, has the advantage of reducing the time and effort required for an editor to manually search for and remove unnecessary speech elements by having a server perform speech recognition on audio data of a video received from a client to generate text conversion data and automatically detect edit point candidates therefrom.

[0021] In addition, the present invention has the advantage of being able to select editing targets based on consistent criteria without relying on the editor's subjective judgment, as the editing point analysis unit calculates the reliability for each editing point candidate and determines the editing target based on the calculated reliability.

[0022] Furthermore, the present invention has the advantage of being able to concentrate the computational load associated with speech recognition and edit point analysis on the server and reduce the processing burden on the client side by adopting a structure that transmits edit point information generated on the server to the client.

[0023] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0024] FIG. 1 is a diagram showing the overall configuration of a video editing system according to one embodiment of the present invention. FIG. 2 is a diagram showing a context-based reliability calculation algorithm performed by an edit point analysis unit in a video editing system according to an embodiment of the present invention. FIG. 3 is a diagram showing an edit point detection algorithm performed by an edit point analysis unit in an image editing system according to an embodiment of the present invention. FIG. 4 is a diagram showing the detailed process of step (b-1) of the edit point detection algorithm in an image editing system according to one embodiment of the present invention. FIG. 5 is a diagram showing the detailed process of step (b-2) of the edit point detection algorithm in an image editing system according to one embodiment of the present invention. FIG. 6 is a diagram showing the detailed process of step (b-3) of the edit point detection algorithm in an image editing system according to one embodiment of the present invention. FIG. 7 is a diagram showing an artificial intelligence verification algorithm performed by an artificial intelligence verification unit in an image editing system according to an embodiment of the present invention. FIG. 8 is a diagram showing a feedback learning algorithm performed by a feedback learning unit in an image editing system according to an embodiment of the present invention. Specific details for implementing the invention

[0025] In this specification, where a component (or region, layer, part, etc.) is described as being "on," "connected," or "combined" with another component, it means that it may be directly placed / connected / combined with the other component, or that a third component may be placed between them.

[0026] Identical reference numerals denote identical components. Additionally, in the drawings, the thicknesses, proportions, and dimensions of the components are exaggerated for the effective illustration of the technical content.

[0027] "And / or" includes all one or more combinations that the associated configurations can define.

[0028] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0029] Additionally, terms such as "below," "lower side," "above," and "upper side" are used to describe the relationships between the components depicted in the drawings. These terms are relative concepts and are described based on the directions indicated in the drawings.

[0030] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Additionally, terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and are explicitly defined herein unless interpreted in an ideal or overly formal sense.

[0031] Terms such as "include" or "have" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] Furthermore, when it is stated in this specification that a first component operates or is executed on (ON) a second component, it should be understood that the first component operates or is executed in an environment where the second component operates or is executed, or operates or is executed through direct or indirect interaction with the second component.

[0033] Where any component, device, or system is described as including a component consisting of a program or software, it should be understood that, even without explicit mention, that component, device, or system includes hardware (e.g., memory, CPU, etc.) or other programs or software (e.g., an operating system or drivers required to run the hardware) necessary for the execution or operation of that program or software.

[0034] Furthermore, unless otherwise specified regarding the implementation of a component, it should be understood that the component may be implemented in software, hardware, or in any form that combines both software and hardware.

[0035] Furthermore, the terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components to the mentioned components.

[0036] Additionally, in this specification, terms such as 'part', 'device', etc., may be intended to refer to hardware and the functional and structural combination of software driven by said hardware or for driving said hardware. For example, the hardware here may be a data processing device including a CPU or other processor. Furthermore, the software driven by the hardware may refer to a running process, object, executable, thread of execution, program, etc.

[0037] Furthermore, it can be easily inferred by an average expert in the art of the present invention that the above terms may refer to a specific code and a logical unit of hardware resources for executing the said specific code, and do not necessarily refer to physically connected code or a single type of hardware.

[0038] FIG. 1 is a diagram showing the overall configuration of a video editing system according to one embodiment of the present invention.

[0039] As illustrated in FIG. 1, the video editing system in this embodiment may include a client (100) and a server (200). The client (100) and the server (200) can transmit and receive data to and from each other via a network, and the client (100) can operate in conjunction with video editing software.

[0040] In this embodiment, the client (100) can transmit audio data of the video to the server (200) and receive edit point information from the server (200). The client (100) is a computing device on which video editing software is installed and executed, and can be implemented as a desktop computer, laptop, workstation, etc. However, the client (100) can be implemented in various forms and is not limited to such forms. The client (100) may include a processor, memory, and a network interface.

[0041] In this embodiment, the client (100) may include an audio extraction unit (110) and an editing execution unit (120).

[0042] The audio extraction unit (110) can extract audio from video editing software and transmit it to the server (200). That is, the audio extraction unit (110) can separate audio data from a video placed on the timeline of the video editing software and transmit the separated audio data to the server (200). At this time, the audio extraction unit (110) can select and extract a track containing the speaker's voice among multiple audio tracks of the video editing software, thereby ensuring that background music or sound effects are not included in the audio data.

[0043] The editing execution unit (120) receives editing point information from the server (200) and can edit the clip of the video editing software based on the editing point information. That is, the editing execution unit (120) can refer to the word-by-word time information included in the editing point information and perform editing processing such as splitting, deleting, or moving the clip corresponding to the time interval on the timeline of the video editing software. In addition, the editing execution unit (120) can receive acceptance or rejection feedback regarding the editing target from the user and transmit it to the server (200).

[0044] The server (200) can analyze audio data of a video received from a client (100) to generate edit point information and transmit the edit point information to the client (100). In this embodiment, the server (200) may include a voice recognition processing unit (210) and an edit point analysis unit (220). Additionally, the server (200) may further include an artificial intelligence verification unit (230) and a feedback learning unit (250). The server (200) may be implemented as a physical server, a cloud-based virtual server, or a distributed processing system composed of multiple servers. However, the server (200) may be implemented in various forms and is not limited to such forms. The server (200) may include a processor, memory, a storage device, and a network interface, and may operate the voice recognition processing unit (210), the edit point analysis unit (220), the artificial intelligence verification unit (230), and the feedback learning unit (250) in software.

[0045] The client (100) and the server (200) may be connected via a wired or wireless network and may transmit and receive data using communication protocols such as HTTP, HTTPS, and WebSocket. However, the communication method between the client (100) and the server (200) may be implemented in various forms and is not limited to such forms.

[0046] The speech recognition processing unit (210) can perform speech recognition on speech data received from the client (100) to generate text conversion data including time information per word. Such text conversion data may include time information regarding the start and end times of speech of each word included in the speech data, and the speech recognition processing unit (210) can generate such text conversion data by utilizing speech-to-text (STT) technology.

[0047] The edit point analysis unit (220) can detect edit point candidates from text conversion data and determine the edit target based on the confidence level calculated for each edit point candidate. Such edit point candidates may include sections corresponding to unintentional utterances, such as interjections, stuttering, unnecessary silent intervals, and word repetitions, and the edit point analysis unit (220) can calculate a confidence level, which is a numerical value indicating the need for editing, for each edit point candidate. Based on the calculated confidence level, the edit point analysis unit (220) can determine the edit point candidates that are judged to require editing as the edit target.

[0048] The artificial intelligence verification unit (230) may request verification from an external artificial intelligence model only for edit point candidates whose reliability calculated by the edit point analysis unit (220) falls within an intermediate range between a predetermined upper threshold and a predetermined lower threshold. Additionally, the artificial intelligence verification unit (230) may determine edit point candidates whose reliability is above the upper threshold as targets for editing, and determine edit point candidates whose reliability is below the lower threshold as unnecessary for editing. Therefore, since the request for verification to the external artificial intelligence model is limited to edit point candidates in the intermediate range, the number of calls to the artificial intelligence model can be reduced.

[0049] The feedback learning unit (250) can receive acceptance or rejection feedback regarding an editing target input through the client (100). Based on the received feedback, the feedback learning unit (250) calculates the acceptance rate of editing point candidates by pattern and can adjust the reliability calculated by the editing point analysis unit (220) based on the calculated acceptance rate. Accordingly, as the user repeatedly accepts or rejects an editing target, the reliability calculation of the editing point analysis unit (220) can be adjusted to match the user's editing tendency.

[0050] Below, the algorithms performed in each of the edit point analysis unit (220), artificial intelligence verification unit (230), and feedback learning unit (250) will be described in detail.

[0051] FIG. 2 is a diagram showing a context-based reliability calculation algorithm performed by an edit point analysis unit (220) in an image editing system according to one embodiment of the present invention.

[0052] As illustrated in FIG. 2, in this embodiment, the edit point analysis unit (220) can perform a context-based reliability calculation algorithm. Such a context-based reliability calculation algorithm may include steps (a-1), (a-2), and (a-3).

[0053] Step (a-1) is a process of obtaining the context before and after each word from the text conversion data. The edit point analysis unit (220) can obtain a sequence of words within a predetermined range located before and after the word as the context before and after for each word included in the text conversion data. For example, if the utterance "Yes, I" follows the utterance "The weather is nice, isn't it?", the context before the word "Yes" can be "The weather is nice, isn't it?" and the context after it can be "I." In this way, in Step (a-1), individual words are not treated independently, and context information before and after the utterance of the word can be obtained together.

[0054] Step (a-2) is a process of determining the context type of each word based on the preceding and succeeding contexts. The edit point analysis unit (220) can determine which context type the word corresponds to by analyzing the preceding and succeeding contexts obtained in Step (a-1). For example, in the case of the word "yes," if the preceding context is a question-type utterance, it can be determined as a context type corresponding to the start of an answer, and if the preceding context is in the middle of a sentence that is not a question-type utterance, it can be determined as a context type corresponding to a habitual interjection. In addition, even if "yes" is uttered alone for a short period of time, it can be determined as a context type corresponding to a habitual interjection. Thus, in Step (a-2), different context types can be assigned to the same word depending on the preceding and succeeding contexts.

[0055] Step (a-3) is a process of calculating different reliability for the same word depending on the context type. The edit point analysis unit (220) can calculate the reliability of each word based on the context type determined in step (a-2). For example, if the word "yes" is located immediately after a question and is determined to be the start of an answer, a low reliability may be calculated, and in this case, the word may not be classified as an edit target. On the other hand, if the word "yes" is located in the middle of a sentence and is determined to be a habitual interjection, a high reliability may be calculated, and in this case, the word may be more likely to be classified as an edit target. Therefore, according to the context-based reliability calculation algorithm, different reliability is calculated for the same word depending on the utterance context, so the determination of the edit target can be subdivided compared to a method that processes uniformly without considering the context.

[0056] FIG. 3 is a diagram showing an edit point detection algorithm performed by an edit point analysis unit (220) in an image editing system according to one embodiment of the present invention.

[0057] As illustrated in FIG. 3, in this embodiment, the edit point analysis unit (220) can perform an edit point detection algorithm. Such an edit point detection algorithm may include steps (b-1), (b-2), (b-3), and (b-4).

[0058] Step (b-1) is a process of detecting silent intervals from text conversion data and classifying silent intervals that exceed a dynamic threshold calculated based on the speaker's speech pattern as edit point candidates. The edit point analysis unit (220) can calculate the time interval between consecutive words by referring to the word-by-word time information included in the text conversion data, and can detect intervals where such time interval is greater than a predetermined length as silent intervals. At this time, the edit point analysis unit (220) can calculate a dynamic threshold based on the speaker's speech pattern instead of a fixed threshold. Such a dynamic threshold can be calculated based on the speaker's average time interval between words and its distribution, and different dynamic thresholds can be applied to speakers with fast speech speeds and speakers with slow speech speeds, respectively. The edit point analysis unit (220) can classify silent intervals that exceed the dynamic threshold as edit point candidates. The detailed process of Step (b-1) will be described later with reference to FIG. 4.

[0059] Step (b-2) is a process of detecting words corresponding to interjections from text conversion data and classifying them into a predetermined category. The edit point analysis unit (220) can detect words corresponding to interjections by analyzing each word included in the text conversion data. Such interjections may include filling expressions such as "um," "uh," "ah," etc., softening expressions such as "uh," "what," "somewhat," etc., response expressions such as "yes," "yes," restart expressions such as "no, no, no," etc., and thought expressions such as "well," "there." The edit point analysis unit (220) can classify the detected interjections into a predetermined category, and the degree of editing recommendation may be set differently depending on each category. The detailed process of Step (b-2) will be described later with reference to FIG. 5.

[0060] Step (b-3) is a process of detecting utterance repetition by analyzing the similarity between consecutive words in text conversion data. The edit point analysis unit (220) can analyze the similarity between words that appear consecutively in the text conversion data. For example, a pattern in which a complete word follows the repetition of the first syllable, such as when "ga" is followed by "price," or a pattern in which a complete word follows a partial utterance, such as when "geureo" is followed by "geureo," can be detected as utterance repetition. Additionally, utterance repetition can be detected when the same word is repeated consecutively. The edit point analysis unit (220) can classify the detected utterance repetitions as edit point candidates. The detailed process of Step (b-3) will be described later with reference to FIG. 6.

[0061] Step (b-4) is a process of synthesizing the results detected in steps (b-1) through (b-3) as edit point candidates. The edit point analysis unit (220) can synthesize the silent segments classified in step (b-1), the interjections classified in step (b-2), and the utterance repetitions detected in step (b-3) as edit point candidates. For each of the edit point candidates synthesized in this way, the edit point analysis unit (220) can assign an edit point type and reliability, and the final edit target can be determined through additional verification by a context-based reliability calculation algorithm or an artificial intelligence verification unit (230).

[0062] FIG. 4 is a diagram showing the detailed process of step (b-1) of the edit point detection algorithm in an image editing system according to one embodiment of the present invention.

[0063] As illustrated in FIG. 4, in this embodiment, step (b-1) may include, in detail, step (b-1-1), step (b-1-2), step (b-1-3), and step (b-1-4).

[0064] Step (b-1-1) is a process of calculating the time interval between consecutive words from the word-by-word time information included in the text conversion data. The edit point analysis unit (220) can calculate the difference between the end time of each word's utterance and the start time of the next word's utterance in the text conversion data as a time interval. Such a time interval can correspond to the length of the silent interval existing between words.

[0065] Step (b-1-2) is a process of calculating the average value and standard deviation of time intervals for each speaker. The edit point analysis unit (220) classifies the time intervals calculated in step (b-1-1) by speaker and can calculate the average value and standard deviation of time intervals for each speaker. Such average values ​​and standard deviations can reflect each speaker's unique speech speed and speech pattern.

[0066] Step (b-1-3) is a process of calculating a dynamic threshold value for each speaker based on the mean value and standard deviation. The edit point analysis unit (220) can calculate a dynamic threshold value for each speaker using the mean value and standard deviation calculated in Step (b-1-2). For example, the dynamic threshold value can be calculated by adding a predetermined multiple of the standard deviation to the mean value. Accordingly, a relatively short dynamic threshold value can be calculated for speakers with a fast speaking speed, and a relatively long dynamic threshold value can be calculated for speakers with a slow speaking speed.

[0067] Step (b-1-4) is a process of classifying silent intervals that exceed a dynamic threshold as edit point candidates. The edit point analysis unit (220) can identify intervals among the time intervals calculated in step (b-1-1) that exceed the dynamic threshold calculated in step (b-1-3) as unnecessary silent intervals and classify them as edit point candidates. In this way, by applying different dynamic thresholds for each speaker, silent interval detection that is adaptive to the speaker's speech characteristics can be performed compared to the case where a fixed threshold is used.

[0068] FIG. 5 is a diagram showing the detailed process of step (b-2) of the edit point detection algorithm in an image editing system according to one embodiment of the present invention.

[0069] As illustrated in FIG. 5, in this embodiment, step (b-2) may include, in detail, step (b-2-1), step (b-2-2), and step (b-2-3).

[0070] Step (b-2-1) is a process of determining whether each word included in the text conversion data corresponds to an interjection by comparing it with a predefined list of interjections. The edit point analysis unit (220) can extract each word from the text conversion data and compare the extracted word with a predefined list of interjections. Such a list of interjections may include interjections that frequently appear in Korean speech, and the edit point analysis unit (220) can determine whether each word corresponds to an interjection based on the comparison result.

[0071] Step (b-2-2) is a process of classifying the identified interjection into one of the following categories: filling expressions, relaxation expressions, response expressions, restart expressions, or thought expressions. The edit point analysis unit (220) can classify the word identified as an interjection in Step (b-2-1) into a predetermined category. For example, "um," "uh," "ah," etc. can be classified as filling expressions, and "geu," "mwo," "jom," "salgan," etc. can be classified as relaxation expressions. Additionally, "yes," "yes," "ung," etc. can be classified as response expressions, "no," "no no," etc. can be classified as restart expressions, and "well," "hey," etc. can be classified as thought expressions.

[0072] Step (b-2-3) is a process of assigning different initial confidence levels to each interjection according to category. The edit point analysis unit (220) can assign an initial confidence level to each interjection based on the category classified in Step (b-2-2). For example, a relatively high initial confidence level may be assigned to interjections corresponding to fill expressions or restart expressions, and an intermediate level of initial confidence level may be assigned to interjections corresponding to relaxation expressions or thought expressions, as additional judgment based on context is required. Additionally, a relatively low initial confidence level may be assigned to interjections corresponding to response expressions, as they are sometimes used as responses to questions. Such initial confidence levels can be adjusted later by a context-based confidence calculation algorithm.

[0073] FIG. 6 is a diagram showing the detailed process of step (b-3) of the edit point detection algorithm in an image editing system according to one embodiment of the present invention.

[0074] As illustrated in FIG. 6, in this embodiment, step (b-3) may include, in detail, step (b-3-1), step (b-3-2), and step (b-3-3).

[0075] Step (b-3-1) is a process of extracting an initial consonant from each of the consecutive words in the text conversion data. The edit point analysis unit (220) can extract the initial consonant of the syllable constituting the word for each of the consecutive words appearing in the text conversion data. In the case of Korean, each syllable consists of a combination of an initial consonant, a medial vowel, and a final consonant, and the edit point analysis unit (220) can separate and extract the initial consonant from each syllable through Unicode operations, etc.

[0076] Step (b-3-2) is a process of determining the type of utterance repetition, such as syllable restart, word restart, or similar sound repetition, based on whether the extracted initial consonants match. The edit point analysis unit (220) can determine whether there is utterance repetition between consecutive words by comparing the initial consonants extracted in Step (b-3-1). For example, a pattern in which a complete word follows the repetition of the first syllable, such as when "Ga" is followed by "Gapga", can be determined as a syllable restart. Also, a pattern in which a complete word follows a partial utterance, such as when "Geureu" is followed by "Geureu-nae," can be determined as a word restart. Furthermore, a pattern in which similar pronunciations with identical initial consonants follow one another, such as when "Geu" is followed by "Geo," can be determined as a similar sound repetition.

[0077] Step (b-3-3) is a process of classifying a preceding incomplete utterance among words corresponding to the utterance repetition type as a candidate for an edit point. When the utterance repetition type is determined in step (b-3-2), the edit point analysis unit (220) can classify a preceding incomplete utterance among consecutive words as a candidate for an edit point. For example, if "price" is uttered after "ga," the preceding "ga" can be classified as a candidate for an edit point as an incomplete utterance, and if "geureo" is uttered after "geureo," the preceding "geureo" can be classified as a candidate for an edit point as an incomplete utterance. In this way, the edit point analysis unit (220) can maintain the subsequent completed utterance and classify only the preceding incomplete utterance as a candidate for an edit point.

[0078] FIG. 7 is a diagram showing an artificial intelligence verification algorithm performed by an artificial intelligence verification unit (230) in an image editing system according to one embodiment of the present invention.

[0079] As illustrated in FIG. 7, in this embodiment, the artificial intelligence verification unit (230) can perform an artificial intelligence verification algorithm. Such an artificial intelligence verification algorithm may include steps (c-1), (c-2), (c-3), and (c-4).

[0080] Step (c-1) is a process of determining the domain type of the edit point candidate. The artificial intelligence verification unit (230) can determine the domain type based on the content characteristics of the video or audio data currently being processed. Such domain types may include interviews, lectures, podcasts, etc., and the criteria for unnecessary utterances may differ depending on the domain type. For example, in the interview domain, "yes" as a response to a question may be retained, but in the lecture domain, the same "yes" may be classified as a habitual interjection.

[0081] Step (c-2) is a process of selecting an AI prompt corresponding to a domain type. The AI ​​verification unit (230) can select an AI prompt corresponding to the domain type identified in Step (c-1). Such AI prompts can be predefined for each domain and may include guidelines for utterance types recommended for editing and utterance types not recommended for editing in the corresponding domain. For example, in an AI prompt corresponding to an interview domain, interjections such as habitual “um” and “uh” or stuttering may be designated as targets for recommended editing, while responses after a question such as “yes” or emotional expressions such as “wa” and “ah” may be designated as targets not recommended for editing.

[0082] Step (c-3) is a process of grouping edit point candidates corresponding to the intermediate interval into a predetermined unit. The artificial intelligence verification unit (230) can group edit point candidates corresponding to the intermediate interval between the upper threshold and the lower threshold of the reliability calculated by the edit point analysis unit (220) into a predetermined unit. For example, if there are multiple edit point candidates corresponding to the intermediate interval, the artificial intelligence verification unit (230) can group them into a predetermined number of units. Through such grouping, the number of requests to the external artificial intelligence model can be reduced, and verification results for multiple edit point candidates can be received with a single request.

[0083] Step (c-4) is a process of requesting verification from an external AI model along with an AI prompt on a batch group basis. The AI ​​verification unit (230) may request verification from an external AI model along with an AI prompt selected in Step (c-2) for each batch group grouped in Step (c-3). The external AI model may determine whether to edit each edit point candidate by referring to the instructions included in the AI ​​prompt and the text and contextual information of each edit point candidate. Based on the verification results received from the external AI model, the AI ​​verification unit (230) may determine whether the edit point candidate corresponding to the intermediate section is subject to editing or does not require editing.

[0084] FIG. 8 is a diagram showing a feedback learning algorithm performed by a feedback learning unit (250) in a video editing system according to one embodiment of the present invention.

[0085] As illustrated in FIG. 8, in this embodiment, the feedback learning unit (250) can perform a feedback learning algorithm. Such a feedback learning algorithm may include steps (d-1), (d-2), (d-3), and (d-4).

[0086] Step (d-1) is a process of storing training data that includes text patterns of edit point candidates, edit point types, context before and after, domain, and user identification information. When a user inputs feedback of acceptance or rejection for each edit target through the client (100), the feedback learning unit (250) can store the text patterns, edit point types, context before and after, domain, and user identification information corresponding to the edit target as training data. For example, if a user inputs rejection feedback for an edit target that is "Yes," the feedback learning unit (250) can store training data that includes "Yes" as a text pattern, an interjection as an edit point type, utterance content before and after the word as context before and after, an interview as a domain, and user identification information of the user. Such training data can be stored cumulatively whenever the user inputs feedback.

[0087] Step (d-2) is a process of individually calculating the acceptance rate for each text pattern by user identification information and domain from the training data. The feedback learning unit (250) can refer to the training data stored in Step (d-1), aggregate the number of acceptance feedbacks for each text pattern and the total number of feedbacks, and calculate the acceptance rate for each text pattern. Since the calculation of such acceptance rates can be performed individually by user identification information and domain, different acceptance rates may be calculated depending on the user and domain, even for the same text pattern. For example, regarding the text pattern "Yes," if 3 out of 10 total feedbacks in a specific user's interview domain are acceptances and 7 are rejections, the acceptance rate under that condition may be calculated as 30%.

[0088] Step (d-3) is a process of classifying patterns with an acceptance rate above a predetermined upper limit as patterns recommended for editing, and patterns with an acceptance rate below a predetermined lower limit as patterns recommended for retention. Based on the acceptance rates for each text pattern calculated in Step (d-2), the feedback learning unit (250) can classify text patterns with an acceptance rate above a predetermined upper limit as patterns recommended for editing. Such patterns recommended for editing are patterns that are frequently accepted by the user, and thus have a higher probability of being classified as a target for editing when the same pattern is detected later. Additionally, the feedback learning unit (250) can classify text patterns with an acceptance rate below a predetermined lower limit as patterns recommended for retention. Such patterns recommended for retention are patterns that are frequently rejected by the user, and thus have a higher probability of being excluded from the target for editing when the same pattern is detected later.

[0089] Step (d-4) is a process of increasing the reliability for editing recommended patterns and decreasing the reliability for maintenance recommended patterns. The feedback learning unit (250) can increase the reliability of an editing point candidate corresponding to an editing recommended pattern classified in step (d-3) when the editing point analysis unit (220) detects the candidate. Additionally, the feedback learning unit (250) can decrease the reliability of an editing point candidate corresponding to a maintenance recommended pattern when the candidate is detected. Accordingly, as the user repeatedly inputs feedback, the reliability calculated by the editing point analysis unit (220) can be gradually adjusted to match the user's editing tendencies. For example, if a specific user repeatedly inputs rejection feedback for a pattern of "Yes," the reliability calculated by the editing point analysis unit (220) may be decreased when "Yes" is subsequently detected in the user's video, so that it may not be classified as an editing target.

[0090] For the above, a video editing system having an automatic edit point detection function based on context-based reliability according to an embodiment of the present invention has been described in detail. Below, algorithms that can be additionally applied to the present invention will be described.

[0091] First, in this embodiment, the feedback learning unit (250) can perform a threshold automatic adjustment algorithm that automatically adjusts the upper threshold and lower threshold of the artificial intelligence verification unit (230). Based on the learning data accumulated in the feedback learning unit (250), such a threshold automatic adjustment algorithm can individually adjust the upper threshold and lower threshold of the artificial intelligence verification unit (230) for each user.

[0092] The feedback learning unit (250) can determine whether the total number of feedback accumulated for a specific user is greater than or equal to a predetermined number of feedback accumulation criteria. If the total number of feedback is greater than or equal to the number of feedback accumulation criteria, the feedback learning unit (250) can determine that the user is in a state where threshold adjustment is possible. On the other hand, if the total number of feedback is less than the number of feedback accumulation criteria, the feedback learning unit (250) can maintain the upper threshold and lower threshold of the artificial intelligence verification unit (230) as initial values.

[0093] The feedback learning unit (250) can calculate a feedback match rate from the learning data for a user determined to be in a state where threshold adjustment is possible. This feedback match rate can be calculated as the ratio of the user's actual feedback to the determination of whether to edit based on the reliability calculated by the edit point analysis unit (220). That is, cases where the user inputs acceptance feedback for an edit point candidate determined to be an edit target, and cases where the user inputs rejection feedback for an edit point candidate determined to be unnecessary to edit, are considered as matches, and the ratio of the number of matches to the total number of feedbacks can be calculated as the feedback match rate.

[0094] The feedback learning unit (250) can compare the calculated feedback match rate with a predetermined upper limit standard for the match rate and a predetermined lower limit standard for the match rate. If the feedback match rate is greater than or equal to the upper limit standard for the match rate, the feedback learning unit (250) can determine that the reliability calculation of the edit point analysis unit (220) is sufficiently consistent with the user's editing tendency. In this case, the feedback learning unit (250) can lower the upper threshold of the artificial intelligence verification unit (230) by a predetermined downward adjustment range and raise the lower threshold by a predetermined upward adjustment range. Accordingly, the intermediate interval between the upper threshold and the lower threshold is reduced, and the proportion of edit point candidates for which the artificial intelligence verification unit (230) requests verification from an external artificial intelligence model can be reduced.

[0095] On the other hand, if the feedback match rate is below the lower limit of the match rate standard, the feedback learning unit (250) can determine that the reliability calculation of the edit point analysis unit (220) does not sufficiently match the editing tendency of the user. In this case, the feedback learning unit (250) can raise the upper threshold of the artificial intelligence verification unit (230) by a predetermined upward adjustment range and lower the lower threshold by a predetermined downward adjustment range. Accordingly, the intermediate interval between the upper threshold and the lower threshold is expanded, and the proportion of edit point candidates for which the artificial intelligence verification unit (230) requests verification from an external artificial intelligence model can increase.

[0096] Additionally, if the feedback matching rate is below the upper matching rate threshold and above the lower matching rate threshold, the feedback learning unit (250) can maintain the upper threshold and lower threshold of the artificial intelligence verification unit (230) at their current values.

[0097] The feedback learning unit (250) can restrict the upper threshold value from exceeding a predetermined upper allowable range and the lower threshold value from falling below a predetermined lower allowable range when adjusting the upper threshold value and the lower threshold value. Additionally, the feedback learning unit (250) can restrict the upper threshold value and the lower threshold value to maintain a predetermined minimum interval width between the upper threshold value and the lower threshold value. By such restrictions, the intermediate interval may be prevented from being excessively reduced or expanded.

[0098] With this threshold automatic adjustment algorithm, for users with sufficient accumulated feedback, the rate at which the editing target is determined solely by the reliability adjustment by the editing point analysis unit (220) and the feedback learning unit (250) can be increased, and for users with insufficient accumulated feedback, the verification rate by the artificial intelligence verification unit (230) can be maintained or increased.

[0099] Next, in this embodiment, the edit point analysis unit (220) can perform an edit point clustering algorithm that merges multiple temporally adjacent edit point candidates into a single continuous edit section. Such an edit point clustering algorithm can be performed on the edit point candidates aggregated in step (b-4) of the edit point detection algorithm.

[0100] The edit point analysis unit (220) can sort the aggregated edit point candidates in chronological order and calculate the time interval between adjacent edit point candidates. If the calculated time interval is less than or equal to a predetermined merging standard interval, the edit point analysis unit (220) can merge the corresponding adjacent edit point candidates into a single continuous edit section. For example, if the time interval between the end point of an edit point candidate corresponding to an interjection and the start point of an edit point candidate corresponding to a silent section is less than or equal to the merging standard interval, the edit point analysis unit (220) can merge the time from the start point of the interjection to the end point of the silent section into a single continuous edit section.

[0101] Additionally, even after being merged into a continuous editing section, if the time interval between the end point of the merged continuous editing section and the start point of the next editing point candidate is less than or equal to the merging standard interval, the editing point analysis unit (220) may additionally merge the corresponding editing point candidate into the continuous editing section. This merging process may be performed repeatedly until the time interval between adjacent editing point candidates exceeds the merging standard interval.

[0102] The edit point analysis unit (220) can calculate a single reliability for a merged continuous editing section by combining the reliability of multiple edit point candidates included in the continuous editing section. For example, the edit point analysis unit (220) can calculate the reliability of the continuous editing section by taking the maximum value among the reliability of each edit point candidate included in the continuous editing section, or by taking the average value of the reliability of each edit point candidate as the reliability of the continuous editing section. However, the method of calculating the reliability of the continuous editing section can be implemented in various forms and is not limited to only such forms.

[0103] With such an edit point clustering algorithm, merged consecutive edit sections can be treated as a single edit target, so the number of edit processing operations can be reduced compared to processing individual edit point candidates separately.

[0104] Next, in this embodiment, the server (200) may further include an edit point transmission unit (240) that transmits an edit target determined through an edit point analysis unit (220) and an artificial intelligence verification unit (230) to a client (100) as edit point information. The edit point transmission unit (240) may generate edit point information including word-specific time information, an edit point type, and reliability corresponding to each edit target. The edit point transmission unit (240) may serialize the generated edit point information into a predetermined data format such as JSON or XML, and transmit the serialized edit point information to the client (100) via a network. However, the data format used for serialization may be implemented in various forms and is not limited to such forms.

[0105] Additionally, the edit point transmission unit (240) can perform an edit intensity filtering algorithm that filters the edit target according to the edit intensity level set by the user. Such an edit intensity filtering algorithm can be performed before the generation of edit point information.

[0106] The client (100) can receive input of an editing intensity level from the user. The editing intensity level may be divided into multiple levels, and in this embodiment, may include a first level, a second level, and a third level. The client (100) can transmit the editing intensity level selected by the user to the server (200).

[0107] The edit point transmission unit (240) can determine a filtering criterion reliability corresponding to an edit intensity level received from the client (100). In this embodiment, a predetermined first filtering criterion reliability may correspond to the first level, a predetermined second filtering criterion reliability may correspond to the second level, and a predetermined third filtering criterion reliability may correspond to the third level. At this time, the first filtering criterion reliability may be a value higher than the second filtering criterion reliability, and the second filtering criterion reliability may be a value higher than the third filtering criterion reliability. Therefore, when the first level is selected, only edit targets with a reliability greater than or equal to the first filtering criterion reliability are included in the edit point information, and when the third level is selected, edit targets with a reliability greater than or equal to the third filtering criterion reliability are included in the edit point information, so more edit targets may be included in the third level.

[0108] The edit point transmission unit (240) can select edit targets that have a reliability level greater than or equal to the filtering criterion reliability level determined from among the edit targets determined through the edit point analysis unit (220) and the artificial intelligence verification unit (230). The edit point transmission unit (240) can generate edit point information only for the selected edit targets and transmit it to the client (100).

[0109] Through such an editing intensity filtering algorithm, users can select an editing intensity level based on the nature of the content or the editing intent, and the scope of the editing target can be adjusted according to the selected editing intensity level. However, the number of editing intensity levels and the confidence values ​​of the filtering criteria corresponding to each level can be implemented in various forms and are not limited to this form.

[0110] Preferred embodiments according to the present invention have been described so far, and it is obvious to those skilled in the art that, in addition to the embodiments described above, the present invention may be embodied in other specific forms without departing from the spirit or scope thereof. Therefore, the embodiments described above should be regarded as illustrative rather than restrictive, and accordingly, the present invention is not limited to the description above but may be modified within the scope of the appended claims and their equivalents. Explanation of the symbols

[0111] 100: Client 110: Audio extraction section 120: Edit execution section 200: Server 210: Speech recognition processing unit 220: Edit Point Analysis Section 230: Artificial Intelligence Verification Department 240: Edit point transmission section 250: Feedback Learning Department

Claims

Claim 1 A video editing system comprising a server that analyzes audio data of a video received from a client to generate edit point information and transmits said edit point information to the client, wherein the server comprises: a voice recognition processing unit that performs voice recognition on said voice data to generate text conversion data including time information per word; and an edit point analysis unit that detects edit point candidates from said text conversion data and determines an edit target based on the confidence calculated for each edit point candidate; wherein the edit point analysis unit comprises: (b-1) a step of detecting silent intervals from said text conversion data and classifying silent intervals exceeding a dynamic threshold calculated based on a speech pattern per speaker as edit point candidates; (b-2) a step of detecting words corresponding to interjections from said text conversion data and classifying them into a predetermined category; and (b-3) a step of detecting speech repetition by analyzing the similarity between consecutive words from said text conversion data. A video editing system that performs an edit point detection algorithm comprising: a step (b-4) of synthesizing the results detected in steps (b-1) through (b-3) as edit point candidates. Claim 2 A video editing system according to claim 1, wherein step (b-1) comprises: step (b-1-1) of calculating a time interval between consecutive words from word-by-word time information included in the text conversion data; step (b-1-2) of calculating an average value and a standard deviation of the time interval for each speaker; step (b-1-3) of calculating a dynamic threshold for each speaker based on the average value and the standard deviation; and step (b-1-4) of classifying silent intervals exceeding the dynamic threshold as edit point candidates. Claim 3 A video editing system according to claim 1, wherein step (b-2) comprises: step (b-2-1) of determining whether each word included in the text conversion data corresponds to an interjection by comparing it with a predefined list of interjections; step (b-2-2) of classifying the determined interjections into one of the categories of filler expressions, relief expressions, response expressions, restart expressions, or thought expressions; and step (b-2-3) of assigning different initial confidence levels to each interjection according to the category. Claim 4 A video editing system according to claim 1, wherein step (b-3) comprises: step (b-3-1) of extracting an initial consonant from each of consecutive words in the text conversion data; step (b-3-2) of determining a type of utterance repetition among a syllable restart, a word restart, or a similar sound repetition based on whether the extracted initial consonant matches; and step (b-3-3) of classifying a preceding incomplete utterance among the words corresponding to the utterance repetition type as an edit point candidate. Claim 5 A video editing system comprising a server that analyzes audio data of a video received from a client to generate edit point information and transmits said edit point information to the client, wherein the server comprises: a voice recognition processing unit that performs voice recognition on said audio data to generate text conversion data including word-by-word time information; an edit point analysis unit that detects edit point candidates from said text conversion data and determines an edit target based on a confidence level calculated for each edit point candidate; and an artificial intelligence verification unit that requests verification from an external artificial intelligence model only for edit point candidates whose confidence level falls within an intermediate interval between a predetermined upper threshold and a predetermined lower threshold, confirms edit point candidates above the upper threshold as edit targets, and confirms edit point candidates below the lower threshold as not requiring editing. Claim 6 A video editing system according to claim 5, wherein the artificial intelligence verification unit performs an artificial intelligence verification algorithm comprising: (c-1) a step of determining the domain type of the edit point candidate; (c-2) a step of selecting an artificial intelligence prompt corresponding to the domain type; (c-3) a step of grouping the edit point candidates corresponding to the intermediate section into a predetermined unit; and (c-4) a step of requesting verification from the external artificial intelligence model together with the artificial intelligence prompt in the unit of the group. Claim 7 A video editing system comprising a server that analyzes audio data of a video received from a client to generate edit point information and transmits said edit point information to the client, wherein the server comprises: a voice recognition processing unit that performs voice recognition on said audio data to generate text conversion data including word-by-word time information; an edit point analysis unit that detects edit point candidates from said text conversion data and determines an edit target based on a confidence level calculated for each edit point candidate; and a feedback learning unit that receives acceptance or rejection feedback regarding the edit target input through said client, calculates a pattern-by-pattern acceptance rate of the edit point candidates based on said feedback, and adjusts the confidence level calculated by said edit point analysis unit based on said acceptance rate. Claim 8 In claim 7, the feedback learning unit performs a feedback learning algorithm comprising: (d-1) a step of storing learning data including text pattern of the edit point candidate, edit point type, preceding and succeeding context, domain, and user identification information; (d-2) a step of individually calculating an acceptance rate for the text pattern for each user identification information and domain from the learning data; (d-3) a step of classifying a pattern with an acceptance rate above a predetermined upper limit criterion as an edit recommended pattern and a pattern with an acceptance rate below a predetermined lower limit criterion as a retention recommended pattern; and (d-4) a step of increasing the confidence for the edit recommended pattern and decreasing the confidence for the retention recommended pattern. Claim 9 A video editing system that performs a context-based reliability calculation algorithm, wherein, in any one of claims 1, 5, or 7, the editing point analysis unit comprises: (a-1) a step of obtaining the context before and after each word from the text conversion data; (a-2) a step of determining the context type of each word based on the context before and after; and (a-3) a step of calculating different reliability for the same word according to the context type. Claim 10 delete

Images