Apparatus for correcting original text in image by using natural language processing processor

Abstract

The present invention provides an apparatus for correcting original text in an image by using a natural language processing processor. The apparatus for correcting original text in an image by using a natural language processing processor comprises a natural language processing processor and a memory, wherein the memory stores instructions that, when executed, cause the natural language processing processor to: identify text data converted from image data; classify the text data by sentence on the basis of morphemes; identify a first sentence in which a subject and / or an object is missing in the text data classified by sentence; input the first sentence and the domain of the image data into a pre-trained deep learning model; receive first modified data from the deep learning model; identify a first final sentence by including the first modified data in the first sentence; and generate modified text data for translation by including the first final sentence, instead of the first sentence, in the text data.

Description

Video text editing device using a natural language processing processor

[0001] The present invention relates to a device for modifying an original text before translation is performed on voice data extracted from image data using a natural language processing processor.

[0002]

[0003] The content described in this section merely provides background information for the present embodiment and does not constitute prior art.

[0004] With the recent increase in video usage, the need for subtitles for videos produced in multiple languages ​​is increasing. Specifically, providing subtitles in a language different from the original language requires translation of audio data extracted from the video data.

[0005] Translation can be performed after converting audio data into text. However, simply performing machine translation can lead to low translation accuracy. Furthermore, if the original text contains sentences missing at least one of the subject, object, or predicate, performing the translation as-is can lower the accuracy and completeness of the translation. Furthermore, if the video data contains domain-specific terminology (e.g., jargon, abbreviations, etc.), the likelihood of mistranslation can increase.

[0006] Therefore, there was a need for a device that could improve the accuracy of translation.

[0007]

[0008] The purpose of the present invention is to provide an image original text correction device using a natural language processing processor that can perform translation by generating corrected text data that converts a sentence in which at least one of a subject and an object is missing into a complete sentence using a pre-learned deep learning model.

[0009] In addition, an object of the present invention is to provide an image source text correction device using a natural language processing processor capable of converting a sentence with a missing predicate into a complete sentence.

[0010] In addition, an object of the present invention is to provide an image original text modification device using a natural language processing processor capable of modifying text data before performing translation and then performing translation on the modified text data.

[0011] The objectives of the present invention are not limited to those mentioned above. Other objectives and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0012]

[0013] An image original text correction device using a natural language processing processor according to an embodiment of the present invention includes a natural language processing processor and a memory operatively connected to the natural language processing processor, wherein the memory stores instructions that, when executed, cause the natural language processing processor to identify text data converted from image data, segment the text data into sentences, identify a first sentence in which at least one of a subject and an object is missing from the text data segmented into sentences, input the first sentence and a domain of the image data into a pre-trained deep learning model, receive first correction data corresponding to at least one of the subject and the object of the first sentence that is missing from the deep learning model, include the first correction data in the first sentence, identify a first final sentence in which at least one of the subject and the object of the first sentence that is missing is supplemented, and include the first final sentence in the text data instead of the first sentence, thereby generating corrected text data for translation.

[0014] In addition, the instructions cause the natural language processing processor to identify a second sentence in which a predicate is missing from the text data classified by sentences, identify a portion of image data between a predetermined time from the end time of the second sentence in the image data, identify second correction data corresponding to the missing predicate of the second sentence based on facial expressions and actions of characters in the portion of the image data, include the second correction data in the second sentence, identify a second final sentence in which the missing predicate of the second sentence is supplemented, and include the second final sentence in the text data instead of the second sentence, thereby generating the corrected text data.

[0015] In addition, the instructions cause the natural language processing processor to identify a second sentence having a missing predicate in the text data divided by the sentences, identify a third sentence located after the second sentence and a fourth sentence located before the second sentence in the text data, identify the predicate of the third sentence and the predicate of the fourth sentence, respectively, determine a context based on the predicate of the third sentence and the predicate of the fourth sentence, identify second correction data corresponding to the missing predicate of the second sentence based on the determined context, include the second correction data in the second sentence, identify a second final sentence in which the missing predicate of the second sentence is supplemented, and include the second final sentence in the text data instead of the second sentence, thereby generating the corrected text data.

[0016] In addition, the instructions cause the natural language processing processor to identify a second sentence having a missing predicate in the text data classified by sentence, input the domain of the image data, the second sentence, and the text data into the pre-trained deep learning model, receive second correction data corresponding to the missing predicate of the second sentence from the deep learning model, include the second correction data in the second sentence, identify a second final sentence having the missing predicate of the second sentence supplemented, and include the second final sentence in the text data instead of the second sentence, thereby generating the corrected text data, and the deep learning model determines a context based on the second sentence and the text data and outputs the second correction data.

[0017] In addition, the instructions cause the natural language processing processor to train the deep learning model using training data including a domain of the image data, a training sentence with at least one of a subject and an object missing, and a final training sentence, wherein the final training sentence is a complete sentence in which at least one of the subject and object of the training sentence that is missing is supplemented.

[0018] Additionally, the instructions cause the natural language processing processor to monitor whether the image data is uploaded when the user has registered a channel to upload the image data, and to identify the image data when the image data is uploaded to the channel.

[0019] Additionally, the instructions cause the natural language processing processor to identify whether the text data classified by sentences includes a term related to the domain of the image data, and, based on the inclusion of the term in the text data classified by sentences, convert the term into a substitute term, and then identify the first sentence after converting the term into the substitute term.

[0020] In addition, the instructions cause the natural language processing processor to identify some text data corresponding to a time from the start of the image data to a set time from the text data classified by sentence, input the some text data into the pre-trained deep learning model, and receive the domain of the image data from the deep learning model.

[0021] Additionally, the instructions cause the natural language processing processor to train the deep learning model using domain learning sentences containing words by which the domain can be identified and domain learning data containing the domain.

[0022] Additionally, the instructions cause the natural language processing processor to identify a second sentence containing a prohibited word in the text data classified by sentence, and to exclude a subtitle for the prohibited word in the second sentence.

[0023]

[0024] The video original text correction device using the natural language processing processor of the present invention can improve the accuracy and completeness of translation by converting a sentence in which at least one of the subject and object is missing into a complete sentence using a pre-learned deep learning model and then performing translation.

[0025] In addition, the image original text correction device using the natural language processing processor of the present invention can improve the accuracy and completeness of translation by converting a sentence with a missing predicate into a complete sentence and then performing translation.

[0026] In addition, the image original text modification device using the natural language processing processor of the present invention can improve the accuracy and completeness of translation by performing translation after modifying text data before performing translation.

[0027] In addition to the above-described contents, the specific effects of the present invention are described together with the specific matters for carrying out the invention below.

[0028]

[0029] FIG. 1 is a drawing for explaining an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0030] FIG. 2 is a flowchart for explaining the operation of an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0031] Figure 3 is a drawing for explaining step (S105) of Figure 2.

[0032] FIG. 4 is a diagram for explaining an image original text modification device and a deep learning model using a natural language processing processor according to an embodiment of the present invention.

[0033] FIG. 5 is a drawing for explaining an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0034] Figure 6 is a diagram illustrating an example of the configuration of the deep learning model of Figure 4.

[0035] FIG. 7 is a drawing for explaining the operation of an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0036] FIG. 8 and FIG. 9 are drawings for explaining step (S203), step (S205), and step (S207) of FIG. 7.

[0037] FIG. 10 is a drawing for explaining steps (S203), (S205) and (S207) of FIG. 7.

[0038] FIG. 11 is a drawing for explaining steps (S203), (S205) and (S207) of FIG. 7.

[0039] FIG. 12 is a drawing for explaining the operation of an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0040] FIG. 13 is a drawing for explaining an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0041] FIG. 14 and FIG. 15 are drawings for explaining the operation of an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0042] FIG. 16 is a diagram for explaining a hardware implementation of an image source text modification device using a natural language processing processor according to some embodiments of the present invention.

[0043]

[0044] The terms and words used in this specification and claims should not be interpreted based on their general or dictionary meanings. In accordance with the principle that inventors can define the concepts of terms and words to best describe their inventions, they should be interpreted in a way that is consistent with the technical concept of the present invention. Furthermore, the embodiments described in this specification and the configurations depicted in the drawings are merely examples of how the present invention can be realized and do not fully represent the technical concept of the present invention. Therefore, it should be understood that various equivalents, modifications, and applicable examples may exist as of the time of filing.

[0045] The terms first, second, A, B, etc. used in this specification and claims may be used to describe various components, but the components should not be limited by these terms. These terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term "and / or" includes any combination of a plurality of related listed items or any item among a plurality of related listed items.

[0046] The terminology used in this specification and claims is for the purpose of describing specific embodiments only and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "comprise" or "have" in this application do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0047] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0048] Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings within the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined herein. Furthermore, each component, process, procedure, or method included in each embodiment of the present invention may be shared within the scope of non-contradictory technical aspects.

[0049]

[0050] Hereinafter, a video original text modification device using a natural language processing processor according to an embodiment of the present invention will be described with reference to FIGS. 1 to 6.

[0051] FIG. 1 is a diagram for explaining an image original text modification device using a natural language processing processor according to an embodiment of the present invention. FIG. 2 is a flowchart for explaining the operation of an image original text modification device using a natural language processing processor according to an embodiment of the present invention. FIG. 3 is a diagram for explaining step (S105) of FIG. 2. FIG. 4 is a diagram for explaining an image original text modification device using a natural language processing processor according to an embodiment of the present invention and a deep learning model. FIG. 5 is a diagram for explaining an image original text modification device using a natural language processing processor according to an embodiment of the present invention. FIG. 6 is a diagram exemplarily illustrating the configuration of the deep learning model of FIG. 4.

[0052] Referring to FIGS. 1 and 2, an image source text modification device (100) using a natural language processing processor according to some embodiments of the present invention may include a natural language processing processor (110) and a memory (120). The image source text modification device (100) using a natural language processing processor may include at least one additional component in addition to the components illustrated in FIG. 1. For example, the image source text modification device (100) using a natural language processing processor may further include at least one of a communication module or a display. For example, the components of the image source text modification device (100) using a natural language processing processor may be the same entity or may constitute separate entities.

[0053] The memory (120) can store various data used by at least one component (e.g., the natural language processing processor (120)) of the image source text modification device (100) using the natural language processing processor. The data can include, for example, input data or output data for software (e.g., a program) and commands related thereto. The memory (120) can include volatile memory or non-volatile memory.

[0054] The memory (120) may store commands, information, or data related to the operations of components included in the image text modification device (100) using a natural language processing processor. For example, the memory (120) may store instructions that, when executed, enable the natural language processing processor (110) to perform various operations described in this document.

[0055] The natural language processing processor (110) may be operatively coupled to a memory (120) to perform the overall function of the image source text modification device (100) using the natural language processing processor. The natural language processing processor (110) may include, for example, one or more natural language processing processors. The one or more natural language processing processors may include, for example, an image signal processor (ISP), an application processor (AP), or a communication processor (CP).

[0056] The natural language processing processor (110) may, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the image original text modification device (100) using the natural language processing processor connected to the natural language processing processor (110), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the natural language processing processor (110) may load a command or data received from another component (e.g., a communication module) into the memory (120), process the command or data stored in the memory (120), and store the result data in the memory (120). According to one embodiment, the natural language processing processor (110) may include a main natural language processing processor (e.g., a central processing unit or an application natural language processing processor), and an auxiliary natural language processing processor (e.g., a graphics processing unit, an image signal natural language processing processor, a sensor hub natural language processing processor, or a communication natural language processing processor) that can operate independently or together therewith. Additionally or alternatively, the auxiliary natural language processing processor may be configured to use less power than the main natural language processing processor or to be specialized for a specific function. The auxiliary natural language processing processor may be implemented separately from the main natural language processing processor or as part of the main natural language processing processor.

[0057] The program may be stored as software in memory (120) and may include, for example, an operating system, middleware, or an application.

[0058] The image original text modification device (100) using a natural language processing processor can exchange data with a deep learning model (200). The deep learning model (200) may be pre-trained. Although the drawing illustrates that the deep learning model (200) is not included in the image original text modification device (100) using a natural language processing processor, the present invention is not limited thereto. For example, the image original text modification device (100) using a natural language processing processor may include the deep learning model (200).

[0059] The natural language processing processor (110) can identify text data converted from image data (S101). The image data may be a target for translation after the original text (text data) has been modified by the image original text modification device (100) using the natural language processing processor according to an embodiment of the present invention. After the translation is complete, subtitles for the translated text can be provided as image data.

[0060] For example, text data may be extracted from video data and converted from voice data. The natural language processing processor (110) may identify video data uploaded via a means for sharing data (e.g., the Internet). Alternatively, the natural language processing processor (110) may identify video data uploaded directly from a user to the video original text modification device (100) using the natural language processing processor.

[0061] In some embodiments, the natural language processing processor (110) may monitor whether video data has been uploaded when a user has registered a channel for uploading video data in the video text editing device (100) using the natural language processing processor. The natural language processing processor (110) may monitor the channel registered by the user and, if video data has been uploaded to the channel, identify the uploaded video data.

[0062] In some embodiments, the natural language processing processor (110) can identify image data registered by a user. The user can upload the image data to a video text editing device (100) using the natural language processing processor to translate the audio of the image data and provide subtitles.

[0063] The natural language processing processor (110) can segment text data into sentences based on morphemes (S105). For example, the natural language processing processor (110) can identify each sentence included in the text data by identifying at least one of a final ending or a sentence ending mark based on the identified morphemes. The sentence ending mark may be, for example, a period, a question mark, an exclamation mark, etc. The natural language processing processor (110) can identify the start time and the end time for each separated sentence. In addition, the natural language processing processor (110) can identify the start time and the end time for each word included in the separated sentence.

[0064] Referring to FIG. 3, the natural language processing processor (110) can divide text data into morpheme units (301) and identify them. The natural language processing processor (110) can analyze the identified morphemes and divide the text data into sentences (302).

[0065] Referring back to FIGS. 1 and 2 , the natural language processing processor (110) can identify a first sentence in which at least one of the subject and object is missing from text data divided into sentences (S107). The text data may be divided into sentences and identified by the natural language processing processor (110). The natural language processing processor (110) can determine whether at least one of the subject and object is missing for each sentence divided separately from the text data.

[0066] The natural language processing processor (110) can input the identified first sentence and the domain of the image data into a pre-trained deep learning model (200) (S109). The domain of the image data may be, for example, a category to which the content of the image data may belong (e.g., fashion, food, etc.).

[0067] In some embodiments, the domain of the image data may be registered by the user.

[0068] In some embodiments, the natural language processing processor (110) can identify the domain of image data using a pre-trained deep learning model. This is described below with reference to FIGS. 13 to 15.

[0069] In some embodiments, the natural language processing processor (110) may analyze a portion of text data to identify a domain of image data. This is described below with reference to FIGS. 13 to 15.

[0070] The natural language processing processor (110) may receive first correction data from the deep learning model (200) (S111). The first correction data may correspond to at least one of the subject and object of the missing first sentence.

[0071] Referring to FIGS. 4, 5, and 6, the natural language processing processor (110) inputs the domain of the first sentence and image data into the deep learning model (200), and first correction data corresponding to at least one of the missing subject and object of the first sentence can be received from the deep learning model (200). The deep learning model (200) may be pre-learned.

[0072] For example, if the first sentence is “I recognize that it is my house,” the natural language processing processor (110) can identify that the first sentence is a sentence with a missing subject in the text data. The natural language processing processor (110) can input the puppy, which is the domain of the image data, and the first sentence into the deep learning model (200). The natural language processing processor (110) can receive first correction data (e.g., “the puppy”) corresponding to the missing subject in the first sentence from the deep learning model (200).

[0073] The natural language processing processor (110) can use training data to enable a deep learning model (200) to learn. The training data may include, for example, an image data domain, a training sentence, and a final training sentence. The training sentence may be, for example, a sentence with at least one of a subject and an object missing. The final training sentence may be, for example, a complete sentence with at least one of the subjects and objects of the missing training sentence supplemented.

[0074] For example, the natural language processing processor (110) can enable the deep learning model (200) to learn using learning data including a dog as a domain, “picks up and eats” as a learning sentence, and “the dog picks up and eats” as a final learning sentence.

[0075] For example, the natural language processing processor (110) can enable a deep learning model (200) to learn using training data including mixed martial arts as a domain, “the athlete attempts a knee kick” as a training sentence, and “the athlete attempts a knee kick” as a final training sentence.

[0076] A deep learning model (200) may include an input layer (input) that uses the domain of image data and sentences for training as input nodes, an output layer (Output) that uses the final sentences for training as output nodes, and M hidden layers placed between the input layer and the output layer.

[0077] Here, weights can be assigned to the edges connecting the nodes of each layer. These weights or the presence or absence of edges can be added, removed, or updated during the learning process. Therefore, the weights of the nodes and edges between the k input nodes and i output nodes can be updated during the learning process.

[0078] Before the deep learning model (200) performs learning, all nodes and edges can be set to initial values. However, when information is input cumulatively, the weights of the nodes and edges change, and in this process, a matching can be achieved between the parameters input as learning factors (i.e., the domain of the image data and the sentences for learning) and the values ​​assigned to the output nodes (i.e., the final sentences for learning).

[0079] Additionally, when utilizing a cloud server (not shown), the deep learning model (200) can receive and process a large number of parameters. Accordingly, the deep learning model (200) can perform learning based on a large amount of data.

[0080] The weights of the nodes and edges between the input and output nodes that constitute the deep learning model (200) can be updated through the learning process of the deep learning model (200). Furthermore, it goes without saying that the parameters output from the deep learning model (200) can be further expanded with various data in addition to the final sentence for learning.

[0081] Referring back to FIGS. 1 and 2 , the natural language processing processor (110) can identify the first final sentence (S113). For example, the natural language processing processor (110) can identify the first final sentence by including the first modified data in the first sentence. The first final sentence may be a sentence in which at least one of the missing subjects and objects of the first sentence has been supplemented.

[0082] For example, if the first sentence is “I recognize that this is my house,” the domain of the image data is a dog, and the natural language processing processor (110) receives the first correction data (e.g., “the dog”) from the deep learning model (200), the natural language processing processor (110) can identify the first final sentence (e.g., “the dog recognizes that this is its house”) by including the first correction data corresponding to the subject missing from the first sentence in the first sentence.

[0083] The natural language processing processor (110) can generate modified text data (S115). For example, the natural language processing processor (110) can generate modified text data by including the first final sentence in the text data instead of the first sentence. For example, the modified text data may include the first sentence, "I recognize that this is my house," as a modified first final sentence, "The dog recognizes that this is its house."

[0084] The revised text data can be used for translation and subtitle generation. The natural language processing processor (110) divides the original text data into sentences, identifies a first sentence in which at least one of the subject and object is missing from the text data divided into sentences, and replaces the first sentence with a complete sentence in which at least one of the missing subject and object is supplemented to generate revised text data. After that, translation can be performed based on the revised text data. The results of the translation based on the revised text data can be provided as subtitles.

[0085] In some embodiments, the natural language processing processor (110) may prevent subtitles from being provided for a prohibited word (e.g., a swear word) if any of the original text data and the modified text data includes the prohibited word. Alternatively, the natural language processing processor (110) may cause a preset symbol to be displayed in the subtitles in correspondence with the prohibited word. The natural language processing processor (110) may indicate that a translation corresponding to the prohibited word is not provided as a subtitle because any of the original text data and the modified text data includes the prohibited word. The natural language processing processor (110) may notify the user that any of the original text data and the modified text data includes the prohibited word.

[0086] If the original text data contains sentences to be corrected, it is difficult to grasp the context of the original text when correcting it after translation, and furthermore, words that do not fit the context may be corrected, which may increase the possibility of mistranslation. The video original text correction device (100) using a natural language processing processor according to an embodiment of the present invention performs translation using the corrected text data after correcting the original text data, and generates subtitles based on the translation results, thereby improving the completeness and accuracy of the translation.

[0087] Each of the steps described in FIG. 2 can be performed through at least one module that can be included in the natural language processing processor (110).

[0088]

[0089] Hereinafter, with reference to FIGS. 1, 7, and 9, a video source text modification device using a natural language processing processor according to an embodiment of the present invention will be described. For clarity, any details that overlap with those previously described will be simplified or omitted.

[0090] FIG. 7 is a diagram for explaining the operation of an image source text modification device using a natural language processing processor according to an embodiment of the present invention. FIG. 8 and FIG. 9 are diagrams for explaining steps (S203), (S205), and (S207) of FIG. 7.

[0091] Referring to FIGS. 1 and 7, the natural language processing processor (110) can identify text data converted from image data (S101). The natural language processing processor (110) can classify text data into sentences based on morphemes (S105).

[0092] The natural language processing processor (110) can determine whether at least one of the subject and object is missing for each sentence of the separated text data (S201).

[0093] For example, the natural language processing processor (110) can identify a first sentence in which at least one of the subject and object is missing from text data divided by sentence (Y in S201).

[0094] The natural language processing processor (110) can input the domain of the first sentence and image data into a pre-trained deep learning model (200) (S109). The natural language processing processor (110) can receive first correction data from the deep learning model (200) (S111). The first correction data may correspond to at least one of the subject and object missing from the first sentence. The natural language processing processor (110) can identify a first final sentence, which is a completed sentence, by supplementing the first sentence based on the first correction data (S113). The natural language processing processor (110) can generate corrected text data by including the first final sentence instead of the first sentence in the text data (S115).

[0095] The natural language processing processor (110) can determine whether a sentence including a subject and an object includes a predicate (S203). For example, the natural language processing processor (110) can identify a second sentence including a subject and an object among the sentences of each separated text data (N in S201). The natural language processing processor (110) can identify that the second sentence is missing a predicate (Y in S203). For example, the natural language processing processor (110) can identify the second sentence, “The puppy is fed,” in image data related to a puppy.

[0096] The natural language processing processor (110) can identify second modified data related to the second sentence (S205).

[0097] Referring to FIGS. 8 and 9, the natural language processing processor (110) can identify a second sentence (402) in which a predicate is missing from text data (400).

[0098] The natural language processing processor (110) can identify the end time (402b) of the second sentence (402) and a certain time (403a) from the end time (402b) of the second sentence (402). For example, the end time (402b) of the second sentence (402) can be identified based on the final ending (4021).

[0099] The video data may be playable video data from the start time (501) to the end time (503) of the video data. The natural language processing processor (110) may identify some video data from the video data between the end time (402b) of the second sentence (402) and a certain period of time (403a).

[0100] The natural language processing processor (110) can identify second correction data corresponding to the predicate of the missing second sentence (402) based on the facial expressions and actions of characters appearing in some video data. For example, the natural language processing processor (110) can identify the second correction data "eat" for the second sentence ("the dog is fed") with the missing predicate.

[0101] Referring back to FIGS. 1 and 7 , the natural language processing processor (110) can identify the second final sentence (S207). The natural language processing processor (110) can identify the second final sentence with the missing predicate supplemented by including the second correction data in the second sentence. For example, the natural language processing processor (110) can identify the second final sentence (“The dog is eating”) by including the second correction data (“It eats”) in the second sentence with the missing predicate (“The dog is eating”).

[0102] The natural language processing processor (110) can generate modified text data by including the second final sentence instead of the second sentence in the text data (S115).

[0103] The natural language processing processor (110) can identify that a sentence containing a subject and an object among the sentences of each separated text data also contains a predicate (N in S203).

[0104] The natural language processing processor (110) can perform translation on modified text data including a first final sentence with the first sentence supplemented and a second final sentence with the second sentence supplemented. The results of the translation can be provided as subtitles.

[0105] The image original text modification device (100) using a natural language processing processor according to an embodiment of the present invention can improve the completeness and accuracy of translation by performing translation using modified text data after modifying the original text data.

[0106]

[0107] Hereinafter, an image source text modification device using a natural language processing processor according to an embodiment of the present invention will be described with reference to FIGS. 1, 7, and 10. For clarity, any details that overlap with those previously described will be simplified or omitted.

[0108] FIG. 10 is a drawing for explaining steps (S203), (S205) and (S207) of FIG. 7.

[0109] Referring to FIGS. 1 and 7, the natural language processing processor (110) can identify a second sentence containing a subject and an object among the sentences of each separated text data (N in S201). The natural language processing processor (110) can identify that the second sentence is missing a predicate (Y in S203).

[0110] The natural language processing processor (110) can identify second modified data related to the second sentence (S205).

[0111] Referring to FIG. 10, the natural language processing processor (110) can identify a third sentence (403) located after a second sentence (402) in which a predicate is missing in text data (400), and a fourth sentence (404) located before the second sentence (402).

[0112] The natural language processing processor (110) can identify a part of the third sentence (403) including the predicate (4035), which is the final ending of the third sentence (403), and a part of the fourth sentence (404) including the predicate (4044), which is the final ending of the fourth sentence (404).

[0113] For example, if the fourth sentence (404), which is the sentence immediately preceding the second sentence (402), does not contain a predicate, a part of the fifth sentence (405), which includes a predicate (4054), can be identified in the fifth sentence (405), which is the sentence immediately preceding the fourth sentence (404). For example, if the third sentence (403), which is the sentence immediately preceding the second sentence (402), does not contain a predicate, a part of the sixth sentence (406), which includes a predicate (4064), can be identified in the sixth sentence (406), which is the sentence immediately preceding the fourth sentence (404).

[0114] The natural language processing processor (110) can determine the context based on a part including the predicate (4035) of the third sentence (403) and a part including the predicate (4044) of the fourth sentence (404).

[0115] For example, the second sentence may be “The dog is fed.” For example, if the fourth sentence (404) is “The dog needs to be fed three times a day,” and the third sentence (403) is “The dog may be thirsty after eating,” the natural language processing processor (110) can determine the context and identify the second modified data (“eat”) corresponding to the missing predicate in the second sentence (“The dog is fed”).

[0116] The natural language processing processor (110) can identify a second final sentence modified into a complete sentence using the second modified data (S207) and generate modified text data by including the second final sentence in the text data.

[0117]

[0118] Hereinafter, an image source text modification device using a natural language processing processor according to an embodiment of the present invention will be described with reference to FIGS. 1, 5, 7, and 11. For clarity of explanation, any details that overlap with those previously described will be simplified or omitted.

[0119] FIG. 11 is a drawing for explaining steps (S203), (S205) and (S207) of FIG. 7.

[0120] Referring to FIGS. 1 and 7, the natural language processing processor (110) can identify a second sentence containing a subject and an object among the sentences of each separated text data (N in S201). The natural language processing processor (110) can identify that the second sentence is missing a predicate (Y in S203).

[0121] The natural language processing processor (110) can identify second modified data related to the second sentence (S205).

[0122] Referring to FIG. 11, a natural language processing processor (110) can input the domain of image data, a second sentence with a missing predicate, and text data into a pre-trained deep learning model (200).

[0123] The deep learning model (200) can output second modified data by judging the context based on the second sentence and text data. The deep learning model (200) may be pre-trained.

[0124] Referring to FIG. 5, the natural language processing processor (110) can enable a deep learning model (200) to learn using training data. The training data may include, for example, a domain of image data, text data, training sentences, and final training sentences. The training sentences may be, for example, sentences with missing predicates. The final training sentences may be, for example, sentences completed by supplementing the missing predicates of training sentences.

[0125] The deep learning model (200) can determine the context using learning data and output the missing predicate, which is the second correction data, based on the context.

[0126] The natural language processing processor (110) can receive second correction data corresponding to the predicate of the second sentence that is missing from the deep learning model (200).

[0127] In some embodiments, the deep learning model (200) may include at least one model. For example, the deep learning model (200) may include a first model for complementing at least one of a subject and an object, and a second model for complementing a predicate.

[0128]

[0129] Hereinafter, with reference to FIGS. 1 and 12, a video source text modification device using a natural language processing processor according to an embodiment of the present invention will be described. For clarity, any details that overlap with those previously described will be simplified or omitted.

[0130] FIG. 12 is a drawing for explaining the operation of an image original text modification device using a natural language processing processor according to an embodiment of the present invention.

[0131] Referring to FIG. 1 and FIG. 12, the natural language processing processor (110) can identify text data converted from image data (S101) and classify the text data into sentences based on morphemes (S105).

[0132] The natural language processing processor (110) can identify whether a term is included in text data divided into sentences (S301). The term may be, for example, a term related to the domain of the image data. The term may, for example, be used only in a specific domain and not generally used. The term may be, for example, an abbreviation or a new word. The natural language processing processor (110) can identify whether a term is included in each divided sentence of the text data.

[0133] The natural language processing processor (110) can store terms by domain in a database, for example. Based on the terms stored in the database and the domains associated therewith, the natural language processing processor (110) can identify whether the terms are included in text data divided by sentence.

[0134] For example, the natural language processing processor (110) can identify a sentence containing the term “Yeokkula” among each distinct sentence of text data, where the image domain is makeup.

[0135] The natural language processing processor (110) can convert a term into a substitute term (S303) based on whether the term is included (Y in S301). Before performing translation, the natural language processing processor (110) can change a term included in the original text data into a substitute term. The substitute term may be, for example, a common word that can replace the term.

[0136] For example, the natural language processing processor (110) may, for a sentence containing the term “Yeokkula,” convert the term “Yeokkula” into the alternative term “summer cool tone light.”

[0137] The natural language processing processor (110) can identify sentences in which at least one of the subject and object is missing from text data separated by sentence after converting the term into a substitute term (S201).

[0138] The natural language processing processor (110) can perform step (S201) based on the fact that the text data divided by sentence does not contain a term (N in S301).

[0139]

[0140] Hereinafter, a video source text modification device using a natural language processing processor according to an embodiment of the present invention will be described with reference to FIGS. 5, 13, and 15. For clarity, any details that overlap with those previously described will be simplified or omitted.

[0141] FIG. 13 is a diagram illustrating an image source text modification device using a natural language processing processor according to an embodiment of the present invention. FIG. 14 and FIG. 15 are diagrams illustrating the operation of an image source text modification device using a natural language processing processor according to an embodiment of the present invention.

[0142] Referring to FIGS. 13, 14, and 15, the natural language processing processor (110) of the image original text modification device (100) using the natural language processing processor can identify the domain of image data using the deep learning model (200).

[0143] The natural language processing processor (110) can identify some text data from text data divided by sentence. The some text data may be part of the text data corresponding to a set time from the start time of the image data.

[0144] The video data may be video data that is played from the start point (501) to the end point (503).

[0145] The natural language processing processor (110) can identify image data for domain verification from the start time (501) of the image data to the set time (510). The natural language processing processor (110) can identify some text data corresponding to the start time (501) of the image data to the set time (510) in the text data (400).

[0146] For example, the first sentence of text data (400) may be the 7th sentence (407), and the 8th sentence (408) may be located after the 7th sentence (407). The natural language processing processor (110) may identify the starting point (407a) of the 7th sentence (407) corresponding to the starting point (501) of the image data, and may identify the starting point (408c) of the 8th sentence (408) corresponding to the set point (510). The natural language processing processor (110) may identify some text data from the starting point (407a) of the 7th sentence (407) to the starting point (408c) of the 8th sentence (408).

[0147] Some text data may include only a portion of a specific sentence in text data (400) separated by sentence.

[0148] The natural language processing processor (110) can input some text data into the deep learning model (200). The natural language processing processor (110) can receive the domain of image data from the deep learning model (200).

[0149] The deep learning model (200) may be pre-trained.

[0150] Referring to FIG. 5, the natural language processing processor (110) can enable a deep learning model (200) to learn using training data. The training data may include, for example, sentences for domain learning and the domain of image data. The sentences for domain learning may be sentences containing words that can identify the domain of the image data.

[0151] In some embodiments, the natural language processing processor (110) can identify a domain of image data without using a deep learning model (200). The natural language processing processor (110) can determine whether some text data contains words that can identify the domain. The natural language processing processor (110) can identify a domain of image data based on whether some text data contains words that can identify the domain.

[0152] The natural language processing processor (110) can store words that can identify a domain in a database. When identifying a domain of image data from some text data, the natural language processing processor (110) can identify the domain of image data based on data stored in the database.

[0153]

[0154] Hereinafter, with reference to FIG. 16, an image source text modification device using a natural language processing processor according to some embodiments of the present invention will be described. For clarity, any details that overlap with those previously described will be omitted.

[0155] FIG. 16 is a diagram for explaining a hardware implementation of an image source text modification device using a natural language processing processor according to some embodiments of the present invention.

[0156] Referring to FIG. 16, an image source text modification device (100) using a natural language processing processor according to some embodiments of the present invention may be implemented as an electronic device (1000). The electronic device (1000) may include a processor (1010), an input / output device (1020, I / O), a memory (1030, memory), an interface (1040), a storage (1050, storage), and a bus (1060, bus). The processor (1010), the input / output device (1020), the memory (1030), the interface (1040), and / or the storage (1050) may be coupled to each other via a bus (1060). The bus (1060) corresponds to a path through which data is transferred.

[0157] Specifically, the processor (1010) may include at least one of a Central Processing Unit (CPU), a Micro Processor Unit (MPU), a Micro Controller Unit (MCU), a Graphic Processing Unit (GPU), a microprocessor, a digital signal processor, a microcontroller, an application processor (AP), and logic elements capable of performing functions similar thereto.

[0158] The input / output device (1020) may include at least one of a keypad, a keyboard, a touchscreen, and a display device.

[0159] The memory (1030) can load data and / or programs, etc. At this time, the memory (1030) is an operating memory for improving the operation of the processor (1010) and may include high-speed DRAM and / or SRAM. The memory (1030) may include one or more volatile memory devices such as DDR SDRAM (Double Data Rate Static DRAM) and SDR SDRAM (Single Data Rate SDRAM) and / or one or more non-volatile memory devices such as EEPROM (Electrically Erasable Programmable ROM) and flash memory.

[0160] The interface (1040) may perform a function of transmitting data to or receiving data from a communication network. The interface (1040) may be wired or wireless. For example, the interface (1040) may include an antenna or a wired or wireless transceiver.

[0161] Storage (1050) can store and preserve data and / or programs. Storage (1050) may include one or more non-volatile memory devices, such as a solid state drive (SSD), a hard drive, or flash memory. In the present invention, storage (1050) may store a computer program comprising instructions for performing a method for detecting abnormal transactions.

[0162] Alternatively, the image source text modification device (100) using a natural language processing processor according to embodiments of the present invention may be a system formed by connecting multiple electronic devices (1000) to each other via a network. In this case, each module or combination of modules may be implemented as an electronic device (1000). However, the present embodiment is not limited thereto.

[0163]

[0164] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more commands stored in a storage medium (e.g., a memory (120)) readable by a machine (e.g., an image text modification device (100) using a natural language processing processor). For example, the natural language processing processor (110) of the machine (e.g., an image text modification device (100) using a natural language processing processor) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0165] According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0166] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0167]

[0168] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

Claims

1. In a video original text modification device using a natural language processing processor, natural language processing processor; and comprising a memory operatively connected to the natural language processing processor; The above memory, when executed, causes the natural language processing processor to: Identify text data converted from image data, Divide the above text data into sentences, In the text data classified by sentence, identify a first sentence in which at least one of the subject and object is missing, The domain of the first sentence and the image data is input into a pre-trained deep learning model, Receive first correction data corresponding to at least one of the subject and object of the first sentence that is missing from the deep learning model, Including the first modified data in the first sentence, identifying a first final sentence in which at least one of the missing subject and object of the first sentence is supplemented, Store instructions for generating modified text data for translation by including the first final sentence instead of the first sentence in the text data. A device for editing original text of an image using a natural language processing processor.

2. In paragraph 1, The above instructions cause the natural language processing processor to: In the text data separated by the above sentences, identify the second sentence with a missing predicate, In the above image data, identify some image data between a certain time from the end time of the second sentence, Based on the facial expressions and actions of the characters in the above-mentioned part of the video data, the second modified data corresponding to the predicate of the missing second sentence is identified, By including the second modified data in the second sentence, a second final sentence is identified in which the missing predicate of the second sentence is supplemented, Generating the modified text data by including the second final sentence in the text data instead of the second sentence A device for editing original text of an image using a natural language processing processor.

3. In paragraph 1, The above instructions cause the natural language processing processor to: In the text data separated by the above sentences, identify the second sentence with a missing predicate, Identify the third sentence located after the second sentence in the above text data, and the fourth sentence located before the second sentence, Identify the predicate of the third sentence above and the predicate of the fourth sentence above, respectively, Based on the predicate of the third sentence above and the predicate of the fourth sentence above, determine the context, Identify the second modified data corresponding to the predicate of the second sentence that is missing based on the determined context, By including the second modified data in the second sentence, a second final sentence is identified in which the missing predicate of the second sentence is supplemented, Generating the modified text data by including the second final sentence in the text data instead of the second sentence A device for editing original text of an image using a natural language processing processor.

4. In paragraph 1, The above instructions cause the natural language processing processor to: In the text data separated by the above sentences, identify the second sentence with a missing predicate, The domain of the above image data, the second sentence, and the text data are input into the pre-trained deep learning model, Receive second modified data corresponding to the predicate of the second sentence that is missing from the deep learning model, By including the second modified data in the second sentence, a second final sentence is identified in which the missing predicate of the second sentence is supplemented, Generating the modified text data by including the second final sentence in the text data instead of the second sentence, The above deep learning model judges the context based on the second sentence and the text data and outputs the second modified data. A device for editing original text of an image using a natural language processing processor.

5. In paragraph 1, The above instructions cause the natural language processing processor to: Train the deep learning model using training data including a domain of the image data, a training sentence with at least one subject and object missing, and a final training sentence, The final sentence for learning above is a complete sentence in which at least one of the subjects and objects of the missing sentence for learning above is supplemented. A device for editing original text of an image using a natural language processing processor.

6. In paragraph 1, The above instructions cause the natural language processing processor to: If a user has registered a channel to upload the above video data, monitor whether the above video data is uploaded, When the above video data is uploaded to the above channel, the above video data is identified A device for editing original text of an image using a natural language processing processor.

7. In paragraph 1, The above instructions cause the natural language processing processor to: Identify whether the text data classified by the above sentences contains a term related to the domain of the image data, Based on the inclusion of the term in the text data classified by the sentences above, the term is converted to a substitute term, After converting the above term to the above alternative term, identifying the first sentence A device for editing original text of an image using a natural language processing processor.

8. In paragraph 1, The above instructions cause the natural language processing processor to: In the text data divided by the above sentences, some text data corresponding to the time from the start of the video data to the set time is identified, Input some of the above text data into the pre-trained deep learning model, Receiving the domain of the image data from the deep learning model A device for editing original text of an image using a natural language processing processor.

9. In paragraph 8, The above instructions cause the natural language processing processor to: The deep learning model is trained using domain learning sentences containing words that can identify the domain and domain learning data containing the domain. A device for editing original text of an image using a natural language processing processor.

10. In paragraph 1, The above instructions cause the natural language processing processor to: In the text data separated by the above sentences, identify the second sentence containing the prohibited word, The subtitles for the above-mentioned prohibited words in the second sentence are excluded. A device for editing original text of an image using a natural language processing processor.

Images