Device and method for deciding whether to concatenate between character strings

Abstract

A method for determining whether strings are connected is disclosed. The method for determining whether strings are connected according to the present invention comprises: a step of obtaining first location information of a first area box placed on a specific screen, a first string within the first area box, and a first image within the first area box; a step of obtaining second location information of a second area box placed on the specific screen separated from the first area box, a second string within the second area box, and a second image within the second area box; a step of calculating, based on an artificial intelligence model, a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string; and a step of determining whether the first area box and the second area box are connected based on the location connection probability, the image connection probability, and the string connection probability.

Description

Technology Field

[0001] The present invention relates to an apparatus and method for determining whether subtitles separated and placed in video content are connected by considering the positional relationship between the strings, the similarity of the images constituting the strings, and the semantic relationship between the strings. Background Technology

[0002] Recently, subtitles are frequently inserted into video content, and particularly in variety shows, entire sentences or words are often separated into individual strings and placed on the screen to add entertainment or emphasis.

[0003] For example, referring to Fig. 1a, the sentence “Please help me” is separated and arranged into individual strings such as “do,” “wa,” “ju,” “se,” and “yo.”

[0004] For another example, refer to Fig. 1b, where the sentence “Oh, I’ve been caught” is separated into individual strings “Oh” and “I’ve been caught”.

[0005] Meanwhile, there have already been technologies that extract words from images and recognize their meanings, such as OCR recognition devices. However, since OCR recognition devices recognize a single word based on the premise that the characters constituting a word are connected continuously, there was a problem in that they could not recognize the entire sentence or the entire word by connecting the separated strings in situations where strings are separated and arranged as shown in FIGS. 1a and FIGS. 1b.

[0006] In other words, due to the diversity of string arrangements, the diversity of fonts, colors, and other image features, and the complexity of string structures, there were limitations in the recognition device determining whether to connect separated subtitles on the screen using word or sentence recognition based simply on rules. The problem to be solved

[0007] The present invention aims to solve the aforementioned problems, and the objective of the present invention is to provide an apparatus and method for determining whether subtitles separated and placed in video content are connected by considering the positional relationship between strings, the similarity of images constituting the strings, and the semantic relationship between strings. means of solving the problem

[0008] A method for determining whether strings are connected according to the present invention comprises: a step of obtaining first location information of a first area box placed on a specific screen, a first string within the first area box, and a first image within the first area box; a step of obtaining second location information of a second area box placed separately from the first area box on the specific screen, a second string within the second area box, and a second image within the second area box; a step of calculating, based on an artificial intelligence model, a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string; and a step of determining whether the first area box and the second area box are connected based on the location connection probability, the image connection probability, and the string connection probability.

[0009] In this case, the first string and the second string may be placed separately from each other on the same screen of the video content as subtitles inserted into the video content.

[0010] Meanwhile, the step of calculating the location connection probability between the first location information and the second location information may include: a step in which a first location feature extraction model outputs a first location feature vector obtained from the first location information based on weights set in the first location feature extraction model; a step in which a second location feature extraction model outputs a second location feature vector obtained from the second location information based on weights identical to those set in the first location feature extraction model; and a step in which a location comparison model calculates the location connection probability between the first location information and the second location information using the first location feature vector and the second location feature vector.

[0011] In this case, the step of calculating the location connection probability between the first location information and the second location information using the first location feature vector and the second location feature vector can be performed by the location comparison model multiplying the 'difference between the j-th vector value of the first location feature vector and the j-th vector value of the second location feature vector' by a weight corresponding to the j-th, thereby calculating the location connection probability between the first location information and the second location information.

[0012] Meanwhile, the step of calculating the image connection probability between the first image and the second image may include: a first image feature extraction model outputting a first image feature vector obtained from the first image based on weights set in the first image feature extraction model; a second image feature extraction model outputting a second image feature vector obtained from the second image based on weights identical to those set in the first image feature extraction model; and an image comparison model calculating the image connection probability between the first image and the second image using the first image feature vector and the second image feature vector.

[0013] In this case, the step of calculating the image connection probability between the first image and the second image using the first image feature vector and the second image feature vector is such that the image comparison model can calculate the positional connection probability between the first image and the second image by multiplying the 'difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector' by a weight corresponding to the j-th.

[0014] Meanwhile, the step of calculating the probability of string connection between the first string and the second string can calculate the probability that the second string follows the first string based on a language model.

[0015] Meanwhile, the step of determining whether the first area box and the second area box are connected can be performed by the final judgment model applying corresponding weights to the location connection probability, the image connection probability, and the string connection probability, respectively, and then summing them up to calculate the connection probability between the first area box and the second area box.

[0016] Meanwhile, a step of extracting a plurality of area boxes from the above specific screen,

[0017] The method may further include the step of selecting the first area box among the plurality of area boxes as the reference area box, and, prior to the step of acquiring the first image, the step of selecting the second area box located outside the upper-left area of ​​the reference area box among the plurality of area boxes as the comparison area box.

[0018] Meanwhile, the device for determining whether strings are connected according to the present invention comprises: an image acquisition unit for acquiring image content; and a control unit for acquiring first location information of a first area box placed on a specific screen of the image content, a first string within the first area box, and a first image within the first area box, and acquiring second location information of a second area box placed separately from the first area box on the specific screen, a second string within the second area box, and a second image within the second area box, and calculating, based on an artificial intelligence model, a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string, and determining whether the first area box and the second area box are connected based on the location connection probability, the image connection probability, and the string connection probability.

[0019] In this case, the first string and the second string may be placed separately from each other on the same screen of the video content as subtitles inserted into the video content.

[0020] Meanwhile, the artificial intelligence model may include a first position feature extraction model that outputs a first position feature vector obtained from the first position information based on weights set in the first position feature extraction model, a second position feature extraction model that outputs a second position feature vector obtained from the second position information based on weights identical to those set in the first position feature extraction model, and a position comparison model that calculates a position connection probability between the first position information and the second position information using the first position feature vector and the second position feature vector.

[0021] In this case, the above-mentioned position comparison model can calculate the position connection probability between the first position information and the second position information by multiplying the 'difference between the j-th vector value of the first position feature vector and the j-th vector value of the second position feature vector' by a weight corresponding to the j-th.

[0022] Meanwhile, the artificial intelligence model may include a first image feature extraction model that outputs a first image feature vector obtained from the first image based on weights set in the first image feature extraction model, a second image feature extraction model that outputs a second image feature vector obtained from the second image based on weights identical to those set in the first image feature extraction model, and an image comparison model that calculates an image connection probability between the first image and the second image using the first image feature vector and the second image feature vector.

[0023] In this case, the image comparison model can calculate the positional connection probability between the first image and the second image by multiplying the 'difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector' by a weight corresponding to the j-th.

[0024] Meanwhile, the artificial intelligence model may include a string-to-string probability distribution determination model that calculates the probability that the second string follows the first string based on a language model.

[0025] Meanwhile, the artificial intelligence model may include a final decision model that calculates the connection probability between the first area box and the second area box by applying corresponding weights to the location connection probability, the image connection probability, and the string connection probability, respectively, and then summing them.

[0026] Meanwhile, the control unit may select the first area box among the plurality of area boxes as the reference area box, and select the second area box located outside the upper-left area of ​​the reference area box among the plurality of area boxes as the comparison area box.

[0027] Meanwhile, a computer program stored in a medium according to the present invention performs a method for determining whether strings are connected, comprising the steps of: acquiring first location information of a first area box placed on a specific screen, a first string within the first area box, and a first image within the first area box; acquiring second location information of a second area box placed separately from the first area box on the specific screen, a second string within the second area box, and a second image within the second area box; calculating, based on an artificial intelligence model, a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string; and determining whether the first area box and the second area box are connected based on the location connection probability, the image connection probability, and the string connection probability. Effects of the invention

[0028] Since humans comprehensively consider context, semantic relationships between strings, arrangement of strings, fonts, and colors to recognize separated strings on a screen as a single continuous character, they can easily recognize separated strings on a single screen as connected words or sentences. Furthermore, the present invention has the advantage of predicting the connectivity of strings arranged in an image by implementing this human intuitive judgment method into an artificial intelligence model.

[0029] In particular, in conventional OCR recognition, it was very difficult to recognize separated strings by connecting them to each other, and especially when strings were not arranged horizontally or vertically to each other as in FIG. 1a, they could not be recognized by connecting them to each other. However, the present invention has the advantage of improving the accuracy of determining whether strings can be connected by learning the pattern of how strings that can be connected as one are arranged on the screen.

[0030] In addition, conventional OCR recognition only recognizes strings as text and does not consider font type, color, etc. However, strings that can be connected as one mostly have the same font type, a specific color pattern, or a similar background. Therefore, the present invention has the advantage of improving the accuracy of determining whether a string is connected by comparing the string as an image and reflecting it in the final judgment.

[0031] In addition, unlike standardized documents, strings are dynamically arranged in various ways in media subtitle recognition and advertising screen recognition, so in such cases, manual processing by humans was required. Furthermore, when processing simply based on rules, it was difficult to respond flexibly to various string separations and string arrangement forms. However, according to the present invention, by training an artificial intelligence model using a large amount of actual screens (e.g., frame video with subtitles inserted in a broadcast program), there is an advantage of being able to accurately determine whether strings are connected without human intervention.

[0032] Accordingly, the present invention has the advantage of being able to provide various intelligent media services, such as video search, automatic highlight scene generation, and clip video generation.

[0033] When subtitle creators separate and arrange sentences or words, they arrange them from left to right and from top to bottom to ensure readability. Therefore, the comparison area box that must be connected after the reference area box is unlikely to be located at the top-left corner of the reference area box. Accordingly, the present invention has the advantage of minimizing computational load by excluding the area box located at the top-left corner of the reference area box from the candidate group for the comparison area box. Brief explanation of the drawing

[0034] Figure 1 is a diagram illustrating subtitles inserted into video content. FIG. 2 is a block diagram illustrating a device for determining whether strings are connected according to the present invention. FIG. 3 is a flowchart illustrating a method for determining whether strings are connected according to the present invention. FIG. 4 is a diagram schematically illustrating the artificial intelligence model used in the present invention. FIG. 5 is a drawing for explaining a plurality of area boxes according to the present invention. FIG. 6 is a diagram illustrating the operation of a region box relationship determination model according to the present invention. Figure 7 is a diagram illustrating the operation of an image similarity judgment model. Figure 8 is a diagram illustrating a method for selecting a candidate group of comparison area boxes. Figure 9 is a diagram illustrating a method for determining whether strings are connected in various subtitle arrangements. Specific details for implementing the invention

[0035] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0036] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0037] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0038] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0039] For convenience of explanation in implementing the present invention, the components may be described in detail; however, these components may be implemented within a single device or module, or a single component may be divided and implemented across multiple devices or modules.

[0040] FIG. 2 is a block diagram illustrating a device for determining whether strings are connected according to the present invention.

[0041] The device (100) for determining whether to connect strings may refer to a device that provides TV services, such as a TV or a set-top box (STB). Here, TV services may refer to broadcasting services such as cable TV or IPTV, VOD services, content recommendation services, search services, etc. In this case, the device (100) for determining whether to connect strings may include, in addition to the components shown in FIG. 2, a display unit (in the case of a TV) or an interface for connecting to a TV (in the case of a set-top box).

[0042] In addition, the device (100) for determining whether strings are connected may be an IPTV system that provides various multimedia content, such as movies or broadcast programs, internet search, and other additional services to a television receiver using an ultra-high-speed internet network.

[0043] However, it is not limited to this, and the device (100) for determining whether strings are connected may be various devices capable of recognizing strings from video content.

[0044] The components of the device (100) for determining whether strings are connected, as described in FIG. 2, are not essential for implementing the operation according to the present invention, so some of the components may be omitted or other components may be added.

[0045] A device (100) for determining whether strings are connected may include an image acquisition unit (110), a control unit (120), and a memory (130).

[0046] The video acquisition unit (110) can acquire video content.

[0047] Specifically, the image acquisition unit (110) includes a communication unit, and in this case, the image acquisition unit (110) can support wired or wireless communication between the device (100) determining whether to connect strings and an external device.

[0048] More specifically, the image acquisition unit (110) may provide an interface for connecting the device (100) determining whether strings are connected to a wired / wireless network including the Internet network. Additionally, the image acquisition unit (110) may receive image content from an external device through a network connected to the image acquisition unit (110) or another network linked to the connected network.

[0049] The control unit (120) can control the overall operation of the device (100) for determining whether strings are connected.

[0050] In addition, the control unit (120) can extract multiple area boxes from video content and determine whether to connect the multiple area boxes.

[0051] The memory (130) can store programs, content, signal-processed images, voice or data signals, etc. for each signal processing and control within the control unit (120).

[0052] Additionally, the memory (130) can store an application or data for the operation of the device (100) for determining whether strings are connected.

[0053] FIG. 3 is a flowchart illustrating a method for determining whether strings are connected according to the present invention.

[0054] A method (300) for determining whether strings are connected according to the present invention may include: a step (S310) of obtaining first location information of a first area box placed on a specific screen, a first string within the first area box, and a first image within the first area box; a step (S320) of obtaining second location information of a second area box placed separately from the first area box on a specific screen, a second string within the second area box, and a second image within the second area box; a step (S350) of calculating a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string; and a step (S370) of determining whether the first string and the second string are connected based on the location connection probability, the image connection probability, and the string connection probability.

[0055] FIG. 4 is a diagram schematically illustrating the artificial intelligence model used in the present invention.

[0056] An artificial neural network (neural network) models the operating principles of biological neurons and the relationships between them; it is an information processing system in which multiple neurons, called nodes or processing elements, are connected in a layered structure.

[0057] Artificial neural networks can be trained using training data. Here, training refers to the process of determining the parameters of an artificial neural network using training data to achieve objectives such as classifying, regressing, or clustering input data. Typical examples of artificial neural network parameters include weights assigned to synapses or biases applied to neurons.

[0058] An artificial neural network trained using training data may be referred to as an artificial intelligence model in this specification.

[0059] Among the learning methods of artificial neural networks, supervised learning is a machine learning approach designed to infer a function from training data. Furthermore, predicting and outputting the class of an input vector from these inferred functions can be referred to as classification.

[0060] In supervised learning, artificial neural networks are trained with labels provided for the training data. Here, a label refers to the correct answer (or result) that the artificial neural network must infer when the training data is input.

[0061] In this specification, the correct answer (or result) that an artificial neural network must infer when training data is input is referred to as labeling data.

[0062] The loss function can be used as an indicator (criterion) to determine optimal model parameters during the training process of an artificial neural network. In artificial neural networks, training refers to the process of manipulating the network's parameters (weights, biases, etc.) to reduce the loss function, and the objective of training can be viewed as determining the parameters that minimize the loss function.

[0063] The loss function may primarily use the Mean Squared Error (MSE) or Cross Entropy Error (CEE), but the present invention is not limited thereto.

[0064] Meanwhile, where the term “model” is used in this specification, it may mean that the corresponding model is composed of an artificial neural network.

[0065] Because humans comprehensively consider context, semantic relationships between strings, arrangement of strings, fonts, and colors to recognize separated strings on a screen as a single continuous character, they can easily recognize separated strings on a single screen as connected words or sentences.

[0066] Accordingly, the artificial intelligence model (400) according to the present invention may include a region box relationship judgment model (410), an image similarity judgment model (420), a probability distribution judgment model between strings (430), and a final judgment model (440).

[0067] Meanwhile, the instructions constituting the artificial intelligence model (400) may be stored in memory (130). In this case, the control unit (120) can read the instructions stored in memory (130) and perform the operation of the artificial intelligence model (400) described below, so the operation of the artificial intelligence model (400) can be seen as the operation of the control unit (120).

[0068] The area box relationship determination model (410) may be a model that determines the possibility of a connection between the first area box and the second area box by considering the arrangement of the first area box and the second area box.

[0069] Accordingly, the area box relationship judgment model (410) receives the first location information of the first area box and the second location information of the second area box as input values, and can output an output value called the location connection probability (P1) between the first location information and the second location information.

[0070] The image similarity judgment model (420) may be a model that determines the possibility of a connection between a first area box and a second area box by considering the similarity or other correlation between a first area box as an image and a second area box as an image.

[0071] Accordingly, the image similarity judgment model (420) can receive the first image within the first area box and the second image within the second area box as input values ​​and output an output value called the image connection probability (P2) between the first image and the second image.

[0072] The probability distribution determination model (430) between strings may be a model that determines the possibility of a connection between the first area box and the second area box by considering the semantic relationship or context between the string in the first area box and the string in the second area box.

[0073] Accordingly, the probability distribution determination model between strings (430) can receive the first string within the first area box and the second string within the second area box as input values ​​and calculate the probability of string connection between the first string and the second string.

[0074] The final judgment model (440) may be a model for finally determining whether the first area box and the second area box are connected by considering all possibilities for connection considering the arrangement of the first area box and the second area box, possibilities for connection considering the similarity or other correlation between the first area box as an image and the second area box as an image, and possibilities for connection considering the semantic relationship or context of the strings of the first area box and the second area box.

[0075] Accordingly, the final judgment model (440) can receive the output value (P1) of the region box relationship judgment model (410), the output value (P2) of the image similarity judgment model (420), and the output value (P3) of the probability distribution between strings (430) as input values, and output the final connection probability (P) or connection status between the first region box and the second region box.

[0076] FIG. 5 is a drawing for explaining a plurality of area boxes according to the present invention.

[0077] The control unit (120) can extract a plurality of area boxes (510, 520) from a specific screen (500).

[0078] Specifically, the control unit (120) recognizes characters within a specific screen (500) and can obtain one or more strings that can be grouped into units among the characters extracted from the specific screen (500). Here, the criterion for obtaining one or more strings that can be grouped into units may be whether the characters are arranged in a continuous sequence.

[0079] For example, referring to Fig. 5, “Seul” and “Seul” are arranged continuously when considering the spacing between characters and the arrangement of characters, and “Si” and “Dong” are also arranged continuously when considering the spacing between characters and the arrangement of characters. However, “Seul” and “Si” cannot be said to be arranged continuously.

[0080] In this case, the control unit (120) can obtain two strings (“slowly”, “start”) that can be grouped into daily units. Additionally, the control unit (120) can obtain two area boxes (510, 520) that each contain two strings inside.

[0081] That is, the first area box (510) may contain a first string (“slowly”) inside, and the second area box (520) may contain a second string (“starting”) inside. Additionally, a single string may be arranged consecutively.

[0082] In addition, a string of characters contained in a single area box may be composed of multiple characters. For example, referring to FIG. 5, the first string of characters may be composed of multiple characters (“seul-seul”).

[0083] However, this is not limited to this, and a string of characters included in a single area box may consist of a single character. For example, referring again to FIG. 1a, there are 6 area boxes in the screen, and each area box may contain a string of characters consisting of a single character.

[0084] In addition, multiple strings included in each of the multiple area boxes may have different numbers of characters constituting the strings.

[0085] Meanwhile, the control unit (120) can recognize characters placed on the same screen, separate the characters placed on the same screen into multiple strings, and obtain multiple area boxes each containing multiple strings.

[0086] That is, in the present invention, the plurality of strings subject to determination of connection status may be strings that are placed separately from each other on the same screen (for example, the entire video of the same frame in a broadcast program).

[0087] Additionally, multiple strings may be subtitles inserted into video content. For example, multiple strings may be subtitles separately inserted into the filmed video by the producer of a broadcast program.

[0088] The following description assumes that two area boxes (a first area box (510) and a second area box (520)) exist on a specific screen (500). Accordingly, the control unit (120) can obtain a first area box (510) placed on the specific screen (500) and a second area box (520) placed on the specific screen (500) separately from the first area box.

[0089] However, the invention is not limited thereto, and may be applied even when a larger number of area boxes exist.

[0090] FIG. 6 is a diagram illustrating the operation of a region box relationship determination model according to the present invention.

[0091] The control unit (120) can obtain first location information of a first area box (510) and second location information of a second area box (520) placed on a specific screen (500).

[0092] Here, the location information of the area box may mean coordinate information indicating the location of the area box on a specific screen (500).

[0093] For example, the top-left coordinate of a specific screen (500) is set to (0, 0) as a reference coordinate, and the coordinate information of the first area box (510) can be calculated based on the top-left coordinate of the specific screen (500).

[0094] In this case, the coordinate information of the first area box (510) may include the coordinate of the minimum value on the x-axis of the first area box (510) (x1_min), the coordinate of the maximum value on the x-axis of the first area box (510) (x1_max), the coordinate of the minimum value on the y-axis of the first area box (510) (y1_min), and the coordinate of the maximum value on the y-axis of the first area box (510) (y1_max).

[0095] For example, referring to the first area box (510) of FIG. 6A, the coordinates of the first vertex (511) of the first area box (510) may be (x1_min, y1_min), the coordinates of the second vertex (512) of the first area box (510) may be (x1_max, y1_min), the coordinates of the third vertex (513) may be (x1_min, y1_max), and the coordinates of the fourth vertex (514) may be (x1_max, y1_max).

[0096] Meanwhile, coordinate information of the second area box (520) can also be obtained in the same way.

[0097] Meanwhile, before examining the operation of the region box relationship judgment model (410), the structure and training method of the region box relationship judgment model (410) will be explained first.

[0098] The region box relationship judgment model (410) may be composed of a deep neural network (DNN) including multiple hidden layers.

[0099] In this case, the region box relationship determination model (410) may include a first location feature extraction model (411) composed of a fully connected layer with first location information (four coordinates) of a first region box (510) as input values, a second location feature extraction model (412) composed of a fully connected layer with second location information (four coordinates) of a second region box (520) as input values, and a location comparison model (413) that calculates the location connection probability (p1) between the first location information and the second location information using the first location feature vector output by the first location feature extraction model (411) and the second location feature vector output by the second location feature extraction model (412).

[0100] Meanwhile, the string connection determination device (100) can perform training on the area box relationship determination model (410). In this case, the training may be performed by a training device other than the string connection determination device (100).

[0101] In the following, the term “for training” shall be added to data used for training.

[0102] For example, the term "first training area box" is used as a term corresponding to the first area box explained earlier, the term "second training area box" is used as a term corresponding to the second area box, the term "first training location information" is used as a term corresponding to the first location information, and the term "second training location information" is used as a term corresponding to the second location information.

[0103] The control unit (120) can train the area box relationship judgment model (410) by providing the first training location information of the first training area box and the second training location information of the second training area box as input data (input value) to the area box relationship judgment model (410), and by providing the connection status between the first training area box and the second training area box as labeling data (correct answer value) to the area box relationship judgment model (410). In this case, the first training location information and the second training location information can be input after being normalized to a range of 0 to 1.

[0104] Here, the labeling data (correct answer value) can represent one of two classes. The two classes can be connected or unconnected between the two area boxes.

[0105] And the two classes may be correct values ​​that the artificial neural network constituting the area box relationship judgment model (410) must infer using input data (input values).

[0106] Meanwhile, the control unit (120) can input the first training location information of the first training area box into the first location feature extraction model (411). In this case, the first location feature extraction model (411) can extract a first training location feature vector based on the first parameters (weights, biases, etc.) set in the first location feature extraction model (411). In this specification, the meaning of “parameters set in the model” may be a concept including weights or biases applied to synapses or neurons constituting the model.

[0107] Additionally, the control unit (120) can input second training location information of the second training area box into the second location feature extraction model (412). In this case, the second location feature extraction model (412) can extract a second training location feature vector based on the second parameters (weights, biases, etc.) set in the second location feature extraction model (412).

[0108] Meanwhile, the first training feature vector and the second training feature vector can be input into the position comparison model (413). In this case, the position comparison model (413) can calculate the probability of a position connection between the first position information and the second position information using the first training position feature vector and the second training position feature vector based on the third parameter (weights, biases, etc.) set in the position comparison model (413). The operation of the position comparison model (413) will be explained in detail later.

[0109] Meanwhile, when the location connection probability is calculated from the location comparison model (413), the control unit (120) can train the area box relationship determination model (410) based on the difference between the location connection probability calculated from the location comparison model (413) and the labeling data.

[0110] That is, the control unit (120) can calculate a loss value based on a loss function using the difference between the location connection probability and the labeling data, and update the first parameter of the first location feature extraction model (411), the second parameter of the second location feature extraction model (412), and the third parameter set in the location comparison model (413) so that the calculated loss value becomes smaller.

[0111] And the control unit (120) can optimize the first parameter, the second parameter, and the third parameter by repeating this training process.

[0112] Meanwhile, the weights set in the first position feature extraction model (411) and the weights set in the second position feature extraction model (412) may be the same.

[0113] Specifically, the first position feature extraction model (411) and the second position feature extraction model (412) may have the same structure. For example, the first position feature extraction model (411) and the second position feature extraction model (412) may have the same number of layers, the same number of nodes per layer, the same connection structure between nodes, etc.

[0114] Additionally, when the initial parameters of the first position feature extraction model (411) and the second position feature extraction model (412) are identical to each other, the control unit (120) can update the first parameter of the first position feature extraction model (411) and the second parameter of the second position feature extraction model (412) using the same loss value obtained during the training process. Accordingly, the parameters between the first position feature extraction model (411) and the second position feature extraction model (412) can be identical to each other.

[0115] That is, the first position feature extraction model (411) and the second position feature extraction model (412) can have the same structure and the same parameters.

[0116] Meanwhile, when training is completed, the area box relationship judgment model (410) can calculate the probability of location connections between location information even for untrained input data.

[0117] Specifically, the control unit (120) can input the first location information of the first area box (510) and the second location information of the second area box (520) into the area box relationship determination model (410). In this case, the first location information and the second location information can be input after being normalized to a range of 0 to 1.

[0118] More specifically, the control unit (120) can input the first position information (x1_min, x1_max, y1_min, y1_max) of the first area box (510) into the first position feature extraction model (411). In this case, the first position feature extraction model (411) can extract and output a first position feature vector from the first position information based on the first parameter (weight, bias, etc.) set in the first position feature extraction model (411).

[0119] Additionally, the control unit (120) may input the second position information (x2_min, x2_max, y2_min, y2_max) of the second area box (520) into the second position feature extraction model (412). In this case, the second position feature extraction model (412) may extract and output a second feature vector from the second position information based on the second parameter (weight, bias, etc.) set in the second position feature extraction model (412) (i.e., the same parameter as the first parameter set in the first position feature extraction model).

[0120] Meanwhile, the location comparison model (413) can calculate the location connection probability (P1) between the first location information and the second location information using the first location feature vector and the second location feature vector.

[0121] Specifically, the method for calculating the location connection probability (P1) can be expressed by the following mathematical formula.

[0122]

[0123] (σ: sigmoid function, ∝: weight, : The j-th component of the output of the output layer of the first position feature extraction model (411), : j-th component of the output of the output layer of the second position feature extraction model (412)

[0124] The position comparison model (413) may also consist of one or more layers of an artificial neural network, and in this case, a sigmoid function may be used as the activation function of the position comparison model (413), but is not limited thereto.

[0125] Additionally, the location comparison model (413) can constitute the L-th layer of the region box relationship determination model (410). And since the location comparison model (413) is connected to the first location feature extraction model (411) and the second location feature extraction model (412), the output layer of the first location feature extraction model (411) and the output layer of the second location feature extraction model (412) can be represented as the L-1-th layer.

[0126] Meanwhile, the first position feature vector and the second position feature vector can both be composed of n-dimensional vector values. Therefore, the first position feature vector and the second position feature vector can each have n vector values. For example, the first position feature vector is composed of {1-1 vector value, 1-2 vector value ..., 1-n vector value}, and the second position feature vector can also be composed of {2-1 vector value, 2-2 vector value ..., 2-n vector value}.

[0127] And through training, n weights corresponding to each of the n vector values ​​may be set in the position comparison model (413). For example, if the first node of the output layer of the first position feature extraction model (411) outputs a first-1 vector value and the first node of the output layer of the second position feature extraction model (412) outputs a second-1 vector value, a first weight corresponding to the first vector value (i.e., the first-1 vector value and the second-1 vector value) may be set. As another example, if the first node of the output layer of the second position feature extraction model (412) outputs a first-n vector value and the first node of the output layer of the second position feature extraction model (412) outputs a second-n vector value, an n weight corresponding to the n vector value (i.e., the first-n vector value and the second-n vector value) may be set.

[0128] And the position comparison model (413) can calculate the position connection probability between the first position information and the second position information by multiplying the 'difference between the j-th vector value of the first position feature vector and the j-th vector value of the second position feature vector' by a weight corresponding to the j-th. In this case, an absolute value may be applied to the difference between the j-th vector value of the first position feature vector and the j-th vector value of the second position feature vector.

[0129] For example, when j=1, the position comparison model (413) can calculate a first value by multiplying the difference between the first-1 vector value and the second-1 vector value by a first weight corresponding to the first vector value.

[0130] In addition, when j=2, the position comparison model (413) can calculate a second value by multiplying the difference between the first-2 vector value and the second-2 vector value by a second weight corresponding to the second vector value.

[0131] In addition, when j=n, the position comparison model (413) can calculate the nth value by multiplying the difference between the first-n vector value and the second-n vector value by the nth weight corresponding to the nth vector value.

[0132] In this case, the location comparison model (413) can calculate the connection probability (p1) between the first location information and the second location information by summing the first to n values.

[0133] Meanwhile, the training process of the position comparison model (413) described above may include a process of updating the first to nth weights so that the loss value becomes smaller.

[0134] Meanwhile, the term “probability” in this specification is not limited merely to percentiles, but may use various forms of values ​​that represent the possibility of connection between area boxes, such as scores.

[0135] Figure 7 is a diagram illustrating the operation of an image similarity judgment model.

[0136] The training method or operation of the area box relationship judgment model (410) described above in FIG. 6 can also be applied below within a non-contradictory scope.

[0137] The control unit (120) can obtain a first image (710) within a first area box (510) and a second image (720) within a second area box (520) placed on a specific screen (500).

[0138] Specifically, the control unit (120) can extract an image within a first area box (510) in a specific screen (500) to obtain a first image (710) within the first area box (100). Additionally, the control unit (120) can extract an image within a second area box (520) in a specific screen (500) to obtain a second image (720) within the second area box (520).

[0139] Meanwhile, before examining the operation of the image similarity judgment model (420), the structure and training method of the image similarity judgment model (420) will be explained first.

[0140] The image similarity judgment model (420) may be composed of a deep neural network (DNN) including multiple hidden layers.

[0141] Additionally, the image similarity judgment model (420) may include a first image feature extraction model (421), a second image feature extraction model (422), and an image comparison model (423).

[0142] Here, the first image feature extraction model (421) and the second image feature extraction model (422) may include a convolutional neural network (CNN) to extract feature vectors from an image. In this case, the first image feature extraction model (421) may have the first image (710) as an input value, and the second image feature extraction model (422) may have the second image (720) as an input value.

[0143] Meanwhile, the image comparison model (423) can calculate the image connection probability (p2) between the first image and the second image using the first image feature vector output by the first image feature extraction model (421) and the second image feature vector output by the second image feature extraction model (422).

[0144] Meanwhile, the first image may mean a 3D RGB matrix value representing the first image, and the second image may mean a 3D RGB matrix value representing the second image.

[0145] The training process of the image similarity judgment model (420) is described below.

[0146] The control unit (120) provides the first training image of the first training area box and the second training image of the second training area box as input data (input value) to the image similarity judgment model (420), and provides the connection status of the first training area box and the second training area box as labeling data (correct answer value) to the image similarity judgment model (420) so that the image similarity judgment model (420) can be trained. In this case, the first training image and the second training image are converted to the same size and can be normalized for training.

[0147] Here, the labeling data (correct answer value) can represent one of two classes. The two classes can be connected or unconnected between the two area boxes.

[0148] Meanwhile, when a first training image of a first training area box is provided, the first image feature extraction model (421) can extract a first training image feature vector based on the first parameters (weights, biases, etc.) set in the first image feature extraction model (421).

[0149] Additionally, when a second training image of a second training area box is provided, the second image feature extraction model (422) can extract a second training image feature vector based on the second parameters (weights, biases, etc.) set in the second image feature extraction model (422).

[0150] Meanwhile, the image comparison model (423) can calculate the image connection probability between the first image and the second image using the first training image feature vector and the second training image feature vector based on the third parameter (weights, biases, etc.) set in the image comparison model (423). The operation of the image comparison model (423) will be explained in detail later.

[0151] Meanwhile, when the image connection probability is calculated from the image comparison model (423), the control unit (120) can train the image similarity judgment model (420) based on the difference between the image connection probability calculated from the image comparison model (423) and the labeling data.

[0152] And the control unit (120) can optimize the first parameter of the first image feature extraction model (421), the second parameter of the second image feature extraction model (422), and the third parameter of the image comparison model (423) by repeating such a training process.

[0153] Meanwhile, the weights set in the first image feature extraction model (421) and the weights set in the second image feature extraction model (422) may be the same.

[0154] Specifically, the first image feature extraction model (421) and the second image feature extraction model (422) may have the same structure. For example, the first image feature extraction model (421) and the second image feature extraction model (422) may have the same number of layers, the same number of nodes per layer, the same connection structure between nodes, the same kernel structure, etc.

[0155] Additionally, when the initial parameters of the first image feature extraction model (421) and the second image feature extraction model (422) are identical to each other, the control unit (120) can update the first parameter of the first image feature extraction model (421) and the second parameter of the second image feature extraction model (422) using the same loss value obtained during the training process. Accordingly, the parameters between the first image feature extraction model (421) and the second image feature extraction model (422) can be identical to each other.

[0156] That is, the first image feature extraction model (421) and the second image feature extraction model (422) can have the same structure and the same parameters.

[0157] Meanwhile, once training is complete, the image similarity judgment model (420) can calculate the probability of image connections between images even for untrained input data.

[0158] Specifically, the control unit (120) can input the first image (710) of the first area box (510) and the second image (720) of the second area box (520) into the image similarity judgment model (420). In this case, the first image (710) and the second image (720) can be converted to the same size and normalized for training.

[0159] More specifically, the control unit (120) can input the first image (710) of the first area box (510) into the first image feature extraction model (421). In this case, the first image feature extraction model (421) can extract and output a first image feature vector from the first image (710) based on the first parameter (weight, bias, etc.) set in the first image feature extraction model (421).

[0160] Additionally, the control unit (120) may input the second image (720) of the second area box (520) into the second image feature extraction model (422). In this case, the second image feature extraction model (422) may extract and output a second image feature vector from the second image (720) based on the second parameter (weight, bias, etc.) set in the second image feature extraction model (422) (i.e., the same parameter as the first parameter set in the first image feature extraction model).

[0161] Meanwhile, the image comparison model (423) can calculate the image connection probability (P2) between the first image (710) and the first image (720) using the first image feature vector and the second image feature vector.

[0162] Specifically, the method for calculating the image connection probability (P2) can be expressed by the following mathematical formula.

[0163]

[0164] (σ: sigmoid function, ∝: weight, : The j-th component of the output of the output layer of the first image feature extraction model (421), : j-th component of the output of the output layer of the second image feature extraction model (422)

[0165] The image comparison model (423) may also consist of one or more layers of an artificial neural network, and in this case, a sigmoid function may be used as the activation function of the image comparison model (423), but is not limited thereto.

[0166] Additionally, the image comparison model (423) can constitute the L-th layer of the region box relationship determination model (410). And since the image comparison model (423) is connected to the first image feature extraction model (421) and the second image feature extraction model (422), the output layer of the first image feature extraction model (421) and the output layer of the second image feature extraction model (422) can be represented as the L-1-th layer. Additionally, the output layer of the first image feature extraction model (421) and the output layer of the second image feature extraction model (422) can be composed of fully connected layers.

[0167] Meanwhile, the first image feature vector and the second image feature vector are both composed of n-dimensional vector values, and the first image feature vector and the second image feature vector can each have n vector values.

[0168] And through training, the image comparison model (423) may have n weights set corresponding to each of the n vector values.

[0169] And the image comparison model (423) can calculate the image connection probability between the first image (710) and the second image (720) by multiplying the 'difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector' by a weight corresponding to the j-th. In this case, an absolute value may be applied to the difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector.

[0170] And the image comparison model (423) can calculate the connection probability (p2) between the first image (710) and the second image (720) by summing all the values ​​calculated by multiplying the 'difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector' by a weight corresponding to the j-th.

[0171] Meanwhile, the training process of the image comparison model (423) described above may include a process of updating n weights (i.e., the first to nth weights) corresponding to each of the n vector values ​​so that the loss value becomes smaller.

[0172] Next, referring again to FIG. 4, the operation of the probability distribution determination model (430) between strings is explained.

[0173] The probability distribution determination model (430) between strings may be a model that determines the possibility of connection between strings by considering semantic relationships or context between strings.

[0174] Accordingly, the control unit (120) can obtain a first string (“slowly”) within a first area box (510) and a second string (“starting”) within a second area box (520) placed on a specific screen (500). Here, the first string and the second string may mean text.

[0175] And the control unit (120) can provide the first string and the second string to the probability distribution determination model (430) between strings.

[0176] In this case, the probability distribution determination model (430) between strings can measure the probability distribution between strings using the first string and the second string.

[0177] This can be expressed by mathematical formula 3.

[0178]

[0179] The probability distribution determination model (430) between strings can calculate probability values ​​between word sequences by using a known language model such as BERT or GPT trained on a large amount of the corresponding language (e.g., Korean).

[0180] That is, when the first area box (510) is the reference area box and the second area box (520) is the comparison area box, the string probability distribution determination model (430) can calculate the probability that the second string (W2) included in the second area box (520) follows the first string (W1) included in the first area box (510) based on the language model. The calculated probability can be referred to as the string connection probability (P3).

[0181] Meanwhile, the probability distribution determination model between strings (430) can also be trained based on the difference between the string connection probability output from the probability distribution determination model between strings (430) and the labeling data (connection or non-connection between strings).

[0182] Meanwhile, the final judgment model (440) can receive a location connection probability (P1) from the area box relationship judgment model (410), an image connection probability (P2) from the image similarity judgment model (420), and a string connection probability (P3) from the string probability distribution judgment model (430).

[0183] And the final judgment model (440) can determine whether the first area box and the second area box are connected based on the location connection probability, the image connection probability and the string connection probability.

[0184] Meanwhile, before examining the operation of the final judgment model (440), the structure and training method of the final judgment model (440) will be explained first.

[0185] The final decision model (440) may be composed of a neural network including one or more layers. Additionally, the final decision model (440) may be composed of a fully connected layer having location connection probability, image connection probability, and string connection probability as input values.

[0186] Accordingly, during the training process, the training location connection probability, the training image connection probability, and the training string connection probability can be provided to the final judgment model (440) as input values.

[0187] Specifically, the control unit (120) provides the training position connection probability, the training image connection probability, and the training string connection probability as input data (input values) to the final judgment model (440), and provides the connection status of the first training area box and the second training area box as labeling data (correct answer value) to the final judgment model (440) so that the final judgment model (440) can be trained.

[0188] Here, the labeling data (correct answer value) can represent one of two classes. The two classes can be connected or unconnected between the two area boxes.

[0189] In this case, the final judgment model (440) can calculate the connection probability between the first area box and the second area box based on parameters (weights, biases, etc.) set in the final judgment model (440). The operation of the final judgment model (440) will be explained in detail later.

[0190] Meanwhile, when the connection probability between the first area box and the second area box is calculated from the final judgment model (440), the control unit (120) can train the final judgment model (440) based on the difference between the connection probability between the first area box and the second area box and the labeling data.

[0191] That is, the control unit (120) can calculate a loss value based on a loss function using the difference between the connection probability between the first area box and the second area box and the labeling data, and update the parameters set in the final judgment model (440) so that the calculated loss value becomes smaller.

[0192] And the control unit (120) can optimize the parameters set in the final judgment model (440) by repeating this training process.

[0193] Meanwhile, although individual training processes for the area box relationship judgment model (410), image similarity judgment model (420), probability distribution judgment model between strings (430), and final judgment model (440) have been described above, they are not limited thereto, and training for the area box relationship judgment model (410), image similarity judgment model (420), probability distribution judgment model between strings (430), and final judgment model (440) can be performed on the same input data and labeling data.

[0194] That is, the control unit (120) inputs the first training location information of the first training area box and the second training location information of the second training area box into the area box relationship judgment model (410), inputs the first training image of the first training area box and the second training image of the second training area box into the image similarity judgment model (420), and inputs the first training string of the first training area box and the second training string of the second training area box into the string probability distribution judgment model (430).

[0195] In this case, the final judgment model (440) can calculate the connection probability between the first training area box and the second training area box using the training location connection probability output from the area box relationship judgment model (410), the training image connection probability output from the image similarity judgment model (420), and the training string connection probability output from the string probability distribution judgment model (430).

[0196] Additionally, the control unit (120) can update the parameters of each model (410, 420, 430, 440) based on the difference between the connection probability between the first training area box and the second training area box and the labeling data (connected or unconnected area box) given as the correct answer value to the artificial intelligence model (400).

[0197] Meanwhile, it was explained that training is performed on the same input data and labeling data for the region box relationship judgment model (410), image similarity judgment model (420), probability distribution judgment model between strings (430), and final judgment model (440), but this is not limited thereto. The probability distribution judgment model between strings (430) using an existing language model may be excluded from the training target, and the region box relationship judgment model (410), image similarity judgment model (420), and final judgment model (440) may be trained.

[0198] Meanwhile, once training is complete, the artificial intelligence model (400) can calculate the probability of connections between area boxes even for untrained input data.

[0199] Specifically, the control unit (120) can extract a first area box (510) and a second area box (520) from a specific screen.

[0200] Then, the control unit (120) inputs the first location information of the first area box (510) and the second location information of the second area box (520) into the area box relationship judgment model (410), inputs the first image (710) of the first area box (510) and the second image (720) of the second area box (520) into the image similarity judgment model (420), and inputs the first string ("slowly") of the first area box (510) and the second string ("starting") of the second area box (520) into the string probability distribution judgment model (430).

[0201] In this case, the area box relationship judgment model (410) can output the location connection probability (P1), the image similarity judgment model (420) can output the image connection probability (P2), and the string probability distribution judgment model (430) can output the string connection probability (P3).

[0202] Meanwhile, the final judgment model (440) can calculate the connection probability (P) between the first area box and the second area box by applying corresponding weights to the location connection probability (P1), the image connection probability (P2), and the string connection probability (P3), respectively, and then summing them.

[0203] This can be expressed by the following mathematical formula.

[0204]

[0205] (W1: Weight applied to location connection probability, W2: Weight applied to image connection probability, W3: Weight applied to string connection probability)

[0206] Additionally, the final decision model (440) may also consist of one or more layers of an artificial neural network. In this case, the training process of the final decision model (440) may mean a process of updating the weights (W1) applied to the position connection probability, the weights (W2) applied to the image connection probability, and the weights (W3) applied to the string connection probability so that the loss value becomes smaller.

[0207] In this case, a logistic function may be used as the activation function of the final decision model (440), but is not limited thereto. When a logistic function is used as the activation function, the connection between the first area box and the second area box can be expressed by the following mathematical formula.

[0208]

[0209] Meanwhile, if it is determined that the first area box (510) and the second area box (520) are connected, the control unit (120) determines that the first string ("slowly") of the first area box (510) and the second string ("starting") of the second area box (520) are connected, and can connect the first string and the second string.

[0210] In this case, since the probability distribution between strings judgment model (430) has calculated the probability that the second string follows the first string, the control unit (120) can connect the second string after the first string.

[0211] And the control unit (120) can display the result of the first string and the second string being connected, transmit it externally, or store it in memory. For example, the control unit (120) can display the text “starting slowly”. To this end, the device for determining whether strings are connected (100) may include a display unit (not shown).

[0212] In addition, the meaning of the subtitle can be interpreted using the result of concatenating the first string and the second string. For example, the control unit (120) can interpret the meaning of the subtitle based on the combined word “starting slowly”.

[0213] Since humans comprehensively consider context, semantic relationships between strings, arrangement of strings, fonts, and colors to recognize separated strings on a screen as a single continuous character, they can easily recognize separated strings on a single screen as connected words or sentences. Furthermore, the present invention has the advantage of predicting the connectivity of strings arranged in an image by implementing this human intuitive judgment method into an artificial intelligence model.

[0214] In particular, in conventional OCR recognition, it was very difficult to recognize separated strings by connecting them to each other, and especially when strings were not arranged horizontally or vertically to each other as in FIG. 1a, they could not be recognized by connecting them to each other. However, the present invention has the advantage of improving the accuracy of determining whether strings can be connected by learning the pattern of how strings that can be connected as one are arranged on the screen.

[0215] In addition, conventional OCR recognition only recognizes strings as text and does not consider font type, color, etc. However, strings that can be connected as one mostly have the same font type, a specific color pattern, or a similar background. Therefore, the present invention has the advantage of improving the accuracy of determining whether a string is connected by comparing the string as an image and reflecting it in the final judgment.

[0216] In addition, unlike standardized documents, strings are dynamically arranged in various ways in media subtitle recognition and advertising screen recognition, so in such cases, manual processing by humans was required. Furthermore, when processing simply based on rules, it was difficult to respond flexibly to various string separations and string arrangement forms. However, according to the present invention, by training an artificial intelligence model using a large amount of actual screens (e.g., frame video with subtitles inserted in a broadcast program), there is an advantage of being able to accurately determine whether strings are connected without human intervention.

[0217] Accordingly, the present invention has the advantage of being able to provide various intelligent media services, such as video search, automatic highlight scene generation, and clip video generation.

[0218] On the other hand, a standard OCR reader recognizes text on the screen and outputs the OCR result; in this case, it outputs an area box (location information of the area box) and a character recognition result (text of the string).

[0219] Accordingly, the control unit (120) can recognize text on the screen using an OCR recognizer and perform a method (300) for determining whether strings are connected using the area box (location information of the area box) and the text of the string output by the OCR recognizer.

[0220] The following explains the method for selecting the reference area box and the comparison area box.

[0221] Referring again to FIG. 1a, three or more area boxes (a total of six) are extracted within the screen. In this case, the control unit (120) can select two of the three or more area boxes.

[0222] In this case, the control unit (120) can select one of the two area boxes as the reference area box and the other area box as the comparison area box.

[0223] Meanwhile, the device for determining whether a string is connected (100) can determine whether a comparison area box is connected after a reference box when calculating the probability of a connection between area boxes.

[0224] For example, when the reference area box is the first area box (510) and the comparison area box is the second area box (520), the probability distribution determination model between strings (430) calculates the probability that the second string follows the first string. Therefore, when the reference area box is the first area box (510) and the comparison area box is the second area box (520), whether the second area box (520) is connected after the first area box (510) (or the probability of being connected) can be calculated.

[0225] The following explains how to select candidates for comparison area boxes that are subject to determining connection status.

[0226] Figure 8 is a diagram illustrating a method for selecting a candidate group of comparison area boxes.

[0227] A specific screen (800) can be divided into a candidate group exclusion area (860) and a candidate group extraction area (870).

[0228] Here, the candidate exclusion area (860) may refer to the upper-left area of ​​the reference area box (810).

[0229] Here, the upper-left area of ​​the reference area box (810) may refer to the area located at the upper-left from the reference point of the reference area box (810).

[0230] Here, the reference point of the reference area box (810) may mean a point that has the minimum x-coordinate (x_min) and the minimum y-coordinate (y_min) among the points constituting the reference area box (810).

[0231] For example, the first vertex (811) of the reference area box (810) is the leftmost and uppermost vertex among the points constituting the reference area box (810), and may have a minimum x-coordinate (x_min) and a minimum y-coordinate (y_min). In this case, the first vertex (811) may be selected as the reference point of the reference area box (810).

[0232] In addition, the upper-left area of ​​the reference area box (810) may refer to an area formed by points where the x-coordinate is less than or equal to the x-coordinate (x_min) of the reference point and the y-coordinate is less than or equal to the y-coordinate (y_min) of the reference point. Also, the upper-left area of ​​the reference area box (810) may be selected as the candidate exclusion area (860).

[0233] Meanwhile, when selecting a comparison area box to determine whether to connect with the reference area box (810), the control unit (120) may exclude the area box located in the upper-left area of ​​the reference area box from the selection target.

[0234] Here, the area box located in the upper-left area (860) of the reference area box (810) may mean that the reference point of the area box (the point having the minimum x-coordinate (x_min) and the minimum y-coordinate (y_min) among the points constituting the area box) is located in the upper-left area (860).

[0235] On the other hand, when selecting a comparison area box to determine whether to connect with the reference area box (810), the control unit (120) may select an area box located outside the upper-left area (860) of the reference area box as the comparison area box.

[0236] That is, the area excluding the candidate group exclusion area (860) in a specific screen (800) becomes the candidate group extraction area (870), and the control unit (120) can select the area box located in the candidate group extraction area (870) as the comparison area box.

[0237] Here, the area box located outside the upper-left area (860) of the reference area box (i.e., located in the candidate extraction area (870)) may mean that the reference point of the area box (the point having the minimum x-coordinate (x_min) and the minimum y-coordinate (y_min) among the points constituting the area box) is located outside the upper-left area (860) of the reference area box.

[0238] In conclusion, the control unit (120) can extract a plurality of area boxes from a specific screen, select a first area box among the plurality of area boxes as a reference area box, and select a second area box located outside the upper-left area of ​​the reference area box among the plurality of area boxes as a comparison area box.

[0239] Meanwhile, when three or more area boxes (e.g., Area Box A, Area Box B, Area Box C) are extracted from a specific screen and two or more area boxes (Area Box B, Area Box C) are located outside the upper-left area of ​​the reference area box (Area Box A), the control unit (120) may select all of the two or more area boxes (e.g., Area Box B and Area Box C) as comparison area boxes. In this case, the control unit (120) may determine whether Area Box B is connected after Area Box A and whether Area Box C is connected after Area Box A.

[0240] Figure 9 is a diagram illustrating a method for determining whether strings are connected in various subtitle arrangements.

[0241] Referring to FIG. 9a, the control unit (120) can extract a plurality of area boxes (910, 920, 930, 940) from a specific screen (900).

[0242] First, the case where the first area box (910) is selected as the reference area box is explained.

[0243] With the first area box (910) selected as the reference area box, the control unit (120) can select a comparison area box. Specifically, the control unit (120) can select an area box located outside the upper-left area of ​​the first area box (910) among a plurality of area boxes (910, 920, 930, 940) as the comparison area box. Accordingly, the second area box (920), the third area box (930), and the fourth area box (940) can be selected as comparison area boxes.

[0244] In this case, the control unit (120) can determine whether a second area box (920) is connected after the first area box (910), determine whether a third area box (930) is connected after the first area box (910), and determine whether a fourth area box (940) is connected after the first area box (910).

[0245] Next, the case where the second area box (920) is selected as the reference area box is described.

[0246] With the second area box (920) selected as the reference area box, the control unit (120) can select a comparison area box. Specifically, the control unit (120) can select an area box located outside the upper-left area of ​​the second area box (920) among a plurality of area boxes (910, 920, 930, 940) as the comparison area box. Accordingly, the third area box (930) and the fourth area box (940) can be selected as comparison area boxes.

[0247] In this case, the control unit (120) can determine whether a third area box (930) is connected after the second area box (920) and whether a fourth area box (940) is connected after the second area box (920).

[0248] In the same principle, when the third area box (930) is selected as the reference area box, the second area box (920) and the fourth area box (940) can be selected as comparison area boxes.

[0249] In the same way, when the fourth area box (940) is selected as the reference area box, the comparison area box may not be selected.

[0250] Meanwhile, it is also possible to select a reference area box based on the positions of the area boxes. This will be explained with reference to FIG. 9a. In FIG. 9a, the explanation assumes that subtitles are inserted in such a way that the meaning is connected in the order of the first area box (910), the second area box (920), the third area box (930), and the fourth area box (940).

[0251] The control unit (120) can select the area box located at the top left among the plurality of area boxes (910, 920, 930, 940) as the reference area box.

[0252] Specifically, the control unit (120) can select a first reference point (911) among a plurality of reference points (911, 921, 931, 941) corresponding to a plurality of area boxes (910, 920, 930, 940) in which both the x-coordinate and y-coordinate have minimum values ​​compared to other reference points, and select a first area box (910) having the first reference point (911) as a reference area box.

[0253] Then, the control unit (120) can determine that the second area box (920) is connected after the first area box (910) through a comparison between the reference area box and the comparison area boxes.

[0254] Meanwhile, it is also possible to exclude area boxes that have already been selected as standard area boxes from being selected for comparison area boxes.

[0255] Specifically, the control unit (120) can select the second area box (920) as the reference area box. Then, with the second area box (920) selected as the reference area box, the control unit (120) can select the comparison area box. In this case, the control unit (120) can exclude the first area box (910), which was previously selected as the reference area box, from the selection of the comparison area box. Accordingly, with the second area box (920) selected as the reference area box, the control unit (120) can select the third area box (930) and the fourth area box (940) as comparison area boxes.

[0256] Referring to FIG. 9b, the control unit (120) can extract a plurality of area boxes (1010, 1020, 1030, 1040) from a specific screen (1000).

[0257] First, the case where the first area box (1010) is selected as the reference area box is explained.

[0258] With the first area box (1010) selected as the reference area box, the control unit (120) can select a comparison area box. Specifically, the control unit (120) can select an area box located outside the upper-left area of ​​the first area box (1010) among a plurality of area boxes (1010, 1020, 1030, 1040) as the comparison area box. Accordingly, the second area box (1020), the third area box (1030), and the fourth area box (1040) can be selected as comparison area boxes.

[0259] In this case, the control unit (120) can determine whether a second area box (1020) is connected after the first area box (1010), determine whether a third area box (1030) is connected after the first area box (1010), and determine whether a fourth area box (1040) is connected after the first area box (1010).

[0260] Next, the case where the second area box (1020) is selected as the reference area box is described.

[0261] With the second area box (1020) selected as the reference area box, the control unit (120) can select a comparison area box. Specifically, the control unit (120) can select an area box located outside the upper-left area of ​​the second area box (1020) among a plurality of area boxes (1010, 1020, 1030, 1040) as the comparison area box. Accordingly, the first area box (1010), the third area box (1030), and the fourth area box (1040) can be selected as comparison area boxes.

[0262] In this case, the control unit (120) can determine whether the first area box (1010) is connected after the second area box (1020), determine whether the third area box (1030) is connected after the second area box (1020), and determine whether the fourth area box (1040) is connected after the second area box (920).

[0263] In the same way, when the third area box (1030) is selected as the reference area box, the fourth area box (1040) can be selected as the comparison area box.

[0264] In the same way, when the fourth area box (1040) is selected as the reference area box, the comparison area box may not be selected.

[0265] Meanwhile, it is also possible to exclude area boxes confirmed to be connected to the next area box from the selection criteria for comparison area boxes.

[0266] Specifically, the second area box (1020) has been selected as the reference area box, and the first area box (1010) is already confirmed to be connected to the second area box (1020).

[0267] In this case, the control unit (120) obtains the first area box (1010), the third area box (1030), and the fourth area box (1040) located outside the upper-left area of ​​the second area box (920) among the plurality of area boxes (1010, 1020, 1030, 1040), and may exclude the first area box (1010) among them. Accordingly, the third area box (1030) and the fourth area box (1040) may be selected as comparison target boxes.

[0268] When subtitle creators separate and arrange sentences or words, they arrange them from left to right and from top to bottom to ensure readability. Therefore, a comparison area box that must be connected after a reference area box is unlikely to be located at the top-left corner of the reference area box. Accordingly, the present invention has the advantage of minimizing computational load by excluding area boxes located at the top-left corner of the reference area box from the candidate pool of comparison area boxes. In other words, area boxes placed at the top-left corner will have more comparison candidates, while area boxes placed at the bottom-right corner will have fewer comparison candidates.

[0269] The foregoing invention may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer may include a control unit. Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention. Explanation of the symbols

[0270] 110: Image acquisition unit 120: Control unit 130: Memory

Claims

Claim 1 A method for determining whether strings are connected, comprising: a step of obtaining first location information of a first area box placed on a specific screen, a first string within the first area box, and a first image within the first area box; a step of obtaining second location information of a second area box placed separately from the first area box on the specific screen, a second string within the second area box, and a second image within the second area box; a step of calculating, based on an artificial intelligence model, a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string; and a step of determining whether the first area box and the second area box are connected based on the location connection probability, the image connection probability, and the string connection probability; wherein the step of determining whether the first area box and the second area box are connected calculates the connection probability between the first area box and the second area box by applying corresponding weights to the location connection probability, the image connection probability, and the string connection probability, respectively, and then summing them. Claim 2 In claim 1, the first string and the second string are subtitles inserted into video content, and the method for determining whether strings are connected is a method for determining whether strings are separated and placed on the same screen of the video content. Claim 3 A method for determining whether strings are connected, wherein the step of calculating the probability of a location connection between the first location information and the second location information comprises: a step in which a first location feature extraction model outputs a first location feature vector obtained from the first location information based on a weight set in the first location feature extraction model; a step in which a second location feature extraction model outputs a second location feature vector obtained from the second location information based on a weight identical to the weight set in the first location feature extraction model; and a step in which a location comparison model calculates the probability of a location connection between the first location information and the second location information using the first location feature vector and the second location feature vector. Claim 4 In claim 3, the step of calculating the probability of a position connection between the first position information and the second position information using the first position feature vector and the second position feature vector comprises: the position comparison model calculating the probability of a position connection between the first position information and the second position information by multiplying the 'difference between the j-th vector value of the first position feature vector and the j-th vector value of the second position feature vector' by a weight corresponding to the j-th; a method for determining whether there is a connection between strings. Claim 5 A method for determining whether strings are connected, wherein the step of calculating the image connection probability between the first image and the second image comprises: a step in which a first image feature extraction model outputs a first image feature vector obtained from the first image based on weights set in the first image feature extraction model; a step in which a second image feature extraction model outputs a second image feature vector obtained from the second image based on weights identical to those set in the first image feature extraction model; and a step in which an image comparison model calculates the image connection probability between the first image and the second image using the first image feature vector and the second image feature vector. Claim 6 In claim 5, the step of calculating the image connection probability between the first image and the second image using the first image feature vector and the second image feature vector is a method for determining whether strings are connected, wherein the image comparison model calculates the positional connection probability between the first image and the second image by multiplying the 'difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector' by a weight corresponding to the j-th. Claim 7 In claim 1, the step of calculating the probability of string connection between the first string and the second string is a method for determining whether strings are connected, which calculates the probability that the second string follows the first string based on a language model. Claim 8 delete Claim 9 A method for determining whether strings are connected, comprising: a step of extracting a plurality of area boxes from a specific screen; a step of selecting a first area box among the plurality of area boxes as a reference area box; and, prior to the step of acquiring a first image, a step of selecting a second area box located outside the upper-left area of ​​the reference area box among the plurality of area boxes as a comparison area box. Claim 10 A device for determining whether strings are connected, comprising: a video acquisition unit for acquiring video content; and a control unit for acquiring first location information of a first area box placed on a specific screen of the video content, a first string within the first area box, and a first image within the first area box, and acquiring second location information of a second area box placed separately from the first area box on the specific screen, a second string within the second area box, and a second image within the second area box, and, based on an artificial intelligence model, calculating a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string, and determining whether the first area box and the second area box are connected based on the location connection probability, the image connection probability, and the string connection probability; wherein the control unit calculates the connection probability between the first area box and the second area box by applying corresponding weights to the location connection probability, the image connection probability, and the string connection probability, respectively, and summing them. Claim 11 In claim 10, the first string and the second string are subtitles inserted into video content, and a device for determining whether strings are connected to each other, which are placed separately on the same screen of the video content. Claim 12 In claim 10, the artificial intelligence model comprises: a first position feature extraction model that outputs a first position feature vector obtained from the first position information based on weights set in the first position feature extraction model; a second position feature extraction model that outputs a second position feature vector obtained from the second position information based on weights identical to those set in the first position feature extraction model; and a position comparison model that calculates a position connection probability between the first position information and the second position information using the first position feature vector and the second position feature vector; a device for determining whether strings are connected. Claim 13 In claim 12, the above-mentioned position comparison model is a device for determining whether a string is connected, which calculates a position connection probability between the first position information and the second position information by multiplying the 'difference between the j-th vector value of the first position feature vector and the j-th vector value of the second position feature vector' by a weight corresponding to the j-th. Claim 14 In claim 10, the artificial intelligence model comprises: a first image feature extraction model that outputs a first image feature vector obtained from the first image based on weights set in the first image feature extraction model; a second image feature extraction model that outputs a second image feature vector obtained from the second image based on weights identical to those set in the first image feature extraction model; and an image comparison model that calculates an image connection probability between the first image and the second image using the first image feature vector and the second image feature vector; a device for determining whether strings are connected. Claim 15 In claim 14, the image comparison model is a device for determining whether strings are connected, which calculates the positional connection probability between the first image and the second image by multiplying the 'difference between the j-th vector value of the first image feature vector and the j-th vector value of the second image feature vector' by a weight corresponding to the j-th. Claim 16 In claim 10, the artificial intelligence model comprises a string-to-string probability distribution determination model that calculates the probability that the second string follows the first string based on a language model, and a string-to-string connection determination device. Claim 17 delete Claim 18 In claim 10, the control unit selects the first area box among the plurality of area boxes as a reference area box, and selects the second area box located outside the upper-left area of ​​the reference area box among the plurality of area boxes as a comparison area box, a device for determining whether there is a connection between strings. Claim 19 A step of obtaining first location information of a first area box placed on a specific screen, a first string within the first area box, and a first image within the first area box; a step of obtaining second location information of a second area box placed on the specific screen separated from the first area box, a second string within the second area box, and a second image within the second area box; a step of calculating, based on an artificial intelligence model, a location connection probability between the first location information and the second location information, an image connection probability between the first image and the second image, and a string connection probability between the first string and the second string. A computer program stored on a medium for performing a method for determining whether a string is connected, comprising: a step of determining whether a first area box and a second area box are connected based on the position connection probability, the image connection probability, and the string connection probability; wherein the step of determining whether a first area box and a second area box are connected calculates the connection probability between the first area box and the second area box by applying corresponding weights to the position connection probability, the image connection probability, and the string connection probability, respectively, and then summing them.

Images