Apparatus and method for processing correlation betwenn query and visual media

Abstract

The present invention relates to a text and image association processing device and a text and image association processing method, wherein the text and image association processing device may include an association modeling unit that acquires an image token, a text token, and a dummy token, generates an attention weight based on the image token, the text token, and the dummy token, and acquires a result token using the attention weight; an importance processing unit that acquires a candidate weight vector based on the image token and an importance candidate group, and acquires an importance token based on the candidate weight vector; and an encoding unit that performs encoding for the result token and the importance token.

Description

Technology Field

[0001] This relates to a device for processing associations between text and images and a method for processing associations between text and images. Background Technology

[0002] Videos consist of a combination of a significant number of video frames. Therefore, searching for a specific scene or frame desired by a user within a video is by no means easy, despite the necessity. Accordingly, research and development of technologies capable of more effectively searching for specific scenes within a given video have been continuously conducted. Video temporal grounding is a process of detecting temporal moments that match language requests specified by the user; it is a technology that can significantly improve the user experience by helping to automatically find corresponding scenes when the user inputs a desired scene as a keyword.

[0003] In the past, for each scene or frame within a video, keywords related to that scene or frame were added in the form of metadata, and scene search was performed by comparing keywords entered by the user with keywords stored in the metadata format. However, this had the problem that the production period, labor input, and costs were inevitably excessive because corresponding keywords had to be selected and entered manually for each scene.

[0004] Recently, artificial intelligence technologies are being introduced into keyword analysis and scene search to address these issues. In particular, there is a growing trend of utilizing attention mechanisms, which possess significantly superior natural language processing performance, and Transformer models based on them. Typically, determining whether each video clip corresponds well to text requires properly aligning the representation spaces of multiple modalities; Transformers are gaining increasing attention because they can improve search performance by enabling the easier integration of such multimodal representations.

[0005] Nevertheless, many AI technologies currently on the market suffer from poor search accuracy because they fail to adequately reflect the correlation between natural language text and video frames. In other words, there was a frequent problem of outputting search results for scenes that did not match the user's request. Specifically, conventional technologies utilize attention mechanism-based transformers to extract integrated video and natural language expressions and are trained to predict correlations between the extracted expressions and the input natural language text within a single video; however, it was common practice not to consider the correlation between the natural language and video frames when extracting these integrated expressions. Consequently, discrepancies frequently occurred between the similarity between the extracted natural language and the video expression and the actual similarity, leading to misidentification of scenes. Therefore, there was a demand for improvements in the accuracy of scene search. Prior art literature

[0006] [Patent Document 0001] "Method for searching video scenes and scene search system using the same" (Republic of Korea Registered Patent Publication No. 10-0721409 (Applicant: Naver Corporation, Date of Publication: May 17, 2007)

[0007] [Non-patent Literature 0001] "Training data-efficient image transformers & distillation through attention" (Location: https: / / doi.org / 10.48550 / arXiv.2012.12877, Authors: Hugo Touvron et al., Publication Date: Dec. 23, 2020) The problem to be solved

[0008] The problem is to provide a text-image association processing device and method capable of searching for and detecting images corresponding to input text, such as natural language, and more accurately and appropriately determining and predicting the degree of correlation between the image and the natural language. means of solving the problem

[0009] To solve the aforementioned problem, a text-to-image association processing device and a text-to-image association processing method are provided.

[0010] The text and image association processing device may include an association modeling unit that acquires an image token, a text token, and a dummy token, generates attention weights based on the image token, the text token, and the dummy token, and acquires a result token using the attention weights; an importance processing unit that acquires a candidate weight vector based on the image token and an importance candidate group, and acquires an importance token based on the candidate weight vector; and an encoding unit that performs encoding for the result token and the importance token.

[0011] The above attention weight may be generated by using the image token as a query and a combination of the text token and the dummy token as a key.

[0012] The importance processing unit may be configured to obtain the average of the image tokens, obtain a corrected image token by subtracting the average from the image tokens, input the combined result of the corrected image tokens and the importance candidate group into an activation function, and obtain a combination result by combining the output of the activation function and a text relevance score.

[0013] The text relevance score above may be calculated through the sum of the respective values ​​corresponding to each text token that correspond to the same image token in the attention weights above.

[0014] The importance processing unit obtains the candidate weight vector by summing the values ​​corresponding to each image token that correspond to the same importance from the combination result, obtains the processed candidate weight vector by removing elements of low importance from the candidate weight vector, obtains the viewpoint description token by combining the processed candidate weight vector with the importance candidate group, and obtains the importance token by adding image context to the viewpoint description token, wherein the viewpoint description token may be for encoding whether the image clip corresponding to the image token has matching information.

[0015] The text and image association processing device may further include an image-text association processing unit that generates a guide map using an image token that matches the context, a projected text token, and a projected dummy token, and performs distillation on the attention weight based on the guide map.

[0016] The above image-text association processing unit may be configured to obtain a first moment prototype corresponding to an image and an image token corresponding to the context of the text and an image token corresponding to the context of the text based on the image token and the above image token, and to obtain a prototype token corresponding to the text and a dummy token, a text token projected on the text token, a dummy token projected on the dummy token, a second moment prototype corresponding to the text token, and a second non-moment prototype corresponding to the dummy token based on the text token and the above dummy token.

[0017] The above image-text association processing unit may generate the guide map based on an image token corresponding to the context of the text, the projected text token, and the projected dummy token, and obtain a distilled attention weight by performing distillation based on the guide map for the portion of the attention weight corresponding to the text token.

[0018] The above association modeling unit can also generate the attention weight by reflecting the distilled attention weight.

[0019] A method for processing associations between text and video may include the steps of: acquiring a video token, a text token, and a dummy token; generating attention weights based on the video token, the text token, and the dummy token; acquiring a result token using the attention weights; acquiring a candidate weight vector based on the video token and a group of importance candidates; acquiring an importance token based on the candidate weight vector; and performing encoding on the result token and the importance token to acquire a video or video clip for the text.

[0020] The above attention weight may be generated using the image token as a query and a combination of the text token and the dummy token as a key.

[0021] The step of obtaining a candidate weight vector based on the image token and the importance candidate group may include: obtaining the average of the image token; obtaining a corrected image token by subtracting the average from the image token; inputting the combined result of the corrected image token and the importance candidate group into an activation function; obtaining a combination result by combining the output of the activation function and a text relevance score; and obtaining the candidate weight vector by summing the values ​​corresponding to each image token that correspond to the same importance from the combination result.

[0022] The text relevance score above may be calculated through the sum of the respective values ​​corresponding to each text token that correspond to the same image token in the attention weights above.

[0023] Based on the above candidate weight vector, the step of obtaining an importance token may include: a step of obtaining a processed candidate weight vector by removing low-importance elements from the above candidate weight vector; a step of obtaining a viewpoint description token by combining the processed candidate weight vector with the above importance candidate group; and a step of obtaining the importance token by adding video context to the viewpoint description token, wherein the viewpoint description token may be for encoding whether a video clip corresponding to the video token has matching information.

[0024] The method for processing associations between text and images may further include the step of generating a guide map using context-matched image tokens, projected text tokens, and projected dummy tokens, and performing distillation on the attention weights based on the guide map.

[0025] The step of generating a guide map using an image token corresponding to the above context, a projected text token, and a projected dummy token, and performing distillation on the attention weight based on the guide map may also include the step of obtaining a first moment prototype corresponding to an image token and an image token corresponding to the context of the text and an image token corresponding to the context of the text based on the image token, and the step of obtaining a prototype token corresponding to the text and a dummy token, a projected text token for the text token, a projected dummy token for the dummy token, a second moment prototype corresponding to the text token, and a second non-moment prototype corresponding to the dummy token based on the text token and the dummy token.

[0026] The step of generating a guide map using an image token corresponding to the context, a projected text token, and a projected dummy token, and performing distillation on the attention weights based on the guide map may further include the step of generating the guide map based on an image token corresponding to the context of the text, the projected text token, and the projected dummy token, and the step of obtaining distilled attention weights by performing distillation based on the guide map for the portion of the attention weights corresponding to the text token.

[0027] The step of generating attention weights based on the image token, the text token, and the dummy token may include the step of generating attention weights by reflecting the distilled attention weights. Effects of the invention

[0028] According to the above-described device and method for processing association between text and images, one can obtain the advantage of appropriately searching for images corresponding to input text, such as natural language, and performing a more accurate judgment and prediction regarding the degree of association between the image and the natural language.

[0029] According to the above-described device and method for processing association between text and video, when searching for at least one scene within a video having multiple frames based on natural language and predicting its importance, the problem of the association between natural language text and video not being easily modeled can be resolved.

[0030] According to the above-described device and method for processing association between text and video, it is possible to implement a framework that is robust to various natural language inputs and capable of performing scene search within a video that aligns with user intent.

[0031] According to the above-described device and method for processing associations between text and video, users can more easily and quickly search for scenes, people, objects, expressions and / or actions, etc. that correspond to their requests within movies, television video content, and educational content. Brief explanation of the drawing

[0032] FIG. 1 is a block diagram of a text and image association processing device according to one embodiment. FIG. 2 is a block diagram illustrating an association modeling unit, an importance processing unit, and an encoding unit according to one embodiment. FIG. 3 is a diagram illustrating attention weights and text relevance scores (query correspondence scores) according to one embodiment. FIG. 4 is a block diagram illustrating an image text association processing unit according to one embodiment. FIG. 5 is a diagram illustrating the operation of generating a prototype token based on a moment token and a video token according to one embodiment. FIG. 6 is a diagram illustrating the operation of generating a prototype token based on a sentence token, a query token, and a dummy token according to one embodiment. FIG. 7 is a first figure for illustrating contrast learning according to one embodiment. FIG. 8 is a second figure illustrating contrast learning according to one embodiment. FIG. 9 is a diagram illustrating the distillation of a guide map according to one embodiment. FIG. 10 is a diagram showing the relationship between an importance processing unit, an encoding unit, and an association model processing unit according to one embodiment. FIG. 11 is a block diagram of an importance token generation unit according to one embodiment. FIG. 12 is a diagram illustrating the operation of an encoding unit and a decoding unit according to one embodiment. FIG. 13 is another block diagram of a text and image association processing device according to one embodiment. FIG. 14 is a flowchart of a method for processing association between text and images according to one embodiment. Specific details for implementing the invention

[0033] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0034] The terms used in this specification will be briefly explained, and the present invention will be described in detail. The terms used in this invention have been selected to be as widely used as possible, taking into account their functions in the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, terms used in this invention should be defined not merely by their names, but based on the meanings they possess and the content of the invention as a whole. Throughout the specification, when it is stated that a part is connected to another part, this may mean that the part and the other part are physically connected to each other, and / or electrically connected. Furthermore, when it is stated that a part includes another part, unless specifically stated otherwise, this does not mean that another part other than the one mentioned is excluded, but rather that additional parts may be included depending on the embodiment, etc. Terms such as “part,” “module,” and “unit” as used in the specification refer to a unit corresponding to all or part of at least one device, system, method, structure, and material, etc., and may perform a specified function or operation depending on the circumstances. The “part,” “module,” and “unit,” etc. may be implemented in software, in hardware such as an FPGA or ASIC, or as a combination of software and hardware, depending on the designer, manager, or user. However, the “part,” “module,” and “unit,” etc. are not limited solely to software or hardware. The “part,” “module,” and “unit,” etc. may be configured to reside in an addressable storage medium or configured to operate one or more processors.Accordingly, as an example, terms such as "part," "module," and "unit" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. According to an embodiment, a single "part," "module," "unit," etc. may be implemented as a single physical or logical configuration, or as a plurality of physical or logical configurations. Additionally, it is possible for a plurality of "parts," "modules," "units," etc., to be implemented as a single physical or logical configuration. Expressions such as the first to N (where N is a natural number greater than or equal to 1) are used to distinguish at least one part(s) from other part(s) for convenience of explanation, and may be arbitrarily selected and added to the components. For example, a component named as the first component may be otherwise named as the second component, and the second A component named as a component may also be otherwise named as the first component. Furthermore, expressions such as "first to Nth," unless otherwise specified, do not imply that these components are sequential. The term "and / or" may include a combination of multiple related items or any one of the multiple related items, but does not exclude a combination of two or more of the multiple related items. A singular expression may include a plural expression unless there is an obvious exception in the context. Additionally, an underscore (_) typically indicates that a character added after the underscore is a subscript of the character located before the underscore, and a caret (^) indicates that a character added after the caret is a superscript of the character located before the caret; however, depending on the situation, they may be used with a different meaning.

[0035] Hereinafter, an embodiment of a text and image association processing device and a system including the same will be described with reference to FIGS. 1 and 2.

[0036] FIG. 1 is a block diagram of a text and image association processing device according to one embodiment.

[0037] Referring to an embodiment illustrated in FIG. 1, a text and image association processing device (100, hereinafter referred to as the processing device) is configured to acquire at least one image (10) and at least one text (11), predict the importance between at least one part (10a, hereinafter referred to as the image clip) of the image (10) and the text (11), and acquire a detection result (99) based thereon.

[0038] At least one image (10) or image clip (10a) may include video, and depending on the embodiment, may include still images. An image clip (10a) is a relatively short video corresponding to a part of the overall image (10), and may be a set of consecutive video frames containing a specific scene or moment. An image clip (10a) may be predefined by a user or may be extracted from the image (10) by a processing device (100). In the latter case, the processing device (100) may acquire one or more image clips (10a) from at least one image (10) using a predetermined learning model.

[0039] The detection result (99) may include one or more video clips (10a) corresponding to the text (11) among at least one video clip(s) (10a) belonging to a predetermined video (10). Additionally, the detection result (99) may include a location within the video (10) for the corresponding video clip (10a) (e.g., a temporal location such as a specific timecode or a timecode of a certain range). Here, the location for the video clip (10a) may include a specific temporal location (e.g., a timecode of the point that is the center of the video clip) and a span associated therewith. A span may refer to the temporal range, i.e., the length of a segment, from the start to the end of a specific scene when that scene appears in the video.

[0040] According to an embodiment, the processing device (100) may predict the importance between each given image (10) itself and the text (11), and based on the predicted importance, obtain one or more images (10) corresponding to the text (11) among a plurality of images (10) as a detection result (99). In other words, the processing device (100) may be designed to select the optimal image (10) from a plurality of images (10). Below, the operation of the processing device (100) will be described based on an embodiment in which an image clip (10a) corresponding to the text (11) is obtained as a detection result (9), but this is not limited only to cases where an image clip (10a) corresponding to the text (11) is obtained. The configuration or operation described may also be applied to obtaining the optimal image (10) corresponding to the text (11) among a plurality of images (10), in the same way as described below or in a partially modified manner.

[0041] According to one embodiment, the processing device (100) may include, as illustrated in FIG. 1, an image encoding unit (101), a text encoding unit (102), a dummy encoding unit (103), an association modeling unit (110), an image-text association processing unit (130), an importance processing unit (150), an encoding unit (170), a decoding unit (180), and a result acquisition unit (190). At least two of these image encoding units to result acquisition units (101 to 190) may be configured to transmit commands or data, etc., unilaterally or bilaterally through a circuit, a cable, and / or a wireless communication network. Depending on the choice of a designer or user, at least one of these image encoding units to result acquisition units (101 to 190) may be omitted. According to an embodiment, at least two of the image encoding units to result acquisition units (101 to 190) may be physically separated or logically separated. In the case of physical separation, at least two of the image encoding unit to result acquisition unit (101 to 190) may be implemented using different physical processing devices (e.g., multiple physically separated central processing units (CPUs) or multiple physically separated server hardware devices). In addition, in the case of logical separation, at least two of the image encoding unit to result acquisition unit (101 to 190) may be implemented by a single physical processing device, for example, a single central processing unit.

[0042] The video encoding unit (101) receives at least one video (10) and can obtain one or more video tokens (20) corresponding to at least one video (10). According to one embodiment, the video encoding unit (101) may generate one or more video tokens (20) corresponding to each video clip (10a) by performing encoding on at least one video clip (10a) within the video (10) and extracting features of the video clip (10a). If the video (10) includes a plurality of video clips (10a), the video encoding unit (101) may obtain a plurality of video tokens (20) corresponding to each of the plurality of video clips (10a). That is, the video encoding unit (101) may generate the same number (i.e., L_v) of video tokens (20) for a predetermined number (e.g., L_v) of video clips. Here, the plurality of video clips (10a) may correspond to each of a plurality of scenes within the video (10). According to one embodiment, the image token (20) may be an embedding vector for a corresponding image (10) or a part thereof. If necessary, the image encoding unit (101) may perform the process of acquiring at least one image clip (10a) from the image (10) in advance. The generated image token (20) may be transmitted to at least one of the association modeling unit (110) and the importance processing unit (150) simultaneously or separately. According to an embodiment, when a plurality of images (10) are received, the image encoding unit (101) may generate a plurality of image tokens (20) corresponding to each of the plurality of images (10), and / or may generate a plurality of image tokens (20) corresponding to each image clip belonging to the plurality of images (10).

[0043] The text encoding unit (102) may receive at least one text (11) and obtain at least one text token (21) corresponding to at least one text (11). Specifically, the text encoding unit (102) may obtain one or more text tokens (21) corresponding to each of at least one text (11) by encoding the received at least one text (11) and extracting features of the text (11). Here, the text (11) may include a combination of one or more words. For example, the text (11) may be a sentence or phrase prepared by including words such as “Woman”, “wearing”, “Yankees”, , “beach”. According to an embodiment, the text token (21) may be generated for all words in the text (11) or for some words. For example, in the case of a sentence consisting of multiple words (e.g., L_q words), the text encoding unit (102) may output the same number (i.e., L_q) text tokens (21) or fewer text tokens (21). According to an embodiment, the text token (21) may include an embedding vector corresponding to the text (11). The generated text token (21) may be transmitted to at least one of the association modeling unit (110) and the importance processing unit (150) as needed. Additionally, the generated text token (21) may be transmitted to the dummy encoding unit (103).

[0044] The dummy encoding unit (103) outputs a dummy token (22), but may receive a text token (21) generated by the text encoding unit (102) and generate at least one dummy token (22) based thereon. Here, the dummy token (22) is a token obtained by modeling a meaning unrelated to or different from the text (20), for example, it may be generated by modeling a meaning opposite to that of the natural language (each word) that makes up the text (20). In other words, the dummy token (22) may be implemented based on an expression that excludes the context of the text token (21) based on the text token (21), and may be obtained by encoding a word(s) (e.g., "woman") included in the text (20) and a word(s) (e.g., "man") that is semantically different from the word(s) (e.g., "woman") included in the text (20). According to one embodiment, the dummy encoding unit (103) may provide the transmitted text token (21) as an input value to a predetermined learning layer (e.g., an attention layer) to generate a predetermined number (e.g., L_d) of learnable dummy tokens (22). Here, the dummy token (22) may be generated based on a context implemented by excluding the context of the text corresponding to the text token (21). Accordingly, the dummy token (22) may be instance-adaptive dummy tokens generated based on the text token (21). One or more generated dummy tokens (22) may be transmitted to the association modeling unit (110).

[0045] As described below, the dummy token (22) is connected and combined with the text token (21) and forms a part of the attention weight (30, which may be represented as an attention map). In this case, the dummy token (22) may be included in the attention weight (30) in inverse proportion to the information of the mutual relationship between the text (11) and the video clip (10a). The attention weight (30) formed by including the dummy token (22) is used in the cross-attention process, and these dummy tokens (22) allow the weights to be distributed more effectively in the cross-attention process and allow the degree of cross-modal attention for each video clip (10a) to be distinguished. Accordingly, the dummy token (22) enables the improvement of the interaction between features by utilizing the relationship between multiple different modalities (e.g., video (10) and text (11)).

[0046] FIG. 2 is a block diagram illustrating an association modeling unit, an importance processing unit, and an encoding unit according to one embodiment, and FIG. 3 is a diagram illustrating an attention weight and a text relevance score (query correspondence score) according to one embodiment.

[0047] Referring to FIGS. 2 and 3, the association modeling unit (110) can obtain an image token (20), a text token (21), and a dummy token (22), and obtain an attention weight (30) based thereon, and, if necessary, may further obtain a processing result (81, hereinafter referred to as a result token) based on the attention weight (30). The attention weight (30) may be provided to the image-text association processing unit (130), and the result token (81) may be transmitted to the encoding unit (170).

[0048] According to one embodiment, the association modeling unit (110) can obtain an attention weight (30) as illustrated in FIG. 3 by using an image token (20) as a query and a combination of a text token (21) and a dummy token (22) as a key. More specifically, for example, the association modeling unit (110) can obtain a query (Q=[p_Q(v_1), p_Q(v_2), ..., p_Q(v_Lv)]) by projecting the transmitted image token (20). Here, v_i means an image token (20), and Lv means the number of image tokens (20). p_Q(·) means a projection layer for the query. Additionally, the association modeling unit (110) interconnects and combines text tokens (21) and dummy tokens (22) as shown on the left side of FIG. 3, and projects the combination of text tokens (21) and dummy tokens (22) to obtain a corresponding key (K=[p_K(t_1), p_K(t_2), ..., p_K(t_Lq), p_K( D_1), p_K( D_2), ..., p_K( D_Ld)]) can be obtained. Here, t_i represents a text token (21), Lq represents the number of text tokens (21), ~D_i represents a dummy token (22) encoded based on the text token (21), and LD represents the number of dummy tokens (22). p_K(·) represents a projection layer for the key. Meanwhile, the text token (21) may also be used as a value. In this case, the association modeling unit (110) can obtain the corresponding value (V = [p_V(t_1), p_V(t_2), ..., p_V(t_Lq)]) by projecting the text token (21) as is (without combining it with the dummy token (22). Here, p_V(·) represents a projection layer for the value. In this case, the adaptive cross attention (ACA()) implemented using a dummy token (22) can be expressed as Equation 1 below for the i-th video clip.

[0049] [Mathematical Formula 1]

[0050]

[0051] In Equation 1, W_i,j represents the attention weight (30), and Q_i, K_k, and V_j represent the query, key, and value, respectively. exp() represents an exponential function and may be replaced with other activation functions, such as a sigmoid function, as needed. As described above, since text tokens (21) are used as values ​​(V_j), the number of values ​​(V_j) corresponds to the number of text tokens (21) (L_q). Also, since a combination of text tokens (21) and dummy tokens (22) is used as a key, the number of keys corresponds to the sum of the number of text tokens (21) and dummy tokens (22) (L_q+L_d). Meanwhile, in Equation 1, h represents the projected hidden dimension, and ⊙ represents the inner product. Referring further to the illustration in FIG. 3, the top row of all rows corresponds to a text token (21), the bottom row corresponds to a dummy token (22), and each column corresponds to an image token (20). According to Equation 1, a value (V_j) containing information about a text token (21) with a high value is summed based on the inner product between a query (Q_i) containing information about an image clip (10a) and a key (K_k) containing information about dummy tokens (22) that have opposite meanings to natural language words. In this case, the total sum of the attention values ​​containing information about the text token (21) is not fixed to 1 even if it is applied to a softmax function. Therefore, attention of different sizes can be applied depending on the degree of association, and through this, the contribution of the text (11) to the integrated representation between the image (10) and the text (11) can be increased by the degree of association with the image (10).

[0052] According to one embodiment, the association modeling unit (110) may generate an attention weight (30) by performing learning that further reflects the distilled result by the image-text association processing unit (130) as described below. Specifically, the image-text association processing unit (130) may obtain a guide map (60 in FIG. 9) by combining a projected image token (41 in FIG. 9), a projected text token (43 in FIG. 9), and a projected dummy token (44 in FIG. 9), distill the previously obtained attention weight (30) based on the guide map (60), and then generate a new attention weight (30) based on the distillation result. In other words, the association modeling unit (110) may generate an attention weight (30) by reflecting the distillation result based on the guide map (60).

[0053] According to one embodiment, the association modeling unit (110) may further calculate a text relevance score (31) for each video clip (e.g., each video token (20)) as illustrated in FIG. 3. The text relevance score (31) is provided such that the dummy token (22) is reflected in the attention weight (30) in inverse proportion to the saliency score, and may be defined, for example, as in Equation 2 below.

[0054] [Mathematical Formula 2]

[0055]

[0056] That is, the text relevance score (31) is obtained for each image token (20, i.e., v^-_1 to v^-_Lv) corresponding to each image token (20), and can be given as the sum of each value (i.e., t_1 to t_Lq) corresponding to each text token (21) that corresponds to the same image token (20) in the attention weight (30) prior to the inner product operation. In other words, the text relevance score (31) may be obtained as the sum of the values ​​corresponding to the remaining text tokens (21), excluding the value corresponding to the dummy token (22) from the column values ​​for a given image token (20). The text relevance score (31) may be learned for scoring the importance of each image clip (10a). This will be described later.

[0057] The image text association processing unit (130) can obtain associations between the image clip (10a) and each word within the text (11), and may be trained for this purpose. In the past, it was common practice to obtain a descriptor that appropriately represents the context within the text (11) and perform processing based on it. However, while each word within the text (11) usually represents a single context, there is a probability that it may have a different meaning depending on the situation. This means that each word may have a different association for each image frame or image clip (10a). Therefore, to solve this problem, the image text association processing unit (130) obtains an appropriate attention weight that represents the correlation between the image clip (10a) and the word.

[0058] FIG. 4 is a block diagram illustrating an image text association processing unit according to one embodiment. FIG. 5 is a diagram illustrating an operation of generating a prototype token based on moment tokens and video tokens according to one embodiment, and FIG. 6 is a diagram illustrating an operation of generating a prototype token based on sentence tokens, query tokens and dummy tokens according to one embodiment.

[0059] Specifically, the image-text association processing unit (130) may include an image-text descriptor generation unit (131), an alignment space learning unit (135), and a distillation unit (138), as illustrated in FIG. 4.

[0060] The image-text representation generation unit (131) can generate moment prototypes (20-2, 21-2, moment prototype) and non-moment prototypes (20-3, 21-3, non-moment prototype) corresponding to each modality domain, respectively, by using prototype tokens, for example, moment tokens (20-1) and sentence tokens (21-1), for each modality domain (i.e., image (10) and text (20)) to align the embeddings of two modalities, i.e., image (10) and text (20), as illustrated in FIGS. 5 and 6.

[0061] Specifically, an image-text descriptor generation unit (131) according to one embodiment may first obtain an image token (20) for a plurality of image clips (10a) as shown in FIG. 5, and obtain a learnable moment token (20-1) corresponding to the image token (20). Here, the moment token (20-1) may include a token representing an important point in time (e.g., an image clip (10a), a timecode, a span, etc.) within the image (10) or image clip (10a). Next, the image-text descriptor generation unit (131) obtains a result (40) by combining a moment token (20-1) and an image token corresponding to the context (41, i.e., v^1+_i, where i is a natural number between 1 and L_v), and a result (40-1) by combining a prototype token (20-1) and an image token not corresponding to the context (41-1, i.e., v^1-_i, where i is a natural number between 1 and L_v), and inputs each to a moment encoder (132). Accordingly, a first moment prototype (20-2) and a first non-moment prototype (20-3) corresponding to each are obtained. Here, the first moment prototype (20-2) is output in correspondence with the image token (41) corresponding to the context, and the first non-moment prototype (20-3) may be output together with the image token (41-1) not corresponding to the context. These can be expressed as Equations 3 and 4 below, respectively.

[0062] [Mathematical Formula 3]

[0063]

[0064] [Mathematical Formula 4]

[0065]

[0066] In mathematical equations 3 and 4, ^M^b+ is the first moment prototype (20-2), and ^M^b- is the first non-moment prototype (20-3). ; signifies concatenation, and hat (^, hat) signifies the output projected through self-attention.

[0067] Additionally, an image-text descriptor generation unit (131) according to one embodiment may first acquire a plurality of text tokens (21) and dummy tokens (22) as shown in FIG. 6, and acquire a learnable sentence token (21-1) corresponding to the text tokens (21) and dummy tokens (22). The sentence token (21-1) may be a token representing an important word or phrase within the text (1). In the case of natural language, since there is no part that does not correspond to the context, the image-text descriptor generation unit (131) may be configured to use a dummy token (22) to replace this part. The image-text descriptor generation unit (131) obtains a result (42) of combining a sentence token (21-1) and a text token (21), and a result (42-1) of combining a sentence token (21-1) and a dummy token (22), and then inputs these into a moment encoder (132) to obtain a second moment prototype (21-2) and a second non-moment prototype (21-3). Here, the second moment prototype (21-2) is output together with a projected text token (43), and the second non-moment prototype (21-3) is output together with a projected dummy token (44). These can be expressed by the following mathematical formulas 5 and 6, respectively.

[0068] [Mathematical Formula 5]

[0069]

[0070] [Mathematical Formula 6]

[0071]

[0072] In mathematical formulas 5 and 6, ^S^b+ represents the second moment prototype (21-2), and ^S^b- represents the second non-moment prototype (21-3).

[0073] FIG. 7 is a first figure for illustrating contrast learning according to one embodiment, and FIG. 8 is a second figure for illustrating contrast learning according to one embodiment.

[0074] The alignment space learning unit (135) can be trained to interconnect related parts through contrastive learning.

[0075] Specifically, referring to the illustration in FIG. 7, the first moment prototype (20-2, e.g., ^M^1+, ^M^2+, ^M^3+, ���) and the corresponding second moment prototype (21-2, e.g., ^S^1+, ^S^2+, ^S^3+, ���) are treated as positive pairs so as to be mutually related, and the rest are treated as negative pairs. That is, among the plurality of first moment prototypes (20-2), a predetermined first moment prototype (e.g., ^M^1+) is defined as a positive pair with the corresponding second moment prototype (e.g., ^S^1+) among the plurality of second moment prototypes (21-2), but is defined as a negative pair with the rest. Additionally, the portion corresponding to the first non-moment prototype (20-3) is treated as a negative pair in relation to the second moment prototype (21-2). In other words, among the tokens (20) for the video clip (10a), the video token (20) corresponding to the projected text token (43) is treated as having high relevance to the text token (43) (i.e., as a positive pair), and otherwise is treated as having low relevance to the text token (43) (i.e., as a negative pair). Here, the text token (43) may be a projected text token whose corresponding text is related to the video clip (10a).

[0076] Meanwhile, referring to the illustration in FIG. 8, the first moment prototype (20-2, e.g., ^M^1+, ^M^2+, ^M^3+, 쪋) is treated as a negative pair so that its relevance is evaluated as low with respect to the second non-moment prototype (21-3, i.e., a prototype derived from a dummy token (22), e.g., ^S^1-, ^S^1-, ^S^1-, 쪋) corresponding to the second moment prototype (21-2, e.g., ^S^1+, ^S^1+, 쪋), and the remaining part is treated as a positive pair. In other words, among a plurality of first moment prototypes (20-2), a specific first moment prototype (e.g., ^M^1+) is defined as a negative pair with a second non-moment prototype (e.g., ^S^1-) related to the first moment prototype, and as a positive pair with the rest. Here, the related second non-moment prototype (^S^1-) may correspond to a second moment prototype (e.g., ^S^1+) corresponding to a specific first moment prototype (e.g., ^M^1+) among a plurality of second moment prototypes (21-2). The reason the remaining parts are treated as positive pairs is that it is difficult to determine an obvious antonym for a given word or sentence. In particular, since each word can have various antonyms in relation to a sentence composed of multiple words, the remaining words (i.e., the complement of a specific word) excluding the word with a specific meaning are assigned to a dummy token (22) for processing. On the other hand, the part corresponding to the first non-moment prototype (20-3) is treated as a positive pair in relation to the second non-moment prototype (21-3). Thus, an image token (20) that is highly relevant to a given projected text token (43) is treated as a negative pair because it is less relevant to the projected dummy token (44), and an image token (20) that is less relevant to a given text token (21) is treated as highly relevant to the dummy token (22) (i.e., as a positive pair).

[0077] In other words, the image token (20) is processed as highly relevant to the corresponding text token (21), and as highly relevant to the dummy token (22) in the case of non-relevance.

[0078] According to one embodiment, the alignment space learning unit (135) may perform contrast learning based on a loss function such as the following mathematical formula 7.

[0079] [Mathematical Formula 7]

[0080]

[0081] In Equation 7, L_align is a loss function for spatial alignment of image tokens (20) and text tokens (21) at a high level, L^+_align is intended to emphasize approximation to the first moment prototype (20-2, ^M^b+) and the second moment prototype (21-2, ^S^b+), and L^-_align is related to the first non-moment prototype (20-3) and the second non-moment prototype (21-3), and is intended to assign image clips (10a) with weak relationships to dummy tokens (22). In this case, the loss function L^+_align can be defined, for example, as in Equation 8 below.

[0082] [Mathematical Formula 8]

[0083]

[0084] In addition, L^-_align can also be defined by the following mathematical formula 9.

[0085] [Mathematical Formula 9]

[0086]

[0087] In Equations 8 and 9, B is a set of indices within the total batch. τ is a scaling parameter for contrastive learning. In this case, Equation 8 strengthens the proximity of the relationship between the first moment prototype (20-2) and the second moment prototype (21-2), and Equation 9 strengthens the semantic representation of the dummy token (22).

[0088] The distillation unit (138) can infer the correlation between the image (10, 10a) and the text (11) at a more granular level and distill the inferred correlation into the attention weight (30) used in the cross-attention layer of the association modeling unit (110). That is, the distillation unit (138) is configured to obtain the similarity of the projected image token (41), projected text token (43), and projected dummy token (44) that correspond to the context, and to reflect this in the attention weight (30).

[0089] FIG. 9 is a diagram illustrating the distillation of a guide map according to one embodiment.

[0090] According to one embodiment, the distillation unit (138) may first combine a projected text token (43) and a projected dummy token (44), as shown in FIG. 9, and obtain at least one guide map (60) by taking an inner product with a projected image token (41) corresponding to the context. The process may also be expressed as shown in Equation 10 below.

[0091] [Mathematical Formula 10]

[0092]

[0093] In mathematical formula 10, ^v^+_i refers to the token that corresponds to the context of the i-th video clip (10a) among the projected tokens (41) for the video clip (10a). This guide map (60) can represent the inference result regarding the correlation between the video clip (10a) and the text (20). When the guide map (60) is obtained, the distillation unit (138) can distill the previously obtained attention weight (30) based on the guide map (60) to obtain the distilled attention weight (33). According to one embodiment, the partial attention weight (32), consisting only of the remaining part (i.e., the part corresponding to the text token (21)) excluding the part corresponding to the dummy token (22) from the attention weight (30), may be the subject of distillation. In this case, the distillation unit (138) first obtains partial attention weights (32) by removing the part corresponding to the dummy token (22) from the attention weight (30), and distills the obtained partial attention weights (32) using the guide map (60). In this case, the distillation unit (138) may, in detail, extract expressions corresponding to each text (20) and image (10) in the aligned space, measure the similarity between mutually corresponding expressions, and distill the similarity to the model. Accordingly, the finally distilled attention weight (33) is obtained. The distilled attention weight (33) contains information about how important the image clip (10a) is. The distilled attention weight (33) is reflected in the attention weight (30) of the association modeling unit (110). In other words, as described above, the association modeling unit (110) generates the attention weight (30) by reflecting the distillation result.

[0094] Meanwhile, distillation loss can be defined as shown in the following mathematical formula 11.

[0095] [Mathematical Formula 11]

[0096]

[0097] FIG. 10 is a diagram showing the relationship between an importance processing unit, an encoding unit, and an association model processing unit according to one embodiment.

[0098] The importance processing unit (150) can acquire an importance token (80) as illustrated in FIG. 10. According to one embodiment, the importance processing unit (150) may include an importance token generation unit (151) that receives an image token (20) and acquires an importance token (80-1) based thereon, and a projection unit (159) that performs a projection on the acquired importance token (80-1) to acquire a projected importance token (80) and transmits it to an encoding unit (170).

[0099] FIG. 11 is a block diagram of an importance token generation unit according to one embodiment.

[0100] Referring to FIG. 11, an importance token generation unit (151) according to one embodiment may include a token combination unit (152), a candidate weight vector processing unit (155), an importance candidate combination unit (156), and an image context combination unit (157).

[0101] The token combination unit (152) first obtains a video token (21), calculates the average of the video token (21) through the average calculation unit (153) to obtain a video context (153-1), and obtains a corrected video token (24) by subtracting the video context (153-1) from the video token (21). The video context (153-1) is used to prevent abnormal values ​​from occurring when calculating similarity to obtain the importance score result (171 in FIG. 12) by the encoding unit (170) described later. Specifically, the video context (153-1) is subtracted in the token combination unit (152) and is reflected in the viewpoint description token (74) in the video context combination unit (157) as described later. The corrected video token (24) shows how much the token for a specific video clip (10a) differs based on the context (153-1). The corrected image token (24) corresponds to setting the average of the domain in which all image clips (10a) are represented to 0, and as described below, a fixed number of importance candidates (P_1, P_2, P_lv) can represent all image clips (10a).

[0102] Next, a transpose (24-1) may be performed on the corrected image token (24). The transposed image token (24) is inner productd with the importance candidate group (70) and then input into the activation function (154). Here, the importance candidate group (70) may include a plurality of importance candidates (P_1, P_2, P_lv). The importance candidate group (70) may be predefined by a user or designer and is stored in a separately provided storage unit and provided to the token combination unit (152).

[0103] Meanwhile, the token combination unit (152) can obtain the text relevance score (31) of FIG. 3, perform transpose (31-1) on the text relevance score (31), and then combine (e.g., multiply) the transpose (31-1) processed text relevance score (31) with the output result of the activation function (154) to obtain a combination result (71). Here, the text relevance score (31) is combined to reflect which token among the given image tokens (21) is more relevant. The combination result (71) may have the form of a matrix, and the number of rows of the combination result (71) may correspond to the number of image clips (10a) (L_v), and the number of columns may correspond to the number of importance candidates (L_p) of the importance candidate group (70).

[0104] The candidate weight vector processing unit (155) receives a combination result (71) from the token combination unit (152) and can obtain a candidate weight vector (72) from the combination result (71). The candidate weight vector (72) may be obtained, for example, by summing the value(s) corresponding to the same importance candidates (P_1, P_2, P_lv) (i.e., each value belonging to the same column within the combination result (71)). Subsequently, the candidate weight vector processing unit (155) can obtain a processed candidate weight vector (73) by selecting all or some of the elements of the candidate weight vector (72). In this case, the processed candidate weight vector (73) may be obtained by selecting only the elements of the candidate weight vector (72) whose values ​​are greater than a certain value, or by selecting from the element with the largest value up to the element with the Kth largest value (i.e., the top-K elements). The processed candidate weight vector (73) is passed to the importance candidate combination unit (156).

[0105] The importance candidate combination unit (156) receives the processed candidate weight vector (73) and can obtain a corresponding result (74, hereinafter referred to as the time-view explanation token) by performing a dot product of the processed candidate weight vector (73) and the importance candidate group (70). The time-view explanation token (74) is a token for encoding whether each video clip (10a) has appropriately matching information. The time-view explanation token (74) is transmitted to the video context combination unit (157).

[0106] The image context combining unit (157) obtains an importance token (80-1) by combining the image context (153-1) described above with the viewpoint description token (74). That is, the image context combining unit (157) is configured to restore the previously deducted image context (153-1).

[0107] The above-described process can be expressed as shown in the following mathematical formulas 12 and 13.

[0108] [Mathematical Formula 12]

[0109]

[0110] In Equation 12, C_j is a candidate weight vector (72), and V_ctx is a context token for the image context (153-1). P_j and P_k represent importance candidates (P_1, P_2, P_lv). a_i is a text relevance score (31).

[0111] [Mathematical Formula 13]

[0112]

[0113] In mathematical formula 13, T represents the importance token (80-1), and C_(k) represents the k-th largest element within the candidate weight vector (72). 1 is the indicator.

[0114] The importance token (80-1) can be transmitted to the projection unit (159) as shown in FIG. 10 and output to correspond to the output space of the adaptive cross-attention. Accordingly, the projected importance token (80) is obtained. The projection unit (159) may be provided by sharing the same parameters as the projection layer of the adaptive cross-attention of the association modeling unit (110). Such an importance token (80) is encoded similarly to the image token (20), and the problem of modality gap between multiple tokens (20, 21) can be eliminated.

[0115] FIG. 12 is a diagram illustrating the operation of an encoding unit and a decoding unit according to one embodiment.

[0116] Referring to FIGS. 1, 10 and 12, the encoding unit (170) may obtain an importance token (80, which may be an importance token prior to projection (80-1) according to the embodiment) from the importance processing unit (150), obtain a result token (81) from the association modeling unit (110), and perform encoding on each of these to obtain corresponding encoding results (82, 83). According to one embodiment, the encoding unit (170) may include a transformer encoder.

[0117] The encoding results (82, 83) are combined, and accordingly, a result with a score for importance can be output. In this case, important parts and / or relatively important video clips (10a) in the video (10) or video clip (10a) are scored to have relatively high values, and unimportant parts are scored to have relatively low values. Here, important parts or important video clips (10a) refer to parts of the video or clips that correspond to the text, so it is possible to identify important parts or corresponding video clips (10a) within the video (10) and to determine their locations.

[0118] The encoding result (83) corresponding to the result token (81) can be transmitted to the decoding unit (180). The decoding unit (180) can generate a prediction result (91) using the encoding result (83) and the decoder query (90). Depending on the embodiment, the decoding unit (180) may be implemented using a transformer decoder. Hungarian matching may be performed on the prediction result (91) as needed. Hungarian matching can optimize the matching between the prediction result (91) and the ground truth. The processing result (91) of the decoding unit (180) is transmitted to the result acquisition unit (190).

[0119] The result acquisition unit (190) acquires a time point corresponding to the prediction result (91). Here, the time point corresponding to the prediction result (91) may include a time code for a part of the image (10) that best matches the input text (11), for example, an image clip (10a). In other words, the result acquisition unit (190) is able to derive an image clip (10a) that matches the text (11).

[0120] Meanwhile, at least one loss function may be used for scene detection and the temporal location detection of the scene of the processing device (100) described above. For example, a cross-entropy loss function may be used for the detection of the viewpoint of the scene, and a margin ranking loss function, a rank contrastive loss function, and / or an entropy loss function may be used for the scoring result of importance. The text relevance score (31) may also be trained using a predetermined loss function. In this case, the overall loss function may be expressed as shown in Equation 14 below.

[0121] [Mathematical Formula 14]

[0122]

[0123] In Equation 14, L_obj represents the total loss function, L_mr is the loss function for detecting viewpoints for a scene, L_hl is the loss function for scoring results of importance, and L_attn is the loss function for text relevance scores (31). L_bce is the binary cross-entropy loss function, which can be given as Equation 15 below. This is used to distinguish between moments and non-moments within each instance.

[0124] [Mathematical Formula 15]

[0125]

[0126] Additionally, L_ortho is intended to enhance orthogonality between embeddings and is provided so that encoded dummy tokens (22) do not play the same role. L_ortho can be given, for example, as in Equation 16 below.

[0127] [Mathematical Formula 16]

[0128]

[0129] L_align is a loss function for spatial alignment, which can be given as Equation 7 above, and L_distill is a distillation loss function, which can be given as Equation 11 above.

[0130] FIG. 13 is another block diagram of a text and image association processing device according to one embodiment.

[0131] Referring to FIG. 13, the text and image association processing device (200, hereinafter referred to as the processing device) in one embodiment may include an input unit (201) that receives at least one of an image (91) and a text (92), a storage unit (203) that stores the image (91), the text (92), or the processing result of the processor (205) (e.g., a scene location search result (171) or a corresponding image (191)), a processor (205) that receives the image (91) and the text (92), and acquires an image corresponding to a given text (92) among a plurality of images (91) and / or acquires a part corresponding to a given text (92) within the given image (91) (e.g., at least one image frame, etc.), and an output unit (207) that outputs data stored in the storage unit (203) or the processing result of the processor (205) to the outside. Here, at least two of the input unit (201), storage unit (203), processor (205), and output unit (207) may be connected via a circuit, cable, or wireless communication network. Depending on the situation, at least one of the input unit (201), storage unit (203), and output unit (207) may be omitted.

[0132] The input unit (201) may receive one or more images (91) and may receive one or more texts (92) simultaneously or sequentially. At least one of the received images (91) and texts (92) may be transferred to a storage unit (203) or to a processor (205). The input unit (201) may include, for example, at least one of a keyboard, mouse, tablet, touch screen, touch pad, track ball, trackpad, scanner device, image capturing module, motion detection sensor, light receiving sensor, pressure sensor, proximity sensor, microphone, data input / output terminal, and a communication module (e.g., a LAN card, a short-range communication module, or a mobile communication module).

[0133] The storage unit (203) may temporarily or non-temporarily store one or more images (91) and / or one or more texts (92) obtained through the input unit (201), etc., and may provide one or more images (91) and / or one or more texts (92) requested according to the call of the processor (205) or a predefined setting, etc. Additionally, the storage unit (203) may store at least one program (which may be referred to as an app, application, or software) provided to be executed by the processor (205) and to be performed by the processor (205). Here, the program stored in the storage unit (205) may be one that was directly written or modified by a designer and then stored in the storage unit (205), one that was received and stored from another physical recording medium (an external memory device or a compact disc (CD), etc.), and / or one that was obtained or updated through an electronic software distribution network accessible via a wired or wireless communication network. The storage unit (205) may include, for example, at least one of a register, a cache memory, a main memory, and an auxiliary memory. These may be implemented using, for example, a magnetic disk or a semiconductor device.

[0134] In one embodiment, the processor (205) may be configured to perform all or part of the operations or functions of the previously described image encoding unit (101), text encoding unit (102), dummy encoding unit (103), association modeling unit (110), image text association processing unit (130), importance processing unit (150), encoding unit (170), decoding unit (180), and result acquisition unit (190) in the same order as described above or in a different order. Depending on the embodiment, the processor (205) may be implemented using a single physical processing unit, e.g., a single central processing unit, or may be implemented using two or more separate physical processing units, e.g., a central processing unit and a graphics processing unit (GPU). In the former case, the previously described image encoding unit to result acquisition unit (101 to 190) may all be implemented by a central processing unit or a graphics processing unit. In the latter case, some of the aforementioned image encoding unit or result acquisition unit (101 to 190) may be implemented by a central processing unit, and the remaining parts may be implemented by a graphics processing unit.

[0135] When a plurality of images (10) are input, the processor (205) may select at least one image (10) or a part of an image (10) (i.e., an image frame, image clip, cut, or scene, etc.) that corresponds to a given text (11) among the plurality of images (10). When a single image (10) is input, the processor (205) may select a part of a single image (10) that corresponds to a given text (11). Additionally, the processor (205) may obtain information about the location of a part of an image (10) that corresponds to a given text (11).

[0136] The processor (205) may execute a program stored in the storage unit (230) to perform operations corresponding to the video encoding unit or result acquisition unit (101 to 190) described above.

[0137] The processor (205), depending on the embodiment, may include a central processing unit, a graphics processor, a microcontroller unit (MCU), an application processor (AP), an electronic control unit (ECU) and / or a microcomputer (Micom), and may also include at least one other electronic device capable of performing computation and control. These devices may be implemented using one or more semiconductor chips, circuits, or related components alone or in combination.

[0138] The output unit (207) can visually and / or audibly output an image (10) or a part of an image (10) that is determined by the processor (205) to correspond to the text (11). For example, the output unit (207) may output at least one image (10) selected from a plurality of images (10), and / or output a part of at least one image (10) (i.e., an image frame, an image clip, a cut, or a scene, etc.). Additionally, the output unit (207) may indicate the location of a part of an image (10) that corresponds to the given text (11), and may further output information about the image (10) that is determined to correspond to the text (11) among the plurality of images (10). The output unit (207) may include, but is not limited to, a display, a printer device, a speaker device, an image output terminal, a data input / output terminal and / or a communication module.

[0139] The processing device (100, 200) described above may be implemented using a device specifically designed to perform certain operations and controls, such as modeling the association between text and images as described above, learning the correlation, measuring importance, and selecting or acquiring at least one image corresponding to the text based on this, or generating a dummy token for this purpose, or it may be implemented using at least one information processing device alone or in combination. Here, at least one information processing device may include, for example, a desktop computer, a laptop computer, a server hardware device, a smartphone, a tablet PC, a smart watch, a smart band, a head-mounted display (HMD) device, a portable game console, a navigation device, a video recording device (such as a camcorder or action cam), a remote control device (such as a remote control), a digital television, a set-top box, a digital media player device, a media streaming device, a sound playback device (such as an artificial intelligence speaker), a home appliance, a manned or unmanned mobile vehicle (for example, a vehicle, a mobile robot, a wireless model vehicle or a robot vacuum cleaner, etc.), a manned or unmanned aerial vehicle (for example, an aircraft, a helicopter, a drone, a model airplane or a model helicopter, etc.), a household / industrial / military robot, an industrial / military machine device or a medical device, but is not limited thereto. Depending on the situation or conditions, a designer or user, etc., may consider and adopt at least one of the various devices capable of computational processing and control of information, in addition to the information processing device described above, as the processing device (100, 200) described above.

[0140] The processing device (100, 200) described above can be used in various fields to achieve various purposes. For example, the processing device (100, 200) enables the producer of a video to quickly search for desired scenes and perform video editing more conveniently, thereby reducing the production cost and time of the video. In addition, the processing device (100, 200) enables viewers of movies or television content to search for scenes in which specific actors, specific actions, and / or specific dialogue appear or are expressed according to their intentions, and enables trainees to easily and quickly search for parts corresponding to specific lectures or topics, or parts explaining desired content, within educational video content. Furthermore, the processing device (100, 200) described above enables advertisers, advertising planners, or marketing experts to automatically and accurately identify scenes in which a product to be marketed appears within videos such as advertisements or movies, thereby enabling them to efficiently evaluate marketing effects and reduce related costs. Furthermore, the operation of the processing unit (100, 200) may also be used to search for specific events, etc., in security camera footage. In addition, the processing unit (100, 200) has the advantage of being widely used in various fields related to video.

[0141] An embodiment of a method for processing association between text and images will be described below with reference to FIG. 14.

[0142] FIG. 14 is a flowchart of a method for processing association between text and images according to one embodiment.

[0143] Referring to FIG. 14, at least one of an image token, a text token, and a dummy token may first be obtained (400). Here, the image token may be obtained by performing encoding on a given image or each clip of the image, and the text token may be obtained by performing encoding on a given text (e.g., each word in the text). The dummy token may be generated based on the obtained text token, but may be obtained by encapsulating a meaning unrelated to the text corresponding to the obtained text token (e.g., antonyms of each word in the text). The dummy token may be combined with the text token.

[0144] Next, a model for association can be constructed (402). The model for association can be implemented based on cross-attention, in which case the attention weight can be constructed using an image token as a query and a combination of a text token and a dummy token as a key. Meanwhile, a text token is used as a value. Here, the attention weight may be obtained by reflecting the distillation result described later. A result token may be obtained based on the combination of the attention weight and the value. The result token may be used for obtaining detection results and scoring each part (or image clip). According to an embodiment, at least one additional text relevance score may be obtained during the process of constructing the association model. The text relevance score may be obtained by summing the respective values ​​corresponding to each text token that correspond to the same image token within the attention weight.

[0145] Meanwhile, a guide map is generated, and attention weights may be distilled based on the generated guide map (404).

[0146] According to one embodiment, for the generation of a guide map, a prototype token corresponding to an image is first obtained, and a first moment prototype and a first non-moment prototype may be obtained based on the obtained prototype token. Here, the first moment prototype corresponds to an image token that corresponds to the context of the text, and the first non-moment prototype corresponds to an image token that does not correspond to the context of the text. Additionally, a prototype token corresponding to the text and a dummy token may be obtained, and a second moment prototype and a second non-moment prototype may be obtained based thereon. The acquisition of the first and second moment prototypes and the acquisition of the first and second non-moment prototypes may be performed based on a predetermined encoder, for example, a moment encoder. Meanwhile, along with the acquisition of the first moment prototype, a context-corresponding image token is acquired, along with the acquisition of the second moment prototype, a projected text token is acquired, and along with the acquisition of the second non-moment prototype, a projected dummy token is acquired.

[0147] Meanwhile, through contrastive learning, related parts can be interconnected. In this case, regarding the second moment prototype, the second moment prototype corresponding to the first moment prototype is processed as a positive pair to ensure mutual correlation (i.e., treated as similar), while the remainder is processed as a negative pair (i.e., treated as dissimilar). The first non-moment prototype is processed as a negative pair in relation to the second moment prototype. Meanwhile, regarding the second non-moment prototype, the second non-moment prototype corresponding to the first moment prototype is processed as a negative pair to ensure low mutual correlation (i.e., treated as dissimilar), while the remainder is processed as a positive pair (i.e., treated as similar). Accordingly, effective learning regarding the similarity between image tokens and text tokens becomes possible.

[0148] A guide map can be obtained based on context-corresponding image tokens, projected text tokens, and projected dummy tokens. The guide map can be used for distilling the aforementioned attention weights. In this case, only the portion of the attention weights corresponding to the text tokens may be subject to guide map-based distillation. The distilled attention weights may be reflected in generating attention weights during the association modeling process described above.

[0149] In addition, an importance token may be obtained (406). To obtain the importance token, first, the average of the image tokens, i.e., the image context, is subtracted from the image token to obtain a corrected image token, and then the transpose result of the corrected image token is inner producted with a pre-prepared importance candidate group, input into an activation function, and the output of the activation function is combined with the transpose result of the text relevance score to obtain a combination result. Then, a candidate weight vector is obtained from the combination result. The candidate weight vector can be obtained by summing the value(s) corresponding to each image token while corresponding to the same importance candidate. Among the elements of the candidate weight vector, elements with low importance are removed, and only elements with high importance are left to obtain a processed candidate weight vector, and the processed candidate weight vector is inner producted with the importance candidate group, and accordingly, a viewpoint description token is obtained. An image context may be added to the obtained viewpoint description token, and accordingly, an importance token is obtained. The importance token can be projected, and as a result, the projected importance token is output.

[0150] The aforementioned association modeling process (402), guide map generation and attention weight distillation process (404), and importance token acquisition process (406) may be performed simultaneously and / or sequentially, depending on the situation. Additionally, some of these processes (402, 404, 406) may be initiated before others, simultaneously with others, or last.

[0151] Subsequently, detection results can be obtained based on importance tokens and result tokens (404). First, an encoding result corresponding to each is obtained according to the encoding of the importance tokens and result tokens based on a transformer encoder, and a result in which importance is scored can be obtained through the combination of these. Additionally, a prediction result is obtained according to the decoding of the encoding result for the result token, and the prediction result may be obtained using a transformer decoder based on a decoder query. If necessary, Hungarian matching may be further performed on the prediction result. Based on this, at least one time point (which may include a segment) within the video corresponding to the prediction result can be obtained. Accordingly, a part of the video or a video clip that matches the text can be obtained.

[0152] The method for processing associations between text and images according to the above-described embodiment may be implemented in the form of a program that can be executed by a computer device. The program may include instructions, libraries, data files and / or data structures, etc., either individually or in combination, and may be designed and manufactured using machine code or high-level language code. The program may be specially designed to implement the above-described method, or it may be implemented using various functions or definitions that are known and available to a person skilled in the art in the field of computer software. Furthermore, the computer device may be implemented to include a processor or memory, etc., that enable the realization of the program's functions, and may further include a communication device as needed. The program for implementing the above-described method for processing associations between text and images may be recorded on a recording medium readable by a device such as a computer. A computer-readable recording medium may include at least one type of physical storage medium capable of temporarily or non-temporarily storing one or more programs executed upon the call of a device such as a computer, such as a semiconductor storage medium such as ROM, RAM, SD card or flash memory (e.g., a solid-state drive (SSD)), a magnetic disk storage medium such as a hard disk or floppy disk, an optical recording medium such as a compact disk or DVD, or a magneto-optical recording medium such as a floptical disk.

[0153] Although various embodiments of a text-to-image association processing device and a text-to-image association processing method have been described above, the device or method is not limited to the embodiments described above. Various other devices or methods that can be implemented by a person skilled in the art by modifying and varying the embodiments described above may also be embodiments of the text-to-image association processing device or text-to-image association processing method described above. For example, even if the described method(s) are performed in a different order than described, and / or the described component(s) such as a system, structure, device, or circuit are combined, connected, or combined in a form different from described, or replaced or substituted by other components or equivalents, they may still be embodiments of the text-to-image association processing device and / or text-to-image association processing method described above.

[0154] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential nature of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0155] 100: Text and Image Association Processing Device 101: Video encoding unit 102: Text encoding unit 103: Dummy Encoding Section 110: Association Modeling Section 130: Image-Text Correlation Processing Unit 150: Importance Processing Unit 170: Encoding section 180: Decoding section 190: Result Acquisition Section

Claims

Claim 1 A text and image association processing device comprising: an association modeling unit that acquires an image token, a text token, and a dummy token, generates an attention weight based on the image token, the text token, and the dummy token, and acquires a result token using the attention weight, wherein the dummy token is generated by modeling a meaning opposite to that of the natural language constituting the text for the text token; an importance processing unit that acquires a candidate weight vector based on the image token and an importance candidate group, and acquires an importance token based on the candidate weight vector; and an encoding unit that performs encoding for the result token and the importance token. Claim 2 An association processing device between text and image comprising: an association modeling unit that acquires an image token, a text token, and a dummy token, generates an attention weight based on the image token, the text token, and the dummy token, and acquires a result token using the attention weight; an importance processing unit that acquires a candidate weight vector based on the image token and a group of importance candidates, and acquires an importance token based on the candidate weight vector; and an encoding unit that performs encoding on the result token and the importance token; wherein the attention weight is generated by using the image token as a query and the combination of the text token and the dummy token as a key. Claim 3 A text and image association processing device comprising: an association modeling unit that acquires an image token, a text token, and a dummy token, generates an attention weight based on the image token, the text token, and the dummy token, and acquires a result token using the attention weight; an importance processing unit that acquires a candidate weight vector based on the image token and an importance candidate group, and acquires an importance token based on the candidate weight vector; and an encoding unit that performs encoding for the result token and the importance token; wherein the importance processing unit acquires an average of the image token, acquires a corrected image token by subtracting the average from the image token, inputs the combined result of the corrected image token and the importance candidate group into an activation function, and acquires a combination result by combining the output of the activation function and a text relevance score. Claim 4 A text and image association processing device according to paragraph 3, wherein the text relevance score is calculated through the sum of respective values ​​corresponding to each text token that correspond to the same image token in the attention weight. Claim 5 In paragraph 3, the importance processing unit obtains the candidate weight vector by summing the values ​​corresponding to each image token while corresponding to the same importance candidate from the combination result, obtains the processed candidate weight vector by removing elements of low importance from the candidate weight vector, obtains the viewpoint description token by combining the processed candidate weight vector with the importance candidate group, and obtains the importance token by adding image context to the viewpoint description token, wherein the viewpoint description token is for encoding whether the image clip corresponding to the image token has matching information. Claim 6 A text and image association processing device comprising: an association modeling unit that acquires an image token, a text token, and a dummy token, generates an attention weight based on the image token, the text token, and the dummy token, and acquires a result token using the attention weight; an importance processing unit that acquires a candidate weight vector based on the image token and a group of importance candidates, and acquires an importance token based on the candidate weight vector; and an encoding unit that performs encoding for the result token and the importance token; and further comprising an image-text association processing unit that generates a guide map using an image token that matches the context, a projected text token, and a projected dummy token, and performs distillation on the attention weight based on the guide map. Claim 7 In claim 6, the image-text association processing unit obtains a prototype token corresponding to an image and, based on the image token, an image token corresponding to the context of the text and a first moment prototype corresponding to the image token corresponding to the context of the text, and, based on the text token and the dummy token, a prototype token corresponding to the text and a dummy token, a projected text token for the text token, a projected dummy token for the dummy token, a second moment prototype corresponding to the text token, and a second non-moment prototype corresponding to the dummy token. Claim 8 In claim 7, the image-text association processing unit generates the guide map based on an image token corresponding to the context of the text, the projected text token, and the projected dummy token, and performs distillation based on the guide map for the portion of the attention weights corresponding to the text token to obtain distilled attention weights. Claim 9 In claim 8, the association modeling unit is a text and image association processing device that generates the attention weight by reflecting the distilled attention weight. Claim 10 A method for processing association between text and image, comprising: a step of obtaining an image token, a text token, and a dummy token, wherein the dummy token is generated by modeling a meaning opposite to that of the natural language constituting the text for the text token; a step of generating attention weights based on the image token, the text token, and the dummy token; a step of obtaining a result token using the attention weights; a step of obtaining a candidate weight vector based on the image token and a group of importance candidates; a step of obtaining an importance token based on the candidate weight vector; and a step of performing encoding on the result token and the importance token to obtain an image or image clip for the text. Claim 11 A method for processing association between text and image, comprising: a step of obtaining an image token, a text token, and a dummy token; a step of generating an attention weight based on the image token, the text token, and the dummy token; a step of obtaining a result token using the attention weight; a step of obtaining a candidate weight vector based on the image token and a group of importance candidates; a step of obtaining an importance token based on the candidate weight vector; and a step of performing encoding on the result token and the importance token to obtain an image or image clip for the text; wherein the attention weight is generated by using the image token as a query and the combination of the text token and the dummy token as a key. Claim 12 A method for processing association between text and image, comprising: a step of obtaining an image token, a text token, and a dummy token; a step of generating attention weights based on the image token, the text token, and the dummy token; a step of obtaining a result token using the attention weights; a step of obtaining a candidate weight vector based on the image token and a group of importance candidates; a step of obtaining an importance token based on the candidate weight vector; and a step of performing encoding on the result token and the importance token to obtain an image or image clip for the text; wherein the step of obtaining a candidate weight vector based on the image token and the group of importance candidates includes: a step of obtaining an average of the image tokens; a step of obtaining a corrected image token by subtracting the average from the image tokens; a step of inputting the combined result of the corrected image token and the group of importance candidates into an activation function; a step of obtaining a combination result by combining the output of the activation function and a text relevance score; and a step of obtaining the candidate weight vector by summing the values ​​corresponding to each image token that correspond to the same importance candidate from the combination result. Claim 13 A method for processing association between text and image, wherein, in claim 12, the text relevance score is calculated through the sum of respective values ​​corresponding to each text token that correspond to the same image token in the attention weight. Claim 14 A method for processing association between text and video according to claim 12, wherein the step of obtaining an importance token based on the above candidate weight vector comprises: a step of obtaining a processed candidate weight vector by removing low-importance elements from the above candidate weight vector; a step of obtaining a viewpoint description token by combining the processed candidate weight vector with the above importance candidate group; and a step of obtaining the importance token by adding video context to the viewpoint description token; wherein the viewpoint description token is for encoding whether a video clip corresponding to the video token has matching information. Claim 15 A method for processing association between text and images, comprising: a step of obtaining an image token, a text token, and a dummy token; a step of generating an attention weight based on the image token, the text token, and the dummy token; a step of obtaining a result token using the attention weight; a step of obtaining a candidate weight vector based on the image token and a group of importance candidates; a step of obtaining an importance token based on the candidate weight vector; and a step of performing encoding on the result token and the importance token to obtain an image or image clip for the text; and further comprising a step of generating a guide map using a context-matching image token, a projected text token, and a projected dummy token, and performing distillation on the attention weight based on the guide map. Claim 16 In claim 15, the step of generating a guide map using an image token corresponding to the context, a projected text token, and a projected dummy token, and performing distillation on the attention weight based on the guide map comprises: a step of obtaining a first moment prototype corresponding to an image token corresponding to the context of the text and an image token corresponding to the context of the text based on the image token and the image token; a step of obtaining a prototype token corresponding to the text and a dummy token, a projected text token for the text token, a projected dummy token for the dummy token, a second moment prototype corresponding to the text token, and a second non-moment prototype corresponding to the dummy token based on the text token and the dummy token. Claim 17 In claim 16, the step of generating a guide map using an image token corresponding to the context, a projected text token, and a projected dummy token, and performing distillation on the attention weight based on the guide map further comprises: generating the guide map based on an image token corresponding to the context of the text, the projected text token, and the projected dummy token; and obtaining a distilled attention weight by performing distillation based on the guide map for the portion of the attention weight corresponding to the text token. Claim 18 In claim 17, the step of generating attention weights based on the image token, the text token, and the dummy token comprises the step of generating attention weights by reflecting the distilled attention weights; a method for processing association between text and image.

Images