In-stream advertisement matching network for implementing search for advertisement space on stream of video content

Abstract

According to the present invention, disclosed is an in-stream advertisement matching network for implementing search for an advertisement space on a stream of video content. According to the present invention, provided is an in-stream advertisement matching network comprising: an advertisement space detection model for detecting a scene-change point on a stream of video content and selecting the scene-change point as an advertisement insertion position; and an advertisement selection model for selecting an advertisement that enables smooth transition to an in-stream advertisement from the selected advertisement insertion position, so that implementation of selecting the advertisement insertion position at which the in-stream advertisement is to be reproduced and selecting, from a group of advertisement candidates, an advertisement capable of inducing smooth transition from the selected advertisement insertion position can be achieved, thereby enabling reduction of discomfort caused by advertisement exposure without interrupting concentration on the video content.

Description

An in-stream advertising matching network that implements ad space discovery on streams of video content.

[0001] The present invention relates to an in-stream advertising matching network that implements search for advertising spaces on a stream of video content.

[0002] Mid-roll or in-stream advertisements that are inserted into a stream of video content suddenly appear in the stream of video content that do not match the atmosphere of the stream, so that a smooth transition from the video content is not implemented, causing discomfort to viewers due to exposure to the advertisement, and hindering concentration on the video content after exposure to the advertisement. Therefore, there is a need to develop a technical means to resolve problems caused by advertisement exposure and to implement a smooth transition between the video content and the advertisement.

[0003] One embodiment of the present invention includes an advertisement space detection model for detecting a scene transition point in a stream of video content and selecting the scene transition point as an advertisement interposition point, and an advertisement selection model for selecting an advertisement that can smoothly transition to an in-stream advertisement from the selected advertisement interposition point, thereby enabling selection of an advertisement interposition point at which an in-stream advertisement is to be reproduced and selection of an advertisement that can induce a smooth transition from the selected advertisement interposition point among a group of advertisement candidates, so as to reduce discomfort caused by advertisement exposure while not disturbing concentration on the video content.

[0004] In order to solve the above and other problems, an in-stream advertising matching network according to one embodiment of the present invention,

[0005] An advertisement space detection model that captures scene transition points between preceding and succeeding video frames in a stream of video content and selects them as advertisement insertion locations; and

[0006] For each advertisement candidate forming a group of advertisement candidates, an advertisement selection model may be included that calculates a mutual matching fitness with an advertisement intervention position selected from the advertisement space detection model and selects an advertisement candidate to be played at the advertisement intervention position from among the group of advertisement candidates.

[0007] According to one embodiment of the present invention, an in-stream advertisement matching network can be provided, which can implement selection of an advertisement placement location where an in-stream advertisement is to be reproduced and selection of an advertisement capable of inducing a smooth transition from an advertisement placement location selected from a group of advertisement candidates, so as to reduce discomfort caused by advertisement exposure while not disturbing concentration on the video content, including an advertisement space detection model for detecting a scene transition location in a stream of video content and selecting the scene transition location as an advertisement intervention location, and an advertisement selection model for selecting an advertisement capable of smoothly transitioning from the selected advertisement intervention location to an in-stream advertisement, and an advertisement selection model for selecting an advertisement capable of smoothly transitioning from the selected advertisement intervention location to an in-stream advertisement.

[0008] FIG. 1 is a diagram illustrating an overall configuration of an in-stream advertisement matching network, which includes an advertisement space search model for selecting an advertisement intervention location by inputting a stream of video content, and an advertisement selection model for selecting an in-stream advertisement to be played at the selected advertisement intervention location, in one embodiment of the present invention.

[0009] FIG. 2 illustrates a diagram for explaining contrastive language image pre-training (CLIP), which trains a text encoder and an image encoder to match each other by inputting a plurality of images forming a pair and text corresponding to captioning describing the plurality of images, and maximizing the cosine similarity between pairs of text embeddings output from each text encoder and image embeddings output from the image encoder, and minimizing the cosine similarity between pairs that do not match each other.

[0010] FIG. 3 is a diagram showing the architecture of a GPT (Generative Pre-trained Transformer) model as an architecture of a language model that can be applied to an in-stream advertisement matching network according to one embodiment of the present invention, and is a diagram showing natural language processing that calculates each word expression as a sum of weights according to the degree of association with previously input words (e.g., calculating an attention score) according to the input order of a word sequence as an architecture of a GPT model that stacks a plurality of decoder blocks.

[0011] Figure 4a shows an example of a spectrogram, which is divided into unit times along the time axis and expressed with different frequency intensities for each time unit, by applying a Fourier transform to decompose the voice forming the speech into different frequency components for each time unit by dividing it into fixed time units.

[0012] Figure 4b shows an example of a Mel-spectrogram generated through Mel scaling, which gives higher weights to low-frequency sounds in consideration of auditory characteristics sensitive to low frequencies.

[0013] Figure 5 is a diagram showing the architecture of an ARS model for extracting voice features from the voices forming a speech and extracting voice features that are widely distributed along the time axis and shared.

[0014] FIG. 6 is a diagram illustrating, by way of example, contrastive predictive coding (CPC) learning for extracting shared voice features from voice features extracted from previous time steps in the ASR model illustrated in FIG. 5.

[0015] Figure 7 is a diagram illustrating an architecture of a network for implementing a similarity analysis that synthesizes multi-modal data of images and speech based on a similarity analysis of image data and speech data.

[0016] FIG. 8 is a diagram illustrating a language model or embedding model that takes text as input and maps it to a text embedding space, and illustrates mapping of input texts to embedding spaces at adjacent locations according to their similarity.

[0017] Figure 9a illustrates a diagram for explaining the extraction of styles corresponding to the correlation between different features (features 1 to 4) produced from the application of different kernels to extract different features from specific layers of a CNN network into which different images are input, for the purpose of analyzing the similarity between different images.

[0018] Figure 9b illustrates a diagram for explaining the extraction of a Gram matrix regarding the correlation or style between different features (features 1 to 4) extracted from different images.

[0019] FIG. 10 is a diagram illustrating a histogram that shows different pixel values ​​and the number of pixels in which each different pixel value appears in R, G, and B images synthesized at the channel level to form each image for similarity analysis between different images.

[0020] Figure 11a illustrates a diagram for explaining an R-CNN network as an architecture for explaining object recognition that predicts the boundary of an object expressed in an image in the form of a bounding box.

[0021] Figure 11b illustrates a diagram for explaining the Fast R-CNN network as another architecture for explaining object recognition that predicts the boundaries of objects expressed on an image in the form of bounding boxes.

[0022] Figure 11c illustrates a diagram for explaining the Faster R-CNN network as another architecture for explaining object recognition that predicts the boundaries of objects expressed on an image in the form of bounding boxes.

[0023] FIG. 11d is a diagram illustrating an example of a Region Proposal Network (RPN) of the network illustrated in FIG. 11c.

[0024] Figure 12 illustrates an exemplary architecture of a network that implements comprehensive sentiment classification based on multi-modal data of speech and image by combining predictions from a speech-based sentiment classification model and an image-based sentiment classification model.

[0025] Figure 13 is a diagram illustrating a space of emotion classification including 12 radial levels of emotion classification, with active and passive at both ends along the Arousal axis and negative and positive at both ends along the Valence axis, on a two-dimensional plane having an Arousal axis and a Valence axis.

[0026] FIG. 14 is a diagram illustrating HDMI transmission, which includes R, G, B channels through which audio data and a synchronization signal are transmitted together during a blanking period of video data, and a clock channel (CLOCK) through which a channel clock is transmitted, as different channels through which video content is transmitted in one embodiment of the present invention.

[0027] An in-stream advertising matching network according to one embodiment of the present invention,

[0028] An advertisement space detection model that captures scene transition points between preceding and succeeding video frames in a stream of video content and selects them as advertisement insertion locations; and

[0029] For each advertisement candidate forming a group of advertisement candidates, an advertisement selection model may be included that calculates a mutual matching fitness with an advertisement intervention position selected from the advertisement space detection model and selects an advertisement candidate to be played at the advertisement intervention position from among the group of advertisement candidates.

[0030] For example, the above advertisement space detection model can determine the location between a preceding video frame and a succeeding video frame, which have relatively low similarity, as an advertisement intervention location.

[0031] For example, the above advertising space detection model calculates the similarity between a preceding video frame and a succeeding video frame for a plurality of frame sequences including a preceding video frame and a succeeding video frame taken in a chronological relationship in a stream of video content, and

[0032] Among the plurality of frame sequences that are the target of the above similarity calculation, the position between the preceding and succeeding video frames of the frame sequence with the lowest similarity can be determined as the advertisement intervention position.

[0033] For example, the above advertising space detection model can calculate the similarity between multi-modal data including video data and audio data of a preceding video frame and multi-modal data including video data and audio data of a succeeding video frame.

[0034] For example, the above advertising space detection model,

[0035] It may include a multi-modal AI model that inputs image data or images of the preceding and succeeding image frames, and outputs a text embedding matching the image embedding of the input image in a multi-modal embedding space of image-text, a first context vector including context information of the input image, a latent vector that maps the input image onto the embedding space of the image, or a first context vector corresponding to the image embedding.

[0036] For example, the above advertising space detection model,

[0037] It may include an ASR (auto speech recognition) model that inputs voice data or speech of the preceding video frame and the succeeding video frame, and outputs a second context vector including text about the input voice data or context information of the input speech, or a latent vector that maps the input voice data to a voice embedding space, or a second context vector corresponding to the voice embedding.

[0038] For example, the above advertising space detection model,

[0039] A first similarity calculated from natural language processing between a first context vector regarding a preceding image of the preceding image frame and a first context vector regarding a succeeding image of the succeeding image frame; and

[0040] The advertisement intervention location can be selected based on the result of collating the second similarity calculated from natural language processing between the second context vector regarding the preceding speech of the preceding video frame and the second context vector regarding the succeeding speech of the succeeding video frame.

[0041] For example, the above advertising space search model,

[0042] The first and second similarities above can be combined to calculate a comprehensive similarity score, and the advertisement intervention location can be selected from the calculated similarity score.

[0043] For example, the above advertising space search model,

[0044] The similarity between the preceding video frame and the succeeding video frame can be analyzed from a similarity analysis between the style forming the image of the preceding video frame and the style forming the image of the succeeding video frame.

[0045] For example, the above advertising space search model,

[0046] A Gram matrix is ​​generated that expresses the correlation between different features extracted from the image of the preceding video frame and the image of the succeeding video frame,

[0047] The similarity between the preceding image frame and the succeeding image frame can be analyzed from the differential values ​​of each element forming the Gram matrix extracted from the image of the preceding image frame and the Gram matrix extracted from the image of the succeeding image frame.

[0048] For example, the above advertising space search model,

[0049] The similarity between the preceding video frame and the succeeding video frame can be analyzed based on information about the color tone or color sensation that forms the image of the preceding video frame and the image of the succeeding video frame.

[0050] For example, the above advertising space search model,

[0051] For each of the three-channel images that are synthesized with respect to each other to form the image of the preceding image frame and the image of the succeeding image frame, the similarity between the preceding image frame and the succeeding image frame can be analyzed from histogram or histogram information that expresses the number of pixels appearing for each pixel value.

[0052] For example, the above advertising space search model,

[0053] Based on the histogram or histogram information extracted from the image of the preceding image frame and the image of the succeeding image frame, the similarity between the preceding image frame and the succeeding image frame can be analyzed from the difference value of the number of pixels for each pixel value.

[0054] For example, the above advertising space search model,

[0055] Based on the first to third histograms or the first to third histogram information directly extracted from each of the three-channel images forming the image of the preceding image frame and the image of the succeeding image frame, the similarity between the preceding image frame and the succeeding image frame can be analyzed by collecting the differential values ​​of the number of pixels for each pixel.

[0056] For example, the above advertising space search model,

[0057] Includes an emotion classification model for implementing emotion recognition from the preceding and following image frames,

[0058] The above emotional classification model can classify each of the input preceding and succeeding video frames into a class of a pre-set emotional classification by using data of the input preceding and succeeding video frames as input.

[0059] For example, the above sentiment classification model,

[0060] An image data-based emotion classification model that inputs two-dimensional matrix data of pixel sets forming images of the preceding and succeeding image frames from the image data of the preceding and succeeding image frames; and

[0061] The preprocessed data may include a voice data or speech data of the preceding and succeeding video frames, and may include a voice data-based emotion classification model that inputs two-dimensional matrix data regarding a spectrogram in which different frequency intensities are expressed for each time unit.

[0062] For example, the above advertising space detection model performs a similarity analysis between the preceding video frame and the following video frame,

[0063] The similarity between a preceding video frame and a succeeding video frame taken from a stream of video content down-sampled at a frequency or period (sampling rate) delayed from the frame rate (fps, frames per second) of the above video content can be calculated.

[0064] For example, the above advertising space search model,

[0065] In order to extract context information from audio data transmitted at a relatively low data rate while changing to a relatively low frequency than video data transmitted at a high data rate while changing to a relatively high frame rate (fps, frames per second) on the stream of the above video content, the similarity between the preceding video frame and the succeeding video frame can be calculated for each time step based on the chronological relationship between the video frame taken at the current time step, the video frame taken at the previous time step, and the video frame taken at the subsequent time step from the stream of the downsampled video content.

[0066] For example, the above advertising space search model,

[0067] A downsampled video frame can be input from a stream of video content at a certain frequency delayed from the frame rate (fps, frames per second) of the video content or at an interval according to a synchronization signal for synchronizing audio data and video data with each other.

[0068] For example, the above advertising space search model,

[0069] Similarity analysis between preceding and succeeding video frames can be performed based on audio data that changes slowly across a number of video frames at a relatively low frequency in the stream of the above video content.

[0070] Hereinafter, with reference to the attached drawings, an in-stream advertisement matching network based on video content analysis according to one embodiment of the present invention will be described.

[0071] FIG. 1 is a diagram illustrating an overall configuration of an in-stream advertisement matching network, which includes an advertisement space search model for selecting an advertisement intervention location by inputting a stream of video content, and an advertisement selection model for selecting an in-stream advertisement to be played at the selected advertisement intervention location, in one embodiment of the present invention.

[0072] An in-stream advertisement matching network based on video content analysis according to one embodiment of the present invention can capture scene transition points in a stream of video content and select the scene transition points as in-stream advertisement intervention locations.

[0073] In one embodiment of the present invention, the in-stream advertisement matching network can calculate a mutual matching suitability between the position of an in-stream advertisement in a stream of video content, the position of the corresponding in-stream advertisement, and a candidate group of in-stream advertisements, and select an in-stream advertisement based on the calculated result.

[0074] In one aspect of an in-stream advertisement matching network according to one embodiment of the present invention, a location where a scene or mood changes between a preceding video frame and a succeeding video frame in a stream of video content can be searched for, and the searched location can be selected as an in-stream advertisement interposition location. For example, the network according to one embodiment of the present invention may include an in-stream advertisement space search model for searching for an in-stream advertisement interposition location.

[0075] In one aspect of an in-stream advertisement matching network according to one embodiment of the present invention, a mutual matching suitability is calculated from a one-to-one relationship between a preceding video frame and a group of advertisement candidates, selectively among preceding video frames and subsequent video frames before and after an advertisement interposition position selected from an advertisement space search model, and an advertisement candidate having the highest mutual matching suitability relative to a preceding video frame preceding the selected advertisement interposition position can be selected as an in-stream advertisement to be played at the selected advertisement interposition position, so as to enable a smooth transition from the preceding video frame to the in-stream advertisement. To this end, the in-stream advertisement matching network according to one embodiment of the present invention may include an advertisement selection model for selecting an in-stream advertisement to be played at the selected advertisement interposition position from among the group of advertisement candidates.

[0076] For example, the in-stream advertisement space search model can analyze preceding and succeeding video frames in a stream of video content based on the content context, and can capture a position where the content context-based similarity between the preceding and succeeding video frames is relatively low based on the content context-based analysis result. As described below, in one embodiment of the present invention, preceding and succeeding video frames are each taken from a frame sequence down-sampled at a sampling rate lower than the frame rate (fps, frames per second) of the video content in a stream of video content, and the similarity between these preceding and succeeding video frames can be analyzed. For example, the similarity between the preceding and succeeding video frames can be calculated for a plurality of frame sequences each including a pair of adjacent video frames (a pair of preceding and succeeding video frames), and the position between the preceding and succeeding video frames of the frame sequence having the lowest similarity among the plurality of frame sequences to be subjected to the similarity calculation can be determined as the advertisement interposition position.

[0077] In one embodiment of the present invention, as a configuration for selecting a location where an in-stream advertisement is interposed on a stream of video content, an advertisement space search model can calculate a content context-based similarity between a preceding video frame and a succeeding video frame, and for example, a scene change location or an atmosphere change location where the similarity between the preceding video frame and the succeeding video frame is relatively low can be captured and selected as an advertisement interposition location where an in-stream advertisement is to be played. In one embodiment of the present invention, the advertisement space search model for performing a similarity analysis between the preceding video frame and the succeeding video frame can include calculating a similarity between multi-modal data including a video image (hereinafter, preceding image) and speech (hereinafter, preceding speech) of a preceding video frame, and multi-modal data including a video image (hereinafter, following image) and speech (hereinafter, following speech) of a succeeding video frame.

[0078] Multimodal AI Model

[0079] FIG. 2 illustrates a diagram for explaining contrastive language image pre-training (CLIP), which trains a text encoder and an image encoder to match each other by inputting a plurality of images forming a pair and text corresponding to captioning describing the plurality of images, and maximizing the cosine similarity between pairs of text embeddings output from each text encoder and image embeddings output from the image encoder, and minimizing the cosine similarity between pairs that do not match each other.

[0080] In one embodiment of the present invention, in the similarity analysis between a preceding image and a succeeding image (similarity analysis according to natural language processing), a multi-modal AI model or a multi-modal embedding that has learned a multi-modal embedding space of text-images can be applied, and the multi-modal AI model can learn a multi-modal embedding space of images and texts, and can output a captioning or text for describing the input image from a text embedding or a text representation (language representation) that matches or is most closely mapped to an input image or an image representation (visual representation) or an image embedding for the input image on the learned multi-modal embedding space of text-images. For example, in one embodiment of the present invention, the multi-modal AI model can take a preceding image as an input, convert it into an image representation or image embedding for the input preceding image, and then infer a text for describing the preceding image from a text embedding or text representation that matches or most closely matches the image representation or image embedding for the preceding image in a multi-modal embedding space of image-text. Similarly, the multi-modal AI model can take a succeeding image as an input, convert it into an image representation or image embedding for the input succeeding image, and then output a text for describing the succeeding image from a text embedding or text representation that matches or most closely matches the image representation or image embedding for the succeeding image in a multi-modal embedding space of image-text.

[0081] In one embodiment of the present invention, context information including association information between each word forming each preceding word sequence and each succeeding word sequence can be produced from a preceding word sequence and a succeeding word sequence forming texts for a preceding image and a succeeding image output from a multi-modal AI model that inputs a preceding image and a succeeding image, respectively, and a first context vector for the preceding word sequence and a second context vector for the succeeding word sequence can be extracted from the context-based embedding produced in this manner, and a cosine similarity analysis can be performed between the extracted first context vector and the second context vector to produce the similarity between the preceding image and the succeeding image as a result.

[0082] FIG. 2 illustrates a diagram for explaining contrastive language image pre-training (CLIP), which trains a text encoder and an image encoder to match each other by inputting a plurality of images forming a pair and text corresponding to captioning describing the plurality of images, and maximizing the cosine similarity between pairs of text embeddings output from each text encoder and image embeddings output from the image encoder, and minimizing the cosine similarity between pairs that do not match each other.

[0083] For example, in one embodiment of the present invention, the multi-modal AI model can convert an image and a text into embeddings respectively, and then predict the text embedding that is closest to the image embedding, or can reverse the input-output relationship to predict the image embedding that is closest to the text embedding. For example, in one embodiment of the present invention, CLIP (contrastive language image pre-training) can be applied as a multi-modal AI model trained to connect image-text pairs, and in one embodiment of the present invention, the multi-modal AI model can use web-based raw data, that is, a large-scale raw data without human annotation, as a training data set, and for example, can be trained using a web-based image and a captioning text describing the image as training data as image-text pairs, and can be trained using contrastive learning.

[0084] The above multi-modal AI model may include an image encoder for producing an image representation (visual representation) for an input image and a text encoder for producing a text representation (language representation) for an input text, and inputs n pairs of images and texts (image-text pairs) forming each pair to the image encoder and the text encoder, respectively, to generate a mini-batch (n training data) of images and a mini-batch (n training data) of texts, and then sets the pairs of images and texts that form each pair to be positive pairs, and sets the pairs of images and texts that do not form each other to be negative pairs, so that the cosine similarity for the n positive pairs is maximized, and n 2 - The cosine similarity for n negative pairs can be learned to be minimal, and the image encoder and the text encoder can be trained together from such contrastive learning. For example, in one embodiment of the present invention, the multi-modal AI model can learn the multi-modal embedding space of image-text, and for example, the image encoder and the text encoder can be trained to map the respective image embeddings and text embeddings from the image encoder that embeds the input image into the image space and the text encoder that embeds the input text into the text space to each other.

[0085] In one embodiment of the present invention, a multi-modal AI model that inputs an image of a preceding video frame (a preceding image) and an image of a succeeding video frame (a succeeding image) can output a text embedding for the preceding image and a text embedding for the succeeding image that match the preceding image and the succeeding image, respectively, and the text embedding for the preceding image and the text embedding for the succeeding image output from the multi-modal AI model that has learned the multi-modal embedding space in this way can be understood as being expressed as a context-based embedding that includes each contextual information in the embedding space of the text, and for example, in one embodiment of the present invention, the text embedding for the preceding image and the text embedding for the succeeding image are each made into a first context vector for the preceding image and a second context vector for the succeeding image, and a stream of video content is generated from a similarity analysis (similarity analysis according to natural language processing) between the first context vector for the preceding image and the second context vector for the succeeding image. In the above, scene transition points where the similarity between the preceding and following video frames is relatively low can be captured and in-stream advertisements can be inserted by making the scene transition points spaces for in-stream advertisements (ad space search model).For example, in one embodiment of the present invention, a similarity analysis (similarity analysis according to natural language processing) can be performed between the text embedding for the preceding image output from the multi-modal AI model and the text embedding for the succeeding image, and in various embodiments of the present invention, the text embedding for the preceding image and the succeeding image output from the multi-modal AI model can be input, and a context vector can be generated that contains association information or context information between words or word expressions forming the text embedding for each of the preceding and succeeding images through an embedding model or a language model (LM, language model) that maps the text embeddings output from these AI models to another text space or text embedding space, and a large language model (LLM, large language model) that takes the text embedding for the preceding and succeeding images as input, such as GPT, which is a forward language model, or BERT, which is a bi-directional language model including forward and backward, can be generated for each of the preceding and succeeding images. It is possible to derive association information (e.g., self-attention) between each word or word expression that forms a text embedding for an image and a text embedding for a subsequent image.

[0086] FIG. 3 is a diagram showing the architecture of a GPT (Generative Pre-trained Transformer) model as an architecture of a language model (LLM, large language model) that can be applied to an in-stream advertisement matching network according to one embodiment of the present invention, and is a diagram showing natural language processing that calculates each word expression as a sum of weights according to the degree of association with previously input words (e.g., calculating an attention score) according to the input order of a word sequence as an architecture of a GPT model that has a plurality of decoder blocks (DECODERs) stacked.

[0087] For example, the text embedding inferred from the multi-modal AI model can be input to a GPT model (see FIG. 3), and in the GPT model, a decoder block (DECODER) including a multi-head attention and a feedforward network that are stacked in a plurality of attention blocks (DECODER) are accumulated and stacked in a plurality of stacks, and a plurality of decoder blocks can be connected so that the result value of each decoder block is input to the next connected decoder block, and a word or word expression to come out after the word or token input to the first decoder block can be predicted from the result value of the last decoder block.

[0088] The above GPT model (see FIG. 3) does not include the encoder structure of the transformer and only includes the decoder structure of the transformer, so it may not include the encoder-decoder cross-attention of the BERT model and may include masked self-attention (see FIG. 3) as a forward language model. That is, the above GPT model may include an architecture in which a plurality of decoder blocks (DECODER) including masked multi-headed self-attention (see FIG. 3) and a feedforward network (see FIG. 3) are accumulated and stacked. For example, in each decoder block (DECODER) included in the above GPT model (see FIG. 3), the parameters of the mask multi-head self-attention (weight vectors operated with each word or token to produce a query, key, and value) may have different values.

[0089] In the above mask self-attention, words or tokens after the word or token to be predicted can be masked according to the input order, and in the above mask self-attention, a score is calculated from a scaled dot product between a query vector and a key vector corresponding to the word or token to be predicted, and a softmax function is applied to the calculated score to calculate the result value of the query vector from the weighted sum of the value vectors with the normalized score as a weight. In this case, for the masked word or token, the score before the application of the softmax function can be replaced with an infinite negative value so that the normalized score practically converges to zero.

[0090] As a language model according to one embodiment of the present invention, in the GPT model, an input embedding that is a sum of a token embedding, a positional embedding, and a segment embedding can be input, and the GPT model can input a special token corresponding to the beginning of a sentence sequence (for example, <sos>) from which a special token (e.g., <eos>) can be repeated until the next word or token is produced, and the probability for the next word or token can be calculated, and the next word or token can be predicted based on the calculated probability. For example, the GPT model may include parameters learned to maximize the objective function with the likelihood of the next word or token appearing as the objective function.

[0091] As a language model according to one embodiment of the present invention, the GPT model can predict the probability of each token forming a token embedding matrix to appear next from an operation between a result value from the last decoder block, that is, an expression value for each word or token, and a token embedding matrix including the entire vocabulary, and can predict the token with the highest prediction probability by applying a softmax function as the word or token to appear next.

[0092] In one embodiment of the present invention, a text representation including association information (e.g., self-attention) between each word forming the text embedding can be generated from the text embedding for the preceding image and the text embedding for the succeeding image output from the multi-modal AI model (see FIG. 2), and a context vector can be extracted from this text representation. In this way, the text embedding for the preceding image and the text embedding for the succeeding image can be extracted from the multi-modal AI model (see FIG. 2) and the language model (e.g., the GPT model, see FIG. 3), and first and second context vectors including context information for the preceding image and the succeeding image can be extracted from them, and a scene transition point having a relatively low similarity between the preceding image (preceding image frame) and the succeeding image (afterward image frame) can be captured through a similarity analysis (similarity analysis according to natural language processing) between these first and second context vectors. In one embodiment of the present invention, similarity analysis between context vectors for a preceding image and a succeeding image (similarity analysis according to natural language processing) can be performed by calculating a Euclidean distance or cosine similarity between the context vector for each preceding image and the context vector for each succeeding image.

[0093] <ASR, auto speech recognition>

[0094] Figure 4a shows an example of a spectrogram, which is divided into unit times along the time axis and expressed with different frequency intensities for each time unit, by applying a Fourier transform to decompose the voice forming the speech into different frequency components for each time unit by dividing it into fixed time units.

[0095] Figure 4b shows an example of a Mel-spectrogram generated through Mel scaling, which gives higher weights to low-frequency sounds in consideration of auditory characteristics sensitive to low frequencies.

[0096] Figure 5 is a diagram showing the architecture of an ARS model for extracting voice features from the voices forming a speech and extracting voice features that are widely distributed along the time axis and shared.

[0097] FIG. 6 is a diagram illustrating, by way of example, contrastive predictive coding (CPC) learning for extracting shared voice features from voice features extracted from previous time steps in the ASR model illustrated in FIG. 5.

[0098] In one embodiment of the present invention, speech of a preceding video frame (preceding speech) and speech of a succeeding video frame (following speech) in a stream of video content capable of providing an intervention position of an in-stream advertisement may include voice data or speech data of the video content, and in one embodiment of the present invention, in a similarity analysis (similarity analysis according to natural language processing) between the preceding speech and the succeeding speech, an ASR (auto speech recognition) model may be applied to input voice data or speech data of the preceding speech and the succeeding speech and output text related to the voice data or speech data. The above ASR model can generate a spectrogram in which the voice data or utterance data of speech are divided into fixed time units (e.g., 15 msec, etc.) and the intensity (or strength) of each frequency is expressed for each time unit by applying a Fourier transform, thereby dividing the voice data or utterance data of speech into different frequency components for each time unit (see FIG. 4a). In this way, the ASR model can perform preprocessing of voice data to convert the voice data or utterance data of speech into a vector form or a two-dimensional matrix form suitable for DNN (deep neural network) processing. In various embodiments of the present invention, in the preprocessing of the voice data of the speech, a Mel-spectrogram can be generated through Mel scaling that gives a higher weight to the voice data of low frequencies in consideration of the auditory characteristics that are sensitive to low frequencies (see FIG. 4b).

[0099] In one embodiment of the present invention, the ASR model may include an encoder (see FIG. 5) that takes as input speech data or utterance data or a spectrogram (see FIGS. 4A and 4B) preprocessed from the speech data or utterance data as described above and outputs a compressed vector representation. More specifically, the ASR model may include an encoder (see FIG. 5) of the convolution neural network (CNN) family and an aggregator of the recurrent neural network (RNN) family.

[0100] For the aggregator of the above ASR model (see FIG. 5), considering that the voice data or utterance data of the speech may include information corresponding to the extraction target from the ASR model as noise components that change rapidly along the time axis and information that changes slowly along the time axis and is widely distributed along the time axis and shared, the aggregator (see FIG. 5) may apply CPC (contrastive predictive coding, see FIG. 6) to the feature vector (Zt, see FIG. 6) of the voice output from the encoder (see FIG. 5) for each time step in order to extract the shared information of the voice that is widely distributed along the time axis of the speech, and may be trained to extract the shared vector (Ct) of the voice that is generally shared up to the previous time step. For example, in the above CPC, the feature vector (Xt) of the voice output from the encoder in the subsequent time step can be learned from the feature vector (Zt) of the voice output from the encoder (see FIG. 6) up to the previous time step, and through such CPC (contrastive predictive coding), the aggregator (see FIG. 6) can be learned to be able to extract the shared vector (Ct) of the voice shared in the previous time step and the subsequent time step well.More specifically, in the above CPC, it can be learned according to a loss function that is configured as a positive pair for the shared vector (Ct) of the voice of the aggregator (see FIG. 6) at the current time step and the feature vector (Zt+1) of the voice output from the encoder at a future time step close to the current time step according to the time position, and a negative pair for the shared vector (Ct) of the voice of the aggregator (see FIG. 6) at the current time step and the feature vector (e.g., Zt-3, Zt+3) of the voice output from the encoder at a future time step relatively far from the current time step.

[0101] For example, in one embodiment of the present invention, in the CPC, the weights of the transformation matrix can be learned so that the matrix product between the output vector (Xt) of the encoder (see FIG. 6) and the transformation matrix at each time step and the shared vector (Ct) of the aggregator (see FIG. 6) have relatively high or relatively low similarity depending on the distance according to the time step, and the shared vector (Ct) of the voice that well extracts the shared features from the feature vector (Xt) of the voice output from the encoder (see FIG. 6) can be produced from the product of the transformation matrix having the learned weights. Referring to FIG. 5, an ASR model applicable to an in-stream advertisement matching network according to an embodiment of the present invention may be input as an acoustic model that takes as input a shared vector of speech extracted from an aggregator, and may predict the probability of a next word among words forming a vocabulary from an acoustic model including parameters learned using speech and text aligned with respect to each other as training data, for example, the ASR model may include a language model including parameters learned only from text, together with an acoustic model learned using speech and text aligned with respect to each other as training data, and may calculate the probability of each candidate word appearing as the next word by considering the overall context for about five candidate words predicted with a relatively high probability from the acoustic model, and may predict the next word based on the result of adding up the predictions of the acoustic model and the predictions of the language model with different weights, and as a result, text may be extracted from speech forming a speech of a preceding video frame and a speech of a subsequent video frame.

[0102] Referring to FIG. 5, in one embodiment of the present invention, the ASR model can perform discretization or quantization to extract a discrete integer sequence from a continuous vector, such as speech data or utterance data, and for example, a discrete integer sequence (quantization, see FIG. 5) can be extracted from speech data or utterance data through quantization, and a continuous vector (Z) can be extracted again from the discrete integer sequence. ∧ After extracting the feature vector (Xt, see Fig. 5), as described above, by applying the CPC loss function, a transformation matrix can be trained in which a matrix product with the feature vector (Xt) output from the encoder is calculated to extract a shared vector (Ct), and in the inference step, a shared vector (Ct) of voice that well extracts shared features from the feature vector (Xt) of voice can be extracted from the matrix product of the feature vector (Xt) of voice output from the encoder and the transformation matrix of the learned weights.

[0103] For example, the ASR model can take speech data or utterance data as input, add some noise to the feature vector of the speech (Xt, see FIG. 5) output from the encoder (see FIG. 5), and quantize it (Gumbel-softmax) or apply clustering (K-means clustering) to quantize it, and convert it into a discrete array through such quantization. That is, in one embodiment of the present invention, speech data or utterance data can be taken as input, and a feature vector of the speech (Xt, see FIG. 5) can be output by compressing the speech data to a certain size from the encoder (see FIG. 5), and Gumbel-softmax can be applied to discretize or quantize the feature vector of the speech (Xt, see FIG. 5) output from the encoder. In the above Gumbel-softmax, noise (noise) sampled from uniform distribution u(0, 1) and then logarithmically processed is added to the feature vector of the voice output from the encoder (Xt, see Fig. 5), and softmax is taken to derive the probability for each element forming the feature vector of the voice (Xt, see Fig. 5), and the temperature function is applied as a scaling factor. A discrete representation similar to a one-hot vector can be produced. If the embedding matrix is ​​vector-multiplied by this one-hot vector, a continuous vector (Z) is generated again. ∧ t, see Fig. 5) can be produced, and the continuous vector produced in this way (Z ∧ By applying the CPC loss function to the feature vector (Xt) of the voice output from the encoder (see Fig. 5), a transformation matrix can be learned in which a matrix product is calculated with the feature vector (Xt) of the voice output from the encoder to extract a shared vector (Ct) of the voice, and in the inference step, a shared vector (Ct) of the voice that well extracts shared features from the feature vector (Xt) of the voice can be extracted from the matrix product of the feature vector (Xt) of the voice output from the encoder and the transformation matrix of the learned weights.

[0104] In one embodiment of the present invention, a speech feature vector (Xt, see FIG. 5) that compresses the speech data to a certain size from an encoder can be output by inputting speech data or utterance data, and K-means clustering can be applied to discretize or quantize the speech feature vector (Xt, see FIG. 5) output from the encoder. For example, in the K-means clustering, a discrete representation can be produced by assigning an index that is most closely mapped to the speech feature vector (Xt, see FIG. 5) output from the encoder among the indices of an embedding matrix or a code book, and a continuous vector (Z) is generated again from the index of this discrete representation and the embedding matrix of the code book. ∧ t, see Fig. 5) can be produced, and the continuous vector produced in this way (Z ∧ By applying the CPC loss function to the feature vector (Xt) of the voice output from the encoder (see Fig. 5), a transformation matrix can be learned in which a matrix product is calculated with the feature vector (Xt) of the voice output from the encoder to extract a shared vector (Ct) of the voice, and in the inference step, a shared vector (Ct) of the voice that well extracts shared features from the feature vector (Xt) of the voice can be extracted from the matrix product of the feature vector (Xt) of the voice output from the encoder and the transformation matrix of the learned weights.

[0105] <Aggregation of the first similarity between the preceding and succeeding images and the second similarity between the preceding and succeeding speeches>

[0106] Figure 7 is a diagram illustrating an architecture of a network for implementing a similarity analysis that synthesizes multi-modal data of images and speech based on a similarity analysis of image data and speech data.

[0107] In one embodiment of the present invention, a scene transition point having relatively low similarity between a preceding video frame and a succeeding video frame can be captured through a similarity analysis (multimodal similarity analysis, similarity analysis according to natural language processing) between multimodal data regarding a preceding video frame including an image (preceding image) and speech (preceding speech) of a preceding video frame and multimodal data regarding a succeeding video frame including an image (postceding image) and speech (postceding speech) of a succeeding video frame. For example, in one embodiment of the present invention, the advertising space search model can calculate a comprehensive similarity score (see FIG. 7) by collecting the similarity (first similarity) of the video data (image) between the preceding image and the succeeding image and the similarity (second similarity) of the voice data (utterance) between the preceding speech and the succeeding speech, and can calculate a comprehensive similarity score (see FIG. 7) by collecting the output from the multi-modal AI model that calculates the similarity (first similarity) of the video data (image) forming the multi-modal data of the preceding video frame and the succeeding video frame and the output of the ASR model, and for example, in one embodiment of the present invention, the output of the multi-modal AI model and the output of the ASR model are passed through a layer (see FIG. 7) for concatenating and flattening, a linear fully connected layer (see FIG. 7), and a layer for normalization (see FIG. 7) to calculate a comprehensive similarity score between the preceding video frame and the succeeding video frame. Can be produced.In other words, in one embodiment of the present invention, the advertisement space detection model may include a multi-modal AI model and an ASR model, and the multi-modal AI model may be capable of calculating a first similarity calculated from natural language processing between a first context vector regarding a preceding image of the preceding video frame and a first context vector regarding a succeeding image of the succeeding video frame, and the ASR model may be capable of calculating a second similarity calculated from natural language processing between a second context vector regarding a preceding speech of the preceding video frame and a second context vector regarding a succeeding speech of the succeeding video frame, and an advertisement intervention location may be selected based on a result of collecting the first similarity and the second similarity calculated from the multi-modal AI model and the ASR model, respectively.

[0108] For example, in one embodiment of the present invention, by comparing the magnitude of scores derived from a plurality of frame sequences including preceding and succeeding video frames taken from a stream of video content, the space between the preceding and succeeding video frames having the lowest similarity score can be selected as an advertisement space (advertisement insertion location).

[0109] <Sampling of preceding and succeeding video frames>

[0110] Referring to FIG. 1, in one embodiment of the present invention, the advertisement space search model may, for example, select a space on a stream of video content for which a relatively low similarity is calculated while performing a multi-modal similarity analysis (similarity analysis according to natural language processing) between the preceding frame and the succeeding frame, by setting the video frame of the preceding time step and the video frame of the succeeding time step forming a stream of video content as the preceding frame and the succeeding frame, as an advertisement insertion location.

[0111] In one embodiment of the present invention, the video content can be played back at a frame rate of, for example, 60 frames per second (fps) or 30 fps, depending on the resolution of the video content or the performance of an encoder for video compression of a display source that transmits the video content, and the rendering performance of a display device that receives and plays back the video content transmitted from the display source, and the video frame of a preceding previous time step and the video frame of a succeeding subsequent time step in the stream of the video content can be taken as a preceding video frame and a succeeding video frame, respectively, and a similarity analysis of these preceding and succeeding video frames can be performed from an advertisement space search model that inputs these preceding and succeeding video frames. In one embodiment of the present invention, the advertisement space search model may perform an operation (an operation for a first similarity analysis of image data and an operation for a second similarity analysis of audio data) of the advertisement space search model for analyzing similarity between the preceding image frame and the succeeding image frame by taking the image frame of the current time step as the preceding image frame and using the image frame of the previous time step as the preceding image frame, and further, perform an operation (an operation for a first similarity analysis of image data and an operation for a second similarity analysis of audio data) of the advertisement space search model for analyzing similarity between the preceding image frame and the succeeding image frame by taking the image frame of the current time step as the preceding image frame and using the image frame of the subsequent time step as the succeeding image frame.

[0112] In one embodiment of the present invention, the advertising space search model may take a video frame of each current time step from a stream of video content output from a display device at a frame rate of 60 fps or 30 fps, and sampled at a sampling period (frame rate) later than the frame rate, and may use a video frame of each current time step as a subsequent video frame for a video frame of a previous time step and a preceding video frame for a video frame of a subsequent time step, and may perform a similarity analysis between the preceding video frame and the succeeding video frame for each frame sequence, for a frame sequence including each preceding video frame - the video frame of the current time step (corresponding to the succeeding video frame), and another frame sequence including the video frame of the current time step (corresponding to the preceding video frame) - the succeeding video frame.For example, in one embodiment of the present invention, when considering the computational processing time of an advertisement space search model for performing a similarity analysis between a preceding video frame and a succeeding video frame for each frame sequence forming a preceding video frame and a succeeding video frame by taking a preceding video frame and a succeeding video frame from a stream of video content forming a sequence of video frames, the advertisement space search model may perform down sampling from the display device at a sampling rate delayed from the frame rate of the display device, for example, at a sampling rate delayed from the frame rate of 60 fps or 30 fps of the display device, rather than receiving all video frames output from the display device and performing a similarity analysis on all of these video frames, and taking a video frame of a current time step from the stream of downsampled video content, and performing a similarity analysis on a frame sequence including a preceding video frame-successor video frame with respect to a video frame of a previous time step, and also performing a similarity analysis on a frame including a preceding video frame-successor video frame with respect to a video frame of a subsequent time step. Similarity analysis can be performed on sequences.

[0113] For example, in one embodiment of the present invention, for a frame sequence including a video frame taken at a current time step and previous and subsequent video frames from a sequence of down-sampled video frames delayed from the frame rate of a display device, a similarity analysis between a preceding video frame and a succeeding video frame may be performed at each time step, in other words, a similarity analysis may be performed for each frame sequence including a preceding video frame and a succeeding video frame according to the chronological relationship of the down-sampled video frames or video frames from the display device. In addition, a similarity analysis may be performed for each frame sequence from the computational processing of an advertisement space search model, and a position or space between a preceding video frame and a succeeding video frame having the lowest similarity calculated according to the similarity analysis calculated at each time step may be selected as a position (an advertisement interposition position or an advertisement interposition space) at which the in-stream advertisement is to be played.

[0114] In one embodiment of the present invention, the video data on the stream of the video content may correspond to high-frequency data having a relatively high frame rate (fps, frames per second), for example, the video data may be transmitted at a relatively high data transmission rate, and in contrast, the audio data may correspond to low-frequency data having a relatively low rate, for example, the audio data may be transmitted at a relatively low data transmission rate, and the video data and the audio data may be synchronized with each other.

[0115] FIG. 14 is a diagram illustrating HDMI transmission, which includes R, G, B channels through which audio data and a synchronization signal are transmitted together during a blanking period of video data, and a clock channel (CLOCK) through which a channel clock is transmitted, as different channels through which video content is transmitted in one embodiment of the present invention.

[0116] In one embodiment of the present invention, in data transmission according to HDMI, data can be transmitted through a total of four channels, including three channels for transmitting video data (see FIG. 14, CHANNEL 0 to 2, R, G, B three channels) and one channel for transmitting a channel clock signal (see FIG. 14, CLOCK), and during the blanking period (see FIG. 14) between scan lines of the three channels for transmitting video data (see FIG. 14, CHANNEL 0 to 2, R, G, B three channels), AUDIO PACKETS (see FIG. 14) or AUDIO HEADER (see FIG. 14) for audio data can be transmitted, and in this way, audio data can be transmitted at a relatively lower data transmission rate than video data while being transmitted during the blanking period (see FIG. 14) corresponding to the interval between scan lines of video data, and for example, video data and audio data can be transmitted at different data transmission rates while being transmitted during the blanking period between scan lines of video data. Can be synchronized. The synchronization signal for synchronizing audio data and video data through this specification can comprehensively mean various synchronization signals for synchronizing audio data and video data, and for example, it can comprehensively mean synchronization signals for actually implementing synchronization of audio data and video data, such as AUDIO PACKETS or AUDIO HEADER for audio data and the BLANKING section in which these AUDIO PACKETS or AUDIO HEADER are transmitted.

[0117] In one embodiment of the present invention, for video data with a relatively high data transmission rate, considering the computational processing speed of the advertisement space search model, a similarity analysis (first similarity analysis) between a preceding video frame (preceding image) and a succeeding video frame (following image) may be performed on down-sampled video frames (or video data) that are delayed from the frame rate of the video frame (or video data) output from the display device, and, unlike video data, for audio data transmitted with a relatively low data transmission rate, a similarity analysis (second similarity analysis) between a preceding video frame (preceding speech) and a succeeding video frame (following speech) may be performed on the entire audio data of the video content without resorting to down-sampling. For example, in a stream of video content, audio data or speech data changes at a relatively low frequency and changes slowly at a relatively low data transmission rate, and therefore, in terms of the efficiency of computational processing for identifying contextual information in the stream of video content, a second similarity analysis based on audio data may be more advantageous than a first similarity analysis based on audio data. In this sense, for the purpose of identifying contextual information of voice data on the stream of the video content, the sampling rate or speed of the video data, which needs to be synchronized with the voice data and for which similarity analysis needs to be performed, can be adjusted, and for example, similarity analysis can be performed at each time step according to the before-after relationship of the time step from the multi-modal data of the video data and voice data of the video frame sampled at a down-sampling speed (e.g., 1 fps or less) that is more delayed than the down-sampling speed (e.g., 5 fps, etc.) considering the computational processing speed of the advertising space search model.

[0118] In various embodiments of the present invention, the advertising space search model performs similarity analysis on audio data transmitted at a relatively low data transmission rate rather than on each video frame from video data transmitted at a relatively high data transmission rate, thereby reducing the computational processing burden per unit time, and by performing similarity analysis between a preceding video frame and a succeeding video frame based on audio data that changes slowly over a plurality of video frames at a relatively low frequency rather than on video data that changes at a relatively high frequency or frame rate, context information can be easily extracted from a stream of video content, and for example, based on the overall context information of the preceding video frame and the succeeding video frame, a scene change point or an atmosphere change point with relatively low similarity between them can be captured.

[0119] For example, in one embodiment of the present invention, the advertising space search model can perform a similarity analysis of a preceding video frame - a succeeding video frame based on audio data, and in various embodiments of the present invention, according to a synchronization signal for synchronization between audio data and video data, a similarity analysis of a preceding video frame - a succeeding video frame can also be performed based on video data synchronized with audio data, and for example, a video frame that is downsampled so that contextual information of audio data or speech can be extracted, for example, a video frame that is downsampled to a constant frequency or a relatively low frequency based on a synchronization signal for synchronization of audio data and video data, can be downsampled to a frequency lower than a frame rate of video content. However, in various embodiments of the present invention, considering a situation in which the story development phase (scene change or mood change) changes according to changes in video data rather than voice data or speech data at various locations on a stream of video content, the location of the scene change or the location of the mood change can be captured from changes in video data or images rather than voice data or speech data as the story develops, and for example, for video frames sampled at a constant sampling frequency delayed from the frame rate of the video content, a similarity analysis based on video data is performed and / or a similarity analysis based on voice data is performed to capture the location of the scene change or the location of the mood change based on the context on a stream of video content.

[0120] In one embodiment of the present invention, depending on the performance of the encoder for video compression of the display source transmitting the video content or the resolution of the video content, or the rendering performance of the display device that receives and plays the video content transmitted from the display source, for example, without affecting the frame rate of the video content of, for example, 30 fps to 60 fps, during the process of playing according to the frame rate (fps, frame per second) of the video content that can be typically set, downsampling is performed to a sampling frequency (sampling rate) that is delayed from the corresponding frame rate (fps, frame per second), thereby selecting an advertisement interposition position according to one embodiment of the present invention and selecting an in-stream advertisement to be played at the selected advertisement interposition position, so that, for example, the selection of the advertisement interposition position and the selection of the in-stream advertisement can be performed during the process of playing the video content at the playback speed of the typical video content, and accordingly, for example, in one embodiment of the present invention, the implementation of the in-stream advertisement matching network of the present invention can be made without additional configuration of hardware for separate processing of the stream of the video content.

[0121] In one embodiment of the present invention, a context vector including context information of image data (preceding image) and audio data (preceding speech) of a preceding image frame, and image data (postceding image) and audio data (postceding speech) of a succeeding image frame, may be referred to as a context vector including context information of each image in the sense that it encapsulates each preceding image and each succeeding image in text form and encapsulates context information across the entirety of each preceding image and each succeeding image, and similarly, it may be referred to as a context vector including context information of each speech in the sense that it encapsulates each preceding speech and each succeeding speech in text form and encapsulates context information across the entirety of each preceding speech and each succeeding speech. In one embodiment of the present invention, the information of the image data (image) input from the multi-modal AI model (see FIG. 2) and the voice data (speech) input to the ASR model (see FIG. 5) is expressed in the form of text, and thus can be referred to as a context vector. Throughout this specification, the context vector refers to a context vector that reflects attention (e.g., self-attention) information through another language model (see FIG. 3, for example, a large-scale language model such as GPT or BERT) to imply association information between each word or word expression (token) forming each text or text embedding output from the multi-modal AI model (see FIG. 2) and the ASR model (see FIG. 5). The context vector can also be referred to as a context vector in the sense that it includes context information, which is association information (e.g., self-attention) between different words forming the text, and for example, in various embodiments of the present invention, the context output from the multi-modal AI model (see FIG. 2) or the ASR model (see FIG. 5) A vector is a video data (image) or audio data (speech,) of a preceding video frame and a succeeding video frame.It can comprehensively mean a context vector output from a multi-modal AI model (see FIG. 2) or an ASR model (see FIG. 5) to include overall context information of an image or speech throughout the entirety of an utterance), and a context vector output from the multi-modal AI model (see FIG. 2) or an ASR model (see FIG. 5) or a text expressed by the context vector as input, and a context vector output from the multi-modal AI model (see FIG. 2) or an ASR model (see FIG. 5) to include context information of the text corresponding to association information between words or word expressions forming the text, and a context vector output from a language model (see FIG. 3, for example, a large-scale language model such as GPT or BERT) to include context information of the text corresponding to association information between words or word expressions forming the text.

[0122] <Advertisement Selection Mutual Matching Suitability>

[0123] Referring to FIG. 1, in one embodiment of the present invention, a scene transition point having a relatively low similarity between a preceding video frame and a succeeding video frame in a stream of video content can be captured based on an output of an advertisement space search model that takes a stream of video content as an input, and the captured scene transition point can be selected as an advertisement interposition point. At this time, an in-stream advertisement having a relatively high mutual matching suitability with the selected advertisement interposition point among a group of advertisement candidates can be selected as an in-stream advertisement to be played at the selected advertisement interposition point based on the mutual matching suitability with the selected advertisement interposition point.

[0124] An in-stream advertisement matching network according to one embodiment of the present invention may include, as described above, an advertisement space search model (see FIG. 1) for performing a similarity analysis (similarity analysis based on natural language processing) between a preceding video frame and a succeeding video frame in a stream of video content, and an advertisement selection model (see FIG. 1) for selecting an in-stream advertisement to be played at an advertisement interposition position selected from a group of advertisement candidates based on the similarity analysis (similarity analysis based on natural language processing) of the advertisement space search model. For example, in one embodiment of the present invention, the advertisement selection model may calculate a mutual matching suitability between a preceding video frame and a group of advertisement candidates, selectively from among the preceding video frames and the succeeding video frames before and after the selected advertisement interposition position, and may select an advertisement candidate having a relatively high mutual matching suitability from among the group of advertisement candidates as an in-stream advertisement to be played at the selected advertisement interposition position.

[0125] In one embodiment of the present invention, the selection of the advertisement interposition position is based on a multi-modal similarity analysis between a preceding video frame and a succeeding video frame in a stream of video content, and a scene transition point having a relatively low multi-modal similarity can be captured, and the selection of the in-stream advertisement to be played at the selected advertisement interposition position can calculate a mutual matching suitability with the preceding video frame selectively among the preceding video frames and the succeeding video frames before and after the selected advertisement interposition position, and for example, the preceding video frame can calculate a mutual matching suitability one-to-one with each advertisement candidate forming a group of advertisement candidates, and among the group of advertisement candidates, an advertisement candidate having a relatively high mutual matching suitability, for example, the highest mutual matching suitability, can be selected as the in-stream advertisement to be played at the corresponding advertisement interposition position.

[0126] As such, in one embodiment of the present invention, as a target for calculating the mutual matching suitability with a group of advertisement candidates, the mutual matching suitability is calculated selectively between a preceding video frame and a succeeding video frame before and after the selected advertisement interposition position, in order to induce a smooth transition between the video content and the advertisement without causing the advertisement to be unpleasant, such as by interrupting the focus of the video content, breaking the flow of the video content, or causing a sudden emotional change. For example, in one embodiment of the present invention, a scene change point with a relatively low multi-modal similarity between a preceding video frame and a succeeding video frame in a stream of video content can be captured and selected as an advertisement interposition position, and accordingly, an in-stream advertisement can be selected from a group of advertisement candidates by considering only the mutual matching suitability with the preceding video frame, and the mutual matching suitability with a succeeding video frame after the in-stream advertisement is played can be not considered. In other words, since a scene change or mood change occurs between a preceding video frame and a succeeding video frame before and after a scene change point where an in-stream advertisement is played, there may be no need to consider a scene change or mood change after the in-stream advertisement is played, and since a scene or mood change occurs between a preceding video frame and a succeeding video frame regardless of whether an in-stream advertisement is played, there may be no need to consider a smooth transition to a succeeding video frame until after the in-stream advertisement is played.In this way, in one embodiment of the present invention, among the preceding video frames and the succeeding video frames before and after the selected advertisement intervention position, the mutual matching suitability between the preceding video frame and a group of advertisement candidates may be calculated to implement a smooth transition from the preceding video frame to the playback of the in-stream advertisement, but the mutual matching suitability between the group of advertisement candidates and the succeeding video frame may not be calculated to implement a smooth transition to the succeeding video frame after the playback of the in-stream advertisement.

[0127] An in-stream advertisement matching network according to one embodiment of the present invention may include an advertisement selection model (see FIG. 1) for calculating a mutual matching fit between a preceding video frame preceding a selected advertisement interposition position and a group of candidate advertisements. The advertisement selection model may selectively apply any one type of mutual matching fit among various types of mutual matching fit described below, or may apply a combination of different types of mutual matching fit, for example, by applying different weights to the different types of mutual matching fit described below and combining them, or by applying a network for collecting different types of mutual matching fit (including layers for concatenating and flattening different types of mutual matching fit and linear fully connected layers, etc.) to calculate a score for a comprehensive mutual matching fit that collects these different types of mutual matching fit (see, for example, FIG. 7). For example, in various embodiments of the present invention, an advertisement to be played at a corresponding advertisement intervention position may be selected based on a comprehensively compiled mutual matching suitability according to priorities (or different weights) for different types of mutual matching suitability.

[0128] <선행 영상 프레임과 광고 후보 간의 상호 매칭 적합도 제1 유형>

[0129] In one embodiment of the present invention, considering that the context vector regarding the preceding image or preceding speech output from the advertisement space search model that inputs the stream of the video content contains context information of the preceding video frame, the context vector of the preceding video frame that has already been calculated before the operation of the advertisement selection model (the context vector regarding the preceding image and the context vector regarding the preceding speech, for example, the text embedding or context vector output from a multi-modal AI model as illustrated in FIG. 2 using the preceding image as input, or the text or context vector output from an ASR model as illustrated in FIG. 5 using the preceding speech as input, or the text or text embedding output from these multi-modal AI models or ASR models as input, the context vector output from a language model as illustrated in FIG. 3 - the context vector containing information on associations between different words forming a word sequence - self-attention) can be directly utilized in the processing for advertisement selection, thereby reducing the computational resources and computational time required for the processing for advertisement selection.For example, in one embodiment of the present invention, each context vector extracted from the image (preceding image) and speech (preceding speech) of the preceding video frame may include context information of the preceding video frame, and the mutual matching suitability between the preceding video frame and the group of advertisement candidates may be analyzed by calculating the similarity between each context vector (the context vector regarding the preceding image and the context vector regarding the preceding speech) including the context information of the preceding video frame and the entity name or brand information regarding the advertisement target object stored in the database in connection with the group of advertisement candidates, and the mutual matching suitability between the preceding video frame and the group of advertisement candidates may be performed one-to-one for each advertisement forming the group of advertisement candidates in a relationship with a specific preceding video frame preceding the selected advertisement interposition position, and the advertisement having the highest mutual matching suitability may be selected according to the similarity calculated for each advertisement candidate, and may be selected as an in-stream advertisement to be played at the corresponding advertisement interposition position.

[0130] In a first type of mutual matching suitability applicable to one embodiment of the present invention, multi-modal data regarding preceding video frames extracted from a search for an advertisement intervention position prior to advertisement selection, such as a context vector regarding each preceding image (preceding image) and a context vector regarding the preceding speech (preceding speech), and a similarity analysis (similarity analysis according to natural language processing) between the named entity and brand information of an advertisement target object stored in connection with each of a group of advertisement candidates, can be used to select an advertisement to be played at the corresponding advertisement intervention position. For example, the named entity of the advertisement target object in this specification may mean a general name used to classify and recognize the class of the object that is the advertisement target of the in-stream advertisement, and the brand information may comprehensively mean brand-related information provided by the advertiser of each in-stream advertisement, including brand information (brand name), and, for example, may comprehensively mean various brand-related information including images that the advertiser of each in-stream advertisement intends to promote or spread about its own brand.

[0131] FIG. 8 is a diagram illustrating a language model or embedding model that takes text as input and maps it to a text embedding space, and illustrates mapping of input texts to embedding spaces at adjacent locations according to their similarity.

[0132] In various embodiments of the present invention, context vectors regarding images (preceding images) and speeches (preceding speeches) as multi-modal data forming the preceding video frame may each include context information of the preceding video frame, and may be expressed as text embeddings on the same text space (see FIG. 8) or the same text embedding space (see FIG. 8). The named entity and brand information of the advertisement target object for each of the above group of advertisement candidates can also be converted into text embedding to calculate the similarity with the context vector containing the context information of the preceding video frame (for example, as shown in FIG. 8, the named entity and brand information of the advertisement target object can be mapped onto a text space or a text embedding space from an embedding model or a language model), and in various embodiments of the present invention, the named entity and brand information of the advertisement target object for each of the above group of advertisement candidates can be mapped onto a text embedding space (see FIG. 8) in which a context vector containing context information on the preceding video frame (a context vector on the preceding image and a context vector on the preceding speech) is embedded, and according to the Euclidean distance or cosine similarity between the context vector on the preceding video frame and the named entity and brand information of the advertisement target object, which are mapped onto the same text embedding space (see FIG. 8), the preceding video preceding the selected advertisement interposition position The similarity between a context vector containing context information of a frame and each advertisement candidate forming a group of advertisement candidates can be calculated.

[0133] In various embodiments of the present invention, a multi-modal AI model (see FIG. 2) that takes as input the image data (image) of the preceding image frame and learns the bi-modal embedding space of image-text can output a text embedding that matches the corresponding image embedding from the image data or image of the input preceding image frame, and can output a text embedding that includes a vector representation of the text. In addition, an ASR model (see FIG. 5) that takes as input the audio data (speech or utterance) of the preceding image frame can output a text embedding that includes a vector representation of the text as a text embedding from the utterance data of the preceding image frame. In one embodiment of the present invention, text (e.g., word sequence) or text embedding output from the multi-modal AI model (see FIG. 2) and ASR model (see FIG. 5) can be input to a separate language model (see FIG. 3) so as to express context information including association information (e.g., self-attention) between each word forming the corresponding text (e.g., word sequence), and the language model (see FIG. 3) can perform an attention (self-attention) operation on each word forming the input text or word sequence.

[0134] In a first type of mutual matching suitability between a preceding video frame and an advertisement candidate that can be applied to one embodiment of the present invention, the text embedding (text embedding for the preceding image) output from each multi-modal AI model (see FIG. 2), the text embedding (text embedding for the preceding speech) output from the ASR model (see FIG. 5), and the entity name or brand information of the advertisement target object of the advertisement candidate can be mapped onto the same text space (see FIG. 8) or text embedding space (see FIG. 8) (e.g., the embedding model of FIG. 8), and, for example, the Euclidean distance or cosine similarity between different text embeddings mapped onto the same text embedding space (see FIG. 8) can be calculated, that is, the mutual relationship between different texts or text embeddings can be expressed, by mapping to the same text embedding space (see FIG. 8) to which a dense representation (distributed representation) or dense embedding is applied rather than a sparse representation, and for this purpose, Each text embedding (text embedding for preceding images and preceding speeches) output from the multi-modal AI model (see FIG. 2) and ASR model (see FIG. 5) and the entity name and brand information of the advertising target object can be mapped to the same embedding space (see FIG. 8) by performing an operation with an embedding matrix regarding the mapping between vocabulary and word tokens (see the embedding model of FIG. 8).

[0135] In this way, in the first type of mutual matching suitability between a preceding video frame and a group of advertisement candidates that can be applied in one embodiment of the present invention, in order to calculate mutual matching suitability between the multi-modal video data and voice data forming the preceding video frame and the entity name and brand information of the advertisement target object of the advertisement candidate, the context vectors (the context vector regarding the preceding image and the context vector regarding the preceding speech) output from each of the multi-modal AI models (see FIG. 2) and ASR models (see FIG. 5) can be input into a network (embedding model, see FIG. 8) that can be mapped to the same text space (see FIG. 8) or text embedding space (see FIG. 8) for mutual matching suitability or similarity analysis with a group of advertisement candidates, and through the network (embedding model, see FIG. 8), a group of advertisement candidates that are the target of mutual matching suitability or similarity analysis with the preceding video frame can also be mapped onto the space (see FIG. 8) in which the multi-modal data forming the preceding video frame is embedded, and the same text It can be embedded into a space (see Fig. 8) or a text embedding space (see Fig. 8).

[0136] For example, in a first type of mutual matching suitability applicable to one embodiment of the present invention, as multi-modal data forming a preceding video frame, each of the video data (preceding image) and the voice data (preceding speech, utterance) may be similarly mapped to a text embedding or context vector that matches or is most closely mapped to an image embedding of the preceding image input from a multi-modal AI model (see FIG. 2), a text embedding or context vector output from the preceding speech input from an ASR model (see FIGS. 5 and 6), and a context vector regarding the entity name or brand information of the advertising target object of the group of advertisement candidates, thereby calculating the mutual matching suitability between the preceding video frame and the group of advertisement candidates. In this way, in a first type of mutual matching suitability applicable to one embodiment of the present invention, a context vector output from a multi-modal AI model (see FIG. 2) to include context information throughout the preceding image, and a context vector output from an ASR model (see FIG. 5) to include context information throughout the preceding speech, A similarity analysis based on natural language processing can be performed between a context vector and the entity name and brand information of an advertisement target object of a group of advertisement candidates, for example, a similarity analysis based on natural language processing can be performed between a context vector including context information across the entire preceding image or preceding speech as text or text-embedded context vector, and a context vector including context information across the entire preceding image or preceding speech and a text embedding of the entity name and brand information of an advertisement target object of a group of advertisement candidates, that is, a context vector regarding a preceding image embedded in the same text space (see FIG. 8) or text embedding space (see FIG. 8),The mutual matching suitability between the preceding video frame and a group of advertisement candidates can be calculated by natural language processing that calculates the Euclidian distance or cosine similarity between the context vector regarding the preceding speech and the text regarding the entity name and brand information of the advertisement target object after the advertisement. In other words, in the first type of mutual matching suitability applicable to one embodiment of the present invention, the mutual matching suitability can be calculated based on natural language processing, and for example, the shape features of the object may be extracted from the image forming the preceding video frame and the image of the advertisement candidate itself, and a similarity analysis on the features on the images may not be performed.

[0137] <Type 2 Mutual Matching Fit Between Preceding Video Frames and Ad Candidates>

[0138] Figure 9a illustrates a diagram for explaining the extraction of styles corresponding to the correlation between different features (features 1 to 4) produced from the application of different kernels to extract different features from specific layers of a CNN network into which different images are input, for the purpose of analyzing the similarity between different images.

[0139] Figure 9b illustrates a diagram for explaining the extraction of a Gram matrix regarding the correlation or style between different features (features 1 to 4) extracted from different images.

[0140] In one embodiment of the present invention, in calculating the mutual matching suitability between a preceding video frame preceding a selected advertisement intervention location and a group of advertisement candidates, a style forming an image of the preceding video frame (preceding image) and a style forming an image of each advertisement candidate forming the group of advertisement candidates are extracted, and by analyzing the similarity between the styles extracted from different images in this way, the mutual matching suitability between the preceding video frame and the group of advertisement candidates can be calculated. For example, in one embodiment of the present invention, a style may be defined as a correlation between different features forming an image, and for example, the different features forming an image may include different features forming an image, such as a profile such as a line or a surface, or a color tone. For example, in one embodiment of the present invention, style may mean that when feature 1 has a high activation value among different features 1 and 2 extracted from an image of a preceding video frame or an image of an advertisement candidate, feature 2 also has a high activation value, and for example, a Gram matrix containing information about style is calculated by taking a dot product of vectors representing the activation values ​​of feature 1 and feature 2, and the similarity between the Gram matrices is analyzed, that is, the mutual matching suitability between the image of the preceding video frame and the image of the advertisement candidate can be calculated from the similarity analysis between the Gram matrix containing style information extracted from the image of the preceding video frame and the Gram matrix containing style information extracted from the image of the advertisement candidate.

[0141] More specifically, in a CNN (convolution neural network) network that performs a classification task of taking an image as input and extracting shape features (outlines, color tones, etc.) from the input image and calculating the probability (confidence) corresponding to each class, as the layers deepen from the input end where the image is input toward the output end, the number of channels forming each layer increases while the matrix dimension of each channel (e.g., the matrix dimension expressing a set of pixels) decreases, and at this time, the feature maps (feature maps, see FIG. 9a) forming different channels forming a specific layer of the CNN network can be understood as extracting different features (different outlines, different color tones, etc., features 1 to 4) of the input image, and in one embodiment of the present invention, the result values ​​of a specific layer of a specific CNN network, that is, a CNN network with fixed weights (e.g., different channels forming a specific layer or a synthesis product to which different filters or different kernels are applied, for example, each channel, filter, and kernel are each different from each other) The correlation between different features (features 1 to 4) expressed by the feature map (see Fig. 9a) that forms the feature map (which can extract different outlines or color tones, etc.) can be defined as a style, and for example, in one embodiment of the present invention, a Gram matrix (see Fig. 9b) expressing the correlation between different features (features 1 to 4) is generated, and the generated Gram matrix itself can be understood as containing style information of the corresponding image.For example, referring to FIG. 9a, the image of the preceding video frame and the image of the advertisement candidate can be input into a CNN series network (e.g., VGG net) with the same parameters or weights fixed, and at this time, different filters or kernels for extracting different features (features 1 to 4) from specific layers along the depth direction of the CNN series network are applied, so that feature maps in which different features (features 1 to 4) are extracted from the preceding video frame and the image of the advertisement candidate can be generated, and a Gram matrix (see FIG. 9b) in which the correlation between each feature (correlation, inner dot product between features 1 to 4) is expressed between the features (features 1 to 4) extracted from these linear video frames and the image of the advertisement candidate can be produced.

[0142] In one embodiment of the present invention, the Gram matrix (see FIG. 9b, G ij ) can be expressed as follows, and can be expressed in the form of a matrix whose elements are the dot products between each feature i and j to express the correlation between different features i and j for the total number k of channels forming a specific layer L of a specific CNN network.

[0143]

[0144] For example, the Gram matrix (see FIG. 9b) selects a specific layer of a specific CNN network (e.g., VGG net as a type of CNN network) and generates a Gram matrix that is generated as an inner product of different channels or different features i and j that form a specific layer, and analyzes the similarity between the image of the preceding video frame and the image of the advertisement candidate based on the difference values ​​between each element of the different Gram matrices of the Gram matrix generated from the image of each preceding video frame and the Gram matrix generated from the image of the advertisement candidate, and more specifically, the mutual matching suitability between the preceding video frame and the image of the advertisement candidate can be calculated by dividing the sum of the difference values ​​between each element of the different Gram matrices of the Gram matrix generated from the image of the preceding video frame and the Gram matrix generated from the image of the advertisement candidate by a scaling factor or calculating the mean squared error (MSE) that is the average of the sum of the difference values ​​between each element of the different Gram matrices of the Gram matrix generated from the image of the preceding video frame or the sum of the squares of the difference values ​​between each element of the Gram matrix or the sum of the squares of the difference values ​​between each circle of the Gram matrix by a scaling factor, and calculating the MSE (mean squared error) between each element of the Gram matrix that contains the style information of the preceding video frame and the advertisement candidate. The advertisement candidate that minimizes the sum of the difference values ​​(or the sum of the squares of the difference values, etc.) can be selected as the advertisement to be played at the selected advertisement intervention position.

[0145] <Type 3 Mutual Matching Fit Between Preceding Video Frames and Ad Candidates>

[0146] FIG. 10 is a diagram illustrating a histogram that shows different pixel values ​​and the number of pixels in which each different pixel value appears in R, G, and B images synthesized at the channel level to form each image for similarity analysis between different images.

[0147] In the second type of mutual matching suitability as described above, style information is extracted from each of the preceding video frames preceding the selected advertisement interposition location and a group of advertisement candidates (see FIGS. 9a and 9b), and similarity analysis is performed between the extracted style information, thereby calculating the mutual matching suitability between the preceding video frames and the group of advertisement candidates. As described above, calculating a Gram matrix (see FIG. 9b) regarding the correlation between different features expressed on each image as style information, and calculating the similarity (for example, the sum of the difference values ​​between each element forming different Gram matrices) between the Gram matrices containing the style information of the images of each preceding video frame and the images of the group of advertisement candidates may be based on defining the style for each image as the correlation between different features (features 1 to 4) in one embodiment of the present invention, and here, the different features expressed on the images may include contour information such as lines or planes, as well as images such as color tones or color hues. It may contain color information to form.

[0148] In a third type of mutual matching suitability between a preceding video frame and a group of advertisement candidates that can be applied to one embodiment of the present invention, color tone or color sense as a feature of an image is more specifically defined as information on the distribution of colors forming an image, and as information on the distribution of colors, a histogram (see FIG. 10) expressing the number of pixels in which a corresponding pixel value appears for each pixel value can be extracted from an R, G, B three-channel image (or a Y, Cb, Cr three-channel image) that is synthesized with respect to each other to express different colors can be extracted. For example, in one embodiment of the present invention, a histogram (see FIG. 10) containing color tone or color sense information can be extracted from each image of a preceding video frame and an advertisement candidate, which are targets of calculating mutual matching suitability, and the mutual matching suitability between the corresponding preceding video frame and the group of advertisement candidates can be calculated through a similarity analysis between the histograms (see FIG. 10) extracted from each image.

[0149] In one embodiment of the present invention, a histogram (see FIG. 10) expressing color tone or color tone information from each preceding video frame or image of an advertisement candidate is a different three-channel image that is synthesized with respect to each other to express different colors in each image, and can express the number of pixels in each of the three-channel images in which the corresponding pixel value for each pixel value appears in the R,G,B three-channel image in the R,G,B color space or the Y,Cb,Cr three-channel image in the Y,Cb,Cr color space. For example, in the third type of the mutual matching suitability, the mutual matching suitability between the preceding video frame and the advertisement candidate can be calculated based on the first to third histograms (see FIG. 10, R,G,B) or the first to third histogram information extracted from each of the three-channel images forming the image of the preceding video frame and the image of the advertisement candidate.

[0150] In one embodiment of the present invention, in the third type of mutual matching fitness applicable to calculating the mutual matching fitness between a preceding video frame and a group of advertisement candidates, as in the second type described above, a CNN network (see FIG. 9a, for example, a VGG net, which is a type of CNN network) for extracting a Gram matrix (see FIG. 9b) containing style information from each of the preceding video frames and the images of the group of advertisement candidates is not required, and color tone or color saturation information can be extracted from each image without computational burden, for example, from pixel information of each image itself or different three-channel images that are synthesized with respect to each other to form each image itself.

[0151] In one embodiment of the present invention, the histogram (see FIG. 10) containing the color tone or color saturation information represents the number of pixels in which the corresponding pixel value appears for each pixel value for each 3-channel image, and can be expressed in the form of a histogram matrix with (pixel value, number of pixels) as the coordinates of the matrix, and for example, like a Gram matrix, by calculating the sum of the difference values ​​for each element in the histogram matrix extracted from images of different preceding video frames and advertisement candidates, or the sum of the squares of the difference values ​​for each element (or the MSE mean squared error, which is the average of the sum of the squares of the difference values ​​between each element of the histogram matrix or the sum of the squares of the difference values ​​between each element of the histogram matrix), the mutual matching suitability between the preceding video frame and the advertisement candidate can be calculated. At this time, in one embodiment of the present invention, histograms extracted from different preceding video frames and advertisement candidates can be extracted for each of the three-channel images, and the sum of the difference values ​​or the sum of the squares of the difference values ​​with respect to the number of pixels appearing for each pixel value on the three-channel image, for example, on the first to third channel images forming the three-channel image, in the first to third histogram matrices containing the color tone or color tone information of the first to third channel images, the sum of the squares of the difference values ​​of the different first to third histogram matrices extracted from each of the different preceding video frames and advertisement candidate images can be calculated, thereby calculating the mutual matching suitability between the preceding video frame and the advertisement candidate.In this way, in the third type of mutual matching suitability that can be applied to one embodiment of the present invention, color tone or color information as a feature forming the image of each preceding video frame and the advertisement candidate can be extracted from the number of pixels appearing for each pixel value on the three-channel image forming each image itself from the image of the preceding video frame and the advertisement candidate itself, that is, without performing separate image processing on the image of the preceding video frame and the advertisement candidate, and through a similarity analysis of such color tone or color information, the mutual matching suitability between the preceding video frame and the advertisement candidate can be calculated.

[0152] As described above, in one embodiment of the present invention, in the second type of mutual matching suitability between a preceding video frame and an advertisement candidate, the similarity analysis of the Gram matrix (see FIG. 9b) extracted from each preceding video frame and advertisement candidate, and in the third type of mutual matching suitability between a preceding video frame and an advertisement candidate, the similarity analysis of the histogram matrix (see FIG. 10, histogram) extracted from each preceding video frame and advertisement candidate can be performed based on the difference values ​​between each element of each different matrix (Gram matrix and histogram matrix), and in various embodiments of the present invention, the difference values ​​between each element of each different matrix (Gram matrix and histogram matrix) can comprehensively mean all of the sum of the difference values ​​between each element, the sum of the squares of the difference values ​​between each element, and the mean square error (MSE) obtained by dividing the sum of the difference values ​​between each element by a scaling factor.

[0153] <Mutual Matching Fit Between Preceding Video Frames and Ad Candidates: Type 4>

[0154] Figure 11a illustrates a diagram for explaining an R-CNN network as an architecture for explaining object recognition that predicts the boundary of an object expressed in an image in the form of a bounding box.

[0155] Figure 11b illustrates a diagram for explaining the Fast R-CNN network as another architecture for explaining object recognition that predicts the boundaries of objects expressed on an image in the form of bounding boxes.

[0156] Figure 11c illustrates a diagram for explaining the Faster R-CNN network as another architecture for explaining object recognition that predicts the boundaries of objects expressed on an image in the form of bounding boxes.

[0157] FIG. 11d is a diagram illustrating an example of a Region Proposal Network (RPN) of the network illustrated in FIG. 11c.

[0158] Referring to FIGS. 11a to 11d, the class of an object appearing in a preceding video frame preceding a selected advertisement intervention location and the boundary surrounding the object can be predicted as a bounding box (Bbox), and a similarity analysis is performed between the predicted object class and the object area surrounded by the bounding box (Bbox) and the name of the advertisement target object of the advertisement candidate and the object area in which the advertisement target object appears, and based on the similarity analysis between each preceding video frame and the object area of ​​the advertisement candidate, the mutual matching suitability between the preceding video frame and the advertisement candidate can be calculated.

[0159] In one embodiment of the present invention, in the fourth type of mutual matching suitability between the preceding video frame and the advertisement candidate, object detection or image segmentation can be performed on the image of the preceding video frame (preceding image), and the class of the object appearing on the image of the preceding video frame and the boundary surrounding the object area can be predicted in the form of a bounding box (Bbox) (object recognition) or the object area can be predicted pixel by pixel (image segmentation). To this end, the in-stream advertisement matching network according to one embodiment of the present invention can include a region extraction model for extracting the class of the object appearing on the image of the preceding video frame (preceding image) and the object area (for example, a network of the R-CNN series that performs object recognition as illustrated in FIGS. 11a to 11d or a network such as U-net that performs image segmentation). For example, the class of an object predicted from the above region extraction model may mean, for example, the entity name of an object appearing in a preceding image, and by predicting an object region (e.g., a bounding box Bbox, see FIG. 11a) in which an object appears in the preceding image, and calculating the probability that each object appearing in each predicted object region belongs to a plurality of classes specified in advance, the object appearing in the corresponding object region can be classified as belonging to the class predicted with the highest probability.

[0160] In one embodiment of the present invention, based on a similarity analysis between the class of the object recognized from the image of the preceding video frame (the entity name of the recognized object) and the entity name of the advertisement target object, an advertisement candidate for the advertisement target object having a relatively high similarity to the class of the object recognized from the image of the preceding video frame (the preceding image) preceding the selected advertisement intervention position among a group of advertisement candidates can be determined to have a high degree of mutual matching suitability with the preceding video frame preceding the selected advertisement intervention position.

[0161] In one embodiment of the present invention, an object region (e.g., a bounding box Bbox, see FIG. 11a) in which an object appears is predicted from an image of a preceding video frame (a preceding image), and a similarity analysis of the images between the predicted object region and the object region in which an advertisement target object appears in an image of an advertisement candidate can be used to calculate a mutual matching suitability between the preceding video frame and the advertisement candidate. For example, in one embodiment of the present invention, features are extracted for a plurality of object regions (e.g., a bounding box Bbox, see FIG. 11a) predicted from the preceding image, and an object region in which an advertisement target object of each advertisement candidate forming a group of advertisement candidates appears, and a mutual matching suitability between the preceding video frame and the advertisement candidate can be calculated based on the similarity between the extracted features. For example, in one embodiment of the present invention, the region extraction model can extract a plurality of object regions (e.g., bounding boxes Bbox, see FIG. 11a) in which different objects appear on an image of a preceding video frame (preceding image), can extract features from each object region, and can calculate a mutual matching suitability between the preceding video frame and the advertisement candidate through a similarity analysis between the features extracted from each object region (e.g., bounding boxes Bbox, see FIG. 11a) and the features extracted from the object region of the advertisement candidate in which the advertisement target object appears.

[0162] The above region extraction model inputs an object region extracted from an image of a preceding video frame (preceding image) and an object region of an advertisement target object into a CNN (convolution neural network) network, and at this time, in one embodiment of the present invention, the object region of the preceding image (e.g., bounding box Bbox, see FIG. 11a) and the object region of the advertisement target object can be input into the same CNN network, and for example, the similarity between the feature vector or feature map (see FIG. 9a) extracted therefrom can be determined by inputting them into a CNN network including the same weight. For example, the CNN network can apply different channels (filters or kernels) to extract different features (features 1 to 4) from the input object region, and at the output end, flattening can be performed to arrange the result values ​​of the synthesis multiplication of these different channels into one dimension. At this time, in one embodiment of the present invention, the Euclidean distance between the object region of the preceding image (e.g., the bounding box Bbox, see FIG. 11a) and the one-dimensional feature vector extracted from the object region of the advertisement candidate or the cosine similarity between them can be calculated, or the similarity analysis between the result values ​​of a specific layer of the CNN network, for example, the result values ​​of the synthesis multiplication to which different channels or different filters or different kernels are applied in a specific layer (corresponding to a feature map, see FIG. 9a), can be performed to analyze the similarity between the object image (e.g., the bounding box Bbox, see FIG. 11a) extracted from the preceding image frame and the object image of the advertisement target object.For example, different channels applied in specific layers of the CNN network can extract different features from each object region, for example, different features such as different shapes of outlines among lines, planes, and outlines, or color information regarding color schemes (features 1 to 4, see FIG. 9a). For example, different channels or filters applied in specific layers of the CNN network with the same weights can extract different features (features 1 to 4, see FIG. 9a) from each input object region, and if features (features 1 to 4) extracted from the same CNN network (e.g., VGG net) are arranged in similar positions (the result of the synthetic multiplication to which each channel is applied can include position information of the feature), different objects appearing in substantially different object regions can be determined to be the same similar objects.

[0163] For example, in one embodiment of the present invention, by applying channels or filters having different weights to the object region (e.g., bounding box Bbox, see FIG. 11a) extracted from the image of the preceding video frame (preceding image) and the object region of the advertisement target object, different features (features 1 to 4) are extracted, and by detecting that the same similar features are extracted from the same similar positions on the object region, it is possible to determine that the objects appearing on these different object regions are the same and similar.

[0164] In one embodiment of the present invention, a channel or filter having the same weight is applied to an object region (e.g., a bounding box Bbox, see FIG. 11a) extracted from an image of the preceding video frame and an object region of an advertisement target object, and a feature map (see FIG. 9a) corresponding to a result of synthesizing and multiplying the channels or filters having the same weight is calculated one-to-one for each element forming the feature map (see FIG. 9a), and the difference values ​​are accumulated in the form of summing or squaring the difference values ​​and summing them, and an object region having a relatively small difference value of the feature map (see FIG. 9a) thus calculated can be matched to each other, and for example, a preceding video frame including an object region in which a relatively small difference value is calculated and an advertisement candidate can be determined to have high mutual matching suitability. For example, referring to FIG. 9a, the mutual matching fitness between the preceding video frame and the advertisement candidate can be calculated based on the differential values ​​of each element forming the different features (features 1 to 4) produced by applying different filters or kernels for extracting different features (features 1 to 4) from specific layers of the same CNN network, which include the same parameters or weights, into which the image of the object region of the preceding video frame and the image of the object region of the advertisement candidate are respectively input, and for example, the mutual matching fitness between the preceding video frame and the advertisement candidate can be calculated based on the differential values ​​between the same features (features 1 to 4) extracted from the image of the object region of the preceding video frame and the image of the object region of the advertisement candidate, respectively.

[0165] In one embodiment of the present invention, as described above, unlike the third type of mutual matching fitness in which the mutual matching fitness between the preceding video frame and the advertisement candidate is calculated by calculating the Gram matrix (see FIG. 9b, correlation between features 1 to 4) for the entire image of each preceding video frame and the Gram matrix (see FIG. 9b, correlation between features 1 to 4) for the entire image of the advertisement candidate as the mutual matching fitness between the preceding video frame and the advertisement candidate, and calculating the similarity between the Gram matrix (see FIG. 9b, correlation between features 1 to 4) extracted from the entire image of these preceding video frames and the entire image of the advertisement candidate, the mutual matching fitness between the preceding video frame and the advertisement candidate is calculated by extracting an object region (see FIG. 11a, for example, bounding box Bbox) from the image of the preceding video frame, extracting features (features 1 to 4, see FIG. 9a) from the extracted object region and the object region of the advertisement target object (see FIG. 11a, for example, bounding box Bbox), and considering the positional relationship between the extracted features and the extracted features, the mutual matching fitness between the corresponding preceding video frame and the advertisement candidate is calculated. Can be.For example, in one embodiment of the present invention, the object region (see FIG. 11a, for example, bounding box Bbox) extracted from the image of the preceding video frame and the object region of the advertisement target object (see FIG. 11a, for example, bounding box Bbox) are input to a specific convolution neural network (CNN) network (see FIG. 9a), and the difference value of the feature map (see FIG. 9a, features 1 to 4) including the result value of synthesizing and multiplying different channels or filters for extracting different features (see FIG. 9a, features 1 to 4) applied in a specific layer of the CNN network can be calculated, and the difference value is calculated for each element forming the feature map (see FIG. 9a, features 1 to 4) generated by inputting different object regions, that is, the object region extracted from the image of the preceding video frame and the object region of the advertisement target object (see FIG. 11a, for example, bounding box Bbox) and accumulating the accumulated value in the form of a sum or a form of a square (for example, MSE, mean square error). Based on this, it can be determined that the mutual matching suitability between the different object regions extracted from the feature maps (see Fig. 9a, features 1 to 4) with relatively small difference values ​​or the preceding video frame and the advertisement candidate including each object region is high.

[0166] In the fourth type of mutual matching suitability applicable to one embodiment of the present invention, since only the object region (see FIG. 11a, for example, bounding box, Bbox) in which each object appears is extracted from the preceding video frame and the advertisement candidate, the difference value is calculated for each element of the feature map (see FIG. 9a, features 1 to 4) extracted from the object region of each preceding video frame and the advertisement candidate and the feature map (see FIG. 9a, features 1 to 4) including the positional information of the extracted feature, thereby enabling more precise implementation of the comparison between the shape features of the objects, and the similarity between the object appearing in the object region of the preceding video frame and the advertisement target object as the object appearing in the object region of the advertisement candidate can be inferred with higher precision.

[0167] In this way, in one embodiment of the present invention, by extracting an object region (e.g., bounding box, Bbox, FIG. 11a) in which an object appears rather than the entire region of a preceding video frame, and extracting an object region (e.g., bounding box, Bbox, FIG. 11a) in which an advertisement target object appears rather than the entire advertisement candidate region, the element-by-element difference value of a feature map (e.g., feature 1 to 4, FIG. 9a) including both features (e.g., feature 1 to 4, FIG. 9a) and feature location information of each object region (e.g., bounding box, Bbox, FIG. 11a) can be calculated, and the feature-by-feature location information such as the location of the object where the feature (e.g., feature 1 to 4, FIG. 9a) of the object appearing in each object region (e.g., bounding box, Bbox, FIG. 11a) is extracted can be compared, and the basis data (feature map, FIG. 11a) for more precisely determining whether the object appearing in each object region (e.g., bounding box, Bbox) is similar to each other is provided. 11a, the mutual matching fitness between the preceding video frame and the advertisement candidate can be calculated from the element-wise difference values ​​of features 1 to 4. For example, in the third type of mutual matching fitness between the preceding video frame and the advertisement candidate, the Gram matrix (see Fig. 9b) that is the comparison target between the preceding video frame and the advertisement candidate is calculated by calculating the correlation between different features (see Fig. 9b, features 1 to 4), and such a Gram matrix may not include location information about any one feature (see Fig. 9b, features 1 to 4) other than the correlation between different features.

[0168] However, in one embodiment of the present invention, even in the fourth type of the mutual matching suitability, rather than calculating the difference value of the feature map (see FIG. 9a, features 1 to 4) itself from which each different feature (see FIG. 9a, features 1 to 4) is extracted from each object region (see FIG. 11a, for example, bounding box Bbox) extracted from each preceding video frame and advertisement candidate, a Gram matrix (see FIG. 9b) may be extracted from each object region (see FIG. 11a, for example, bounding box Bbox) extracted from each preceding video frame and advertisement candidate so as to analyze a style that can be defined as a correlation between different features expressed on an image between the preceding video frame and the advertisement candidate, and a similarity analysis may be performed between the Gram matrices (see FIG. 9b) extracted from each object region (see FIG. 11a, for example, bounding box Bbox). In this way, the calculation of the Gram matrix (see FIG. 9b) for the object region (see FIG. 11a, e.g., bounding box Bbox) extracted from each preceding video frame and advertisement candidate is substantially the same as that described in the third type of mutual matching fitness described above, and therefore, a detailed description thereof will be omitted here.

[0169] As described above, in order to extract an object region (e.g., bounding box Bbox, see FIG. 11a) from an image of a preceding video frame and an advertisement candidate, and to input the extracted object region into the same CNN network to extract a feature map (e.g., features 1 to 4, see FIG. 9a) and to calculate a difference value for each element forming the feature map (e.g., FIG. 9a), it may be required that the image of the preceding video frame (preceding image) and the object region (e.g., bounding box Bbox, see FIG. 11a) extracted from the advertisement candidate have the same resolution. In one embodiment of the present invention, image magnification that increases the resolution or image reduction that lowers the resolution can be performed on the object region (e.g., bounding box Bbox, see FIG. 11a) extracted from the preceding video frame and the advertisement candidate so that the number of pixels in the row direction and the column direction forming the image are the same. For example, the above resolution adjustment is to make the size (number of pixels in a row and column) of the array of pixel sets forming the object area of ​​the preceding image frame (e.g., bounding box Bbox in FIG. 11a) and the object area of ​​the advertisement candidate (e.g., bounding box Bbox in FIG. 11a) match each other, and to enable the feature maps (see FIG. 9a, features 1 to 4) having the same screen size (number of pixels in a row and column, resolution) to be output when input to the same CNN network, and can be implemented by a network that implements up-sampling or down-sampling and can map a feature map of a relatively high resolution to a feature map of a relatively high resolution.For example, in one embodiment of the present invention, the up-sampling and down-sampling can be implemented from a CNN series network such as a transposed convolution and a convolution, respectively, and, for example, an output image having a resolution scaled-up from the input image can be generated from a transposed convolution that intervenes an intermediate grid between each pixel forming a row and a column on an input image for an object region (e.g., a bounding box Bbox in FIG. 11a) and applies a filter or kernel.

[0170] <Type 5 Mutual Matching Fit Between Preceding Video Frames and Ad Candidates>

[0171] Figure 12 illustrates an exemplary architecture of a network that implements comprehensive sentiment classification based on multi-modal data of speech and image by combining predictions from a speech-based sentiment classification model and an image-based sentiment classification model.

[0172] Figure 13 is a diagram illustrating a space of emotion classification including 12 radial levels of emotion classification, with active and passive at both ends along the Arousal axis and negative and positive at both ends along the Valence axis, on a two-dimensional plane having an Arousal axis and a Valence axis.

[0173] In one embodiment of the present invention, emotion classification (emotion recognition) can be performed from multi-modal data of video data (image) and audio data (speech, utterance) of a preceding video frame, and for example, emotion classification can be classified into six-stage emotion classes (or AU, action unit) such as anger, disgust, fear, happiness, sadness, and surprise as discretely classified different emotion classes, and can be classified into 12-stage emotion classes radially arranged with active and passive at both ends along the Arousal axis and negative and positive at both ends along the Valence axis on a two-dimensional plane having an Arousal axis and a Valence axis as continuous emotion classification (see FIG. 13).

[0174] In one embodiment of the present invention, an emotion classification model based on image data (image) for recognizing a face region from an image of a preceding image frame and classifying the recognized face region according to a stage of a pre-specified emotion class may be included (see FIG. 13). For example, in one embodiment of the present invention, the emotion classification model based on the image data (see FIG. 13) may include a CNN (convolution neural network) model, and the CNN model may include a plurality of images of a face region as learning data and a weight learned from a target label given as one of classes of a pre-specified emotion classification as a target label for each image of the face region. Referring to FIG. 13, a CNN network as an emotion classification model based on the image data may include an architecture in which a plurality of layers are stacked in the depth direction from an input end where a face region is extracted from a preceding image frame and an image of an advertisement candidate and an image of the extracted face region is input to an output end, and each layer may include a convolution for extracting features from an input image and a pooling (such as max pooling or average pooling) for reducing the number of pixels forming a matrix, and may include a dropout (not shown) for reducing the number of neurons of the convolution to prevent over-fitting. The emotion classification network based on the image data may calculate the probability of belonging to each emotion class classified in advance through a linear fully connected layer.

[0175] In one embodiment of the present invention, the emotional classification model may include an emotional classification model based on audio data (see FIG. 13) together with an emotional classification model based on video data, and may calculate a mutual matching suitability between a corresponding preceding video frame and an advertisement candidate according to multi-modal data of the video data and the audio data. For example, the sentiment classification model based on the speech data (see FIG. 13) divides the speech data or utterance data of the speech of the preceding video frame or the speech of the advertisement candidate into a fixed time unit (15 msec) and applies a Fourier transform to decompose the speech data or utterance data into different frequency components for each time unit, thereby generating a spectrogram (see FIG. 4a) in which the intensity (or strength) of each frequency is expressed for each time unit while being divided into unit time along the time axis, and in this way, the sentiment classification model based on the speech data (see FIG. 13) can perform preprocessing of the speech data to convert the speech data or utterance data of the speech of the preceding video frame or the speech of the advertisement candidate into a vector form (or a two-dimensional matrix form) suitable for DNN (deep neural network) processing, and for example, in various embodiments of the present invention, in the preprocessing of the speech data for the speech data, Mel scaling is performed to give more weight to the speech data of low frequencies in consideration of the auditory characteristics that are sensitive to low frequencies. A spectrogram (see Fig. 4b) can also be generated.For example, in one embodiment of the present invention, a video data-based emotional classification model (see FIG. 13) that predicts emotional classification based on the video data of the preceding video frame and the advertisement candidate (the preceding image and the advertisement candidate image) and a voice data-based emotional classification model (see FIG. 13) that predicts emotional classification based on the voice data of the preceding video frame and the advertisement candidate (the preceding speech and the advertisement candidate speech) may include a CNN series network, and for example, the video data (the preceding image and the advertisement candidate image) input to the video data-based emotional classification model (see FIG. 13) may be expressed as two-dimensional matrix data of a pixel set suitable for processing by a CNN series network, and similarly, the voice data (the preceding speech and the advertisement candidate speech) input to the voice data-based emotional classification model (see FIG. 13) may be preprocessed as two-dimensional matrix data of a spectrogram (see FIGS. 4a and 4b) suitable for processing by a CNN series network.

[0176] In one embodiment of the present invention, the sentiment classification model based on the voice data (see FIG. 13) may include a CNN model that takes as input a spectrogram preprocessed from the voice data or utterance data of speech, and calculates a probability that the spectrogram belongs to a class related to a pre-specified sentiment classification. For example, the CNN model may include a plurality of training data, such as voice data or utterance data, and a weight learned from a target label assigned to each training data as one of the classes of the pre-specified sentiment classification. Referring to FIG. 13, a CNN network as an emotion classification model based on the speech data may include an architecture in which a plurality of layers are stacked in the depth direction from an input end into which two-dimensional data in the form of a spectrogram regarding the speech of a preceding video frame and an advertisement candidate or the speech of a preceding video frame and an advertisement candidate are input to an output end, and each layer may include a convolution for extracting features from an input image and a pooling (such as max pooling or average pooling) for reducing the number of pixels forming a matrix, and may include a dropout (not shown) for reducing the number of neurons of the convolution to prevent over-fitting. The emotion classification network based on the speech data may calculate the probability of belonging to each emotion class classified in advance through a linear fully connected layer.In one embodiment of the present invention, the sentiment classification model based on voice data (see FIG. 13) may include a network for sequence processing to extract context information of a preceding speech or a speech of an advertisement candidate from information input at each time step along the time axis, and to produce an output of each time step (output of a hidden layer) from an input value and an output of a hidden layer of a previous time step, and to produce a final output that cumulatively contains information from a previous time step as a final output while advancing the time step, and may include, for example, a network such as a recurrent neural network (RNN) or a long short term memory (LSTM).

[0177] For example, in one embodiment of the present invention, the sentiment classification model may include a CNN model (see FIG. 13) based on image data having as input an image of a preceding image frame and an image of an advertisement candidate, and a CNN model (see FIG. 13) based on speech data having as input a speech of the preceding image frame and a speech of the advertisement candidate, and the CNN model (see FIG. 13) as the sentiment classification model based on the image data may include a matrix form in which a set of pixels forming an image corresponding to the image of the preceding image frame or the image of the advertisement candidate is arranged in a two-dimensional row and column, and may include a weight learned from a target label assigned to one of the classes of sentiment classification as a plurality of learning data images and a target label assigned to each image. In addition, as an emotion classification model based on the above voice data, the CNN model (see FIG. 13) may take as input a matrix form in which a spectrogram in which the intensity of the frequency is expressed by frequency component along the time axis as a speech of a preceding video frame or a speech of an advertisement candidate is arranged in a two-dimensional row and column, and may include a weight learned from a target label assigned to one of the classes of emotion classification as a target label assigned to each voice data as a plurality of learning data.

[0178] In one embodiment of the present invention, the classes of the emotional classification classified from the emotional classification model based on the video data and the emotional classification model based on the audio data can be collected to determine the mutual matching suitability between the preceding video frame and the advertisement candidate as a whole, and the emotional classification model according to one embodiment of the present invention can classify the preceding video frame and the advertisement candidate into six emotional classes such as anger, disgust, fear, happiness, sadness, and surprise as different emotional classes classified discretely, and it can be determined that the mutual matching suitability is high for the preceding video frame and the advertisement candidate classified into the class of the same emotional classification according to the emotional classification. However, in various embodiments of the present invention, the emotion classification model can set a space of emotion classification in a more continuous form for each discretely classified emotion classification class, and for example, as illustrated in FIG. 13, it can be classified into 12-stage emotion classification classes radially arranged with active and passive at both ends along the Arousal axis and negative and positive at both ends along the Valence axis on a two-dimensional plane having an Arousal axis and a Valence axis, and for each of the 12-stage emotion classification classes, based on the probability predicted from the emotion classification model (see FIG. 13), the emotion classification of each preceding video frame and the emotion classification of the advertisement candidate can be mapped onto the space of the same emotion classification (see FIG. 13), and in this way, based on the Euclidean distance or cosine similarity between the preceding video frame and the advertisement candidate mapped onto the space of the same emotion classification, the emotion classification of the preceding video frame and the advertisement candidate of the closest emotion classification can be determined to have a high degree of mutual matching fitness with the corresponding preceding video frame.In this way, in one embodiment of the present invention, rather than relying on a simplified emotional classification of matching or mismatching between preceding video frames and advertisement candidates classified according to classes of discrete emotional classification, for example, without class matching or mismatching between preceding video frames and advertisement candidates classified into one emotional class predicted with the highest probability among each emotional class for each preceding video frame and advertisement candidate, the advertisement candidate of the emotional classification mapped to the space of the emotional classification (see FIG. 13) according to the predicted probability of belonging to each class so as to be mapped into a space of more detailed or continuous emotional classification (see FIG. 13) and the emotional classification of the preceding video frame mapped to the space of the emotional classification (see FIG. 13) can be determined to have a high mutual matching suitability with the preceding video frame based on the distance between the emotional classification of the preceding video frame and the emotional classification of the advertisement candidate.

[0179] Referring to FIG. 13, a network for classifying each input preceding video frame and advertisement candidate into different emotional classes by inputting multi-modal data of video data and audio data of a preceding video frame and an advertisement candidate can predict a classification regarding a comprehensive emotional class by combining output values ​​from each of an emotional classification model based on video data and an emotional classification model based on audio data, and for example, referring to FIG. 13, a network for classifying a comprehensive emotional class by combining output values ​​of the emotional classification model based on video data and an emotional classification model based on audio data can include a layer for concatenating and flattening the outputs of the emotional classification model based on video data and the emotional classification model based on audio data, a linear fully connected layer, and a normalization layer, and for example, a comprehensive class classification regarding the emotional classification of the preceding video frame or advertisement candidate input from a network including these layers or a probability of belonging to each class can be calculated, and in one embodiment of the present invention, an output in the form of a vector expressing the emotional classification of the preceding video frame or advertisement candidate (for example, the probability of belonging to each emotional class) The emotional classification of the preceding video frame or advertisement candidate can be mapped from the vector-shaped output arranged in a one-dimensional form to the emotional classification space (see Fig. 13).

[0180] Similarity Analysis Between Preceding and Following Video Frames for Searching for Advertisement Intervention Locations: Type 1

[0181] The first to fifth types of mutual matching suitability as described above are calculated by calculating the mutual matching suitability in a one-to-one relationship between a preceding video frame preceding an advertisement interposition selected from an advertisement space detection model and each advertisement candidate forming a group of advertisement candidates. In various embodiments of the present invention, the first to fifth types of mutual matching suitability may also be applied to an advertisement space detection model for implementing a similarity analysis between a preceding video frame and a succeeding video frame to detect a scene change point between the preceding video frame and the succeeding video frame, and for selecting an advertisement interposition point at which an in-stream advertisement will be played by capturing a scene change point having a relatively low similarity between these preceding and succeeding video frames. More specifically, the first to fifth types of mutual matching suitability as described above may be applied to a similarity analysis between a preceding video frame and a succeeding video frame, as described below.

[0182] In the first type of similarity analysis between the preceding video frame and the succeeding video frame, as described above, the first similarity analysis based on natural language processing of the text embedding or the first context vector output from a multi-modal AI model (see FIG. 2) that has learned the multi-modal embedding space of image-text to generate context information of the preceding image and the succeeding image by inputting the video data of the preceding video frame and the succeeding video frame or the preceding image and the succeeding image, and the second similarity analysis based on natural language processing of the text embedding or the second context vector output from an ASR model (see FIGS. 5 and 6) that has the speech data of the preceding video frame and the succeeding video frame or the preceding speech and the succeeding speech as input and generates context information of the preceding speech and the succeeding speech, and a similarity score that synthesizes the results of these first and second similarity analyses can be calculated (see FIG. 7), and according to the similarity score that synthesizes the results of the first and second similarity analyses, the preceding video frame having relatively low similarity and the It can capture the position of advertisement insertion between trailing video frames.

[0183] Similarity Analysis Between Preceding and Following Video Frames for Searching for Advertisement Intervention Locations: Type 2

[0184] In one embodiment of the present invention, an advertisement space search model for capturing a scene transition point on a video stream of a video content layer and selecting the captured scene transition point as an advertisement interposition position where an in-stream advertisement will be played can implement a similarity analysis between a preceding video frame and a succeeding video frame, and can implement a second type of similarity analysis, and in the second type of similarity analysis, a style can be extracted from an image of a preceding video frame (a preceding image), a style can be extracted from an image of a succeeding video frame (a succeeding image), and a similarity analysis can be performed between the extracted style of the preceding image and the style of the succeeding image.For example, in one embodiment of the present invention, the preceding image and the succeeding image are input into the same CNN series network including the same parameters (see FIG. 9a), and a Gram matrix (see FIG. 9b) expressing the correlation between different features (see FIG. 9a, features 1 to 4) can be extracted while forming a specific layer designated in advance of the CNN series network, and a similarity analysis between the style extracted from each preceding image and the style extracted from the succeeding image can be performed based on the sum or mean square error (MSE) of the element-wise difference values ​​forming the matrix of each Gram matrix (see FIG. 9b) in the Gram matrix extracted from the preceding image and the Gram matrix extracted from the succeeding image, for example, targeting a plurality of frame sequences (see FIG. 1) down-sampled on a stream of video content, taking a preceding image frame and a succeeding image frame according to each time step, and analyzing the similarity of the styles extracted from each preceding image frame and the succeeding image frame, for example, different features extracted from each preceding image frame The similarity between the preceding video frame and the succeeding video frame can be estimated from a similarity analysis of the Gram matrix (see FIG. 9b) regarding the correlation (style) between the preceding video frame and the Gram matrix (see FIG. 9b) regarding the correlation (style) between different features extracted from the succeeding video frame, and for example, among a plurality of frame sequences downsampled in a stream of video content (see FIG. 1), the preceding video frame and the succeeding video frame having a relatively low, that is, the lowest, similarity can be selected as an advertisement interposition position for playing an in-stream advertisement.

[0185] Similarity Analysis Between Preceding and Following Video Frames for Searching for Advertisement Intervention Locations: Type 3

[0186] In one embodiment of the present invention, the advertisement space detection model may analyze the similarity between the preceding video frame and the succeeding video frame based on information about the color tone or color sensation of a set of pixels forming the image of the preceding video frame in a third type of similarity analysis between the preceding video frame and the succeeding video frame to capture a scene change point and select an individual advertisement position at which an in-stream advertisement will be played. For example, in the third type of similarity analysis between the preceding video frame and the succeeding video frame, the similarity between the preceding video frame and the succeeding video frame may be analyzed based on a histogram (see FIG. 10) or histogram information expressing the number of pixels appearing for each pixel value in each of the three-channel images synthesized with respect to each other to form the preceding video frame and the succeeding video frame, more specifically, the three-channel images synthesized with respect to each other in the R, G, B color space or the three-channel images synthesized with respect to each other in the Y, Cb, Cr color space.

[0187] For example, in one embodiment of the present invention, in the second type of analysis of the similarity between a preceding video frame and a succeeding video frame for selecting an advertisement intervention position at which an in-stream advertisement is to be played, when extracting different features from each of the preceding video frame and the succeeding video frame and producing a Gram matrix (see FIG. 9b) expressing the correlation of the different features, the different features extracted from the preceding video frame and the succeeding video frame (features 1 to 4, see FIG. 9a) may include different features expressed on the preceding video frame, such as different shapes of outlines, color tones, or color hues. In one embodiment of the present invention, in the third type for analyzing the similarity of a preceding video frame and a succeeding video frame for selecting an advertisement interposition position at which an in-stream advertisement will be played, unlike the second type described above, a histogram (see FIG. 10) or histogram information on color tone or color sensation directly extracted from the images of the preceding video frame and the succeeding video frame is extracted, so that, as in the second type described above, processing of a network that inputs the images of each of the preceding video frame and the succeeding video frame is not required, but from a three-channel image synthesized with respect to each other in the channel dimension to express color information on each of the preceding and succeeding images, a histogram (see FIG. 10) or histogram information on the number of appearances of pixel values ​​for expressing the gradation of each of the R, G, and B colors can be extracted from the histogram (see FIG. 10) on the pixel values ​​and the number of appearances of each pixel forming the images of the preceding and succeeding video frames, thereby extracting color tone or color sensation information of the entire image from the images of the preceding and succeeding video frames.

[0188] More specifically, in the third type of similarity analysis of preceding and succeeding video frames for selecting an advertisement intervention location, for each of the three-channel images extracted from each of the preceding and succeeding images, first to third histograms (see FIG. 10) expressing the number of times each pixel value appears, that is, the number of pixels in which a specific pixel value appears on each of the three-channel images, can be extracted, and based on each of the first to third histogram information, the first to third histograms extracted from the preceding and succeeding images can be expressed in the form of a histogram matrix with (pixel value, number of pixels) as the coordinates of the matrix, and, like the Gram matrix described above (see FIG. 9b), the similarity between the preceding and succeeding images, which are the targets of the similarity analysis, can be calculated based on the sum of the differential values ​​accumulated for each element forming the histogram matrix extracted from each different preceding and succeeding image or the square of the differential value (e.g., MSE, mean square error).

[0189] Similarity Analysis Between Preceding and Following Video Frames for Searching for Advertisement Intervention Locations: Type 4

[0190] In one embodiment of the present invention, in the fourth type of similarity analysis between a preceding video frame and a succeeding video frame for searching an advertisement intervention position where an in-stream advertisement is to be played, object detection or image segmentation can be performed on images of the preceding video frame and the succeeding video frame (the preceding image and the succeeding image), an object region (Fig. 11a, Bbox) can be extracted from each preceding image, and the class of the object appearing in the extracted object region (Fig. 11a, Bbox) can be predicted, and in this way, a similarity analysis between each preceding image and each succeeding image can be implemented by comparing the classes of the objects appearing in the object regions (Fig. 11a, Bbox) extracted from the preceding image and the succeeding image.

[0191] For example, in one embodiment of the present invention, a similarity analysis can be performed between a preceding image and a succeeding image depending on whether the class of an object appearing in a preceding image matches the class of an object appearing in a succeeding image, and, for example, the similarity between a preceding image and a succeeding image can be predicted depending on whether the classes of a plurality of objects appearing in the preceding image and the succeeding image are similar or the number of objects belonging to the same class among a plurality of classes, and more specifically, similar to a preceding image predicted as a scene with a large crowd of people, a succeeding image predicted as a scene with a large crowd of people can be predicted to have high similarity to each other depending on the extraction of an object region (Fig. 11a, Bbox) using the preceding image and the succeeding image as inputs and the class of an object appearing in the extracted object region (Fig. 11a, Bbox) and the number of objects belonging to the same class.For example, in various embodiments of the present invention, in the fourth type of similarity analysis between the preceding image and the succeeding image for selecting the advertisement intervention location where the in-stream advertisement is played, the position information of the object region (Fig. 11a, Bbox) extracted on the preceding image and the succeeding image may be considered together with the number of objects belonging to the same class as the class of the object appearing in the extracted object region (Fig. 11a, Bbox), and for example, in object detection using the preceding image and the succeeding image as input, the object region (Fig. 11a, Bbox) surrounding each object may be predicted to include the center coordinates of the object region (Fig. 11a, Bbox) and the information of the width (w) and the height (h) as the boundary of the object region (Fig. 11a, Bbox), and at this time, the center coordinates of the object region (Fig. 11a, Bbox) and the width (w) and the height (h) of the object region (Fig. 11a, Bbox) From the height (h), it is possible to recognize a major object that appears with a relatively high proportion in each preceding image and each succeeding image, and, depending on whether the classes of the major objects recognized from each preceding image and each succeeding image match, it is possible to predict the similarity between the preceding image and the succeeding image. For example, in one embodiment of the present invention, if an object of the same class as an object expressed over a relatively large object area (Fig. 11a, Bbox) from a relatively central position in the preceding image is expressed over a relatively large object area (Fig. 11a, Bbox) from a relatively central position in the succeeding image, the preceding image and the succeeding image can be predicted to have a high similarity to each other.

[0192] Similarity Analysis Between Preceding and Following Video Frames for Searching for Advertisement Intervention Locations: Type 5

[0193] In the fifth type of similarity analysis between a preceding video frame and a succeeding video frame for searching for an advertisement intervention location, the emotions expressed in the preceding video frame and the emotions expressed in the succeeding video frame are classified into different emotion classes, and the similarity between the preceding video frame and the succeeding video frame can be predicted based on the emotion classification of the preceding and succeeding video frames.

[0194] For example, in the fifth type of similarity analysis of linear video frames and subsequent video frames for searching for an advertisement intervention location according to one embodiment of the present invention, sentiment classification is implemented from multi-modal data of video data and audio data of the preceding video frame, and more specifically, similarity analysis according to sentiment classification between the preceding video frame and the subsequent video frame can be implemented based on the results of combining the similarity analysis based on the video data and the similarity analysis based on the audio data.

[0195] For example, in a sentiment classification model based on image data (Fig. 13), a CNN series network that takes a preceding image and a succeeding image as inputs may be included, and as a sentiment classification model based on image data (see Fig. 13), the CNN series network may include an architecture in which each layer including convolution, pooling, and dropout (not shown) is stacked, and can predict the probability that the input preceding image or succeeding image belongs to each sentiment class by passing through a linear fully connected layer. In a sentiment classification model based on speech data (see Fig. 13), a CNN series network that takes a preceding speech and a succeeding speech as inputs may be included, and as a sentiment classification model based on speech data, the CNN series network may include an architecture in which each layer including convolution, pooling, and dropout (not shown) is stacked, and can predict the probability that the input preceding speech or succeeding speech belongs to each sentiment class by passing through a linear fully connected layer.

[0196] In this way, by using the preceding image frame and the succeeding image frame as inputs, the emotional classification model (see FIG. 13) based on each image data and voice data can generate an output called an emotional classification prediction from an input in the form of a two-dimensional matrix data, and as an input of the emotional classification model (see FIG. 13) based on each image data, the two-dimensional matrix data of a set of pixels forming each preceding image or each succeeding image can be included, and as an input of the emotional classification model (see FIG. 13) based on the voice data, the preceding speech or the succeeding speech can be preprocessed in the form of a two-dimensional matrix that is more suitable for processing by a CNN series network, in the form of a spectrogram (see FIG. 4a) in which the intensity or amplitude of different frequency components along the time axis is expressed, or a MEL-spectrogram (see FIG. 4b) in which a relatively high weight is given to an audible frequency band that takes into account hearing characteristics that are sensitive to relatively low frequency components.

[0197] The emotion classification predicted from the emotion classification model based on the image data (see FIG. 12) and / or the emotion classification model based on the audio data (see FIG. 12) from the image data and / or audio data of the preceding and succeeding video frames can be expressed in a discrete form, and for example, can be classified into six-level emotion classes such as anger, disgust, fear, happiness, sadness, and surprise, or, as shown in FIG. 13, can be classified into 12-level emotion classification classes radially arranged with active and passive at both ends along the Arousal axis and negative and positive at both ends along the Valence axis on a two-dimensional plane having an Arousal axis and a Valence axis, and for each of the 12-level emotion classification classes, based on the probability predicted from the emotion classification model, the emotion classification of each preceding video frame and the emotion classification of the advertisement candidate can be mapped onto the same emotion classification space, and in this way, the Euclidean distance or cosine similarity between the preceding and succeeding video frames mapped onto the same emotion classification space is calculated. Accordingly, the emotional classification of the preceding video frame and the subsequent video frame with the closest emotional classification can be predicted to have a high similarity with the corresponding preceding video frame.

[0198] In this way, in various embodiments of the present invention, in order to capture a scene change point in a stream of video content and select the scene change point as an advertisement interposition position where an in-stream advertisement will be played, for a plurality of frame sequences sampled from a stream of video content, a preceding video frame and a succeeding video frame are taken according to a time step, the similarity between the preceding video frame and the succeeding video frame is calculated, and a scene change point having a relatively low similarity between the preceding video frame and the succeeding video frame can be selected among the plurality of frame sequences. For example, as described above, the first to fifth types regarding the similarity analysis between the preceding video frame and the succeeding video frame for selecting a scene change point may be applied alone or two or more different types may be applied in combination with each other.

[0199] In one embodiment of the present invention, the ad space detection model generates a context vector including context information about each image of a preceding video frame and an image of a succeeding video frame from a multi-modal AI model that has learned a multi-modal image-text embedding space, and through a similarity analysis between the context vectors of the preceding image and the context vectors of the succeeding image, the ad space detection model can select a position where an in-stream ad will be played, which corresponds to a scene change point, between a preceding video frame and a succeeding video frame, which has a relatively low similarity calculated through a similarity analysis between the context vectors of the preceding image and the context vectors of the succeeding image. In addition, the ad space detection model can extract text about the preceding speech and text about the succeeding speech, respectively, through an ASR model that performs a speech-to-text task from the speech of the preceding video frame and the speech of the succeeding video frame, and through a similarity analysis between the context vectors of the preceding speech and the context vectors of the succeeding speech extracted from the ASR model, the ad space detection model can select a position where an in-stream ad will be played, which corresponds to a scene change point, between a preceding video frame and a succeeding video frame, which has a relatively low similarity calculated through a similarity analysis between the context vectors of the preceding speech and the context vectors of the succeeding speech. In this way, in one embodiment of the present invention, the similarity between contexts mapped onto a text embedding space through image-to-text or speech-to-text can be analyzed for the preceding image / trailing image and the preceding speech / trailing speech forming the preceding image frame and the following image frame, respectively.However, in various embodiments of the present invention, instead of mapping the preceding image / following image and the preceding speech / following speech to a text embedding space which is a different mode from the image and the speech, each preceding image / following image may be mapped onto an image embedding space which is the same mode, and for example, each preceding image / following image may be mapped onto an image embedding space such that the image embedding is a latent vector whose dimension is reduced to a lower dimension than that of the preceding image / following image or a context vector which includes implicit information about the entire preceding image / following image, and through a similarity analysis between the image embeddings of each preceding image / following image, a preceding video frame and a succeeding video frame which have a relatively low similarity between the preceding image / following image may be selected as a scene change location or an advertisement interposition location. Similarly, instead of mapping the speech of the preceding video frame and the speech of the succeeding video frame to a text embedding space, which is a different mode from the speech, the preceding speech / following speech can be mapped to a voice embedding space (or audio embedding space) which is the same mode, and each of the preceding speech / following speeches can be mapped onto the voice embedding space as a latent vector whose dimension is reduced to a lower dimension than that of the preceding speech / following speech or a context vector containing implicit information about the entire preceding speech / following speech, and through a similarity analysis between the voice embeddings of the preceding speech / following speeches, a relatively low similarity between the preceding speech / following speech can be calculated and selected as a scene change point or an advertisement insertion point between the preceding video frame and the succeeding video frame.

[0200] As described above, the ad space search model for searching for an ad intervention location can perform a similarity analysis between a preceding image and a succeeding image and a similarity analysis between a preceding speech and a succeeding speech between a preceding image and a succeeding image frame, and in various embodiments of the present invention, the ad space search model can perform a similarity analysis between text embeddings or context vectors mapped onto the text embedding space by mapping the preceding image / the succeeding image onto a text embedding space which is a different mode from the image-to-text, or can perform a similarity analysis between image embeddings mapped onto the image embedding space by mapping the preceding image / the succeeding image onto an image embedding space which is the same mode. Similarly, the above ad space search model can perform a similarity analysis between text embeddings or context vectors mapped onto the text embedding space by mapping the preceding speech / following speech onto a text embedding space that is a different mode from the speech or speech forming voice from speech-to-text, or can perform a similarity analysis between voice embeddings mapped onto the voice embedding space by mapping the preceding speech / following speech onto a voice embedding space (audio embedding space) that is the same mode. In the present specification, a context vector can be understood to encompass, regardless of the mode of the data, an embedding mapped onto a low-dimensional embedding space (a text embedding space or an image embedding space) that encapsulates the entire information of the preceding image / following image, and can be understood to encompass, for example, an embedding mapped onto a low-dimensional embedding space (a text embedding space or a voice embedding space) that encapsulates the entire information of the preceding speech / following speech.

[0201] In various embodiments of the present invention, the advertisement space search model can search for an advertisement interposition location from a similarity analysis between a preceding video frame and a succeeding video frame, and similarly to the advertisement space search model, in the case of the advertisement selection model, in the similarity analysis between the preceding video frame and the advertisement candidate, the image of the preceding video frame and the image of the advertisement candidate can be mapped to a text embedding space different from that of the image, and the similarity between the text embedding or context vector mapped to the text embedding space can be analyzed, or the image of the preceding video frame and the image of the advertisement candidate can be mapped to an image embedding space of the same mode as that of the image, and the similarity analysis can be performed between the image embedding or context vector mapped to the image embedding space. Similarly, the speech of the preceding video frame and the speech of the advertisement candidate can be mapped onto a text embedding space different from the speech or the voice forming the speech, and the similarity between the text embedding or context vector mapped onto the text embedding space can be analyzed, or the speech of the preceding video frame and the speech of the advertisement candidate can be mapped onto a voice embedding space (or audio embedding space) of the same mode as the speech or the voice forming the speech, and the similarity analysis can be performed between the voice embedding or context vector mapped onto the voice embedding space.

[0202] As described above, in various embodiments of the present invention, the multi-modal AI model can map the image of the preceding video frame and the image of the succeeding video frame (or the image of the advertisement candidate) onto a text embedding space of different modes, or map the image of the preceding video frame and the image of the succeeding video frame (or the image of the advertisement candidate) onto the same image embedding space. In this sense, in one embodiment of the present invention, the multi-modal AI model may broadly encompass a network that has learned an image embedding space, rather than a network for implementing image-to-text. Similarly, the ASR model can map the speech of the preceding video frame and the speech of the succeeding video frame (or the speech of the advertisement candidate) onto different modes of text embedding space, or map the speech of the preceding video frame and the speech of the succeeding video frame (or the speech of the advertisement candidate) onto the same speech embedding space. In this sense, in one embodiment of the present invention, the ASR model may broadly encompass a network that has learned a speech embedding space, rather than a network that implements speech-to-text.

[0203] In this specification, similarity analysis between different text embeddings, image embeddings, or voice embeddings mapped onto the same text embedding space, image embedding space, or voice embedding space may comprehensively mean other types of similarity analysis in addition to the Euclidian distance and cosine similarity, and similarity analysis in the present invention may be interpreted in a comprehensive sense rather than being limited to the examples.

[0204] Although the present invention has been described with reference to the embodiments shown in the attached drawings, these are merely exemplary, and those skilled in the art to which the present invention pertains will understand that various modifications and equivalent other embodiments are possible therefrom.

[0205] The present invention can be applied to industries related to mid-roll or in-stream advertising.< / eos> < / sos>

Claims

1. An advertisement space detection model that captures a scene transition point between a preceding video frame and a succeeding video frame in a stream of video content and selects it as an advertisement insertion location; and An in-stream advertisement matching network, comprising an advertisement selection model that calculates a mutual matching fitness with an advertisement intervention position selected from the advertisement space detection model for each advertisement candidate forming a group of advertisement candidates and selects an advertisement candidate to be played at the advertisement intervention position from among the group of advertisement candidates.

2. In paragraph 1, The above advertising space detection model is an in-stream advertising matching network characterized in that it determines an advertising intervention location between a preceding video frame and a succeeding video frame, which have relatively low similarity.

3. In paragraph 2, The above advertising space detection model calculates the similarity between a preceding video frame and a succeeding video frame for a plurality of frame sequences including a preceding video frame and a succeeding video frame taken in a chronological relationship in a stream of video content, and An in-stream advertisement matching network characterized in that the advertisement interposition location is determined to be between a preceding video frame and a succeeding video frame of a frame sequence having the lowest similarity among a plurality of frame sequences that are the targets of the similarity calculation.

4. In paragraph 2, An in-stream advertisement matching network, characterized in that the above advertisement space detection model calculates the similarity between multi-modal data including video data and audio data of a preceding video frame and multi-modal data including video data and audio data of a succeeding video frame.

5. In paragraph 4, The above advertising space detection model is, An in-stream advertising matching network, characterized in that it includes a multi-modal AI model that inputs image data or images of the preceding and succeeding image frames, and outputs a first context vector including a text embedding matching the image embedding of the input image or context information of the input image in a multi-modal embedding space of image-text, or a latent vector that maps the input image onto the embedding space of the image, or a first context vector corresponding to the image embedding.

6. In paragraph 4, The above advertising space detection model is, An in-stream advertising matching network, characterized by including an ASR (auto speech recognition) model that inputs voice data or speech of the preceding video frame and the succeeding video frame, and outputs a second context vector including text regarding the input voice data or context information of the input speech, or a latent vector that maps the input voice data to a voice embedding space, or a second context vector corresponding to the voice embedding.

7. In paragraph 4, The above advertising space detection model is, A first similarity calculated from natural language processing between a first context vector regarding a preceding image of the preceding image frame and a first context vector regarding a succeeding image of the succeeding image frame; and An in-stream advertisement matching network characterized in that the advertisement intervention location is selected based on a result of collating a second similarity calculated from natural language processing between a second context vector regarding a preceding speech of the preceding video frame and a second context vector regarding a succeeding speech of the succeeding video frame.

8. In paragraph 7, The above advertising space search model is, An in-stream advertisement matching network characterized in that the first and second similarities are combined to produce a comprehensive similarity score, and the advertisement intervention location is selected from the produced similarity score.

9. In paragraph 1, The above advertising space search model is, An in-stream advertisement matching network characterized in that the similarity between a preceding video frame and a succeeding video frame is analyzed based on a similarity analysis between a style forming an image of the preceding video frame and a style forming an image of the succeeding video frame.

10. In paragraph 9, The above advertising space search model is, A Gram matrix is ​​generated that expresses the correlation between different features extracted from the image of the preceding video frame and the image of the succeeding video frame, An in-stream advertisement matching network characterized in that the similarity between the preceding video frame and the succeeding video frame is analyzed from the differential values ​​of each element forming the Gram matrix extracted from the image of the preceding video frame and the Gram matrix extracted from the image of the succeeding video frame.

11. In paragraph 1, The above advertising space search model is, An in-stream advertising matching network characterized in that it analyzes the similarity between a preceding video frame and a succeeding video frame based on information about color tone or color sensation that forms the image of the preceding video frame and the image of the succeeding video frame.

12. In paragraph 11, The above advertising space search model is, An in-stream advertising matching network characterized in that the similarity between a preceding video frame and a succeeding video frame is analyzed from a histogram or histogram information expressing the number of pixels appearing for each pixel value for each of the three-channel images that are synthesized with respect to each other to form an image of the preceding video frame and an image of the succeeding video frame.

13. In paragraph 12, The above advertising space search model is, An in-stream advertisement matching network characterized in that the similarity between a preceding video frame and a succeeding video frame is analyzed from the difference value of the number of pixels for each pixel value based on the histogram or histogram information extracted from the image of the preceding video frame and the image of the succeeding video frame.

14. In paragraph 13, The above advertising space search model is, An in-stream advertising matching network characterized in that the similarity between the preceding video frame and the succeeding video frame is analyzed by collecting the differential values ​​of the number of pixels for each pixel based on first to third histograms or first to third histogram information directly extracted from each of the three-channel images forming the image of the preceding video frame and the image of the succeeding video frame.

15. In paragraph 1, The above advertising space search model is, Includes an emotion classification model for implementing emotion recognition from the preceding and following image frames, The above emotional classification model is an in-stream advertisement matching network characterized in that it uses data of input preceding and following video frames as input and classifies each of the input preceding and following video frames into a class of a pre-set emotional classification.

16. In paragraph 15, The above emotional classification model is, An image data-based emotion classification model that inputs two-dimensional matrix data of pixel sets forming images of the preceding and succeeding image frames from the image data of the preceding and succeeding image frames; and An in-stream advertising matching network characterized by including a sentiment classification model based on voice data that inputs two-dimensional matrix data regarding a spectrogram in which intensities for different frequencies are expressed for each time unit, as preprocessed data from voice data or speech data of the preceding and succeeding video frames.

17. In paragraph 1, The above advertising space detection model performs a similarity analysis between the preceding video frame and the following video frame, An in-stream advertising matching network characterized in that it calculates the similarity between a preceding video frame and a succeeding video frame taken from a stream of video content down-sampled at a frequency or period (sampling rate) delayed from the frame rate (fps, frames per second) of the video content.

18. In paragraph 17, The above advertising space search model is, An in-stream advertising matching network characterized in that the similarity between a preceding video frame and a succeeding video frame is calculated at each time step based on the chronological relationship between a video frame taken at a current time step from a stream of downsampled video content, a video frame taken at a previous time step, and a video frame taken at a subsequent time step, so as to extract context information from audio data that is transmitted at a relatively low data rate while changing to a relatively low frequency than video data that is transmitted at a high data rate while changing to a relatively high frame rate (fps, frames per second) on the stream of the video content.

19. In paragraph 17, The above advertising space search model is, An in-stream advertising matching network characterized in that it receives down-sampled video frames from a stream of video content at intervals according to a synchronization signal for synchronizing audio data and video data with a constant frequency delayed from the frame rate (fps, frames per second) of the video content.

20. In paragraph 1, The above advertising space search model is, An in-stream advertising matching network characterized in that it performs a similarity analysis between a preceding video frame and a succeeding video frame based on audio data that changes slowly over a plurality of video frames at a relatively low frequency in the stream of the above video content.

Images