Method for searching video based on video segmentation
By extracting and analyzing image and text representations from video frames using contrastive learning, the method enhances video search accuracy and efficiency by identifying important frames and intervals, addressing the limitations of existing text-based video search methods.
Patent Information
- Application Number
- JP2024071213
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-04-25
AI Technical Summary
Existing video search methods based on text have limitations in accurately identifying specific time segments within videos due to the lack of time-series information, leading to inefficient resource usage and difficulty in finding similar content.
A method that extracts image and text representations from each frame of a video, determines important frames, and constructs a database using contrastive learning between images and text, incorporating speech recognition to generate a comprehensive database for precise video search.
This approach significantly improves video search performance by allowing accurate pinpointing of relevant content within long videos, reducing search time and resource consumption by focusing on important frames and intervals.
Smart Images

Figure 2024160958000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for video segmentation, and more particularly to a technique for determining important points in a video and segmenting the video based thereon. [Background technology]
[0002] There are limitations to video search using only text. Also, the method of searching for an image using text is not suitable for searching a video, which is a collection of serial images. The similarity between an image and text can be calculated using the conventional method of comparing the similarity of the representation vector between the image and text, but since the image of a video contains time sequence information, it is difficult to search for the exact time in the video, making it difficult to calculate the similarity between text and video.
[0003] For example, searching for images using text on a portal site will give more accurate results than searching for videos. In other words, image searches using text work well, but video searches do not. For example, if a video is searched using a conventional image-text search method, since videos are generally 60 fps, the expression vectors in 60 x 60 x 60, i.e., 216,000 images, must be compared with the expression vectors in the search text for a single hour-long video. This requires a lot of resources and a long search time. As another example, when taking a lecture on the Internet, if a user wants to search for a section that explains "Lagrange transform," he or she can search for "Lagrange transform" on a specific site (e.g., a university's open course website). When searching for "Lagrange transform," the search results show the contents of multiple lectures, and it is difficult for the user to identify which of them is the content he or she is looking for. The user must check the entire lecture or check the detailed titles, and even if the user looks at the detailed lecture list, he or she cannot tell which content contains the actual search results. In other words, even if a user performs a text search on a portal site (e.g., YouTube (registered trademark), Google) to find a video that explains a specific topic, it is very difficult to obtain appropriate results related to the desired video.
[0004] Korean Patent Registration No. 10-1940289 (2019.01.14) discloses a content-based video search system. [Prior art documents] [Patent documents]
[0005] Republic of Korea Patent No. 10-1940289 Summary of the Invention [Problem to be solved by the invention]
[0006] The present disclosure aims to provide a method for constructing a database using a video, for example, extracting image and text representations from each frame of the video, determining significant frames, and generating a database of representations associated with the determined significant frames.
[0007] However, the technical problem that the present disclosure is intended to solve is not limited to the above-mentioned technical problem, and may include a variety of technical problems within the scope that is obvious to a person skilled in the art from the contents described below. [Means for solving the problem]
[0008] According to an embodiment of the present disclosure to achieve the above object, a method for constructing a database using a video, executed by a computing device, is disclosed, which may include the steps of: acquiring the video; extracting an image representation from each frame of the acquired video; extracting a text representation from each frame of the acquired video; determining important frames from a plurality of frames of the acquired video based on the image representations and the text representations in the plurality of frames of the acquired video; and generating a database of the representations associated with the determined important frames.
[0009] In one embodiment, at least one of the model for extracting a representation of the image and the model for extracting a representation of the text may be a model trained based on contrastive learning between images and text. In one embodiment, the method may further include a step of performing speech recognition (STT) in a section between a plurality of important frames determined in the captured video to generate STT text information; and a step of extracting a representation related to the STT text information.
[0010] In one embodiment, the database of representations associated with the determined significant frames may include image and text representations extracted from each significant frame; and representations extracted from STT text information generated in intervals between the significant frames.
[0011] In one embodiment, the step of generating a database of a plurality of expressions associated with the determined important frames may include a step of determining an expression extracted from STT text information generated in a section between the plurality of important frames as an expression associated with an important frame located earliest in time series among the plurality of important frames.
[0012] In one embodiment, the database of representations associated with the determined significant frames may include a representation associated with a first significant frame; and a representation associated with a second significant frame subsequent to the first significant frame, and the representations associated with the first significant frame may include a representation of text extracted from the first significant frame; a representation of an image extracted from the first significant frame; and a representation of STT text information extracted from the section between the first significant frame and the second significant frame.
[0013] In one embodiment, the method may further include generating a final search database based on a database of expressions related to the determined significant frames.
[0014] In one embodiment, the final search database may include metadata including information related to the video; image representations indicating static image information contained in the significant frames; text representations indicating static text information contained in the significant frames; and a representation of STT text information including multiple dynamic pieces of information related to the intervals between the multiple significant frames.
[0015] In one embodiment, extracting a textual representation from each frame of the captured video may include extracting the textual representation based on at least one of subtitle information associated with each frame and text information obtained by performing text recognition on an image of each frame.
[0016] In one embodiment, each frame of the captured video is represented as a static representation vector including an image representation and a text representation, and the step of determining an important frame from among a plurality of frames of the captured video may include a step of determining the candidate frame as an important frame based on a calculation of a similarity between the static representation vector of one or more important frames determined most recently in the time series and the static representation vector of the candidate frame. According to an embodiment of the present disclosure to achieve the above object, a video search method executed by a computing device is disclosed, the method includes: obtaining search information related to a video; extracting a representation related to the search information; and determining significant frames related to the search information based on a comparison between the representation extracted from the search information and a plurality of representations for a plurality of significant frames determined in the video, where the plurality of representations of each significant frame may include a representation extracted from an image of each significant frame and a representation extracted from a text of each significant frame.
[0017] In one embodiment, the plurality of representations of each significant frame may further include a representation of STT text information associated with each significant frame.
[0018] In one embodiment, the representation of STT text information for each significant frame may include a representation of STT text information for an interval between each significant frame and a subsequent significant frame.
[0019] In one embodiment, the method further includes providing information related to a significant frame associated with the search information, the information related to the significant frame associated with the search information may include video information corresponding to an interval between the associated significant frame and a significant frame next to the associated significant frame.
[0020] In one embodiment, the step of providing information regarding important frames related to the search information may further include the step of providing a timestamp corresponding to the search information in the STT text information.
[0021] According to an embodiment of the present disclosure for achieving the above object, a method for summarizing a video executed by a computing device is disclosed, which may include: acquiring the video; extracting an image representation from each frame of the acquired video; extracting a text representation from each frame of the acquired video; determining significant frames from a plurality of frames of the acquired video based on the image representations and the text representations in the plurality of frames of the acquired video; and generating summary information for the video based on the plurality of representations associated with the determined significant frames. According to an embodiment of the present disclosure for achieving the above object, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, the one or more processors perform the following operations for constructing a database using a video, which may include: acquiring the video; extracting an image representation from each frame of the acquired video; extracting a text representation from each frame of the acquired video; determining a significant frame from a plurality of frames of the acquired video based on the image representation and the text representation in the plurality of frames of the acquired video; and generating a database of a plurality of representations associated with the determined significant frames.
[0022] In one embodiment, the above operations may further include an operation of performing speech recognition (STT) in a section between a number of important frames determined in the captured video to generate STT text information; and an operation of extracting a representation related to the STT text information. In one embodiment, the database of representations associated with the determined significant frames may include image and text representations extracted from each significant frame; and representations extracted from STT text information generated in intervals between the significant frames.
[0023] In one embodiment, the operations may further include generating a final search database based on a database of expressions associated with the determined significant frames.
[0024] In one embodiment, extracting a textual representation from each frame of the captured video may include extracting the textual representation based on at least one of subtitle information associated with each frame and text information obtained by performing text recognition on an image of each frame.
[0025] In one embodiment, each frame of the captured video is represented as a static representation vector including an image representation and a text representation, and the operation of determining an important frame from among a plurality of frames of the captured video may include an operation of determining the candidate frame as an important frame based on a calculation of a similarity between the static representation vector of one or more important frames determined most recently in the time series and a static representation vector of the candidate frame. In order to solve the above-mentioned problems, a computing device according to an embodiment of the present disclosure is disclosed, the device including at least one processor and a memory, the at least one processor may be configured to: acquire a video; extract an image representation from each frame of the acquired video; extract a text representation from each frame of the acquired video; determine a significant frame from among a plurality of frames of the acquired video based on the image representation and the text representation in the plurality of frames of the acquired video; and generate a database of a plurality of representations associated with the determined significant frames.
[0026] In one embodiment, the at least one processor may be further configured to perform speech recognition (STT) in a section between a plurality of determined important frames in the captured video to generate STT text information; and to extract a representation of the STT text information. In one embodiment, the database of representations associated with the determined significant frames may include image and text representations extracted from each significant frame; and representations extracted from STT text information generated in intervals between the significant frames.
[0027] In one embodiment, the at least one processor may be further configured to generate a final search database based on a database of expressions associated with the determined significant frames. In one embodiment, the at least one processor may be configured to extract the textual representation based on at least one of subtitle information associated with each frame and text information obtained by performing text recognition on the image of each frame.
[0028] In one embodiment, each frame of the captured video is represented as a static representation vector including an image representation and a text representation, and the at least one processor may be configured to determine the candidate frame as a significant frame based on a calculation of a similarity between the static representation vector of one or more significant frames determined most recently in the time series and the static representation vector of the candidate frame. Effect of the Invention
[0029] The present disclosure has the effect of greatly improving the search performance of videos. In addition, the present disclosure can further shorten the user's search time by accurately searching for parts related to the search text in a single long video content. In addition, the present disclosure can determine a relationship between text and videos that did not exist before by building a database of multiple representations related to the determined important frames, thereby making it possible to perform easier and more accurate video searches.
[0030] In addition, the present disclosure determines a section corresponding to an important frame (e.g., the section from the important frame to the next important frame) and then provides pinpoint results based on the STT results of that section, thereby saving resources since not all images are compared during the video search process.
[0031] On the other hand, the effects of the present disclosure are not limited to the effects described above, and may include a variety of effects within a scope that is obvious to a person skilled in the art from the content described below. [Brief description of the drawings]
[0032] [Figure 1] FIG. 2 is a block diagram of a computing device according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a conceptual diagram illustrating a neural network in accordance with an embodiment of the present disclosure. [Diagram 3] 1 is a diagram for explaining a method for contrastive learning between image and text in one embodiment of the present disclosure. [Figure 4] 11 is a diagram for explaining an operation of examining the similarity between an image and text using a model trained by contrastive learning between an image and text in an embodiment of the present disclosure. [Diagram 5] 11 is a diagram for explaining an operation of searching for an image corresponding to a text or image using a model trained by contrastive learning between images and text in one embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram of modules for constructing a video-based database in one embodiment of the present disclosure. [Figure 7] 1 is a diagram illustrating an operation of extracting image and text representations from a video on a frame-by-frame basis in one embodiment of the present disclosure. [Figure 8]1 is a diagram illustrating an example of an embodiment of the present disclosure, in which a static representation vector of a frame at time t is compared with a most recent static representation vector of a plurality of significant frames stored in a database of static representations of significant frames. [Figure 9] 1 is a diagram illustrating an operation of storing static representation vectors of a first frame and a second frame in one embodiment of the present disclosure. [Figure 10] 1 is a diagram illustrating an outline of an operation of determining an important frame from among a plurality of frames from time tm to t+k in an embodiment of the present disclosure. [Figure 11] 1 is a diagram illustrating an outline of an operation of extracting a representation related to STT text information based on information related to the time of an important frame in one embodiment of the present disclosure. [Figure 12] 1 is a diagram illustrating a final representation vector for each important frame, which is a component of a final search database in an embodiment of the present disclosure. [Figure 13] 1 is a diagram illustrating a final search database according to an embodiment of the present disclosure. [Figure 14] FIG. 2 is a block diagram of modules for searching videos in one embodiment of the present disclosure. [Figure 15] 11 is a diagram for explaining an operation of determining top results with high similarity by comparing a search text representation with a final search DB in an embodiment of the present disclosure. [Figure 16] 11 is a diagram for explaining an operation in which several important frames with high similarity are determined from among multiple important frames of related videos based on the final search results in one embodiment of the present disclosure. [Figure 17] This is a diagram for explaining an operation of providing a timestamp containing a search text among STT text information of a section related to a target frame when a user selects at least one of candidate target frames in one embodiment of the present disclosure. [Figure 18] 1 is a flowchart of a method for building a database using videos in accordance with one embodiment of the present disclosure. [Figure 19] 1 is a flowchart of a method for searching for a video in one embodiment of the present disclosure. [Figure 20] 1 is a flowchart of a method for summarizing a video in accordance with one embodiment of the present disclosure. [Figure 21] FIG. 1 is a simplified generalized diagram of an exemplary computing environment in which embodiments of the present disclosure can be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0033] Various embodiments will be described below with reference to the drawings. Various descriptions are provided herein to facilitate understanding of the present disclosure. However, it is apparent that such embodiments can be practiced without such specific descriptions.
[0034] As used herein, terms such as "component," "module," and "system" refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or software execution. For example, a component can be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components can reside within a processor and / or thread of execution. A component can be localized within one computer. A component can be distributed across two or more computers. Such components can also be executed on various computer-readable media having various data structures stored therein. Components can communicate through local and / or remote processes, etc., using signals (e.g., data transmitted over a network such as the Internet to other systems using data and / or signals from one component interacting with other components in a local or distributed system), including, for example, one or more data packets.
[0035] It should be noted that the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified and unless otherwise clear from the context, "X utilizes A or B" is intended to mean one of the natural inclusive permutations. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, then "X utilizes A or B" can apply to any of these. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.
[0036] Additionally, the predicate "comprises" and / or the modifier "comprises" should be understood to mean the presence of the feature and / or component in question. However, the predicate "comprises" and / or the modifier "comprises" should be understood not to exclude the presence or addition of one or more other further features, components and / or groups thereof. Additionally, in the present specification and claims, the singular should generally be construed to mean "one or more," unless a specific number is specified or the context is clear to indicate the singular form.
[0037] Furthermore, the term "at least one of A or B" should be interpreted as meaning "a case in which only A is included," "a case in which only B is included," or "a case in which a combination of A and B is included."
[0038] Those skilled in the art should further appreciate that the various exemplary logical blocks, configurations, modules, circuits, means, logic, and algorithm steps described in accordance with the embodiments disclosed herein can be realized by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various exemplary components, blocks, configurations, means, logic, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design limitations of the overall system. Skilled artisans can implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as departing from the scope of the present disclosure.
[0039] The description of the embodiments set forth herein is provided to enable any person skilled in the art to use or practice the present disclosure. Various modifications to the embodiments will be apparent to those skilled in the art to which the present disclosure pertains. The generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments set forth herein. The present invention is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.
[0040] In this disclosure, the terms network function, artificial neural network, and neural network may be used interchangeably.
[0041] FIG. 1 is a block diagram of a computing device according to an embodiment of the present disclosure.
[0042] 1 is merely a simplified example, and in one embodiment of the present disclosure, the computer device (100) may include other components for implementing the computing environment of the computer device (100), and the computer device (100) may be configured with only some of the disclosed components.
[0043] The computer device (100) may include a processor (110), a memory (130), and a network unit (150).
[0044] In one embodiment of the present disclosure, the processor (100) may be configured with one or more cores and may include a processor for data analysis and deep learning, such as a computing central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). The processor (110) may read a computer program stored in the memory (130) and execute data processing for machine learning in one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform calculations for learning a neural network. In deep learning (DL), the processor (110) may execute calculations for learning a neural network, such as processing input data for learning, extracting features from the input data, calculating errors, and updating the weights of the neural network using backpropagation.
[0045] At least one of the CPU, GPGPU, and TPU of the processor (110) can process the learning of the network function. For example, the CPU and the GPGPU can both perform the learning of the network function or the classification of data using the network function. In one embodiment of the present disclosure, the processors of multiple computing devices can be used together to perform the learning of the network function or the classification of data using the network function. In one embodiment of the present disclosure, the computer program executed in the computing device can be a program executable by the CPU, the GPGPU, or the TPU.
[0046] In one embodiment of the present disclosure, the memory 130 can store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150.
[0047] In one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device (100) may also operate in conjunction with a web storage that performs the storage function of the memory (130) over the Internet. The above description of the memory is merely exemplary, and the present disclosure is not limited thereto.
[0048] In one embodiment of the present disclosure, the network unit (150) can use various wired communication systems such as a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed DSL (VDSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and a local area network (LAN). In addition, the network unit (150) in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.
[0049] The network unit (150) in the present disclosure can be configured regardless of the type of communication, such as wired or wireless, and can be various communication networks such as a short-distance communication network (LAN: Local Area Network), a personal communication network (PAN: Personal Area Network), a wide area communication network (WAN: Wide Area Network), etc. In addition, the network can be the well-known World Wide Web (WWW), or can use wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth (registered trademark). The techniques described herein can be used in the above networks as well as other networks.
[0050] FIG. 2 is a schematic diagram illustrating a network function in one embodiment of the present disclosure.
[0051] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network is often composed of a collection of interconnected computational units, generally called nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes. The nodes (or neurons) that make up a neural network may be interconnected by one or more links.
[0052] In a neural network, one or more nodes connected via links can form a relationship of input node and output node relative to each other. The concept of input node and output node is relative, and any node that is an output node for one node can be an input node in relation to another node, and vice versa. As mentioned above, the relationship between input node and output node can be established around the link. One or more output nodes can be connected to one input node via links, and vice versa.
[0053] In the relationship between an input node and an output node connected via a link, the data of the output node can be determined based on the data input to the input node. Here, the node interconnecting the input node and the output node can have a weight. The weight can be variable and can be changed by a user or an algorithm so that the neural network performs a desired function. For example, when one or more input nodes are interconnected to one output node by each link, the output node can determine the value of the output node based on the value input to the input node connected to the output node and the weight set to the link corresponding to each input node.
[0054] As mentioned above, in a neural network, one or more nodes are interconnected through one or more links to form a relationship between an input node and an output node in the neural network. In a neural network, the characteristics of the neural network can be determined by the number of nodes and links, the correlation between the nodes and links, and the weight value assigned to each link. For example, if there are two neural networks that have the same number of nodes and links but different link weight values, the two neural networks can be recognized as different.
[0055] A neural network may be composed of a set of one or more nodes. A subset of the nodes in a neural network may form a layer. A portion of the nodes in a neural network may form a layer based on their distance from a first input node. For example, a set of nodes whose distance from a first input node is n may form an n-layer. The distance from the first input node may be defined based on the minimum number of links that must be traversed to reach the node from the first input node. However, this definition of a layer is given arbitrarily for the purpose of explanation, and the configuration of layers in a neural network may be defined in a manner different from that described above. For example, a layer of nodes may be defined based on their distance from a final output node.
[0056] The first input node may refer to one or more nodes in a neural network to which data is directly input without passing through a link in relation to other nodes. Alternatively, the first input node may refer to a node that does not have other input nodes connected via a link in relation between nodes based on links in a neural network. Similarly, the final output node may refer to one or more nodes in a neural network that do not have an output node in relation to other nodes. In addition, the hidden node may refer to a node that is neither a first input node nor a final output node and that constitutes a neural network.
[0057] The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is the same as the number of nodes in the output layer, and the number of nodes decreases once and then increases again as the input layer progresses to the hidden layer. The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is smaller than the number of nodes in the output layer, and the number of nodes decreases as the input layer progresses to the hidden layer. Furthermore, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is greater than the number of nodes in the output layer, and the number of nodes increases as the input layer progresses to the hidden layer. The neural network according to another embodiment of the present disclosure may be a neural network in which the above-mentioned neural networks are combined.
[0058] Deep neural networks (DNNs) can refer to neural networks that include multiple hidden layers in addition to input and output layers. Deep neural networks can be used to understand the latent structures of data. In other words, they can understand the latent structures of photos, text, videos, audio, and music (e.g., whether an object is in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.). Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siamese networks, generative adversarial networks (GANs), etc. The above-mentioned deep neural networks are merely examples and the present disclosure is not limited thereto.
[0059] In one embodiment of the present disclosure, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input and output layers. The number of nodes in each layer may decrease from the number of nodes in the input layer toward an intermediate layer called a bottleneck layer (encode), and may be expanded from the bottleneck layer toward the output layer (symmetrical to the input layer) in a manner symmetrical to the reduction. The autoencoder may perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the dimensions after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may decrease as it gets farther from the input data. If the number of nodes in the bottleneck layer (the layer with the fewest number of nodes located between the encoder and the decoder) is too small, there is a possibility that a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (e.g., more than half of the input layer, etc.).
[0060] A neural network can be trained by at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training a neural network can be a process of providing the neural network with knowledge to perform a specific operation.
[0061] Neural networks can be trained in a direction that minimizes the error of the output. In training a neural network, training data is repeatedly input to the neural network, the error of the neural network output and the target regarding the training data is calculated, and the weight value of each node of the neural network is updated by backpropagating the error of the neural network from the output layer of the neural network to the input layer as a direction to reduce the error. In the case of supervised learning, training data in which the correct answer is labeled is used for each training data (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each training data. That is, for example, the training data in supervised learning for data classification can be data in which each training data is labeled with a category. The labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weight value of each node in each layer of the neural network can be updated through backpropagation. The amount of change in the connection weight value of each node to be updated can be determined by a learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The application method of the learning rate can be changed depending on the number of iterations of the neural network learning cycle. For example, in the early stage of neural network learning, the learning rate can be increased to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and in the later stage of learning, the learning rate can be decreased to increase accuracy.
[0062] In the training of a neural network, the training data may generally be a subset of the actual data (i.e., data to be processed using the trained neural network), and therefore there may be a training cycle in which the error associated with the training data decreases, but the error associated with the actual data increases. Overfitting is a phenomenon in which errors increase in the actual data due to excessive learning of the training data. For example, a neural network that has learned about cats by looking at a yellow cat may be unable to recognize that it is a cat when it sees a cat of a color other than yellow, which may be a type of overfitting. Overfitting may cause an increase in errors in machine learning algorithms. To prevent overfitting, various optimization methods may be applied. To prevent overfitting, methods such as increasing the training data, regularization, dropout, which deactivates some of the nodes of the network during the training process, and the use of a batch normalization layer may be applied.
[0063] FIG. 3 is a diagram for explaining an image-text contrastive learning method in one embodiment of the present disclosure, FIG. 4 is a diagram for explaining an operation of examining the similarity between an image and text using a model trained by image-text contrastive learning in one embodiment of the present disclosure, and FIG. 5 is a diagram for explaining an operation of searching for an image corresponding to a text or an image using a model trained by image-text contrastive learning in one embodiment of the present disclosure.
[0064] According to an embodiment of the present disclosure, video segmentation (or video timestamping), which will be described below, is a technique for dividing a video into important scenes. Although there are techniques for searching videos based on this, currently, it is possible to search videos using only text. As an example, it is also possible to perform video search based on a method of searching text using an image or a method of searching for an image similar to an input image, using a representation (e.g., a representation vector used when performing contrastive learning between images and text) described below. The most representative model trained by contrastive learning between images and text is CLIP (Contrastive Language-Image Pretraining).
[0065] According to an embodiment of the present disclosure, image-text contrast learning can be learning using representations. Image-text contrast learning can include 1) learning a model based on the generation of a representation map, 2) searching for text corresponding to an image using the representation map, and 3) searching for an image corresponding to the text or image using the representation map. Note that the representation map refers to a result of conversion from raw data to a vector, matrix, or tensor of a specific dimension using a filter, encoder, or model.
[0066] First, the step of 1) training a model based on the generation of a representation map will be described with reference to FIG. 3. FIG. 3(a) shows the process of converting text into a tensor (representation map) using an encoder. In FIG. 3(a), T1 is the result of converting the initial raw text data ("Pepper the Aussie pup"), which can be represented as a vector, matrix, tensor, etc. As shown in FIG. 3(a), T1 to T NThe representation of N pieces of data is called an N-dimensional representation map. Constructive learning is a method of learning a model based on the similarity between the text representation map and the image representation map. For example, in Figure 3(b), N pieces of text and N pieces of images are arranged so that the most highly related pairs are assigned the same reference number (T1-I1, T2-I2, ... T N -I N ) and texts or images with close reference numbers (representatively, T1 and I1, T2 and I2, ... T N and I N ) is highly similar, and the model can be trained to learn that T1 is a text "Pepper the Aussie pup", I1 is an image corresponding to "Pepper the Aussie pup", and the text "The photo of the Pepper the Aussie pup" is given the reference number 2 because it is similar to T1, and is a text corresponding to T2. In this case, the similarity between T1 and I1 is very high, and the similarity between T2 and I1 is also very high. Cosine similarity can be used as a method for calculating the similarity. Based on the result of the cosine similarity, the model can be trained to have a high similarity between the most related image and text expression and a low similarity between the less related image and text expression. The formula for cosine similarity can be expressed as follows:
[0067] [Number 1] JPEG2024160958000002.jpg28109
[0068] Next, the step 2) of searching for text corresponding to an image using a representation map will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining a method of examining the similarity between an image and text using a model trained by contrastive learning between images and text. The step of searching for text corresponding to an image using a representation map can be performed by converting text examples (plane, car, dog, ..., bird) into representation vectors, converting an input image into a representation vector, and comparing the two converted representation vectors to search for text that is closest to the image (= highly related).
[0069] Next, the step of 3) searching for an image corresponding to a text or image using a representation map will be described with reference to FIG. 5. FIG. 5(a) shows a method of constructing a representation DB learned by contrastive learning as described above with reference to FIG. 3 using a held (pre-stored) image, FIG. 5(b) shows a method of comparing the similarity between an input image and the representation DB, and FIG. 5(c) shows a method of providing an image with the highest similarity. In the step of searching for an image corresponding to a text or image using a representation map, a plurality of images held are converted into representations and stored (e.g., converted into representation vectors using the image-text contrastive learning model described above with reference to FIG. 3 and stored), and when an image is input, the input image is converted (e.g., converted into a representation vector using the image-text contrastive learning model described above with reference to FIG. 3) and the two representation vectors are compared with each other to find an image close to the input image.
[0070] For ease of explanation, (a), (b), and (c) in FIG. 5 are only described for the case where the type of input data is an image, but the operation is similar when text is input. As an example, the process of searching for the most similar image when the input text "puppy" is input will be briefly described below. In the (a) stage, it is possible to convert multiple images already held into a representation vector and build a representation DB. In the (b) stage, it is possible to convert the input text "puppy" into a representation vector using the same model (e.g., a model trained by contrastive learning between images and text like CLIP) used when converting multiple images into a representation vector in the (a) stage. In the (c) stage, it is possible to compare the representation vector converted from "puppy" with the representation DB in (a) and output the image with the highest similarity as the final result.
[0071] CLIP (Contrastive Language-Image Pretrained model), a model trained by contrastive learning between images and text, has all the information between images and text, but general representation (or feature) extraction methods that are not based on such models have a problem that performance drops when performing image search with text. For example, in the process of extracting image representations using general models, there is no information related to related text, and in the process of extracting text representations, there is no information related to related images. Therefore, general text-based image search methods perform searches only based on the similarity between text describing an image (e.g., text in image information, text information related to objects or people in an image, etc.) and the search text, which causes a problem that search performance drops significantly. In other words, models that are not trained by contrastive learning between images and text have a problem that performance drops when performing image search with text, and naturally, video search performance also drops significantly.
[0072] Another problem is that even when a model such as CLIP trained by contrastive learning between images and text is used, the performance is degraded when searching for "videos" that are consecutive images including a time-series flow. In order to solve such problems, the present disclosure is capable of extracting text or image representations based on a model (e.g., CLIP) trained in advance by contrastive learning between images and text, and performing video search based on the extracted representations. This will be described in more detail below with reference to Figs. 7 to 10.
[0073] According to an embodiment of the present disclosure, speech recognition (STT or ASR; Speech To Text, or Automatic Speech Recognition) is a transcription technology that converts speech to text. In other words, speech recognition (STT) is a technology that generates text that corresponds to speech (grammatically and orthographically correct). The input of such speech recognition (STT) may include at least one of a speech signal, a spectrogram converted from the speech signal, or a speech feature. The output of speech recognition (STT) is text in the form of a string. Meanwhile, speech recognition (STT) models can be embodied as various types of models including a neural network model. Also, speech recognition (STT) models can be divided into modularized and non-modularized end-to-end (e2e) models according to the implementation method. Here, the modularized method may include, but is not limited to, a traditional model (e.g., some models of Kaldi toolkit-based ASR, Hybrid-ASR model, etc.) that performs speech recognition by dividing it into an acoustic model (a model that shows in what form a speech signal is expressed), a language model (a model that assigns a probability of occurrence to a word based on a given sentence or word), a pronunciation dictionary, etc. On the other hand, a non-modularized method means an e2e model (e.g., a transformer-based encoder decoder model, etc.), and it is possible to generate a model by learning a lot of data without providing a lower module. On the other hand, a representative decoding method is the beam search method, which is a method that can take into account all possibilities and considers the entire sentence to find the optimal solution rather than predicting only one word that is closest to the correct answer depending on the situation.
[0074] The present disclosure relates to a method for segmenting a video, particularly to a method for determining important points in a video using a text representation, an image representation, or an STT text representation, and for segmenting the video based on the determined important points.
[0075] In addition, the present disclosure can determine important frames using static image information and static text information for each frame, and perform STT for the section between the important frames to extract STT text information. Through this process, the present disclosure can determine static image representation information, static text representation information, and STT text representation information for the section up to the next important frame for each important frame, and can build a representation database (DB) for multiple videos by combining meta information such as the title of the video. As a result, when performing a video search, a representation vector is extracted from the input text, and the extracted representation vector is compared for similarity with the representation database (DB), allowing for a more accurate video search that takes into account detailed content in the video.
[0076] In other words, the present disclosure is characterized by determining important frames by calculating changes in the representation vectors of images and text for each frame, and constructing a representation DB for multiple images by combining representations extracted from the STT result text for multiple sections separated by the important frames with metadata of the images.
[0077] FIG. 6 is a block diagram of multiple modules for constructing a video-based database in one embodiment of the present disclosure.
[0078] Referring to FIG. 6, according to an embodiment of the present disclosure, a computing device (100) may include a related video determination module (10), an input module (20), a significant frame determination module (30), and a search DB generation module (40). Meanwhile, the modules that may be included in such a computing device (100) may be controlled by a processor (110) or may be executed by the operation of the processor (110). Furthermore, the modules that may be included in the computing device (100) for constructing a database using videos are not limited to the modules described above, and may include other modules. Below, exemplary modules for constructing a database using videos will be described in more detail.
[0079] In the following, a "representation" refers to a "representation vector" extracted by a module that can extract features from a representation, such as an encoder. In other words, a set of vectors is stored in a Static Representation DB or a final search database.
[0080] According to an embodiment of the present disclosure, the related video determination module (10) can determine a plurality of videos related to the search text. For the related videos related to the search text, a method similar to an existing video search method can be used. For example, a plurality of videos (hereinafter, "(a plurality of) related videos") related to the text searched by the user can be determined by a search based on a text mainly based on keywords or metadata. The related video determination module (10) can directly perform the process of determining the related videos, but is not limited thereto, and can also determine the related videos through a third party such as YouTube or Google. In addition, depending on the embodiment, the related videos can be determined in advance and input to the following input module (20). For example, in the case of B2B use, a customer can determine all lecture videos that he / she owns as related videos, transmit them to the related video determination module (10) via the customer's terminal, and input them to the following input module (20), thereby constructing a representation database (Representation DB) of a plurality of related videos.
[0081] According to an embodiment of the present disclosure, the input module (20) can receive input of a plurality of related videos determined by the related video determination module (10). As an example, the plurality of related videos are divided into a plurality of frames, and the input module (20) can extract audio (e.g., audio signal or spectrogram, etc.) for performing STT for each video. The input module (20) can also extract various metadata associated with each video. As an example, the metadata can include time information, video URL, video length, video title, video extension, summary information, video provider (e.g., in the case of a YouTube (registered trademark) video, information associated with the account that uploaded the video), etc.
[0082] According to one embodiment of the present disclosure, the significant frame determination module 30 is capable of i) extracting an image representation from each frame of the captured video, ii) extracting a text representation from each frame of the captured video, and iii) determining a significant frame from among a plurality of frames of the captured video based on a plurality of image representations and a plurality of text representations in the plurality of frames of the captured video.
[0083] At least one of the model for extracting an image representation and the model for extracting a text representation can be a model trained based on contrastive learning between images and text. The model trained based on contrastive learning between images and text is a model trained by contrastive learning using large-scale data in advance for the similarity between images and text. In other words, the model for extracting an image representation has already learned information related to the text related to the image. Similarly, the model for extracting a text representation has already learned information related to the image related to the text. As an example, rather than using CLIP (Contrastive Language-Image Pretraining) only when extracting an image representation, it is possible to use OCR-processed CLIP (Contrastive Language-Image Pretraining) related to multiple frames of a related video when extracting a text representation. Therefore, when performing a text search, a search for related images or related text can be performed more appropriately. In addition, an image representation is a vector that a model trained based on contrastive learning between images and text outputs after receiving an image as input, and a text representation is a vector that a model trained based on contrastive learning between images and text outputs after receiving text as input.
[0084] In one embodiment, the important frame determination module 30 may include an image representation extraction module 31 for extracting a representation of an image. The image representation extraction module 31 may extract a representation of an image from each frame of the captured video. For example, the image representation extraction module 31 may be a filter, an encoder, a network model, etc., and may extract a vector, a matrix, a tensor, etc. of a specific dimension. As an example, the image representation extraction module 31 may include Vision Transformers (ViT), ResNet, Contrastive Language-Image Pretraining (CLIP), etc. The image representation may include information related to various patterns related to an image, such as hue information, graphic information, object information, etc., of an image corresponding to the frame.
[0085] Further, in one embodiment, the important frame determination module 30 may include a text representation extraction module 32 for extracting a text representation. The text representation extraction module 32 may extract a text representation from each frame of the captured video. For example, the text representation extraction module 32 may be a filter, an encoder, a network model, etc., and may extract a vector, a matrix, or a tensor of a specific dimension. The text representation extraction module 32 may include a CLIP, an OCR model, an NLP model, or a Transformer encoder, etc. The text representation may include information related to various patterns related to the text, such as information related to the meaning of the text corresponding to the frame. The text representation extraction module 32 may also extract a text representation based on at least one of subtitle information related to each frame and text information obtained by performing text recognition (OCR) on an image of each frame.
[0086] FIG. 7 is a diagram illustrating an operation of extracting image and text representations from a video on a frame-by-frame basis in one embodiment of the present disclosure. For example, referring to FIG. 7, the image representation extraction module (31) can extract an image representation from each frame (e.g., Frame1, Frame2, ..., FrameN) of the video. The text representation extraction module (32) can extract a text representation from each frame (e.g., Frame1, Frame2, ..., FrameN) of the video. Since a video (moving image) is data of a series of frame images, a "static representation vector" consisting of an image representation and a text representation can be generated for each frame image. The "static representation vector" can be composed of multiple static information (e.g., image representation and text representation) obtained in each frame itself. As an example, if the video is 60 fps (frames per second) and is a one-hour long video, a static representation vector corresponding to 60 x 60 x 60 frames can be generated.
[0087] In one embodiment, the significant frame determination module 30 is capable of determining significant frames from among a plurality of frames of the captured video based on iii) a plurality of representations of images and a plurality of representations of text in a plurality of frames of the captured video, where each frame of the captured video can be represented as a static representation vector including an image representation and a text representation.
[0088] The important frame determination module (30) can determine the candidate frame as an important frame based on a calculation of the similarity between the static representation vector of one or more important frames determined at the most recent timing in the time series and the static representation vector of the candidate frame. For example, if the static representation vector of one or more important frames determined at the most recent timing in the time series is significantly different from the static representation vector of the candidate frame, the important frame determination module (30) can determine the frame as an important frame and store it to generate a database (static representation DB) related to multiple representations (Rgepresentation) related to the important frame. As an example, the important frame determination module (30) can determine whether or not there is a significant change by calculating a similarity score, and the method of calculating the similarity score can be, but is not limited to, cosine similarity (see Equation 1).
[0089] FIG. 8 is a diagram that shows, in an embodiment of the present disclosure, an operation of comparing a static representation vector of a frame at time t with the most recent static representation vector of multiple important frames stored in a significant frame static representation database (static representation DB); FIG. 9 is a diagram that shows, in an embodiment of the present disclosure, an operation of storing static representation vectors of the first frame and the second frame; and FIG. 10 is a diagram that shows, in an embodiment of the present disclosure, an operation of determining an important frame from multiple frames from time tm to t+k.
[0090] As an example, referring to FIG. 8, the significant frame determination module (30) can compare the static representation vector of a frame at time t with the static representation vector of the most recent (e.g., three frames) significant frames stored in a database (static representation DB) of multiple representations related to significant frames.
[0091] More specifically, referring to Fig. 9, the significant frame determination module 30 can store the static representation vector of the first frame in the "static representation DB related to multiple representations related to significant frames" without comparison. Also, the significant frame determination module 30 can store the static representation vector of the second frame after comparing it with multiple static representation vectors most recently stored in the "static representation DB related to multiple representations related to significant frames".
[0092] Also, a process of determining an important frame based on the representations at a first time (time tm), a second time (time t), and a third time (time t+k) will be described with reference to FIG. 10. For example, the important frame determination module 30 can calculate a similarity between a static representation vector of a frame at a first time (time tm) and a preset number (e.g., three frames) of important static representation vectors stored in a database (static representation DB) related to a plurality of representations related to important frames. Since the similarity between the static representation vector of a frame at a first time (time tm) and a preset number (e.g., three frames) of important static representation vectors stored in a database (static representation DB) related to a plurality of representations related to important frames is less than a threshold, the important frame determination module 30 can determine the frame at the first time (time tm) as an important frame. In addition, the significant frame determination module (30) can store a static representation vector of the frame at the first time (time tm) determined to be a significant frame in a database (static representation DB) related to multiple representations related to significant frames.
[0093] In an embodiment, the important frame determination module 30 may calculate a similarity between a static representation vector of a frame at a second time (time t) and a preset number (e.g., three frames) of important static representation vectors stored in a database (static representation DB) related to a plurality of representations related to important frames. As an example, the preset number (e.g., three frames) of important frames stored in a database (static representation DB) related to a plurality of representations related to important frames may include a first time (time tm) and two important frames preceding it. Meanwhile, the important frame determination module 30 may determine that a static representation vector of a frame at a second time (time t) has a high similarity with a preset number (e.g., three frames) of important static representation vectors, and may not determine the frame at the second time (time t) as an important frame and not store the frame in the database (static representation DB) related to a plurality of representations related to important frames.
[0094] For example, the important frame determination module 30 can calculate a similarity between a static representation vector of a frame at a third time (time t+k) and a preset number (e.g., three frames) of important static representation vectors stored in a database (static representation DB) related to a plurality of representations related to important frames. As an example, the preset number (e.g., three frames) of important frames stored in a database (static representation DB) related to a plurality of representations related to important frames can include a first time (time tm) and two important frames immediately before it. The important frame determination module 30 can determine the frame at the third time (time t+k) as an important frame because the similarity between the static representation vector of the frame at the third time (time t+k) and the preset number (e.g., three frames) of important static representation vectors stored immediately before is less than a threshold value. In addition, the significant frame determination module (30) can store the static representation vector of the frame at the third time (time t+k) determined as the significant frame in a database (static representation DB) related to multiple representations related to significant frames.
[0095] By the above-mentioned operations described using Figures 8 to 10, the important frame determination module (30) compares the similarity of each frame included in a video, determines frames that are less than a threshold as important frames, and generates a "database (static representation DB) of multiple representations related to important frames" based on the static representation vectors of the determined frames.
[0096] According to an embodiment of the present disclosure, the search DB generation module (40) may include an STT module (41) that performs speech recognition (STT) on audio included in the captured video. The STT module (41) may perform speech recognition (STT) on a section between a plurality of important frames determined in the captured video, and generate STT text information. The STT module (41) may perform speech recognition (STT) on audio included in a plurality of videos, and generate STT text information that is an STT result. In order to perform speech recognition (STT) with higher performance, the STT module (41) may perform pre-processing (or post-processing in some embodiments), such as speech enhancement, noise filtering, voice activity detection (VAD), and speaker diarization. The STT module (41) may also perform post-processing, such as error detection, correction, and insertion of sentence codes, after performing speech recognition (STT). The STT module (41) can also utilize closed caption data instead of STT text information if the captured video contains closed caption data.
[0097] FIG. 11 is a diagram illustrating an operation of extracting a representation related to STT text information based on time-related information in an important frame in an embodiment of the present disclosure. In an embodiment, the search DB generation module 40 may include an STT expression extraction module 42 that extracts a representation related to the STT text information. The STT expression extraction module 42 may divide the STT text information based on information related to time in a plurality of important frames recorded in a "database (static representation DB) related to a plurality of representations related to important frames" and extract a representation related to the STT text information for each divided unit. For example, the STT expression extraction module 42 may be the same module as the text representation extraction module for each frame described above. For example, the STT expression extraction module 42 may be a filter, an encoder, a network model, etc., and may extract a vector, matrix, tensor, etc. of a specific dimension. The STT expression extraction module may include a CLIP, an NLP model, a Transformer encoder, etc.
[0098] On the other hand, the representation extracted from the STT text information (STT Text Representation) is different from the image representation (Image Representation) or the text representation (Text Representation) in that it contains information related to the meaning of the text spoken in the "section" between important frames, and therefore can contain dynamic information (i.e., time-series information) related to the section.
[0099] According to an embodiment of the present disclosure, a database (static representation DB) of representations related to important frames may include image representations and text representations extracted from each important frame and representations extracted from STT text information generated in a section between the important frames. The search DB generation module (40) may determine the representation extracted from the STT text information generated in a section between the important frames as a representation related to an important frame located earliest in time series among the important frames.
[0100] That is, a database (static representation DB) related to a plurality of representations related to important frames may include a plurality of representations related to a first important frame and a plurality of representations related to a second important frame subsequent to the first important frame. Here, the plurality of representations related to the first important frame may include a representation of text extracted from the first important frame, a representation of an image extracted from the first important frame, and a representation related to STT text information extracted from a section between the first important frame and the second important frame. As an example, referring to FIG. 11, in the case of a first important frame (tm important frame) and a second important frame (t+k important frame), a representation extracted from STT text information related to speech in a first time (tm) to second time (t+k) section may be treated as a representation corresponding to a frame at the first time (tm). For example, in FIG. 11, the expression (STT Text Representation) extracted from the rightmost STT text information can be linked to the first time (tm) and stored in the final search database of FIG. 12 described below.
[0101] FIG. 12 is a diagram illustrating a final representation vector for each important frame, which is a component of the final search database in one embodiment of the present disclosure, and FIG. 13 is a diagram illustrating the final search database in one embodiment of the present disclosure.
[0102] In one embodiment, the search DB generation module 40 may include a final search DB generation module 43 for generating a final search database based on a database of a plurality of representations related to the determined important frames. As an example, the final search database may include metadata including information related to the video, an image representation indicating static image information included in the important frames, a text representation indicating static text information included in the important frames, and a representation of STT text information including a plurality of dynamic information related to sections between the plurality of important frames. That is, the final search DB generation module (43) can generate a final search database for all of the multiple related videos by using metadata including the representation related to the STT text information acquired by the STT module (41) and the STT expression extraction module (42) and the information related to the video acquired by the input module (20) in the existing "database (static representation DB) related to multiple representations related to important frames". Referring to FIG. 12 as an example, the final search DB generation module (43) can generate a final representation vector for each time period of multiple important frames for all of the multiple related videos. Note that there can be multiple final representation vectors for one video (for example, when there are multiple important frames), and the final search database can include multiple final representation vectors for all of the related videos.
[0103] Referring to FIG. 13 based on an embodiment, the related videos (images) may include related image 1, related image 2, and related image 3. When related image 1 has two important frames A and B, related image 2 has one important frame C, and related image 3 has two important frames D and E, the final search database may be expressed as shown in FIG. 13. In the case of multiple related images, the first frame is unconditionally determined as an important frame, and a section is formed by determining multiple important frames therebetween. However, in the final search database shown in FIG. 13 including the above-mentioned important frames A, B, C, D, and E, the first frame may be omitted. For example, the time stamp of important frame D in related image 3 may be 03:27, the time stamp of the next important frame E may be 07:54, and the total length of related image 3 may be 12:30. In this case, the section corresponding to the first frame (=time stamp 00:00) of related video 3 is 00:00~03:27, the section corresponding to important frame D (=time stamp 03:27) of related video 3 is 03:27~07:54, and the section corresponding to important frame E (=time stamp 07:54) of related video 3 is 07:54~12:30. Based on this, the "video search method" will be described below.
[0104] FIG. 14 is a block diagram of modules for searching videos in one embodiment of the present disclosure.
[0105] According to an embodiment of the present disclosure, the computing device 100 may include a search text extraction module 50 and a search result providing module 60. Meanwhile, the modules that may be included in the computing device 100 may be controlled by or executed by the processor 110. Furthermore, for video searching, the modules that may be included in the computing device 100 are not limited to the modules described above, and may include additional modules. Exemplary modules for searching videos are described in more detail below.
[0106] According to an embodiment of the present disclosure, the search text extraction module (50) can extract a representation of search information related to the acquired video. The search information can include text information acquired via a user terminal. The search text extraction module (50) can be a model used in the above-mentioned text representation extraction module (32) and STT representation extraction module (42). That is, the search text extraction module (50) can be a filter, an encoder, a network model, etc., and can extract a vector, a matrix, or a tensor of a specific dimension. The search text extraction module (50) can also include a CLIP, an NLP model, a Transformer encoder, etc.
[0107] Although FIG. 14 and the following examples are described with reference to a case where text is received as input, in some embodiments, the input may be in the form of an image, in which case a search image extraction module may be used instead of the search text extraction module (50). The search image extraction module may be a model used in the image representation extraction module (31) described above. That is, the search image extraction module may be a filter, an encoder, a network model, etc., and may extract vectors, matrices, tensors, etc. of a particular dimension. The search image extraction module may also be included in a CLIP, NLP model, Transformer encoder, etc.
[0108] FIG. 15 is a diagram for explaining an operation of determining the top results with the highest similarity by comparing the search text representation with the final search DB in one embodiment of the present disclosure, FIG. 16 is a diagram for explaining an operation of determining several important frames with high similarity from among multiple important frames of related videos based on the final search results in one embodiment of the present disclosure, and FIG. 17 is a diagram for explaining an operation of providing a timestamp containing the search text from the STT text information of the section related to the target frame when a user selects at least one of the candidate target frames in one embodiment of the present disclosure.
[0109] According to an embodiment of the present disclosure, the search result providing module (60) can determine significant frames related to the search information based on a comparison between the representations extracted from the search information and a plurality of representations in the plurality of significant frames determined in the video, where the plurality of representations of each significant frame can include a representation extracted from an image of each significant frame, a representation extracted from a text of each significant frame, and a representation related to STT text information of each significant frame, where the representation of the STT text information of each significant frame can include a representation of STT text information of a section between each significant frame and a subsequent significant frame.
[0110] In an embodiment, the search result providing module 60 can provide information on important frames related to the search information. The information on important frames related to the search information can include video information corresponding to a section between the relevant important frame and a next important frame of the relevant important frame. For example, the search text extracting module 50 can compare a representation extracted from the search information with a final search database to calculate a similarity. When a user searches by image, an image representation extracted from the search image by the search image extracting module can be compared with the final search database to calculate a similarity. The search result providing module 60 can also provide information on important frames that are arranged in order of similarity and are equal to or greater than a threshold value to the user's terminal.
[0111] Referring to FIG. 15 as an example, when the search information related to the video is "Lagrange transform", the search result providing module (60) can calculate the similarity from the relationship between the text representation vector related to the search information extracted from the search text extraction module (50) and the final representation vector for each important frame of a plurality of related videos stored in the final search database. The similarity can be, but is not limited to, a cosine similarity (see Equation 1) as in the above-mentioned various representation extraction modules. The search result providing module (60) can extract the following similarity calculation results as a result of performing a similarity calculation between the search information ("Lagrange transform") and a plurality of final representation vectors for each important frame included in the final database: 0.03 for image 1-frame A, 0.9 for image 1-frame B, 0.001 for image 2-frame C, 0.2 for image 3-frame D, and 0.1 for image 3-frame E. Then, the search result providing module (60) can provide information on important frames to the user's terminal in the order of video 1-frame B, video 3-frame D, and video 3-frame E, sorting them in order of decreasing similarity.
[0112] Also, the search result providing module (60) can provide important frames (hereinafter, target frame candidates) having high similarity among multiple important frames of related videos based on the final search result, as shown in FIG. 16. The search result providing module (60) can provide multiple target frame candidates to the user's terminal. Also, when the user selects one of them (=target frame), the search result providing module (60) can provide the entire video corresponding to the target frame and a corresponding timestamp. Depending on the embodiment, as shown in FIG. 17, in a section corresponding to the target frame (e.g., a section from the target frame to the next important frame of the related video), the timestamp of the position where the actual search text is located can be pinpointed using the STT result text and provided to the user. In other words, when a user selects frame B of related video 1 from among target frame candidates B, D, and E, the search result providing module (60) can pinpoint and provide the timestamp (06:24) containing search information from the STT text information of the section (05:02 to 08:23) corresponding to target frame B.
[0113] According to an embodiment of the present disclosure, the computing device 100 may i) directly determine related videos and perform a video search. For example, this embodiment may be applicable when Action Power directly operates a video sharing platform such as YouTube. The computing device 100 may generate and build a final search database based on a database of multiple expressions related to important frames for videos uploaded to the video sharing platform, and may quickly perform a search based on the database when search information (search text) is input. On the other hand, when the platform is not directly operated, when search information (search text) is input, related videos may be primarily determined using a third party (e.g., Google, YouTube (registered trademark), etc.), and the computing device 100 may build a final search database based on a database of multiple expressions related to important frames for multiple related videos.
[0114] The computing device (100) can also perform a video search even when ii) related videos have already been determined. For example, this embodiment can be considered to apply to a case where a service is provided on a B2B basis, i.e., a case where related videos are provided by a B2B customer (e.g., an Internet lecture provider such as MegaStudy). In one embodiment, all videos can be acquired in advance as related videos from a terminal or server of the customer (e.g., an Internet lecture provider such as MegaStudy). In another embodiment, when a user searches for a text, the B2B customer (e.g., an Internet lecture provider such as MegaStudy) first filters and determines related videos, and the computing device (100) can build a final search database based on a database of multiple expressions related to important frames for the determined multiple related videos and perform a search.
[0115] According to an embodiment of the present disclosure, the above-mentioned video search method can be used in a broadcasting station. For example, in the process of editing videos of a variety program at a broadcasting station, past materials may be used. In this case, it is difficult to quickly find a video to be used, and PDs need to check the past materials directly. As an example, when searching for a specific scene (e.g., a scene in which soccer player Thierry Henry heads a ball filled with water in a variety program called "Infinite Challenge") in multiple past videos, the "video search method" can be used. The computing device (100) can treat all videos in the program (e.g., Infinite Challenge) as related videos and build a final search database based on a database of multiple expressions related to important frames. When the computing device (100) obtains "a scene in which soccer player Thierry Henry heads a ball filled with water" as search information, it can quickly provide the scene of the video in the program (e.g., Infinite Challenge) using the built final search database.
[0116] According to an embodiment of the present disclosure, the above-mentioned video search method can also be used to search for nature documentaries. When there are few lines, such as in nature documentaries, the current text-based video search method may make the search even more difficult. The computing device 100 can also be used for image representation, so it can also be used to search for videos that do not contain speech. For example, if one wants to search for a "scene in which a lion attacks a deer" in a National Geographic video, multiple National Geographic videos can be treated as related videos, a final search database can be constructed, and the desired scene can be searched for.
[0117] According to an embodiment of the present disclosure, the computing device 100 can utilize the operations of the video database construction method and the video search method described above for "video summarization." For example, the computing device 100 can perform video summarization by weighting STT results for a certain period around an important frame timestamp.
[0118] FIG. 18 is a flowchart of a method for building a database using videos in one embodiment of the present disclosure.
[0119] The method for constructing a database using a video shown in Fig. 18 can be performed by the above-mentioned computing device 100. Therefore, even if the content is omitted below, the content described about the computing device 100 can be similarly applied to the description of the method for constructing a database using a video.
[0120] Referring to FIG. 18, a method for constructing a database using a video in one embodiment of the present disclosure may include a step of acquiring the video (S110), a step of extracting an image representation from each frame of the acquired video (S120), a step of extracting a text representation from each frame of the acquired video (S130), a step of determining important frames from a plurality of frames of the acquired video based on the image representations and the text representations in the plurality of frames of the acquired video (S140), and a step of generating a database related to a plurality of representations related to the determined important frames (S150).
[0121] Step S110 is a step of acquiring the video. Step S120 is the step of extracting an image representation from each frame of the captured video.
[0122] Step S130 is a step of extracting a text representation from each frame of the captured video. S130 may include a step of extracting the text representation based on at least one of subtitle information related to each frame and text information obtained by performing text recognition on an image of each frame. Step S140 is determining significant frames from among the plurality of frames of the captured video based on image representations and text representations in the plurality of frames of the captured video.
[0123] Step S150 is a step of generating a database of representations related to the determined important frames, where the database of representations related to the determined important frames may include image representations and text representations extracted from each important frame; and representations extracted from STT text information generated in sections between the important frames.
[0124] According to one embodiment of the present disclosure, the method may further include a step of performing speech recognition (STT) in a section between a plurality of important frames determined in the captured video to generate STT text information; and a step of extracting a representation related to the STT text information.
[0125] The method may also include generating a final search database based on a database of a plurality of representations related to the determined significant frames, wherein the final search database may include metadata including information related to the video, image representations indicating static image information included in the significant frames, text representations indicating static text information included in the significant frames, and a representation of STT text information including a plurality of dynamic information related to sections between the plurality of significant frames.
[0126] FIG. 19 is a flowchart of a method for searching for a video in one embodiment of the present disclosure.
[0127] The video search method shown in Fig. 19 can be performed by the above-mentioned computing device 100. Therefore, even if the content is omitted below, the content described about the computing device 100 can be similarly applied to the description of the video search method.
[0128] Referring to FIG. 19, a video search method in one embodiment of the present disclosure may include a step of obtaining search information related to a video (S210), a step of extracting a representation related to the search information (S220), and a step of determining a significant frame related to the search information based on a comparison between the representation extracted from the search information and a plurality of representations in a plurality of significant frames determined in the video (S230).
[0129] Step S210 is a step of obtaining search information related to the video. According to an embodiment, the search information related to the video can be text or image. Step S220 is a step of extracting a representation related to the search information.
[0130] Step S230 is determining significant frames associated with the search information based on a comparison of representations extracted from the search information with a plurality of representations for a plurality of significant frames determined in the video, where the plurality of representations for each significant frame may include a representation extracted from an image of each significant frame and a representation extracted from text of each significant frame.
[0131] According to an embodiment of the present disclosure, in the method, the plurality of representations of each significant frame may further include a representation of STT text information for each significant frame, where the representation of STT text information for each significant frame may include a representation of STT text information for a section between each significant frame and a subsequent significant frame.
[0132] The method may further include providing information related to a significant frame associated with the search information, the information related to the significant frame associated with the search information may include video information corresponding to an interval between the associated significant frame and a significant frame next to the associated significant frame.
[0133] FIG. 20 is a flowchart of a method for summarizing a video in one embodiment of the present disclosure.
[0134] The method for summarizing a video shown in Fig. 20 can be performed by the above-mentioned computing device 100. Therefore, even if the content is omitted below, the content described about the computing device 100 can be similarly applied to the description about the method for summarizing a video.
[0135] Referring to FIG. 20, a method for summarizing a video in one embodiment of the present disclosure may include a step of acquiring the video (S310), a step of extracting an image representation from each frame of the acquired video (S320), a step of extracting a text representation from each frame of the acquired video (S330), a step of determining important frames from a plurality of frames of the acquired video based on the image representations and the text representations in the plurality of frames of the acquired video (S340), and a step of generating summary information of the video based on a plurality of representations related to the determined important frames (S350).
[0136] Step S310 is a step of acquiring the video. Step S320 is to extract an image representation from each frame of the captured video.
[0137] Step S330 is to extract a text representation from each frame of the captured video.
[0138] Step S340 is determining significant frames from among the plurality of frames of the captured video based on image representations and text representations in the plurality of frames of the captured video.
[0139] Step S350 is generating summary information for the video based on a plurality of representations associated with the determined significant frames.
[0140] In the above description, steps S110 to S150, steps S210 to S230, and steps S310 to S350 may be divided into smaller steps or some steps may be integrated to reduce the number of steps, and some steps may be omitted or the order of steps may be changed as necessary, depending on the embodiment of the present application.
[0141] In accordance with one embodiment of the present disclosure, a computer-readable storage medium having a data structure stored thereon is disclosed.
[0142] A data structure can refer to the organization, management, and storage of data that allows efficient access and modification of the data. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, data modification in the shortest time). A data structure can also be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements can include a connection relationship between data elements as thought by a user. A physical relationship between data elements can include an actual relationship between data elements that is physically stored in a computer-readable storage medium (e.g., a hard disk). A data structure can specifically include a collection of data, a relationship between data, and a function or command that can be applied to the data. An effectively designed data structure allows a computing device to perform calculations while minimizing the use of computing device resources. Specifically, a computing device can increase the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching through an effectively designed data structure.
[0143] Data structures can be classified into linear data structures and non-linear data structures according to the type of the data structure. A linear data structure can be a structure in which only one piece of data is linked after another. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets that are internally ordered. A list can include linked lists. A linked list can be a data structure in which data is linked in a manner in which each piece of data is linked in a row with a pointer. In a linked list, the pointer can contain information about the connection to the next or previous piece of data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circularly linked list depending on the type of the data structure. A stack can be a data list structure that allows limited access to data. A stack can be a linear data structure in which data can be manipulated (e.g., inserted or deleted) only at one end of the data structure. Data stored in a stack can be a LIFO (Last in First Out) data structure in which the last in comes first. A queue is a data structure that allows limited access to data, and unlike a stack, it can be a FIFO (First in First out) data structure where the slowest data is stored is the slowest data is available. A deck can be a data structure that allows data to be processed on both ends of the data structure.
[0144] A nonlinear data structure may be a structure in which multiple pieces of data are concatenated after one piece of data. A nonlinear data structure may include a graph data structure. A graph data structure may be defined by vertices and edges, and the edges may include a line connecting two different vertices. A graph data structure may include a tree data structure. A tree data structure may be a data structure in which a path connecting two different vertices among multiple vertices included in a tree is one data structure. In other words, a graph data structure may be a data structure that does not form a loop.
[0145] Throughout this specification, the terms computation model, neural network, network function, and neural network can be used interchangeably. (Hereinafter, they will be uniformly described as neural network.) The data structure may include a neural network. The data structure including the neural network may be stored in a computer-readable storage medium. The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for training the neural network. The data structure including the neural network may include any of the components of the disclosed configurations. That is, the data structure including the neural network may include all or any combination of data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for training the neural network. In addition to the above configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include all forms of data used or generated in the process of computing the neural network, and is not limited to the above. The computer-readable storage medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be comprised of a collection of interconnected computing units, generally referred to as nodes. Such nodes may be referred to as neurons. A neural network is comprised of at least one or more nodes.
[0146] The data structure may include data to be input to the neural network. The data structure including the data to be input to the neural network may be stored in a computer-readable storage medium. The data to be input to the neural network may include training data input during the training process of the neural network and / or input data to be input to the neural network after training has been completed. The data to be input to the neural network may include data that has undergone pre-processing and / or data to be pre-processed. Pre-processing may include a data processing process for inputting data to the neural network. Thus, the data structure may include data to be pre-processed and data generated in pre-processing. The above data structures are merely examples, and the present disclosure is not limited thereto.
[0147] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored in a computer-readable storage medium. The neural network may include a plurality of weights. The weights are variable and may be varied by a user or an algorithm to allow the neural network to perform a desired function. For example, when one or more input nodes are interconnected to an output node by respective links, the output node may determine an output node value based on values input to the input nodes connected to the output node and parameters set for the links corresponding to the respective input nodes. The above data structures are merely examples, and the present disclosure is not limited thereto.
[0148] By way of example and not limitation, the weights may include weights that are variable during the neural network training process and / or weights at which neural network training has been completed. The weights that are variable during the neural network training process may include weights at the start of a learning cycle and / or weights that are variable during a learning cycle. The weights at which neural network training has been completed may include weights at which a learning cycle has been completed. Thus, a data structure including weights of a neural network may include a data structure including weights that are variable during the neural network training process and / or weights at which neural network training has been completed. Thus, the above-mentioned weights and / or combinations of each weight are included in a data structure including weights of a neural network. The above-mentioned data structures are merely exemplary and the present disclosure is not limited thereto.
[0149] The data structure including the neural network weights may be stored in a computer-readable storage medium (e.g., memory, hard disk) after a serialization process. Serialization may be a process of converting a data structure into a form that can be stored in the same or another computing device and later reconstructed and used. The computing device may serialize the data structure and transmit and receive the data over a network. The serialized data structure including the neural network weights may be reconstructed in the same or another computing device through deserialization. The data structure including the neural network weights is not limited to serialization. Furthermore, the data structure including the neural network weights may include a data structure for increasing the efficiency of calculations while minimizing the use of computing device resources (e.g., nonlinear data structures such as B-Tree, Trie, m-way search tree, AVLtree, Red-Black Tree). The above items are merely examples, and the present disclosure is not limited thereto.
[0150] The data structure may include hyper-parameters of the neural network. The data structure including the hyper-parameters of the neural network may be stored in a computer-readable storage medium. The hyper-parameters may be variables that are changed by a user. The hyper-parameters may include, for example, a learning rate, a cost function, a number of learning cycle iterations, weight initialization (e.g., setting a range of weights to be initialized), and the number of Hidden Units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The above data structures are merely examples, and the present disclosure is not limited thereto.
[0151] FIG. 21 is a simplified general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure can be implemented.
[0152] Although the present disclosure has been described above as generally being embodied in a computing device, those skilled in the art will appreciate that the present disclosure can also be embodied in combination with computer-executable instructions and / or other program modules capable of being executed on one or more computers and / or as a combination of hardware and software.
[0153] Generally, modules herein include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Additionally, those skilled in the art will appreciate that the methods of the present disclosure can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc., each of which can operate in conjunction with one or more associated devices.
[0154] The embodiments described in this disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. A computer includes a variety of computer-readable media. Any medium accessible by a computer can be a computer-readable medium, including volatile and non-volatile media, transitory and non-transitory media, mobile and non-mobile media. By way of example and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, mobile and non-mobile media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store information.
[0155] Computer-readable transmission media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared and other wireless media. Any combination of any of the above media should also be included within the scope of computer-readable transmission media.
[0156] An exemplary environment (1100) for implementing various aspects of the present disclosure is shown including a computer (1102) including a processing unit (1104), a system memory (1106), and a system bus (1108). The system bus (1108) couples system components, including, but not limited to, the system memory (1106), to the processing unit (1104). The processing unit (1104) can be any of a variety of commercially available processors. Dual processors and other multi-processor architectures can also be utilized as the processing unit (1104). The system bus (1108) can be any of several types of bus structures that can be further interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory (1106) includes read only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and contains the basic routines that support the exchange of information between the various components of the computer (1102), such as during start-up. The RAM (1112) can also include high-speed RAM, such as static RAM, for caching data.
[0157] The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA) - although the internal hard disk drive 1114 can also be configured for external use in a suitable chassis (not shown) - a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from and writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., for reading from CD-ROM disks 1122 and for reading from and writing to other high capacity optical media such as DVDs). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 can be connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128, respectively. The interface (1124) for implementing an external drive includes, for example, at least one or both of the Universal Serial Bus (USB) and IEEE 1394 interface technologies. These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of computer 1102, the drives and media accommodate the storage of any data in a suitable digital format. While the above description of computer-readable storage media refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will appreciate that other types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, can also be used in the exemplary operating environment, and further that any such media can contain computer-executable instructions for performing the methods of the present disclosure.
[0158] A number of program modules may be stored in the drives and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules, and / or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented with various commercially available operating systems or combinations of operating systems.
[0159] A user may enter commands and information into the computer 1102 through one or more wired or wireless input devices, such as a keyboard 1138 and a pointing device, such as a mouse 1140. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is connected to the system bus 1108, but may also be connected through other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.
[0160] A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface, such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown), such as speakers, printers, etc.
[0161] The computer 1102 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and / or wireless communications. The remote computer(s) 1148 can be a workstation, a server computer, a router, a personal computer, a handheld computer, a microprocessor-based entertainment device, a peer device or other conventional network node, and typically includes many or all of the components depicted for the computer 1102, although for simplicity, only a memory storage device 1150 is illustrated. The logical connections depicted include wired and wireless connections in a local area network (LAN) 1152 and / or larger networks, such as a wide area network (WAN) 1154. Such LAN and WAN networking environments are commonplace in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global computer network, such as the Internet.
[0162] When used in a LAN networking environment, the computer 1102 connects to the local network 1152 through a wired and / or wireless communication network interface or adapter 1156. The adapter 1156 can facilitate wired or wireless communication to the LAN 1152, which may also include a wireless access point installed thereon for communicating with the wireless adapter 1156. When used in a WAN networking environment, the computer 1102 can include a modem 1158 or have other means for establishing communications over the WAN 1154, such as connecting to a communications server on the WAN 1154 or through the Internet. The modem 1158, which can be internal or external and can be a wired or wireless device, connects to the system bus 1108 through the serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored in a remote memory / storage device 1150. It will be readily appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
[0163] The computer 1102 is operable to communicate with any wireless device or unit that is deployed and operates in wireless communication, such as printers, scanners, desktop and / or handheld computers, portable data assistants (PDAs), communication satellites, any equipment or location associated with a wirelessly detectable tag, and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, communication can be in a predefined structure, such as a traditional network, or simply ad hoc communication between at least two devices.
[0164] Wi-Fi (Wireless Fidelity) allows devices to connect to the Internet without being wired. Wi-Fi is a wireless technology like a cell phone that allows such devices, e.g., computers, to send and receive data indoors and outdoors, anywhere within the range of a base station. Wi-Fi networks use IEEE 802.11 (a, b, g, etc.) radio technology to provide secure, reliable, and fast wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, e.g., at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual bands).
[0165] Those of ordinary skill in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referred to in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields, etc. or particles, optical fields, etc. or particles, or any combination thereof.
[0166] Those of ordinary skill in the art will appreciate that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in the description of the embodiments disclosed herein can be implemented by electronic hardware, various forms of program or design code (for convenience, referred to herein as "software"), or a combination of all of these. To clearly illustrate such interoperability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been generally described above with a focus on their functionality. Whether such functionality is implemented in hardware or software depends on the design constraints imposed on the particular application and the overall system. Those of ordinary skill in the art will appreciate that the described functionality can be implemented in various ways for each particular application, and such implementation decisions should not be interpreted as departing from the scope of the present disclosure.
[0167] The various embodiments disclosed herein may be implemented by a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible by any computer-readable device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROM, cards, sticks, key drives, etc.). Additionally, the various storage media disclosed herein may include one or more devices and / or other machine-readable media for storing information.
[0168] It should be understood that the particular order or hierarchy of the stages in the processes depicted is an example of an exemplary approach. Based on design priorities, it should be understood that the particular order or hierarchy of the stages in the processes can be rearranged within the scope of the present disclosure. The accompanying method claims provide elements of the various stages in a sample order, but are not meant to be limited to the particular order or hierarchy depicted.
[0169] The description of the illustrated embodiments is provided to enable any person of ordinary skill in the art to which the disclosure pertains to the present disclosure to utilize or practice the present disclosure. Various modifications to such embodiments will be apparent to those of ordinary skill in the art to which the disclosure pertains, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited by the embodiments shown herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.
Claims
1. 1. A method for constructing a video database, the method comprising: obtaining said video; extracting a representation of an image from each frame of the captured video; extracting a textual representation from each frame of the captured video; determining significant frames from among the plurality of frames of the captured video based on a plurality of representations of images and a plurality of representations of text in the plurality of frames of the captured video; and generating a database of representations associated with the determined significant frames; Including, method.
2. In claim 1, At least one of the model for extracting an image representation and the model for extracting a text representation is a model trained based on contrastive learning between images and text. method.
3. In claim 1, The method comprises: performing speech recognition (STT) in a section between the determined important frames in the captured video to generate STT text information; and extracting a representation related to the STT text information; Further comprising: method.
4. In claim 3, The database of expressions related to the determined significant frames includes: an image representation and a text representation extracted from each significant frame; and An expression extracted from the STT text information generated in the section between the plurality of important frames. Including, method.
5. In claim 4, The step of generating a database of a plurality of expressions associated with the determined significant frames comprises: determining an expression extracted from STT text information generated in a section between the plurality of important frames as an expression related to an important frame located earliest in a time series among the plurality of important frames; method.
6. In claim 5, The database of expressions related to the determined significant frames includes: A plurality of expressions associated with a first frame of significance; and a plurality of representations associated with second significant frames subsequent to the first significant frame; Including, The plurality of representations associated with the first significant frame include: a representation of the text extracted from said first significant frame; a representation of an image extracted from the first significant frame; and Representation of STT text information extracted from the section between the first important frame and the second important frame. Including, method.
7. In claim 4, The method comprises: generating a final search database based on a database of a plurality of expressions related to the determined important frames; Further comprising: method.
8. In claim 7, The final search database is Metadata comprising information about the video; a representation of an image showing the static image information contained in said significant frame; a textual representation indicating the static textual information contained in said important frame; and A representation of STT text information including a plurality of dynamic information related to a section between the plurality of important frames. Including, method.
9. In claim 1, Extracting a textual representation from each frame of the captured video includes: extracting a text representation based on at least one of subtitle information relating to each frame and text information obtained by performing text recognition on the image of each frame; method.
10. In claim 1, Each frame of the captured video is represented as a static representation vector that includes an image representation and a text representation; The step of determining a significant frame from among a plurality of frames of the captured video comprises: determining the candidate frame as an important frame based on a calculation of a similarity between the static representation vector of the one or more important frames determined most recently in the time series and the static representation vector of the candidate frame; Including, method.
11. 1. A method of video searching performed by a computing device, comprising: obtaining search information relating to the video; Extracting a representation related to the search information; and determining significant frames associated with the search information based on a comparison of representations extracted from the search information to a plurality of representations in a plurality of determined significant frames in the video; Including, The multiple representations of each significant frame include a representation extracted from the image of each significant frame and a representation extracted from the text of each significant frame. method.
12. In claim 11, the plurality of representations of each significant frame further includes a representation of STT text information associated with each significant frame; method.
13. In claim 12, The expression of the STT text information for each important frame is and a representation of STT text information relating to the interval between each of the significant frames and the subsequent significant frame. method.
14. In claim 12, The method comprises: providing information on important frames related to the search information; Further comprising: Information on important frames related to the search information includes: video information corresponding to a section between the relevant significant frame and a significant frame next to the relevant significant frame; Including, method.
15. In claim 14, The step of providing information regarding an important frame related to the search information includes: providing a timestamp corresponding to the search information based on the STT text information; Further comprising: method.
16. 1. A method for summarizing a video, performed by a computing device, comprising: obtaining said video; extracting a representation of an image from each frame of the captured video; extracting a textual representation from each frame of the captured video; determining significant frames from among the plurality of frames of the captured video based on a plurality of representations of images and a plurality of representations of text in the plurality of frames of the captured video; and generating summary information for the video based on a plurality of representations associated with the determined significant frames; Including, method.
17. A computer program stored on a computer readable storage medium, the computer program, when executed on one or more processors, causes the one or more processors to perform the following operations for constructing a video-based database, the operations comprising: obtaining said video; extracting a representation of an image from each frame of the captured video; extracting a textual representation from each frame of the captured video; determining a significant frame from among a plurality of frames of the captured video based on a plurality of representations of images and a plurality of representations of text in a plurality of frames of the captured video; and generating a database of representations associated with the determined significant frames; Including, A computer program stored on a computer readable storage medium.
18. In claim 17, The operation includes: performing speech recognition (STT) in a section between the determined plurality of significant frames in the captured video to generate STT text information; and Extracting Representation from the STT Text Information Further comprising: A computer program stored on a computer readable storage medium.
19. In claim 18, The database of expressions related to the determined significant frames includes: an image representation and a text representation extracted from each significant frame; and An expression extracted from the STT text information generated in the section between the plurality of important frames. Including, A computer program stored on a computer readable storage medium.
20. 20. In claim 19, The operation includes: generating a final search database based on the database of expressions related to the determined important frames; Further comprising: A computer program stored on a computer readable storage medium.
21. In claim 17, The operation of extracting a textual representation from each frame of the captured video comprises: Extracting a text expression based on at least one of subtitle information related to each frame and text information obtained by performing text recognition on the image of each frame. A computer program stored on a computer readable storage medium.
22. In claim 17, Each frame of the captured video is represented as a static representation vector that includes an image representation and a text representation; The operation of determining a significant frame from among a plurality of frames of the captured video includes: determining a candidate frame as an important frame based on a calculation of a similarity between a static representation vector of one or more important frames determined most recently in the time series and a static representation vector of the candidate frame; A computer program stored on a computer readable storage medium.
23. 1. A computing device comprising: At least one processor; and memory Including, The at least one processor Get the video; Extracting a representation of an image from each frame of the captured video; Extracting a textual representation from each frame of the captured video; determining significant frames from among the frames of the captured video based on the representations of images and the representations of text in the frames of the captured video; and and generating a database of representations associated with the determined significant frames. Device.
24. In claim 23, The at least one processor performing speech recognition (STT) in a section between the determined plurality of important frames in the captured video to generate STT text information; and Extracting a representation related to the STT text information It consists of additional Device.
25. 25. In claim 24, The database of expressions related to the determined significant frames includes: an image representation and a text representation extracted from each significant frame; and An expression extracted from the STT text information generated in the section between the plurality of important frames. Including, Device.
26. 26. In claim 25, The at least one processor and further comprising: generating a final search database based on the database of expressions related to the determined important frames. Device.
27. In claim 23, The at least one processor configured to extract a text representation based on at least one of subtitle information related to each frame and text information obtained by performing text recognition on the image of each frame. Device.
28. In claim 23, Each frame of the captured video is represented as a static representation vector that includes an image representation and a text representation; The at least one processor The method is configured to determine a candidate frame as an important frame based on a calculation of a similarity between a static representation vector of one or more important frames determined at a most recent timing in the time series and a static representation vector of the candidate frame. Device.
Citation Information
Patent Citations
Video classification method and device, electronic equipment and computer storage medium
CN112989117A
Techniques for providing similar content
KR102476499B1
Combined-media scene tracking for audio-video summarization
US20030218696A1
Display apparatus, server, system and information-providing methods thereof
US20200334468A1
