Method and device for extracting color scheme from video

By applying quasi-main shot segmentation, color clustering and convex closure technology in movie videos, the problem of difficulty in extracting movie color schemes in the existing technology is solved, and efficient color scheme extraction and the application of movie databases is realized, which improves the Accuracy of movie recommendation system.

CN113255423BActive Publication Date: 2024-11-01KOREA ADVANCED INST OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110022034.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-02-11
Filing Date
2021-01-08
Publication Date
2024-11-01
Estimated Expiration
2041-01-08

AI Technical Summary

Technical Problem

The prior art is difficult to automatically extract accurate color schemes from movie videos, resulting in difficulties in movie recommendation systems when distinguishing and clustering movies, and the existing methods are inefficient in processing complex content in long videos.

Method used

Color schemes are extracted from videos using color clustering and convex cushion enclosure techniques through methods of quasi-main lens segmentation, basic palette extraction and color scheme merging. The specific steps include: segmenting the video into a quasi-main lens, selecting keyframes, clustering colors based on prominent images, merging the basic color palette and using convex hulls to form the final color scheme.

Benefits of technology

It realizes efficient extraction of color schemes from movie videos and is used as descriptors for movie databases, simplifies the application of movie metadata marking and recommendation systems, and improves the visual recognition and clustering accuracy of movies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113255423B_ABST
    Figure CN113255423B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for extracting a color scheme from a video. The device for extracting a color scheme from a video according to an embodiment of the present invention may include: a quasi-master shot segmentation unit, which segments an input video into a plurality of quasi-master shots by detecting the boundaries of the quasi-master shots; a basic color palette extraction unit, which extracts a basic color palette from the segmented quasi-master shots; and a color scheme merging unit, which merges the basic color palettes extracted from the quasi-master shots to extract a color scheme.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The following embodiments relate to a method and apparatus for extracting a color scheme from a video, and more particularly, to a method and apparatus for automatically extracting a color scheme from a video for use as a descriptor. Background Art

[0002] IMDb is the largest and most popular movie database consisting of more than 5 million movies. After 2000, more than 10,000 movies were released in the United States and Canada alone, and the number of releases increased dramatically. As a result, movie recommendations based on user preferences have been more widely studied. Most recommendation systems use social graphs based on movie metadata such as subject matter, directors, keywords, and actors. However, manually marking metadata for each movie requires considerable effort, and when new metadata fields are needed, it takes a considerable amount of time to attach tags again before each movie. Therefore, for an effective recommendation system, it is extremely important to extract descriptors in the video itself.

[0003] Since movies combine various media, they can be represented as multimodal descriptors that include video, text, and audio features. Color is extremely important in visual media, which has an impact on audience perception and preference.

[0004] A color scheme is a combination of colors expressed as a subset of all possible colors to represent a visual identity. Color has a strong influence on human perception and should evoke an emotional response. Visual elements are the first thing humans recognize when watching a video, and color is the most basic element of visual aspects that influences human impressions and emotions.

[0005] Filmmaking strongly considers color, which is controlled by the film, and color specialists control the color of the entire film. Directors use color to support the film's story and create an integrated fictional space. According to cinematography, the color scheme is the photographer's interpretation of the scene, and the color scheme can convey a mood or feeling that remains with the audience even after the film is over. This is because the color scheme is not simply created by the camera, but is a combination of various filmmaking elements, including the backgrounds and sets created by the production designer, the lights installed by the lighting staff, and the costumes.

[0006] Each field of metadata such as genre, director cannot be a primary key to distinguish all videos alone. Similarly, there is no intention to distinguish all movies only by the color scheme proposed below. The color scheme is not a unique feature of each movie, but it is an important factor to cluster movies.

[0007] For example, La La Land (2016) and Whiplash (2014) are drama musical films written and directed by Damien Chazelle. It shares similar metadata, such as director, subject matter, and actors, but gives a different impression due to the intensity of color throughout the period. Whiplash should also be connected to Black Swan (2010) which maintains a similar tone, but there is currently no special metadata to connect the two films. The color scheme can be a very simple and accurate descriptor for quantifying the stage setting of a movie.

[0008] Some previous studies have considered extracting color schemes from images, but there has been little interest in extracting from videos, especially movies. Movies are carefully crafted editing products that directors embody their messages and values. Unlike ordinary videos that are shot continuously without distinguishing between shots or scenes, movies are carefully edited using a variety of objects and heterogeneous content. Movies are usually no longer than 3 hours, but are often longer than regular videos and include 200,000 to 250,000 images (assuming 24fps). It is difficult to extract the main colors from numerous images loaded with complex content, but just as you imagine the basic color palette after watching Anderson's movie, it is the color scheme that dominates the design.

[0009] Prior art literature

[0010] Patent Literature

[0011] Non-patent literature 1: BARALDI L., GRANA C., CUCCHIARA R.: Shot and scene detection via hierarchical clustering for re-using broadcast video. In Computer Analysis of Images and Patterns-16th International Conference, CAIP (2015), pp. 801–811. Summary of the invention

[0012] Embodiments of the present invention relate to a method and apparatus for extracting a color scheme from a video, and more particularly, provide a technology for automatically extracting a color scheme from a video to be used as a descriptor.

[0013] The purpose of an embodiment of the present invention is to provide the following method and device for extracting a color scheme from a video, and to provide a two-step color scheme extraction technology, which extracts a basic color palette from a quasi-master shot and merges it into a single color scheme, and can use color clustering and convex hull encirclement methods to utilize the advantages of each method.

[0014] An apparatus for extracting a color scheme from a video according to an embodiment of the present invention may include: a quasi-master-shot segmentation unit, which segments an input video into a plurality of quasi-master-shots by quasi-master-shot boundary detection (SBD); a basic color palette extraction unit, which extracts a basic color palette from the segmented quasi-master-shots; and a color scheme merging unit, which merges the basic color palettes extracted from the quasi-master-shots to extract a color scheme, wherein the basic color palette extraction unit includes: a key frame selection unit, which selects key frames by using a plurality of saliency maps to reduce the amount of data that needs to be clustered, wherein the saliency maps represent the importance of pixels in each image along a human fixed point, and the key frame selection unit provides an objective function in a manner that the key frames capture the entire shot content well, and the objective function is defined by a weighted sum of saliency, clarity, and representativeness for each frame; and a color clustering unit, which clusters pixels according to distances in a color space, and selects a cluster center as a representative color to extract a basic color palette from a set of selected key frames.

[0015] The quasi-master shot subdivision section may provide a quasi-master shot by combining colors similar to those of consecutive shots taken at the same location.

[0016] The quasi-master shot subdivision unit may extract saliency for each frame in the input video, measure similarity between multiple frames, and perform quasi-master shot boundary detection (SBD) by clustering multiple similar frames to segment the input video into multiple quasi-master shots.

[0017] The quasi-master shot segmentation unit may segment the input video into a plurality of quasi-master shots by using an Imagelab Shot Detector segmentation method that only considers RGB colors.

[0018] The color clustering unit may assign weight values ​​to clusters according to visual attention of pixels using a saliency map of the selected key frame.

[0019] The color scheme merging unit may perform additional clustering on the colors of the basic color palette to extract the final color scheme.

[0020] The color scheme merging unit extracts the color scheme using convex hull enclosing (CHE), which transforms the plurality of basic color palettes into a convex hull shape for an RGB color space, and selects the convex hull boundary vertex as a representative color of the color scheme.

[0021] The color scheme merging unit may extract the color scheme using a convex hull, divide the convex hull into sub-hulls, and select a representative color of each of the sub-hulls as a representative color of the color scheme.

[0022] According to another embodiment of the present invention, a method for extracting a color scheme from a video using an apparatus for extracting a color scheme from a video may include: a segmentation step of segmenting an input video into a plurality of quasi-master shots by detecting a boundary of a quasi-master shot; an extraction step of extracting a basic color palette from the segmented quasi-master shots; and a merging step of merging the basic color palettes extracted from the quasi-master shots to extract a color scheme, wherein the step of extracting the basic color palette from the quasi-master shots includes: a key frame selection step of selecting key frames by using a plurality of saliency maps to reduce the amount of data that needs to be clustered, wherein the saliency maps represent the importance of pixels in each image along a human fixed point, and in the key frame selection step, an objective function is provided in such a way that the key frames can well capture the content of the entire shot, and the objective function is defined by a weighted sum of saliency, clarity, and representativeness for each frame; and a color clustering step of clustering pixels according to distance in a color space, selecting a cluster center as a representative color to extract a basic color palette from a set of selected key frames.

[0023] In the step of dividing the input video into the above-mentioned quasi master shots, the quasi master shot may be provided by combining colors similar to those of consecutive shots shot at the same location.

[0024] In the step of segmenting the input video into the quasi-main shots, saliency may be extracted for each frame in the input video, similarity between multiple frames may be measured, and quasi-main shot boundary detection may be performed by clustering multiple similar frames to segment the input video into multiple quasi-main shots.

[0025] In the step of segmenting the input video into the quasi-master shots, the input video may be segmented into a plurality of quasi-master shots by using an image processing software shot detector segmentation method that only considers RGB colors.

[0026] In the color clustering step, the saliency map of the selected key frame may be used to assign weighted values ​​to the clusters according to the visual attention of the pixels.

[0027] In the step of merging the basic color palettes to extract a color scheme, additional clustering may be performed on the colors of the basic color palettes to extract the final color scheme.

[0028] In the step of merging the basic color palettes to extract the color scheme, the color scheme may be extracted by using a convex hull enclosure, which transforms the multiple basic color palettes into a convex hull shape for the RGB color space, and selects the convex hull boundary vertex as a representative color of the color scheme.

[0029] In the step of merging the basic color palette to extract the color scheme, the color scheme may be extracted by using a convex hull, the convex hull may be divided into sub-hulls, and the representative color of each sub-hull may be selected as the representative color of the color scheme.

[0030] According to an embodiment, the present invention may provide a method and apparatus for automatically extracting a color scheme from a video to be used as a descriptor.

[0031] According to an embodiment, the present invention may provide a method and apparatus for extracting a color scheme from a video, that is, a two-step color scheme extraction technique is provided, wherein the technique extracts a basic color palette from a quasi-master shot and merges it into a single color scheme, and the color clustering and convex hull enclosing methods may be used to utilize the advantages of each method. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A diagram for illustrating color scheme extraction according to an embodiment.

[0033] Figure 2 is a diagram for illustrating a saliency map for color clustering according to an embodiment.

[0034] Figure 3 A diagram for explaining a movie structure according to an embodiment.

[0035] Figure 4 FIG. 1 is a diagram for explaining an apparatus for extracting a color scheme from a video according to an embodiment.

[0036] Figure 5 The flowchart is used to illustrate a method for extracting a color scheme from a video according to an embodiment.

[0037] Figure 6 A diagram for explaining a subdivision section of a quasi-master lens according to an embodiment.

[0038] Figure 7 A diagram for explaining a basic palette extraction unit according to an embodiment.

[0039] Figure 8 It is a diagram for explaining the color scheme merging unit of this embodiment.

[0040] Fig. 9 A diagram for explaining a color scheme merging unit according to another embodiment. DETAILED DESCRIPTION

[0041] The following describes the embodiments with reference to the accompanying drawings. However, the described embodiments can be changed into a variety of different forms, and the scope of the present invention is not limited to the embodiments described below. In addition, the multiple embodiments are provided to more completely describe the present invention to those of ordinary skill in the art to which the present invention belongs. In the drawings, the shapes and sizes of the structural elements are magnified for a clearer description.

[0042] Figure 1 A diagram for illustrating color scheme extraction according to an embodiment.

[0043] Reference Figure 1 In order to extract a color scheme for visual recognition of a video (e.g., a movie), a video 110 is segmented into quasi-master shots and each key frame 120 is selected. A basic color palette 140 may be generated from the key frames 120 selected using a plurality of saliency maps 130. Furthermore, a convex hull enclosing 150 may transform the basic color palette 140 into a convex hull shape for an RGB color space, and the convex hull boundary vertices are selected as representative colors of a color scheme 160.

[0044] A color scheme is a combination of colors represented as a subset of all possible colors to represent visual recognition. The following embodiments propose an automated method for extracting color schemes from videos (e.g., movies). Movies consist of different objects and different types of content that reflect the director's information and values. As mentioned above, since movies are carefully edited videos, unlike ordinary videos that do not distinguish between shooting scenes or scenes, it is a difficult task to extract color schemes at once. Despite these difficulties, the extraction of color schemes still plays a very important role in film production and applications. The color scheme is the film director's interpretation of the scene, and the color scheme can convey the residual atmosphere or feeling to the viewer even after the movie is over. It also acts as a contributing factor to the description of the movie in the metadata fields of the movie, such as subject matter, director, and actors. In addition, unlike metadata, it can be automatically tagged, so it can be easily applied directly to existing movie databases.

[0045] The embodiment can guide the color scheme of a movie from the bottom up to the subdivided scenes. And, color extraction can be formulated as a selection problem, where perceptually important colors are selected using saliency. Moreover, it can provide a quasi-master shot, which is defined as a combination of consecutive shots shot at the same location with similar colors. Among them, the reliability of the method of the proposed embodiment is demonstrated and verified using videos of actual movies.

[0046] Figure 2 is a diagram for illustrating a saliency map for color clustering according to an embodiment.

[0047] Reference Figure 2 , 210 is an input image, 220 is a saliency map, the upper side of 230 is a color scheme without saliency, and the lower side is a saliency color scheme. Although simple color clustering methods are frequently proposed, it is difficult to obtain effective main image colors due to the interference of meaningless colors. In order to extract effective color schemes, saliency maps are crucial to ensure that color selection reflects the importance of objects in the video.

[0048] The overall color scheme of a movie cannot be obtained by simple color clustering methods due to the interference of seemingly repeated but meaningless colors, such as the sky, wall, or black clutter. Saliency maps are a solution to the importance of objects in a movie by color selection. Saliency maps represent the importance of pixels in each image along human fixation points. Since the main pixels dominate the color impression, rather than all pixels, the color scheme in each frame can be obtained using saliency maps.

[0049] The embodiment is the first assignment to summarize the color scheme in a video. Extracting color schemes from images has been studied in computer graphics. This is because the color scheme is the most basic unit for image redrawing and vectorization. The reason why it is difficult to extract the color scheme from a video is as follows, i.e., it should not be driven by a secondary long shot, but should consider the overall color of the video. Therefore, the video is divided into smaller units and the final color scheme is selected in a bottom-up manner.

[0050] Furthermore, an embodiment defines a quasi-master shot as a new unit that combines colors similar to those of a continuous shot taken at the same location. Such a quasi-master shot can be used in video processing processes such as video highlight detection and video thumbnail generation that have been actively studied for decades.

[0051] In addition to simply adopting saliency, embodiments are also considering how to use saliency maps appropriately.The importance (saliency) of each pixel in a frame, the importance of each frame in a shot, and the importance of each shot in a movie are measured at three levels.

[0052] Color quantization of images has been widely studied in the field of computer vision and graphics to address the color loss caused by image compression. This is a limitation of display devices that can only display a limited number of colors as well as color printing. The goal is to represent the original image with fewer colors.

[0053] Recent studies have proposed a variety of color scheme extraction methods for various purposes, including image repainting, image decomposition, image abstraction, digital art, and image indexing. Color scheme extraction is usually done through color clustering, histogram boundaries, image segmentation, and convex hull enclosure.

[0054] In clustering, pixels are grouped based on the distance in color space. The center of the cluster is selected as the representative color. Histogram boundary converts the image into a color space histogram and divides the histogram into multiple parts to identify the main colors. Convex hull surround converts the image into a convex hull shape in a specific color space and selects the convex hull boundary vertices as representative colors. Clustering and histogram methods always select colors that appear in the image, while convex hull surround can well represent the original image, but may select colors that do not necessarily exist in the image.

[0055] The embodiment of the present invention proposes a 2-step color scheme extraction, i.e., extracting the basic color palette from the quasi-master shot and merging into a single color scheme. The advantages of each method can be exploited by using color clustering and convex hull enclosing methods in the proposed pipeline.

[0056] Figure 3 A diagram for explaining a movie structure according to an embodiment.

[0057] like Figure 3 As shown, a movie can be regarded as consisting of a series of linear scenes, where each scene consists of multiple scenes. A scene is a series of interrelated scenes with the same meaning, while a shot is a series of frames taken continuously by a single camera. Since scenes are segmented according to semantic context, the accuracy of scene segmentation is relatively lower than that of shot segmentation. Moreover, since there are many repeated scenes due to video editing, it is inefficient to collect basic color palettes directly from the scenes. Therefore, in order to extract the required color scheme, a new device that can meet the required accuracy and efficiency is needed.

[0058] A master shot is a single shot that includes all the characters and represents the atmosphere of all the spaces filmed. Modern films use master shots during the production phase, but they are often not included in the actual film due to their unique style. Therefore, consecutive shots shot in the same location with similar colors are combined to define a quasi-master shot, rather than an actual master shot.

[0059] Shot changes can occur through sharp or gradual transitions. Abrupt transitions occur in a single frame, as with a camera switch, whereas gradual transitions such as dissolves, fade-ins, fade-outs, and wipes are determined over multiple frames for a variety of video effects. The goal of shot segmentation is to detect such transitions by grouping frames using image similarities.

[0060] Most scene segmentation methods use multimodal features to reduce ambiguity such as video, audio, and text, but usually, since shots are shot once, only video features can be used to detect shot segmentation transitions. Some video characteristics depend on color histograms and local descriptors such as SURF and SIFT. Among them, Baraldi's shot segmentation method (Non-Patent Document 1) is used to define the quasi-master shots of a movie.

[0061] On the other hand, a keyframe is the most representative of the entire shot or scene that protects the scene. Video thumbnails and summaries are also based on a similar principle of keyframe extraction, and machine learning applications have become increasingly popular in recent years.

[0062] The main problem of keyframe extraction is the number of keyframes that must be selected in a shot or scene and the method of selecting these keyframes in a set of shots. Depending on the extraction method, the appropriate number of keyframes can be more than one or more. Simple and fast keyframe selection methods can be used to uniformly or randomly sample frames. However, it is difficult to determine the best frame sampling because such sampling methods will lead to unstable selection results.

[0063] The cost function bounds address the shortcomings of the sampling approach, but a lot of extra computational work is required to compute the cost function for a typical movie (up to 200,000 frames). The computational cost can be reduced by selecting a fixed number of keyframes in the quasi-master shot. Instead of uniform and random sampling, an objective function can be provided to evaluate the importance, clarity, and representativeness of each frame so that the keyframes can better capture the entire shot content.

[0064] In particular, in the case of movies with hundreds of photos, it is somewhat difficult to extract the color scheme captured by the camera. Even if each image in the video is sampled at 240×135 pixels (for example), there can be up to 30,000 colors in each frame, and up to 6 billion colors in a movie. In addition, the above-mentioned movies include complex groups of shots with multiple tones. Therefore, not only the combinations in the entire movie but also the color combinations that appear simultaneously in one scene should be considered. Previous color schemes in movies usually rely on personal suggestions from experienced designers. Therefore, designers take advantage of the tendency to focus attention on clothes, main objects, or prominent backgrounds.

[0065] Previous color scheme extraction methods are usually based on uniformly clustered pixels in all frames. However, as mentioned above, simple clustering tends to promote dark colors, even black. Even if only keyframe pixels are uniformly clustered, this problem will still exist and will tend to select meaningless background and clutter. Therefore, a color scheme extraction including three saliencies is proposed. That is, the relative importance of the shots considered in the sequence, the importance of the frame to the shot, and the importance of the pixel to the frame.

[0066] Figure 4 FIG. 1 is a diagram for explaining an apparatus for extracting a color scheme from a video according to an embodiment.

[0067] Reference Figure 4 The color scheme extraction device of an embodiment may include a quasi-master shot segmentation unit 420, a basic color palette extraction unit 440, and a color scheme merging unit 460. As a result, the color schemes 470 are clustered from bottom to top in the segmented quasi-master shot 430.

[0068] Figure 5 The flowchart is used to illustrate a method for extracting a color scheme from a video according to an embodiment.

[0069] Reference Figure 5 In one embodiment, a method for extracting a color scheme from a video using an apparatus for extracting a color scheme from a video may include: a segmentation step (S110), segmenting an input video into a plurality of quasi-master shots by detecting a boundary of a quasi-master shot; an extraction step (S120), extracting a basic color palette from the segmented quasi-master shots; and a merging step (S130), merging the basic color palettes extracted from the quasi-master shots to extract a color scheme.

[0070] The following describes the steps of a method for extracting a color scheme from a video according to an embodiment.

[0071] The method for extracting a color scheme from a video according to an embodiment can utilize reference Figure 4 The device for extracting a color scheme from a video is described in detail.

[0072] In step S110, the quasi-master shot segmentation unit 420 may segment the input video 410 into a plurality of quasi-master shots 430 by detecting quasi-master shot boundaries. The quasi-master shot segmentation unit 420 may provide the quasi-master shots 430 by combining similar colors of consecutive shots shot at the same location.

[0073] The quasi-master shot subdivision unit 420 may extract saliency for each frame in the input video 410, measure similarity between a plurality of frames, and perform quasi-master shot boundary detection by clustering a plurality of similar frames, thereby segmenting the input video into a plurality of quasi-master shots 430. In particular, the quasi-master shot subdivision unit 420 may segment the input video into a plurality of quasi-master shots 430 by using an Imagelab Shot Detector segmentation method that considers only RGB colors.

[0074] In step S120, the basic color palette extracting unit 440 may extract a basic color palette 450 from the segmented quasi-master shot 430. Step S120 may include: a key frame selecting step of selecting a key frame 442 to reduce the amount of data that needs to be clustered; and a color clustering step of clustering pixels according to a distance in a color space, selecting a cluster center as a representative color to extract a basic color palette 450 from a set of the selected key frames 442.

[0075] More specifically, the basic palette extraction unit 440 may include a key frame selection unit 441 and a color clustering unit 444 .

[0076] The key frame selection unit 441 can select key frames 442 to reduce the amount of data that needs to be clustered. In addition, the key frame selection unit 441 can provide an objective function in a manner that the key frames capture the entire shot content well. The objective function can be defined by a weighted sum of the saliency, clarity, and representativeness of each frame. This is described in detail below.

[0077] Furthermore, the color clustering unit 444 may cluster pixels according to the distance in the color space, select the cluster center as a representative color, and extract a basic color palette 450 from the set of selected key frames 442. The color clustering unit 444 may use the saliency map 443 of the selected key frame 442 to assign weighted values ​​to the clusters according to the visual attention of the pixels.

[0078] In step S130 , the color scheme merging unit 460 may merge the basic color palettes 450 extracted from the respective quasi-master shots 430 to extract a color scheme 470 .

[0079] The color scheme merging unit 460 may perform additional clustering on the colors of the basic color palettes 450 to extract the final color scheme 470. In particular, the color scheme merging unit 460 may use convex hull encirclement to extract the color scheme 470. In this case, the convex hull encirclement may transform the multiple basic color palettes 450 into a convex hull shape for the RGB color space, and the convex hull boundary vertices may be selected as representative colors of the color scheme 470.

[0080] Figure 6A diagram for explaining a subdivision section of a quasi-master lens according to an embodiment.

[0081] Reference Figure 6 , the quasi-main shot subdivision unit 420 can perform quasi-main shot boundary detection, and divide the video into shot groups by combining continuous shots shot at the same position and similar colors, thereby improving the accuracy and effectiveness of segmentation. The quasi-main shot subdivision unit 420 can divide the video 410 input through the quasi-main shot boundary detection into multiple quasi-main shots 430. That is, the quasi-main shot subdivision unit 420 can extract saliency (421) for each frame in the input video 410, measure the similarity between multiple frames (422), and perform quasi-main shot boundary detection (424) by clustering similar frames (423), thereby dividing it into quasi-main shots 430 as subdivided videos.

[0082] Typically, the quasi-primary shots 430 are clustered by color difference using local descriptors for similarity elements such as SIFT or SURF, thus requiring a large amount of computational overhead. In contrast, among others, an image processing software shot detector (non-patent document 1) segmentation method that considers only RGB colors may be employed. The image processing software shot detector may determine the similarity between frames by the sum of two color difference indices, the squared difference between two true corresponding pixels and the chi-squared distance of the RGB color histogram. Similar frames may be clustered using a sliding window that compares frame differences centered on the current frame and moves in one direction.

[0083] Typically, image processing software shot detectors can detect abrupt transitions and gradual transitions separately. i In the case of fi and f i +1 is greater than a threshold value T, and the difference between adjacent shots is greater than T=2, which can be considered a sharp transition. Gradual transitions can be identified by repeating the process of detecting sharp transitions with increasing window size. After detecting the shots, the image processing software shot detector can use hierarchical clustering to group adjacent shots into scenes. To avoid repeated detection of the same transition, two adjacent transitions can be split into frame intervals of more than a specified Ts as a safe area.

[0084] Even if the shots are grouped by (Non-Patent Document 1) and the like, the shot groups segmented by the image processing software shot detector can form a scene by clustering based on color comparison, and the scene can be used as a quasi-master shot for the following two reasons. First, a fixed number of clusters is used to perform a hypothetical clustering scenario when the total number of scenes is known. Second, since the quasi-master shot does not require complete scene segmentation, scene clustering increases computer costs. Therefore, the quasi-master shot with a relaxed T can be used to determine the color difference between shots. T* can be used s Improved functionality of safe zones that are proportional to average shot length rather than T s is proportional to the fixed value.

[0085] Figure 7 A diagram for explaining a basic palette extraction unit according to an embodiment.

[0086] like Figure 7 As shown, the base palette extraction unit 440 (Base Palette Extraction, BPE) can extract the basic palette 450 from the segmented semi-master-shot 430. Although segmented, the quasi-master-shot 430 is still a video, and it is difficult to extract a limited number of colors from a video including many frames. Therefore, the base palette extraction unit 440 can extract the basic palette 450 through two steps. That is, the base palette extraction unit 400 can extract the basic palette 450 through key frame selection and color clustering. In this case, the base palette extraction unit 440 may include a key frame selection unit 441 and a color clustering unit 444.

[0087] Among them, for key frame selection and color clustering, a saliency map 443 related to human visual concerns can be used. The saliency map 443 represents the importance of pixels of human fixation points. Since the main pixels dominate the impression of the movie, rather than all pixels of all frames, a saliency map 443 that helps to identify the optimal basic color palette 450 in each frame image can be used. In this case, the saliency map 443 of the known image can be automatically used without prior knowledge.

[0088] The key frame selection unit 441 can select key frames 442 to extract the basic color palette 450. The key frames best represent the overall content set covering the shot or scene. If the key frames 442 are used, the amount of data that needs to be clustered will be greatly reduced. The key frame selection unit 441 can select a fixed number of key frames 442 in the quasi-master shot 430 to further save computing costs. In order to enable the key frames 442 to capture the entire shot content well, an objective function C(f) is introduced and the importance, clarity and representativeness of each frame are estimated, so that the key frames 442 can capture the entire shot content well. The above objective function C(f) can be expressed as follows.

[0089] C(f)=α s C s (f)+α c C c (f)+α r C r (f). (1)

[0090] Among them, C(f) is expressed by significance (C s (f)), clarity (C c (f)) and representativeness (C r (f)) The weighted sum of the three terms is defined. Based on the experiments used to obtain the equilibrium results, it can be revealed that α s , α c and α r The appropriate coefficient value.

[0091] The saliency condition C(f) can determine the importance of a frame by the average value of the pixels as follows.

[0092]

[0093] Among them, μ p is the saliency of pixel p, and |f| is the total number of pixels in f. The saliency condition scores frames high by giving them high visual attention.

[0094] Representative Condition C r (f) The applicable range of f is evaluated in all frames of the quasi-master shot 430. The representativeness can be expressed as similarity with respect to other frames existing in the same quasi-master shot 430. The representativeness condition can be calculated by the following equation.

[0095]

[0096] Where S is the quasi-master shot to which f belongs, and |S| is the total number of frames in S. The similarity Sim(f, f*) between frames f and f* is defined by HSV histogram comparison, and frames with more f are more similar and thus will increase. For example, a related distance metric of an HSV histogram including 50 bins per channel can be used, and the pairing distance between f and all other frames using the same quasi-master shot 430 can be calculated.

[0097] Clarity, C c (f) A blind / referenceless image spatial quality evaluator (BRISQUE) can be used to measure the clarity of f, that is, the degree of blur, which can be expressed as follows.

[0098] C c (f)=1-0.01*BRISQUE(f). (4)

[0099] BRISQUE scores are usually in the range [0, 100], C c The smaller (f) is, the better the image quality is. For example, the cost of all frames in the quasi-master shot 430 can be investigated by selecting the best three key frames 442.

[0100] The color clustering unit 444 may cluster pixels according to the distance in the color space, select the cluster center as a representative color, and extract a basic color palette 450 from the set of selected key frames 442. In contrast to clustering pixel colors in the same manner, a saliency map 443 of the key frame 442 may be used to assign weighted values ​​to the clusters according to the visual attention of the pixels.

[0101] In the saliency map 443, saliency can be represented as a probability rather than a value itself so that the basic palette 450 does not overfit. The pixel p of the key frame 442 is represented according to the probability ψ(μ p ) is included in the cluster object and generates 0 or 1 as the weight value μ p To provide pixel stability. Therefore, pixels with low weight values ​​may also be selected, but preferably, the cluster group includes pixels with higher stability. The RGB colors of the pixels of all key frames 442 may be used as a single group to perform kmeans clustering. The number k of cluster centers is the same as the number of colors of the basic palette 450, and k=5 is set for the experiment.

[0102] Figure 8 It is a diagram for explaining the color scheme merging unit of this embodiment.

[0103] Reference Figure 8, if all the basic color palettes extracted from the quasi-primary lens 430 are If the basic color palette 450 is combined with the basic color palette 450, two problems may occur. The number of the plurality of quasi-master lenses 430 separated in the input video 410 is large, so a large number of colors are also maintained in the reduced color group, and photos taken in the same environment may be separated from the video during the editing process, so that the color palettes of different quasi-master lenses 430 may overlap. Therefore, if the number of colors in the basic color palette 450 needs to be reduced, a merging process is required.

[0104] The color of the basic color palette 450 is the center of the cluster, that is, it is presented in the quasi-primary shot 430. Therefore, the color scheme merging unit 460 can perform additional clustering on the colors of the basic color palette 450 to obtain a final color scheme 470 that tends to ignore unique but far-from-center colors.

[0105] In order to generate a color scheme 470 including all colors in the basic palette 450, the color scheme merging unit 460 can use a convex hull enclosure 463. The known colors can be arranged in a three-dimensional RGB space and a convex hull surrounding all color points can be generated. Then, while the convex hull surrounds all color points, it can be simplified to a fixed number of vertices equal to the size of the output palette. Different from the clustering method of selecting frequently occurring colors, the convex hull enclosure can include all displayed colors to identify hidden colors.

[0106] The shot length of 461 is not arbitrary, but is in accordance with the director's intention. Figure 1 Therefore, since the duration of important shots tends to be longer, weights can be assigned to the basic palette 450 according to the shot length. The basic palette 450 is copied and inserted into the RGB space according to the shot length 461. Overlapping colors in the RGB space do not affect the convex hull shape, so it is geometrically meaningless to increase the number of the same colors by assigning weighted values ​​in the RGB space. Therefore, the weighted value can be geometrically reflected by jittering 462 of the color position, and can be expressed as follows.

[0107] Π * ={π∈Π|π+Ω(-σ,0)}, (5)

[0108] Wherein, π is the RGB color of the known basic palette 450, Π; Ω(-σ,σ) is a random function for generating RGB vectors through each RGB channel in the range of [-σ,σ], and σ=10 is set for the experiment. The scattered palette is created repeatedly over the lens length 461. In the case where many color points are scattered at positions with high weighted values, more vertices are required in the outer convex hull. Since the vertices of the convex hull constitute the final color scheme, the jitter can accurately reflect the weighted values.

[0109] Fig. 9 A diagram for explaining a color scheme merging unit according to another embodiment.

[0110] Reference Fig. 9 The colors of the basic color palette 450 may be the center of the cluster, and the color scheme merging unit 460 may perform additional clustering on the colors of the basic color palette 450 to obtain a unique final color scheme 470, but tends to ignore colors far from the center.

[0111] The colors of each scene are mostly selected from the movie and therefore cannot be ignored. Therefore, a convex hull enclosure 465 can be used to merge, thereby generating a color scheme 470 including all colors of the basic palette 450. The known colors can be configured in a three-dimensional RGB space to generate a convex hull and enclose all color points. The convex hull can be simplified to a fixed number of vertices 464 and includes all color points.

[0112] However, when the convex hull vertices 464 are directly used as the final color scheme 470, a saturated color may be selected and the color in the middle of the convex hull may be ignored. Fig. 9 As shown, instead of directly using the vertices 464 of the convex hull as a color palette, the convex hull is partitioned into sub-hulls 466 , and a representative color 467 of each sub-hull can be selected as a final color scheme 470 .

[0113] The central color can be obtained by averaging all the colors of the basic palette 450 with the same weighted value, and the above-mentioned vertex can be used as v. The convex hull can be simplified so that the number of faces of the convex hull mesh is the same as the size of the final palette. A sub-package in a triangular pyramid shape can be formed by connecting v and each triangular face. The vertex of the sub-package closest to the weighted value center of the sub-package can be the representative color of the sub-package. The weighted value center can be derived by assigning a weighted value to each color node of the sub-package according to the length of the quasi-main lens to which the node belongs. The center of each sub-package can be the color that constitutes the color scheme 470. This strategy can ease the saturation of the color and prevent the selection of non-existent colors.

[0114] According to an embodiment, a color scheme can be automatically extracted from a video to be used as a descriptor. Also, according to an embodiment, the present invention provides a two-step color scheme extraction technique, that is, a basic color palette can be extracted from a quasi-master shot and merged into a single color scheme, and the color clustering and convex hull enclosing methods can be used to exploit the advantages of each method.

[0115] The above-described device may be embodied as a hardware structural element, a software structural element, and / or a combination of hardware structural elements and software structural elements. For example, the device and structural elements described in the embodiment may be embodied by one or more common computers or special target computers such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that can execute and respond to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the above operating system. In addition, the processing device accesses, stores, operates, processes, and generates data in response to the execution of the software. For ease of understanding, the case of using only one processing device is described, and a person of ordinary skill in the art of the present invention may know that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, it may also be other processing configurations such as parallel processors.

[0116] Software may include a computer program, code, instructions, or a combination of more than one of them, to configure a processing device or to instruct a processing device independently or collectively in a manner to perform work as required. Software and / or data may be embodied in any type of machinery, component, physical device, computer storage medium or device for interpretation by a processing device or for providing instructions or data to a processing device. Software is distributed on computer systems connected by a network so that it can be stored or executed by a distributed method. Software and data may be stored in one or more computer-readable recording media.

[0117] The method of the embodiment is embodied in the form of program instructions that can be executed by a variety of computer units and recorded on a computer-readable medium. The above-mentioned computer-readable medium may include program execution, data files and data structures, etc., alone or in combination. The program instructions recorded on the above-mentioned medium may be specially designed or constructed for the embodiment, or may be instructions known to be used by ordinary technicians in the field of computer software. As an example of a computer-readable recording medium, magnetic media (magnetic media) such as hard disks, floppy disks and magnetic disks, optical recording media (optical media) such as CD-ROMs and DVDs, magneto-optical media (magneto-optical medium) such as optical magnetic floppy disks (floptical disks), and hardware devices specially constructed in a manner of storing instructions and executing such as read-only memories, random access memories, flash memories, etc. As an example of program instructions, mechanical codes formed by compilers and high-level language codes executed by computers through decoders, etc. are included.

[0118] As described above, although multiple embodiments are described through limited embodiments and drawings, as long as a person skilled in the art of the present invention can make various modifications and variations from the above description. For example, even if the described technology is performed in a different order from the described method and / or the structural elements of the described system, structure, device, circuit, etc. can be combined or combined in a different form from the described method, or replaced or substituted by other structural elements or equivalent technical solutions, appropriate results can be achieved.

[0119] Therefore, other examples, other embodiments and contents equivalent to the scope of protection of the invention also fall within the scope of protection of the invention described later.

Claims

1. A device for extracting color schemes from videos, It is characterized in that include: A quasi-main shot segmentation unit, which segments the input video into multiple quasi-main shots by detecting the boundaries of the quasi-main shots; A basic color palette extraction unit extracts a basic color palette from the divided quasi-master shot; and a color scheme merging unit, which merges the above-mentioned basic color palettes extracted from each of the above-mentioned quasi-master shots to extract a color scheme, wherein the above-mentioned basic color palette extraction unit includes: a key frame selection unit, which selects key frames by using multiple saliency maps to reduce the amount of data that needs to be clustered, wherein the above-mentioned saliency map represents the importance of pixels in each image along a human fixation point, and the above-mentioned key frame selection unit provides an objective function in a way that the above-mentioned key frame captures the entire shot content well, and the above-mentioned objective function is defined by a weighted sum of saliency, clarity and representativeness for each frame; and a color clustering unit, which clusters pixels according to distances in color space, and selects cluster centers as representative colors to extract a basic color palette in the set of the above-mentioned selected key frames.

2. The device for extracting color scheme from video according to claim 1, It is characterized in that The quasi-master shot subdivision section provides a quasi-master shot by combining similar colors to consecutive shots taken at the same position.

3. The device for extracting color scheme from video according to claim 1, It is characterized in that The quasi-main shot segmentation unit extracts saliency for each frame in the input video, measures similarity between a plurality of frames, and performs quasi-main shot boundary detection by clustering a plurality of similar frames, thereby segmenting the input video into a plurality of quasi-main shots.

4. The device for extracting color scheme from video according to claim 1, It is characterized in that The quasi-master shot segmentation unit segments the input video into a plurality of quasi-master shots by using an image processing software shot detector segmentation method that only considers RGB colors.

5. The device for extracting color scheme from video according to claim 1, It is characterized in that The color clustering unit assigns weight values ​​to clusters according to visual attention of pixels using the saliency map of the selected key frame.

6. The device for extracting color scheme from video according to claim 1, It is characterized in that The color arrangement merging unit performs additional clustering on the colors of the basic color palette to extract the final color arrangement.

7. The device for extracting color scheme from video according to claim 1, It is characterized in that The color scheme merging unit extracts the color scheme using convex hull encirclement, wherein the convex hull encirclement transforms the plurality of basic color palettes into a convex hull shape for an RGB color space, and selects a convex hull boundary vertex as a representative color of the color scheme.

8. The device for extracting color scheme from video according to claim 1, It is characterized in that The color scheme merging unit extracts the color scheme using a convex hull, divides the convex hull into sub-hulls, and selects a representative color of each of the sub-hulls as a representative color of the color scheme.

9. A method for extracting a color scheme from a video, comprising extracting a color scheme from a video using a device for extracting a color scheme from a video, It is characterized in that include: A segmentation step, segmenting the input video into multiple quasi-main shots by detecting the boundary of the quasi-main shot; An extraction step of extracting a basic color palette from the segmented quasi-master shot; and a merging step of merging the basic color palettes extracted from each of the quasi-master shots to extract a color scheme, wherein the step of extracting the basic color palette in the quasi-master shot comprises: a key frame selection step of selecting key frames by using a plurality of saliency maps to reduce the amount of data that needs to be clustered, wherein the saliency map represents the importance of pixels in each image along a human fixation point, and in the key frame selection step, providing an objective function in such a way that the key frames well capture the entire shot content, the objective function being defined by a weighted sum of saliency, clarity and representativeness for each frame; and a color clustering step of clustering pixels according to distances in a color space, selecting a cluster center as a representative color to extract a basic color palette in a set of the selected key frames.

10. The method for extracting color scheme from video according to claim 9, It is characterized in that In the step of dividing the input video into the above-mentioned quasi master shots, the quasi master shots are provided by combining colors similar to those of consecutive shots shot at the same position.

11. The method for extracting color scheme from video according to claim 9, It is characterized in that In the step of segmenting the input video into the quasi-main shots, saliency is extracted for each frame in the input video, similarity between multiple frames is measured, and quasi-main shot boundary detection is performed by clustering multiple similar frames, thereby segmenting the input video into multiple quasi-main shots.

12. The method for extracting color scheme from video according to claim 9, It is characterized in that In the step of segmenting the input video into the quasi-master shots, the input video is segmented into a plurality of quasi-master shots by using an image processing software shot detector segmentation method that only considers RGB colors.

13. The method for extracting color scheme from video according to claim 9, It is characterized in that In the color clustering step, the saliency map of the selected key frame is used to assign weighted values ​​to the clusters according to the visual attention of the pixels.

14. The method for extracting color scheme from video according to claim 9, It is characterized in that In the step of extracting a color scheme by merging the basic color palettes, additional clustering is performed on the colors of the basic color palettes to extract the final color scheme.

15. The method for extracting color scheme from video according to claim 9, It is characterized in that In the step of merging the basic color palettes to extract the color scheme, the color scheme is extracted using a convex hull enclosure, which transforms the multiple basic color palettes into a convex hull shape for the RGB color space, and selects the convex hull boundary vertex as a representative color of the color scheme.

16. The method for extracting color scheme from video according to claim 9, It is characterized in that In the step of merging the basic color palette to extract the color scheme, the color scheme is extracted by using a convex hull, the convex hull is divided into sub-hulls, and the representative color of each of the sub-hulls is selected as the representative color of the color scheme.