A video special effect rendering method, device and system in a film and television manufacturing process

Abstract

The application relates to the technical field of video special effect rendering, in particular to a video special effect rendering method, device and system in a film and television manufacturing process, which comprises the following steps: frame processing of a film and television video to obtain each frame image; acquiring each corner point and each connected domain in each frame image; acquiring the displacement fluctuation significance of each corner point in each frame image; acquiring the matching connected domain of each connected domain in each frame image in the previous frame image; acquiring the adjacent frame mutation value of each connected domain by analyzing the contour difference and color difference between each connected domain and the matching connected domain, and combining the displacement fluctuation significance of all corner points in each connected domain; and performing clustering analysis on the adjacent frame mutation values of all connected domains in all frame images to obtain each suspected key frame; and acquiring the key coefficient of each suspected key frame and the key change connected domain of each key frame. The application aims to improve the identification accuracy of key frames and the quality of video special effect rendering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video special effect rendering, and particularly relates to a video special effect rendering method, device and system in a film and television manufacturing process. BACKGROUND

[0002] In film and television production, special effects are important techniques for improving the visual effect and realism of works. Currently, key frames in a video are extracted for high-quality special effect rendering. By screening key frames with high information content, computing resources can be concentrated to perform high-density ray tracing or global illumination calculation on complex scenes for high-quality rendering, thereby improving the overall video rendering effect.

[0003] Existing technologies mainly extract key frames based on the texture features and motion changes of images. However, these methods can only extract the overall motion relationship between adjacent frames, and ignore local mutation phenomena between adjacent frames. Therefore, in some complex scenes, key frames are easily missed, and computing resources cannot be concentrated to perform high-quality rendering on the key frames, ultimately affecting the overall video rendering effect. SUMMARY

[0004] In view of the above, it is necessary to provide a video special effect high-quality rendering method, device and system in a film and television manufacturing process. Compared with traditional video special effect high-quality rendering methods, the recognition accuracy of key frames is improved, and the quality of video special effect rendering is improved.

[0005] In a first aspect, the embodiments of the present application provide a video special effect rendering method in a film and television manufacturing process, which comprises the following steps:

[0006] Frame processing is performed on a film and television video to obtain each frame image, and each corner point and each connected domain in each frame image is obtained.

[0007] For each corner point in each frame image, the position change of the corner point in the image and the preset adjacent image, and the displacement fluctuation significance of the corner point are obtained.

[0008] For each connected domain in each frame image, a matching connected domain of the connected domain in an adjacent frame image is obtained, the contour difference and color difference between the connected domain and the matching connected domain are analyzed, the displacement fluctuation significance of all corner points in the connected domain is combined, and the adjacent frame mutation value of the connected domain is obtained.

[0009] Cluster analysis is performed on the adjacent frame mutation values of all connected domains in all frame images to obtain each suspected key frame.

[0010] The time interval between each suspected key frame and its adjacent previous suspected key frame is calculated, and the key coefficient of each suspected key frame is obtained by the contour difference and color difference between each suspected key frame and its adjacent frame image, the change of the time interval, and the distribution of the adjacent frame mutation value of all connected domains in each suspected key frame.

[0011] The key frame and the key change connected domain of each key frame are determined by the distribution of the key coefficient of all suspected key frames and the adjacent frame mutation value of each connected domain of all suspected key frames.

[0012] In one embodiment, the process of obtaining the displacement fluctuation significance is as follows:

[0013] The frame image and its preset adjacent image are combined to form an adjacent image set of the frame image, the moving distance and moving angle of each corner point in the frame image between any two adjacent frame images in the adjacent image set are obtained, and the local change index of each corner point between the any two adjacent frame images is obtained by comparing the moving distance and moving angle of each corner point and its adjacent corner point.

[0014] The calculation formula of the displacement fluctuation significance of the u-th corner point in the i-th frame image is as follows:

[0015] In the formula, is the displacement fluctuation significance of the u-th corner point; , , are the average value of the moving distance, the dispersion of the moving distance, and the dispersion of the local change index of the u-th corner point between the any two adjacent frame images; and n represents the number of adjacent corner points of the u-th corner point.

[0016] In one embodiment, the process of calculating the local change index is as follows:

[0017] The average value of the moving distance of all adjacent corner points of each corner point is calculated, the average value is mapped to a positive number, and the ratio of the moving distance of each corner point to the positive number is calculated.

[0018] The average angle difference between the moving angle of each corner point and that of all adjacent corner points of the corner point is calculated.

[0019] The local change index is the product of the ratio and the average angle difference.

[0020] In one embodiment, the process of obtaining the adjacent frame mutation value is as follows:

[0021] The contour difference between each connected domain and its matched connected domain is calculated.

[0022] obtaining a color histogram of each connected domain, and taking a difference between the color histogram of each connected domain and the matched connected domain as a histogram difference;

[0023] calculating a sum of the contour difference and the histogram difference of each connected domain;

[0024] taking an average of the displacement fluctuation significance of all corner points in each connected domain as a significant average;

[0025] the adjacent frame mutation value is a product of the sum and the significant average.

[0026] In one embodiment, if the matched connected domain of each connected domain cannot be obtained, a color histogram of each frame image is obtained, a contour difference between any two frame images is calculated, and the sum is a sum of the contour difference and the histogram difference between the frame image and the adjacent previous frame image of each connected domain.

[0027] In one embodiment, the process of obtaining the suspected key frame is as follows:

[0028] taking an average of the adjacent frame mutation values of all connected domains in each frame image as a mutation average, obtaining each first clustering cluster of all frame images through the mutation average, and selecting the first clustering cluster with the largest element average;

[0029] counting a maximum value of the adjacent frame mutation values of all connected domains in each frame image, obtaining each second clustering cluster of all frame images through the maximum value, and extracting the second clustering cluster with the largest element average;

[0030] taking each frame image corresponding to the selection result and the extraction result as each suspected key frame.

[0031] In one embodiment, the process of obtaining the key coefficient is as follows:

[0032] calculating a time interval between each suspected key frame and the adjacent previous suspected key frame, taking an average of the time intervals of all suspected key frames as a time difference average, and taking a ratio of the time interval to the time difference average as a time difference ratio;

[0033] the calculation formula of the key coefficient is as follows:

[0034] ; in the formula, is a key coefficient of the jth suspected key frame; is a maximum value function; respectively represent values of the jth suspected key frame in the first and second clustering clusters; is a time difference ratio of the jth suspected key frame; , 、 Respectively represent the sum of the contour difference and color histogram difference between the jth suspected key frame and its adjacent previous frame image, the next frame image, and the previous suspected key frame, Indicates the absolute value operation.

[0035] In one embodiment, the process of screening and determining each key frame and the key change connected domain of each key frame is as follows:

[0036] Obtain the segmentation threshold of the key coefficients of all suspected key frames, and take each suspected key frame whose key coefficient is greater than the segmentation threshold as each key frame; take each connected domain within each key frame whose normalized value of the adjacent frame mutation value is greater than or equal to the preset threshold as each key change connected domain.

[0037] In a second aspect, an embodiment of the present application further provides a video special effects rendering device in a film and television production process, the device comprising:

[0038] The frame processing module is used to process the video frame to obtain each frame image; obtain each corner point and each connected domain in each frame image;

[0039] A key point recognition module is used to combine each frame image and its preset neighboring images into a neighboring image set for each frame image, and to obtain the displacement fluctuation significance of each corner point by analyzing the position change of each corner point in each frame image between any two adjacent frames in the neighboring image set;

[0040] The feature region recognition module is used to obtain the matching connected domains of each connected domain in each frame image in its previous frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, the adjacent frame mutation value of each connected domain is obtained;

[0041] The rendering area recognition module is used to perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame;

[0042] Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame;

[0043] Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

[0044] In a third aspect, the embodiments of the present application further provide a video special effect rendering system in a film manufacturing process, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the video special effect rendering method in the film manufacturing process according to any one of the above embodiments when executing the computer program.

[0045] The present application has at least the following beneficial effects:

[0046] According to the feature that the key frame will have a large motion mutation in the adjacent frame image, the present application constructs a displacement fluctuation significance, which can effectively identify the dynamic change between adjacent frames, so as to extract key feature points, and can make up for the deficiency of the traditional method which only relies on the overall motion relationship.

[0047] Further, considering that the object and scene in the key frame have a large mutation change, the adjacent frame mutation value is constructed in combination with the key feature points, so that the feature region with a key mutation in the frame image can be extracted, and accurate targets are provided for subsequent rendering;

[0048] Further, considering that there are different situations of local mutation and overall mutation of the object in the key frame, the frame images with overall and local significant changes are preliminarily screened through a clustering algorithm, so as to improve the processing efficiency while avoiding the omission of key frames;

[0049] Further, according to the time interval feature of the key frame and in combination with the uniqueness of the key frame, a key coefficient is constructed, which can fully analyze the mutation degree of the overall and local frame images, accurately identify the key frame, and thus improve the quality of the video special effect rendering. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0051] Figure 1 A step flowchart of a video special effect rendering method in a film manufacturing process provided by an embodiment of the present application;

[0052] Figure 2 A flowchart for acquiring suspected key frames;

[0053] Figure 3 A flowchart for acquiring key change connected domains. DETAILED DESCRIPTION

[0054] In the description of the embodiments of the present application, the words "exemplary", "or", "for example" are used to mean serving as an example, instance, or illustration, and not to imply any preference or advantage. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design schemes. In fact, the use of "exemplary", "or", "for example" is intended to present the relevant concept in a specific manner.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. It is to be understood that the use of "or" in the present application is meant to be an open-ended or inclusive "or" and not an exclusive "or".

[0056] In addition, it should be pointed out that the terms "first", "second" in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0057] The specific scheme of the video special effect rendering method, device and system in the film and television manufacturing process provided by the present application will be specifically described below in combination with the drawings.

[0058] Please refer to Figure 1 which shows the step flowchart of a video special effect rendering method in the film and television manufacturing process provided by an embodiment of the present application, and the method comprises the following steps:

[0059] Step S1, frame processing is performed on the film and television video to obtain each frame image, and each corner point and each connected domain in each frame image is obtained.

[0060] The video to be made into a film and television is frame processed to obtain each frame image, each frame image is respectively denoised and converted into a gray image, further, the corner points and edge pixel points in each frame image are respectively obtained through a corner point detection algorithm and an edge detection algorithm, and each connected domain in each frame image is obtained based on the corner points and edge pixel points by using a region growing algorithm. The region growing algorithm is a known technology, and will not be described herein.

[0061] In the present embodiment, when frame processing is performed on the video, the video with a time length of 1 second is divided into 60 frames, wherein 60 is only an embodiment of the present application, and the implementer can set the specific value by himself / herself, and the present application does not have special limitations.

[0062] In this embodiment, the Harris corner point detection algorithm is used to obtain the corner points in each frame image. The Harris corner point detection algorithm is a known technology, and will not be described herein. As an alternative, other existing technologies, such as the Shi-Tomasi corner point detection algorithm, can be used to obtain the corner points in each frame image, without special limitation.

[0063] In this embodiment, the Canny edge detection algorithm is used to obtain the edge pixel points in each frame image. The Canny edge detection algorithm is a known technology, and will not be described herein. As an alternative, other existing technologies, such as the Laplacian operator, can be used to obtain the edge pixel points in each frame image, without special limitation.

[0064] In this embodiment, all the corner points and edge pixel points in each frame image are collectively referred to as feature points. All the feature points are arranged in descending order of the gray value. The first preset number of feature points are used as initial seed points. In the growing process, the absolute value of the difference between the gray values of the growing pixel points and their adjacent pixel points is used as the similarity value. The normalized value of the similarity value is less than a preset threshold. Each growing region obtained by the region growing algorithm is used as a connected domain. The preset number and the preset threshold are both preset by a human being, and can be set by the implementer. In this embodiment, the preset number and the preset threshold are 15 and 0.2, respectively. The hyperbolic tangent function is used to obtain the normalized value of the similarity value.

[0065] In step S2, the displacement fluctuation significance of each corner point is obtained according to the position change of the corner point in the image and the preset adjacent image.

[0066] Since the key frame is a representative frame with significant visual changes in the video, it usually corresponds to the key turning points such as the switching of the scene, the appearance and disappearance of the object, or the significant change of the original object. The non-key frame is a transition frame between adjacent key frames, and the details of the picture change relatively small, and the changes of the object and the scene have continuity. Therefore, compared with the key frame, it does not have unique visual features and important information value. Therefore, the key degree of each frame image can be reflected by analyzing the texture and motion change degree of the object in the adjacent frame image.

[0067] Taking the u-th corner point in the i-th frame image as an example. All the corner points in the i-th frame image and the i-1-th frame image are used as the input of the KTL optical flow method, and the motion vector of each corner point is output, wherein the motion vector is a two-dimensional vector. The length of the motion vector of the corner point can reflect the moving distance of the corner point between the i-1-th frame image and the i-th frame image. The greater the length, the greater the moving distance of the corner point. The KTL optical flow method is a known technology, and will not be described herein.

[0068] It should be noted that: if the i-th frame image is the first frame, the motion vector of the corner point is calculated by the i-th frame image and the i+1-th frame image; if the u-th corner point does not exist in the i-1-th frame image or moves a long distance, resulting in that the motion vector cannot be calculated, the length of the motion vector of the u-th corner point is set to twice the average length of the motion vectors of all corner points in the i-th frame image, and the motion angle of the u-th corner point is set to 0, to represent that the key degree of the u-th corner point is large.

[0069] A window with a size of is constructed with the u-th corner point as the center, and the remaining corner points in the window except the center corner point are used as the near neighbor corner points of the u-th corner point.

[0070] In the embodiment, the value of a is 7, and the value of a is artificially preset, which can be set by the implementer, and the application does not have special restrictions.

[0071] The average value of the moving distances of all the near neighbor corner points of each corner point is calculated, the average value is mapped to a positive number, and the ratio of the moving distance of each corner point to the positive number is calculated; the ratio can reflect the motion amplitude difference between the u-th corner point and its near neighbor corner points. The purpose of mapping the average value to a positive number is to avoid a denominator of 0.

[0072] In the embodiment, the average value is mapped to a positive number by calculating the sum of the average value and a preset positive number, the value of the preset positive number is artificially preset, which can be set by the implementer, and in the embodiment, the value of the preset positive number is 0.01. There are many methods for mapping data to a positive number, and the implementer can select other feasible methods, and the application does not have special restrictions.

[0073] For the u-th corner point and its near neighbor corner points, the motion vector is calculated by the arctan() function to obtain the moving angle of the u-th corner point and its near neighbor corner points, respectively. The average of the difference between the moving angles of the u-th corner point and all its near neighbor corner points is denoted as the average angle difference. The calculation of the angle by the arctan() function and the displacement change is a known technology, and will not be described herein.

[0074] The product of the ratio of the u-th corner point and the average angle difference is used as the local change index of the u-th corner point between the i-1-th frame image and the i-th frame image. The local change index can reflect the motion amplitude difference and the motion angle deviation between the u-th corner point and its near neighbor corner points between adjacent frame images.

[0075] Each frame image and its preset near neighbor image are combined to form a near neighbor image set of each frame image.

[0076] In this embodiment, the preset neighboring images of each frame image are the adjacent previous 3 frame images and the adjacent next 3 frame images of the frame image, wherein 3 is only an embodiment of the present application, and the implementer can set the specific value thereof by himself / herself, and the present application does not make special limitation. When the i-th frame image is located at the beginning or the end of the video, and the number of images in front or behind is not enough, only the actually available images can be used for calculation.

[0077] By analyzing the position change of each corner point in all arbitrary adjacent two frame images in the neighboring image set, the displacement fluctuation significance of each corner point is obtained, and the expression is as follows:

[0078] In the formula, u represents the u-th corner point, and n represents the number of neighboring corner points of the u-th corner point. is the displacement fluctuation significance of the u-th corner point. are respectively the mean value of the moving distance, the dispersion of the moving distance, and the dispersion of the local change index of the u-th corner point between all arbitrary adjacent two frame images; and n represents the number of neighboring corner points of the u-th corner point.

[0079] It should be noted that in the process of calculating the displacement fluctuation significance, the dispersion involved is the coefficient of variation, which is a known technology, and the present application will not be described again. As other implementation manners, on the basis of being able to measure the uneven degree of the distribution of the moving distance and the uneven degree of the distribution of the local change index, the implementer can use other existing technologies to measure, such as variance, standard deviation, etc., and the present application does not make special limitation.

[0080] It should be noted that the displacement fluctuation significance can reflect the fluctuation of the motion state of the u-th corner point between the previous and next multiple continuous frames. The greater the displacement fluctuation significance is, the more obvious the instability of the u-th corner point in the local change and the time sequence is, thereby reflecting that the u-th corner point is more likely to be related to the scene key turning point.

[0081] The displacement fluctuation significance of each corner point in each frame image is calculated by using the same calculation method as the displacement fluctuation significance of the u-th corner point in the i-th frame image.

[0082] In step S3, for each connected domain in each frame image, a matching connected domain of the connected domain in the adjacent frame image is obtained, the contour difference and the color difference between the connected domain and the matching connected domain are analyzed, the displacement fluctuation significance of all corner points in the connected domain is combined, and the adjacent frame mutation value of the connected domain is obtained.

[0083] ​​The displacement fluctuation of the corner point can only quantify the motion change characteristics of a single corner point, but since the determination of the key frame needs to combine the global changes of the objects and scenes in the video, the features of the connected domain need to be combined to further distinguish whether a large mutation of the object has occurred in the image frame.

[0084] Taking the vth connected domain in the i-th image as an example, the i-th image and the i-1-th image are matched by the image matching algorithm to obtain the matching connected domain of the vth connected domain in the i-1-th image.

[0085] In this embodiment, the SIFT algorithm is used to obtain the matching connected domain of the vth connected domain. The SIFT algorithm is a known technology, and will not be described herein. As other embodiments, on the basis of being able to obtain the matching connected domain of the vth connected domain, the implementer can use other existing technologies, such as the SURF algorithm, and the present application does not make special restrictions.

[0086] The contour difference between the vth connected domain and its matching connected domain is calculated. The greater the contour difference, the greater the shape change of the vth connected domain between adjacent image frames.

[0087] In this embodiment, the contour difference between the vth connected domain and its matching connected domain is reflected by the Hausdorff distance between the vth connected domain and its matching connected domain. The greater the Hausdorff distance, the greater the contour difference between the vth connected domain and its matching connected domain. The Hausdorff distance is a known technology, and will not be described herein. As other embodiments, on the basis of being able to measure the contour difference between the connected domains, the implementer can use other existing technologies, such as the Hu moment function, and the present application does not make special restrictions.

[0088] The color histogram of the vth connected domain and its matching connected domain is obtained respectively, and the difference between the two color histograms is denoted as the histogram difference. The color difference between the vth connected domain and its matching connected domain is reflected by the difference between the color histograms. The greater the difference between the color histograms, the greater the color change of the vth connected domain between adjacent image frames. The method for obtaining the color histogram is a known technology, and will not be described herein.

[0089] In this embodiment, the difference between the color histograms is calculated by the Jensen-Shannon divergence. The Jensen-Shannon divergence is a known technology, and will not be described herein. As other embodiments, on the basis of being able to measure the difference between the color histograms, the implementer can use other existing technologies, such as the Kullback-Leibler divergence, and the present application does not make special restrictions.

[0090] The adjacent frame mutation value of each connected domain in each frame image is calculated by using the same calculation method as the adjacent frame mutation value of the vth connected domain in the ith frame image.

[0091] In the formula, Δv represents the adjacent frame mutation value of the vth connected domain; and Δv represents the adjacent frame mutation value of the vth connected domain; and Δv represents the adjacent frame mutation value of the vth connected domain; and Δv represents the adjacent frame mutation value of the vth connected domain; and The significant mean value is recorded as Δv.

[0092] If the matching connected domain of the vth connected domain in the ith frame image cannot be obtained in the (i-1)th frame image, the color histogram of each frame image is obtained, and the contour difference degree between any two frame images is calculated. is the sum of the contour difference degree and the color histogram difference between the ith frame image and the (i-1)th frame image.

[0093] In the embodiment, the contour difference degree between the ith frame image and the (i-1)th frame image is reflected by the Hausdorff distance between the ith frame image and the (i-1)th frame image. As other embodiments, on the basis of measuring the contour difference between the ith frame image and the (i-1)th frame image, the implementer can use other existing technologies, such as the Hu moment function, and the present application does not make special limitations.

[0094] It should be noted that the adjacent frame mutation value can reflect the comprehensive change degree of the vth connected domain in the ith frame image; the greater the adjacent frame mutation value, the greater the difference between the vth connected domain and its matching connected domain in local motion, shape and color, and the more likely the vth connected domain is a key mutation object.

[0095] The adjacent frame mutation values of the connected domains in each frame image are calculated by using the same calculation method as the adjacent frame mutation value of the vth connected domain in the ith frame image.

[0096] In step S4, the adjacent frame mutation values of all connected domains in all frame images are subjected to cluster analysis to obtain each suspected key frame.

[0097] The mean value of the adjacent frame mutation values of all connected domains in each frame image is recorded as a mutation mean value, each first cluster of all frame images is obtained through the mutation mean value, and the first cluster with the largest element mean value is selected.

[0098] The maximum value of the adjacent frame mutation value of all the connected domains in each frame image is counted, each second clustering cluster of all the frame images is obtained through the maximum value, and a second clustering cluster with the maximum element mean is extracted. Each frame image corresponding to the selection result and the extraction result is taken as a suspected key frame. A flowchart of the acquisition of the suspected key frame is shown in FIG. 8. Figure 2

[0099] In this embodiment, the K-means algorithm is used to obtain the first clustering cluster and the second clustering cluster, and the elbow rule is used to obtain the optimal number of clustering clusters. The K-means algorithm is a known technology, and thus will not be described herein. As other embodiments, on the basis of being able to obtain the first clustering cluster and the second clustering cluster, the implementer can use other existing technologies, such as the density peak clustering algorithm, the fuzzy C-means clustering, etc., and the present application does not make special limitations.

[0100] The first clustering cluster can reflect the overall motion change in each frame image, and corresponds to the case of scene switching or large-scale motion of an object. The second clustering cluster corresponds to the case that the overall change in the frame image is not obvious, but the local object has significant changes. Through clustering, the images with large changes can be preliminarily screened, so that the efficiency is improved and the images with general overall changes but significant local changes can be avoided.

[0101] In step 5, the time interval between each suspected key frame and the adjacent previous suspected key frame is calculated, the key coefficient of each suspected key frame is obtained through the contour difference and the color difference between each suspected key frame and the adjacent frame image, the remaining suspected key frames, the change of the time interval, and the distribution of the adjacent frame mutation value of all the connected domains in each suspected key frame.

[0102] Taking the jth suspected key frame as an example. Since there are a large number of transition frames in the video, the time interval between the key frames is large. Therefore, all the suspected key frames are sorted according to the time sequence of the suspected key frames in the video, and the time interval between each suspected key frame and the adjacent previous suspected key frame is calculated. The average value of the time interval of all the suspected key frames is denoted as the time difference average value. The ratio of the time interval of the jth suspected key frame to the time difference average value is denoted as the time difference ratio of the jth suspected key frame. The smaller the time difference ratio, the larger the time interval between the jth suspected key frame and the adjacent previous suspected key frame compared with the average time interval, and the more the jth suspected key frame meets the time interval characteristics of the key frame.

[0103] Further, since the video has significant changes at the key frame, the similarity between the key frame and the adjacent frame images before and after the key frame has a large difference. At the same time, since the content of each key frame has high uniqueness, different key frames do not have similarity. ​

[0104] The key coefficient of each suspected key frame is obtained by the contour difference and color difference between each suspected key frame and its adjacent frame image, the rest of the suspected key frames, in combination with the time difference ratio of each suspected key frame, and the distribution of the adjacent frame mutation value of all connected domains in each suspected key frame, and the expression is:

[0105] In the formula, is the key coefficient of the jth suspected key frame; is the maximum value function; respectively represent the value of the jth suspected key frame in the first and second clustering clusters; is the time difference ratio of the jth suspected key frame; respectively represent the sum of the contour difference and color histogram difference between the jth suspected key frame and its adjacent previous frame image, next frame image, and previous suspected key frame; represents the absolute value operation. If the jth suspected key frame is not in the first clustering cluster, then is 0; if it is not in the second clustering cluster, then is 0.

[0106] In this embodiment, the contour difference involved in the process of calculating the key coefficient is reflected by the Hausdorff distance. As an alternative, on the basis of being able to measure the contour difference between the jth suspected key frame and its adjacent previous frame image, next frame image, and previous suspected key frame, the implementer can use other existing technologies such as the Hu moment function, and the present application does not make special limitations.

[0107] It should be noted that: the maximum significant degree of the jth suspected key frame in the overall change and the local change can be obtained, so as to further accurately select the key frame; the difference change between the jth suspected key frame and its adjacent frame, the rest of the suspected key frames can be reflected, so as to detect the content uniqueness of the suspected key frame; and the time difference ratio can reflect the difference between the time interval of the jth suspected key frame and the average time interval, which is used to detect whether the jth suspected key frame meets the time interval feature of the key frame. The larger the key coefficient is, the more the jth suspected key frame meets the unique features of the key frame, and the more likely it is a key frame.

[0108] The key coefficient of each suspected key frame is calculated by using the same calculation method as the key coefficient of the jth suspected key frame.

[0109] ​​Step S6, screening and determining each key frame and the key change connected domain of each key frame through the distribution of the key coefficients of all suspected key frames and the adjacent frame mutation value of each connected domain of all suspected key frames.

[0110] Obtaining the segmentation threshold of the key coefficients of all suspected key frames, and regarding each suspected key frame with a key coefficient greater than the segmentation threshold as each key frame.

[0111] In this embodiment, the Otsu threshold segmentation algorithm is used to obtain the segmentation threshold of the key coefficients of all suspected key frames. The Otsu threshold segmentation algorithm is a known technology, and will not be described herein. As other embodiments, on the basis of being able to obtain the segmentation threshold of the key coefficients of all suspected key frames, the implementer can use other existing technologies, such as global threshold segmentation, iterative threshold segmentation, etc., which are not specially limited by the present application.

[0112] Regarding each connected domain with a normalized value of the adjacent frame mutation value in each key frame greater than or equal to a preset threshold as each key change connected domain, and regarding each connected domain with a normalized value of the adjacent frame mutation value less than the preset threshold as a background connected domain. The acquisition process of the key change connected domain is shown in FIG. 8. Figure 3

[0113] In this embodiment, the hyperbolic tangent function is used to obtain the normalized value of the adjacent frame mutation value.

[0114] In this embodiment, the value of the preset threshold is 0.5. The value of the preset threshold is preset by a person, and can be set by the implementer, which is not specially limited by the present application.

[0115] Rendering the key change connected domain that needs to be rendered, and after the rendering is completed, performing video synthesis on the rendered key frame and the remaining frame images.

[0116] Based on the same inventive concept as the above method, the present embodiment further provides a video special effect high-quality rendering device in a film and television manufacturing process, comprising:

[0117] A frame processing module is configured to perform frame processing on the film and television video to obtain each frame image, and obtain each corner point and each connected domain in each frame image.

[0118] A key point identification module is configured to group each frame image and a preset adjacent image thereof into an adjacent image set of each frame image, and obtain a displacement fluctuation degree of each corner point by analyzing a position change of each corner point in all adjacent two frame images in the adjacent image set.

[0119] ​The feature region identification module is configured to obtain a matching connected domain of each connected domain in each frame image in a previous frame image, obtain a neighboring frame mutation value of each connected domain by analyzing a contour difference and a color difference between each connected domain and the matching connected domain, and combining a displacement fluctuation significance of all corner points in each connected domain.

[0120] The rendering region identification module is configured to perform clustering analysis on the neighboring frame mutation values of all connected domains in all frame images to obtain each suspected key frame.

[0121] The time interval between each suspected key frame and a neighboring previous suspected key frame is calculated, and a key coefficient of each suspected key frame is obtained by the contour difference and the color difference between each suspected key frame and a neighboring frame image, between the suspected key frame and the remaining suspected key frames, in combination with a change of the time interval and a distribution of the neighboring frame mutation values of all connected domains in the suspected key frame.

[0122] Each key frame and a key change connected domain of each key frame are determined by the distribution of the key coefficients of all suspected key frames, in combination with the neighboring frame mutation values of all connected domains of all suspected key frames.

[0123] Based on the same inventive concept as the above method, the embodiments of the present application also provide a video special effect rendering system in a film and television manufacturing process, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of any one of the above-mentioned video special effect rendering method in the film and television manufacturing process when executing the computer program.

[0124] In summary, according to the feature that the key frame will have a large motion mutation in the neighboring frame image, the displacement fluctuation significance is constructed, which can effectively identify the dynamic change between the neighboring frames, so as to extract the key feature points, and can make up for the deficiency of the traditional method which only relies on the overall motion relationship.

[0125] Further, considering that the object and the scene in the key frame have a large mutation change, the neighboring frame mutation value is constructed in combination with the key feature points, so that the feature region which has a key mutation in the frame image can be extracted, and an accurate target is provided for subsequent rendering.

[0126] Further, considering that there are different situations of local mutation and overall mutation of the object in the key frame, the frame images with overall and local significant changes are preliminarily screened by the clustering algorithm, so that the processing efficiency is improved while avoiding the omission of the key frame.

[0127] Further, according to the time interval feature of the key frame, the key coefficient is constructed in combination with the uniqueness of the key frame, so that the overall and local mutation degrees of the frame image can be fully analyzed, the key frame can be accurately identified, and the quality of the video special effect rendering is improved.

[0128] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flow diagrams and / or block diagrams.

[0129] Obviously, person skilled in the art will recognize that the present application is not limited to the details of the foregoing illustrative embodiments and can be implemented in other ways without departing from the spirit or essential characteristics of the application. The present application thus encompasses modifications obvious to one skilled in the art that are not set forth in detail herein. Furthermore, the present application encompasses other embodiments o that are apparent to one skilled in the art in connection with the foregoing description and accompanying drawings.

Claims

1. A method for rendering video special effects in a film and television production process, characterized in that: The method comprises the following steps: Process the video frame by frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; For each corner point in each frame image, the position change of each corner point in the image in which it is located and the preset neighboring image is obtained to obtain the displacement fluctuation significance of each corner point; For each connected domain in each frame image, obtain the matching connected domain of each connected domain in the adjacent frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, obtain the adjacent frame mutation value of each connected domain; Perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

2. The method for rendering video special effects in a film and television production process according to claim 1, wherein: The process of obtaining the displacement fluctuation significance is as follows: Each frame image and its preset neighboring images are combined into a neighboring image set for each frame image, and the movement distance and movement angle of each corner point in each frame image between any two adjacent frames in the neighboring image set are obtained. By comparing the movement distance and movement angle of each corner point with its neighboring corner points, the local change index of each corner point between the two adjacent frames is obtained; The calculation formula for the displacement fluctuation significance of the u-th corner point in the i-th frame image is: Where, is the displacement fluctuation significance of the u-th corner point; 、 、 are the mean of the moving distance of the u-th corner point between any two adjacent frames, the discreteness of the moving distance, and the discreteness of the local change index; n represents the number of neighboring corner points of the u-th corner point.

3. The method for rendering video special effects in a film and television production process according to claim 2, wherein: The calculation process of the local change index is: Calculate the average of the moving distances of all neighboring corner points of each corner point, map the average to a positive number, and calculate the ratio of the moving distance of each corner point to the positive number; The average of the differences in the moving angles of each corner point and all its neighboring corner points is recorded as the average angle difference; The local variation index is the product of the ratio and the average angle difference.

4. The method for rendering video special effects in a film and television production process according to claim 1, wherein: The process of obtaining the adjacent frame mutation value is as follows: Calculate the contour difference between each connected domain and its matching connected domain; Obtain the color histogram of each connected domain respectively, and record the difference in color histogram between each connected domain and its matching connected domain as the histogram difference; Calculating the sum of the contour difference and the histogram difference of each connected domain; The mean of the displacement fluctuation significance of all corner points in each connected domain is recorded as the significant mean; The adjacent frame mutation value is the product of the sum value and the significant mean value.

5. The method for rendering video special effects in a film and television production process according to claim 4, characterized in that: If the matching connected domains of each connected domain cannot be obtained, the color histograms of each frame image are obtained respectively, and the contour difference between any two frame images is calculated. The sum value is the sum of the contour difference and color histogram difference between the frame image where each connected domain is located and its adjacent previous frame image.

6. The method for rendering video special effects in a film and television production process according to claim 5, wherein: The process of obtaining the suspected key frame is as follows: The average of the mutation values ​​of adjacent frames of all connected domains in each frame image is recorded as the mutation mean, and the first clusters of all frame images are obtained by the mutation mean, and the first cluster with the largest element mean is selected; Counting the maximum value of the mutation value of adjacent frames of all connected domains in each frame image, obtaining each second cluster of all frame images according to the maximum value, and extracting the second cluster with the largest element mean; The image frames corresponding to the selected results and the extracted results are taken as suspected key frames.

7. The method for rendering video special effects in a film and television production process according to claim 6, wherein: The process of obtaining the key coefficients is as follows: Calculate the time interval between each suspected key frame and its adjacent previous suspected key frame, and record the average of the time intervals of all suspected key frames as the time difference average; and record the ratio of the time interval to the time difference average as the time difference ratio; The calculation formula of the key coefficient is: Where, is the key coefficient of the jth suspected key frame; is the maximum value function; Represent the values ​​of the j-th suspected key frame in the first and second clusters respectively; is the time difference ratio of the jth suspected key frame; 、 、 Respectively represent the sum of the contour difference and color histogram difference between the jth suspected key frame and its adjacent previous frame image, the next frame image, and the previous suspected key frame, Indicates the absolute value operation.

8. The method for rendering video special effects in a film and television production process according to claim 1, wherein: The process of screening and determining each key frame and the key change connected domain of each key frame is as follows: Obtain the segmentation threshold of the key coefficients of all suspected key frames, and take each suspected key frame whose key coefficient is greater than the segmentation threshold as each key frame; take each connected domain within each key frame whose normalized value of the adjacent frame mutation value is greater than or equal to the preset threshold as each key change connected domain.

9. A video special effects rendering device in a film and television production process, using a high-quality video special effects rendering method in a film and television production process according to claim 1, characterized in that: The device comprises: The frame processing module is used to process the video frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; Key point recognition module, the position changes of each corner point in its image and preset neighboring images, and the displacement fluctuation significance of each corner point are obtained; The feature region recognition module obtains the matching connected domains of each connected domain in each frame image in the adjacent frame image. By analyzing the contour and color differences between each connected domain and its matching connected domain, and combining the displacement fluctuation significance of all corner points in each connected domain, the adjacent frame mutation value of each connected domain is obtained. The rendering region identification module is used to perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

10. A video special effects rendering system in a film and television production process, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method for high-quality rendering of video special effects in a film and television production process as described in any one of claims 1-8 are implemented.

Images