Video special effect high-quality rendering method, device and system in film and television manufacturing process

Abstract

The invention relates to the technical field of video special effect rendering, in particular to a video special effect high-quality rendering method, device and system in the film and television manufacturing process, and the method comprises the steps: carrying out the framing processing of a film and television video, and obtaining each frame of image; obtaining each angular point and each connected domain in each frame of image; obtaining the displacement fluctuation saliency of each angular point in each frame of image; acquiring a matched connected domain of each connected domain in each frame of image in the previous frame of image, and acquiring an adjacent frame abrupt change value of each connected domain by analyzing the contour difference and color difference between each connected domain and the matched connected domain thereof and combining the displacement fluctuation saliency of all angular points in each connected domain; clustering analysis is carried out through the adjacent frame abrupt change values of all the connected domains in all the frame images, and all suspected key frames are obtained; acquiring a key coefficient of each suspected key frame and a key change connected domain of each key frame; the invention aims to improve the quality of video special effect rendering by improving the recognition precision of the key frame.

Description

Technical Field

[0001] The present application relates to the field of video special effects rendering technology, and specifically to a method, device and system for high-quality rendering of video special effects in a film and television production process. Background Art

[0002] In film and television production, special effects are an important technology for enhancing the visual effects and realism of works. Currently, high-quality special effects rendering is often performed by extracting key frames from videos. By screening key frames with high information content, computing resources can be concentrated to perform high-quality feature rendering such as high-density ray tracing or global illumination calculation on complex scenes, thereby improving the overall special effects rendering effect of the video.

[0003] Existing technologies mainly extract key frames based on the texture features and motion changes of images. However, these methods can usually only extract the overall motion relationship between adjacent frames of the video, while ignoring the local mutation phenomenon between adjacent frames. Therefore, in some complex scenes, it is easy to cause key frames to be missed, and then it is impossible to concentrate computing resources for high-quality rendering of key frames, which ultimately affects the overall video rendering effect. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method, device and system for high-quality rendering of video special effects in the film and television production process. Compared with traditional high-quality rendering methods of video special effects, the quality of video special effects rendering is improved by improving the recognition accuracy of key frames: In a first aspect, an embodiment of the present application provides a method for high-quality rendering of video special effects in a film and television production process, the method comprising the following steps: Process the video frame by frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; For each corner point in each frame image, the position change of each corner point in the image in which it is located and the preset neighboring image is obtained to obtain the displacement fluctuation significance of each corner point; For each connected domain in each frame image, obtain the matching connected domain of each connected domain in the adjacent frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, obtain the adjacent frame mutation value of each connected domain; Perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

[0005] In one embodiment, the process of obtaining the displacement fluctuation significance is as follows: Each frame image and its preset neighboring images are combined into a neighboring image set for each frame image, and the movement distance and movement angle of each corner point in each frame image between any two adjacent frames in the neighboring image set are obtained. By comparing the movement distance and movement angle of each corner point with its neighboring corner points, the local change index of each corner point between the two adjacent frames is obtained; The calculation formula for the displacement fluctuation significance of the u-th corner point in the i-th frame image is: ; Where, is the displacement fluctuation significance of the u-th corner point; 、 、 are respectively the mean of the moving distance of the u-th corner point between all two adjacent frames, the discreteness of the moving distance, and the discreteness of the local change index; n represents the number of neighboring corner points of the u-th corner point.

[0006] In one embodiment, the calculation process of the local variation index is: Calculate the average of the moving distances of all neighboring corner points of each corner point, map the average to a positive number, and calculate the ratio of the moving distance of each corner point to the positive number; The average of the differences in the moving angles of each corner point and all its neighboring corner points is recorded as the average angle difference; The local variation index is the product of the ratio and the average angle difference.

[0007] In one embodiment, the process of obtaining the adjacent frame mutation value is as follows: Calculate the contour difference between each connected domain and its matching connected domain; Obtain the color histogram of each connected domain respectively, and record the difference in color histogram between each connected domain and its matching connected domain as the histogram difference; Calculating the sum of the contour difference and the histogram difference of each connected domain; The mean of the displacement fluctuation significance of all corner points in each connected domain is recorded as the significant mean; The adjacent frame mutation value is the product of the sum value and the significant mean value.

[0008] In one embodiment, if the matching connected domains of each connected domain cannot be obtained, the color histogram of each frame image is obtained respectively, and the contour difference between any two frame images is calculated, and the sum value is the sum of the contour difference and color histogram difference between the frame image where each connected domain is located and its adjacent previous frame image.

[0009] In one embodiment, the process of obtaining the suspected key frame is: The average of the mutation values ​​of adjacent frames of all connected domains in each frame image is recorded as the mutation mean, and the first clusters of all frame images are obtained by the mutation mean, and the first cluster with the largest element mean is selected; Counting the maximum value of the mutation value of adjacent frames of all connected domains in each frame image, obtaining each second cluster of all frame images according to the maximum value, and extracting the second cluster with the largest element mean; The image frames corresponding to the selected results and the extracted results are taken as suspected key frames.

[0010] In one embodiment, the process of obtaining the key coefficient is: Calculate the time interval between each suspected key frame and its adjacent previous suspected key frame, and record the average of the time intervals of all suspected key frames as the time difference average; and record the ratio of the time interval to the time difference average as the time difference ratio; The calculation formula of the key coefficient is: Where, is the key coefficient of the jth suspected key frame; is the maximum value function; Represent the values ​​of the j-th suspected key frame in the first and second clusters respectively; is the time difference ratio of the jth suspected key frame; 、 、 Respectively represent the sum of the contour difference and color histogram difference between the jth suspected key frame and its adjacent previous frame image, the next frame image, and the previous suspected key frame, Indicates the absolute value operation.

[0011] In one embodiment, the process of screening and determining each key frame and the key change connected domain of each key frame is as follows: Obtain the segmentation threshold of the key coefficients of all suspected key frames, and take each suspected key frame whose key coefficient is greater than the segmentation threshold as each key frame; take each connected domain within each key frame whose normalized value of the adjacent frame mutation value is greater than or equal to the preset threshold as each key change connected domain.

[0012] In a second aspect, an embodiment of the present application further provides a high-quality rendering device for video special effects in a film and television production process, the device comprising: The frame processing module is used to process the video frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; A key point recognition module is used to combine each frame image and its preset neighboring images into a neighboring image set for each frame image, and to obtain the displacement fluctuation significance of each corner point by analyzing the position change of each corner point in each frame image between any two adjacent frames in the neighboring image set; The feature region recognition module is used to obtain the matching connected domains of each connected domain in each frame image in its previous frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, the adjacent frame mutation value of each connected domain is obtained; The rendering area recognition module is used to perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

[0013] In a third aspect, an embodiment of the present application also provides a high-quality rendering system for video special effects in a film and television production process, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned methods for high-quality rendering of video special effects in a film and television production process are implemented.

[0014] This application has at least the following beneficial effects: This application constructs displacement fluctuation significance based on the characteristic that key frames will undergo large motion mutations in adjacent frame images. It can effectively identify dynamic changes between adjacent frames and thus extract key feature points, which can make up for the shortcomings of traditional methods that only rely on overall motion relationships. Furthermore, considering that objects and scenes in key frames have large mutation changes, the mutation values ​​of adjacent frames are constructed by combining key feature points, so that the feature areas with key mutations in the frame image can be extracted, providing precise targets for subsequent rendering; Furthermore, considering the different situations of local and global mutations of objects in key frames, a clustering algorithm is used to preliminarily screen the frames with significant global and local changes, thereby improving processing efficiency and avoiding the omission of key frames. Furthermore, based on the time interval characteristics of key frames and the uniqueness of key frames, key coefficients are constructed, which can fully analyze the overall and local mutation levels of frame images and accurately identify key frames, thereby improving the quality of video special effects rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of a method for high-quality rendering of video special effects in a film and television production process provided by one embodiment of the present application; Figure 2 Schematic diagram of the process of obtaining suspected key frames; Figure 3 Schematic diagram of the process of obtaining key change connected domains. DETAILED DESCRIPTION

[0017] In the description of the embodiments of this application, words such as "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "or," and "for example" is intended to present the relevant concepts in a concrete manner.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application relates. The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. It should be understood that, unless otherwise indicated, " / " represents or.

[0019] It should also be noted that the terms "first" and "second" in this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0020] The following describes in detail with reference to the accompanying drawings a specific solution of a method, device and system for high-quality rendering of video special effects in a film and television production process provided by the present application.

[0021] See also Figure 1 , which shows a flowchart of a method for high-quality rendering of video special effects in a film and television production process provided by one embodiment of the present application, the method comprising the following steps: Step S1: Process the video frame by frame to obtain each frame image; obtain each corner point and each connected domain in each frame image.

[0022] The video to be used for film and television production is divided into frames to obtain individual frames. Each frame is then denoised and converted into a grayscale image. Furthermore, corner and edge pixel points within each frame are obtained using a corner detection algorithm and an edge detection algorithm, respectively. Based on these corner and edge pixel points, a region growing algorithm is used to obtain connected domains within each frame. The region growing algorithm is a well-known technique and will not be described in detail in this application.

[0023] In this embodiment, when the video is framed, a video with a duration of 1 second is divided into 60 frames, where 60 is only an embodiment of the present application, and the implementer can set its specific value by himself, and this application does not impose any special restrictions.

[0024] In this embodiment, the Harris corner detection algorithm is used to obtain the corner points in each frame image. The Harris corner detection algorithm is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to obtain the corner points in each frame image, the implementer may adopt other existing technologies, such as the Shi-Tomasi corner detection algorithm, etc., and this application does not impose any special restrictions.

[0025] In this embodiment, the Canny edge detection algorithm is used to obtain edge pixel points in each frame image. The Canny edge detection algorithm is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to obtain edge pixel points in each frame image, the implementer can adopt other existing technologies, such as the Laplacian operator, etc., and this application does not impose any special restrictions.

[0026] In this embodiment, for each frame of an image, all corner points and edge pixels within the image are collectively referred to as feature points. These feature points are arranged from largest to smallest grayscale value, and a preset number of feature points are used as initial seed points. During the growth process, the absolute difference in grayscale value between the growing pixel and its adjacent pixels is used as the similarity value. The growth condition is that the normalized value of the similarity value is less than a preset threshold. The growing regions obtained through the region growing algorithm are considered connected domains. The preset number and threshold values ​​are both manually set and can be customized by the implementer. In this embodiment, the preset number and threshold values ​​are 15 and 0.2, respectively. A hyperbolic tangent function is used to obtain the normalized similarity value.

[0027] Step S2: for each corner point in each frame image, obtain the displacement fluctuation significance of each corner point based on the position change of each corner point in the image in which it is located and the preset neighboring image.

[0028] Keyframes are representative frames in a video that exhibit significant visual changes, typically corresponding to critical turning points such as scene changes, the appearance or disappearance of an object, or significant changes to an existing object. Non-keyframes, on the other hand, are transitional frames between adjacent keyframes. These frames exhibit relatively minor changes in image detail, and the objects and scenes exhibit continuous changes. Therefore, compared to keyframes, non-keyframes lack distinct visual characteristics and significant information value. Therefore, the criticality of each frame can be determined by analyzing the degree of change in texture and motion of objects within adjacent frames.

[0029] Take the uth corner point in the i-th image frame as an example. All corner points in the i-th and i-1-th frames are used as input for the KTL optical flow method, which outputs a motion vector for each corner point. The motion vector is a two-dimensional vector. The modulus of the corner point's motion vector reflects the distance the corner point has moved between the i-1-th and i-th frames. A larger modulus indicates a greater distance the corner point has moved. The KTL optical flow method is well known and will not be described in detail in this application.

[0030] It should be noted that: if the i-th frame image is the first frame, the motion vector of the corner point is calculated using the i-th frame image and the i+1-th frame image; if the u-th corner point does not exist in the i-1-th frame image or moves a long distance, resulting in the motion vector being unable to be calculated, the motion vector modulus of the u-th corner point is set to twice the average modulus of the motion vectors of all corner points in the i-th frame image, and the motion angle of the u-th corner point is set to 0 to indicate that the u-th corner point is more critical.

[0031] With the u-th corner point as the center, the construction size is The window is composed of the corner points except the central corner point in the window, and the remaining corner points in the window are regarded as the neighboring corner points of the u-th corner point.

[0032] In this embodiment, the value of a is 7. The value of a is preset manually and can be set by the implementer. This application does not impose any special restrictions.

[0033] Calculate the average movement distance of all neighboring corner points for each corner point, convert this average to a positive number, and calculate the ratio of each corner point's movement distance to this positive number. This ratio reflects the difference in movement between the uth corner point and its neighboring corner points. Converting this average to a positive number prevents the denominator from being zero.

[0034] In this embodiment, the purpose of mapping the average value to a positive number is achieved by calculating the sum of the average value and a preset positive number. The value of the preset positive number is preset manually and can be set by the implementer. In this embodiment, the value of the preset positive number is 0.01. There are many ways to map data to positive numbers. The implementer can choose other feasible methods on his own. This application does not impose any special restrictions.

[0035] For the uth corner point and its neighboring corner points, the motion vector is calculated using the arctan() function to obtain the movement angles of the uth corner point and its neighboring corner points. The average of the differences in the movement angles of the uth corner point and all its neighboring corner points is recorded as the average angle difference. Calculating angles using the arctan() function and displacement changes is a well-known technique and will not be further described in this application.

[0036] The product of the ratio of the u-th corner point and the average angle difference is used as the local variation index of the u-th corner point between the (i-1)th frame and the i-th frame. The local variation index can reflect the difference in motion amplitude and motion angle deviation between the u-th corner point and its neighboring corner points in adjacent frames.

[0037] Each frame image and its preset neighboring images are combined into a neighboring image set of each frame image.

[0038] In this embodiment, the preset neighboring images for each frame are the three preceding and three following frames of the frame. The value "3" is merely an example of this application, and the implementer may freely set its specific value. This application does not impose any specific restrictions. When the i-th frame is at the beginning or end of the video and the number of preceding or following images is insufficient, the calculation can be performed using only the actually available images.

[0039] By analyzing the position changes of each corner point in each frame image in any two adjacent frames in the neighboring image set, the displacement fluctuation significance of each corner point is obtained, and the expression is: ; Where, is the displacement fluctuation significance of the u-th corner point; 、 、 are respectively the mean of the moving distance of the u-th corner point between all two adjacent frames, the discreteness of the moving distance, and the discreteness of the local change index; n represents the number of neighboring corner points of the u-th corner point.

[0040] It should be noted that: in the process of calculating the significance of displacement fluctuations, the discreteness involved is the coefficient of variation. The coefficient of variation is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to measure the uneven distribution of moving distances and the uneven distribution of local change indexes, implementers can use other existing technologies for measurement, such as variance, standard deviation, etc., and this application does not impose any special restrictions.

[0041] It should be noted that the displacement fluctuation significance can reflect the fluctuation of the motion state of the u-th corner point between multiple consecutive frames. The greater the displacement fluctuation significance, the more obvious the instability of the u-th corner point in local changes and timing, which reflects that the u-th corner point is more likely to be related to the key turning point of the scene.

[0042] The displacement fluctuation significance of each corner point in each frame image is calculated using the same calculation method as that for the displacement fluctuation significance of the u-th corner point in the i-th frame image.

[0043] Step S3: For each connected domain in each frame image, obtain the matching connected domain of each connected domain in the adjacent frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, obtain the adjacent frame mutation value of each connected domain.

[0044] The displacement fluctuation significance of corner points can only quantify the motion change characteristics of a single corner point. However, since the determination of key frames requires the global changes of objects and scenes in the video, it is necessary to combine the characteristics of the connected domain to further distinguish whether there are large mutations in the objects in each frame image.

[0045] Take the vth connected domain in the i-th frame image as an example. Use the image matching algorithm to match the i-th frame image with the i-1-th frame image to obtain the matching connected domain of the vth connected domain in the i-th frame image.

[0046] In this embodiment, the SIFT algorithm is used to obtain the matching connected domain of the v-th connected domain. The SIFT algorithm is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to obtain the matching connected domain of the v-th connected domain, the implementer can adopt other existing technologies, such as the SURF algorithm, etc., and this application does not impose any special restrictions.

[0047] The contour difference between the vth connected domain and its matching connected domain is calculated. The larger the contour difference, the greater the shape change of the vth connected domain between adjacent frame images.

[0048] In this embodiment, the contour difference between the vth connected domain and its matching connected domain is reflected by the Hausdorff distance between the vth connected domain and its matching connected domain. The larger the Hausdorff distance, the greater the contour difference between the vth connected domain and its matching connected domain. The Hausdorff distance is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to measure the contour differences between connected domains, the implementer may adopt other existing technologies, such as Hu moment function, etc., and this application does not impose any special restrictions.

[0049] Color histograms of the vth connected domain and its matching connected domain are obtained respectively, and the difference between the two color histograms is recorded as a histogram difference. The difference between the color histograms reflects the color difference between the vth connected domain and its matching connected domain. The greater the difference between the color histograms, the greater the color change of the vth connected domain between two adjacent frames of the image. The method for obtaining the color histogram is well known in the art and will not be described in detail in this application.

[0050] In this embodiment, the difference between color histograms is calculated by Jensen-Shannon divergence. Jensen-Shannon divergence is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to measure the difference between color histograms, the implementer may adopt other existing technologies, such as Kullback-Leibler divergence, etc., and this application does not impose any special restrictions.

[0051] By analyzing the contour differences and color differences between each connected domain and its matching connected domain, and combining the displacement fluctuation significance of all corner points in each connected domain, the adjacent frame mutation value of each connected domain is obtained. The expression is: Where, Represents the adjacent frame mutation value of the vth connected domain; represents the mean value of the displacement fluctuation significance of all corner points in the vth connected domain; Represents the sum of the contour difference and histogram difference of the vth connected component. Recorded as significant mean.

[0052] If the matching connected domain of the vth connected domain in the i-th frame image cannot be obtained in the i-1th frame image, then the color histogram of each frame image is obtained respectively, and the contour difference between any two frames of image is calculated. It is the sum of the contour difference and color histogram difference between the i-th frame image and the i-1-th frame image.

[0053] In this embodiment, the contour difference between the i-th frame image and the i-1-th frame image is reflected by the Hausdorff distance between the i-th frame image and the i-1-th frame image. As other implementation methods, on the basis of being able to measure the contour difference between the i-th frame image and the i-1-th frame image, the implementer may adopt other existing technologies, such as Hu moment function, etc., and this application does not impose any special restrictions.

[0054] It should be noted that the adjacent frame mutation value can reflect the comprehensive change degree of the vth connected domain in the i-th frame image; the larger the adjacent frame mutation value, the greater the difference between the vth connected domain and its matching connected domain in local motion, shape and color, and the more likely the vth connected domain is to be a key mutation object.

[0055] The same calculation method as that for the adjacent frame mutation value of the vth connected domain in the i-th frame image is used to calculate the adjacent frame mutation value of each connected domain in each frame image.

[0056] Step S4: performing cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame.

[0057] The average of the mutation values ​​of adjacent frames of all connected domains in each frame image is recorded as the mutation mean, and the first clusters of all frame images are obtained by the mutation mean, and the first cluster with the largest element mean is selected; The maximum value of the mutation value of all connected domains in each frame image is counted, and the second clusters of all frame images are obtained by the maximum value, and the second cluster with the largest element mean is extracted. The frame images corresponding to the selection result and the extraction result are regarded as the suspected key frames. The flowchart of the suspected key frame acquisition is as follows: Figure 2 shown.

[0058] In this embodiment, the K-means algorithm is used to obtain the first cluster cluster and the second cluster cluster, and the optimal number of cluster clusters is obtained by the elbow rule. The K-means algorithm is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to obtain the first cluster cluster and the second cluster cluster, the implementer can adopt other existing technologies, such as density peak clustering algorithm, fuzzy C-means clustering, etc., and this application does not impose any special restrictions.

[0059] The first cluster reflects overall motion changes within each frame, corresponding to scene changes and large-scale object movement. The second cluster corresponds to frames where overall changes are subtle but local objects exhibit significant changes. Clustering allows for preliminary screening of images with significant changes, improving efficiency while avoiding missing images with moderate overall changes but significant local changes.

[0060] Step 5: Calculate the time interval between each suspected key frame and its adjacent previous suspected key frame. Obtain the key coefficient of each suspected key frame by analyzing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of adjacent frames in all connected domains within each suspected key frame.

[0061] Take the jth suspected key frame as an example. Since there are a large number of transition frames in a video, there will be a large time interval between key frames. Therefore, all suspected key frames are sorted according to their temporal order in the video, and the time interval between each suspected key frame and its immediately preceding suspected key frame is calculated. The average of these time intervals for all suspected key frames is recorded as the time difference mean. The ratio of the time interval of the jth suspected key frame to the time difference mean is recorded as the time difference ratio of the jth suspected key frame. The smaller the time difference ratio, the larger the time interval between the jth suspected key frame and its immediately preceding suspected key frame is compared to the average time interval, and the more the jth suspected key frame conforms to the time interval characteristics of a key frame.

[0062] Furthermore, since the video will have significant changes at the key frames, the similarity between the key frames and the two adjacent frames will have a large difference; at the same time, since the content of each key frame is highly unique, there is no similarity between different key frames.

[0063] The key coefficient of each suspected key frame is obtained by combining the contour difference and color difference between each suspected key frame and its adjacent frame image and the other suspected key frames, the time difference ratio of each suspected key frame, and the distribution of the adjacent frame mutation values ​​of all connected domains within each suspected key frame. The expression is: Where, is the key coefficient of the jth suspected key frame; is the maximum value function; Represent the values ​​of the j-th suspected key frame in the first and second clusters respectively; is the time difference ratio of the jth suspected key frame; 、 、 Respectively represent the sum of the contour difference and color histogram difference between the jth suspected key frame and its adjacent previous frame image, next frame image, and previous suspected key frame; Indicates the absolute value operation. If the jth suspected key frame is not in the first cluster, then let is 0; if it is not in the second cluster, then let is 0.

[0064] In this embodiment, in the process of calculating the key coefficient, the contour differences involved are all reflected by the Hausdorff distance. As other implementation methods, on the basis of being able to measure the contour differences between the j-th suspected key frame and its adjacent previous frame image, the next frame image, and the previous suspected key frame, the implementer may adopt other existing technologies, such as Hu moment function, etc., and this application does not impose any special restrictions.

[0065] It should be noted that: It is possible to obtain the maximum significance of the jth suspected key frame in the overall change and local change, so as to further refine the selection of key frames; The time difference ratio reflects the difference between the jth suspected key frame and its adjacent frames and the remaining suspected key frames, thereby detecting the uniqueness of the content of the suspected key frame. The time difference ratio reflects the difference between the time interval of the jth suspected key frame and the average time interval, which is used to detect whether the jth suspected key frame meets the time interval characteristics of key frames. The larger the key coefficient, the more the jth suspected key frame meets the unique characteristics of key frames and the more likely it is a key frame.

[0066] The key coefficients of each suspected key frame are calculated using the same calculation method as that of the j-th suspected key frame.

[0067] Step S6 , screening and determining each key frame and its key change connected domain through the distribution of key coefficients of all suspected key frames and combining the adjacent frame mutation values ​​of each connected domain of all suspected key frames.

[0068] A segmentation threshold of key coefficients of all suspected key frames is obtained, and each suspected key frame whose key coefficient is greater than the segmentation threshold is regarded as a key frame.

[0069] In this embodiment, the Otsu threshold segmentation algorithm is used to obtain the segmentation threshold of the key coefficients of all suspected key frames. The Otsu threshold segmentation algorithm is a well-known technology and will not be described in detail in this application. As other implementation methods, on the basis of being able to obtain the segmentation threshold of the key coefficients of all suspected key frames, the implementer can adopt other existing technologies, such as global threshold segmentation, iterative threshold segmentation, etc., and this application does not impose any special restrictions.

[0070] The connected domains whose normalized values ​​of the mutation values ​​of adjacent frames in each key frame are greater than or equal to the preset threshold are regarded as the key change connected domains, and the connected domains whose normalized values ​​of the mutation values ​​of adjacent frames are less than the preset threshold are regarded as the background connected domains. Figure 3 shown.

[0071] In this embodiment, a hyperbolic tangent function is used to obtain normalized values ​​of mutation values ​​of adjacent frames.

[0072] In this embodiment, the value of the preset threshold is 0.5. The value of the preset threshold is preset manually and can be set by the implementer. This application does not impose any special restrictions.

[0073] The key change connected domain that needs to be rendered is rendered, and after the rendering is completed, the rendered key frame is combined with the remaining frame images for video synthesis.

[0074] Based on the same inventive concept as the above method, the embodiment of the present application further provides a high-quality rendering device for video special effects in a film and television production process, comprising: The frame processing module is used to process the video frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; A key point recognition module is used to combine each frame image and its preset neighboring images into a neighboring image set for each frame image, and to obtain the displacement fluctuation significance of each corner point by analyzing the position change of each corner point in each frame image between any two adjacent frames in the neighboring image set; The feature region recognition module is used to obtain the matching connected domains of each connected domain in each frame image in its previous frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, the adjacent frame mutation value of each connected domain is obtained; The rendering area recognition module is used to perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

[0075] Based on the same inventive concept as the above method, an embodiment of the present application also provides a high-quality rendering system for video special effects in a film and television production process, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned methods for high-quality rendering of video special effects in a film and television production process are implemented.

[0076] In summary, this application constructs displacement fluctuation significance based on the characteristic that key frames will undergo large motion mutations in adjacent frame images. This can effectively identify dynamic changes between adjacent frames and thus extract key feature points, which can overcome the shortcomings of traditional methods that only rely on overall motion relationships. Furthermore, considering that objects and scenes in key frames have large mutation changes, the mutation values ​​of adjacent frames are constructed by combining key feature points, so that the feature areas with key mutations in the frame image can be extracted, providing precise targets for subsequent rendering; Furthermore, considering the different situations of local and global mutations of objects in key frames, a clustering algorithm is used to preliminarily screen the frames with significant global and local changes, thereby improving processing efficiency and avoiding the omission of key frames. Furthermore, based on the time interval characteristics of key frames and the uniqueness of key frames, key coefficients are constructed, which can fully analyze the overall and local mutation levels of frame images and accurately identify key frames, thereby improving the quality of video special effects rendering.

[0077] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

[0078] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the basic characteristics of the present application. Therefore, from all perspectives, the above embodiments of the present application should be regarded as exemplary and non-restrictive.

Claims

1. A method for high-quality rendering of video special effects in a film and television production process, characterized in that: The method comprises the following steps: Process the video frame by frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; For each corner point in each frame image, the position change of each corner point in the image in which it is located and the preset neighboring image is obtained to obtain the displacement fluctuation significance of each corner point; For each connected domain in each frame image, obtain the matching connected domain of each connected domain in the adjacent frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, obtain the adjacent frame mutation value of each connected domain; Perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

2. A method for high-quality rendering of video special effects in a film and television production process as claimed in claim 1, characterized in that: The process of obtaining the displacement fluctuation significance is as follows: Each frame image and its preset neighboring images are combined into a neighboring image set for each frame image, and the movement distance and movement angle of each corner point in each frame image between any two adjacent frames in the neighboring image set are obtained. By comparing the movement distance and movement angle of each corner point with its neighboring corner points, the local change index of each corner point between the two adjacent frames is obtained; The calculation formula for the displacement fluctuation significance of the u-th corner point in the i-th frame image is: ； Where, is the displacement fluctuation significance of the u-th corner point; 、 、 are respectively the mean of the moving distance of the u-th corner point between all two adjacent frames, the discreteness of the moving distance, and the discreteness of the local change index; n represents the number of neighboring corner points of the u-th corner point.

3. The method for high-quality rendering of video special effects in a film and television production process as claimed in claim 2, characterized in that: The calculation process of the local change index is: Calculate the average of the moving distances of all neighboring corner points of each corner point, map the average to a positive number, and calculate the ratio of the moving distance of each corner point to the positive number; The average of the differences in the moving angles of each corner point and all its neighboring corner points is recorded as the average angle difference; The local variation index is the product of the ratio and the average angle difference.

4. The method for high-quality rendering of video special effects in a film and television production process according to claim 1, wherein: The process of obtaining the adjacent frame mutation value is as follows: Calculate the contour difference between each connected domain and its matching connected domain; Obtain the color histogram of each connected domain respectively, and record the difference in color histogram between each connected domain and its matching connected domain as the histogram difference; Calculating the sum of the contour difference and the histogram difference of each connected domain; The mean of the displacement fluctuation significance of all corner points in each connected domain is recorded as the significant mean; The adjacent frame mutation value is the product of the sum value and the significant mean value.

5. The method for high-quality rendering of video special effects in a film and television production process as claimed in claim 4, characterized in that: If the matching connected domains of each connected domain cannot be obtained, the color histograms of each frame image are obtained respectively, and the contour difference between any two frame images is calculated. The sum value is the sum of the contour difference and color histogram difference between the frame image where each connected domain is located and its adjacent previous frame image.

6. The method for high-quality rendering of video special effects in a film and television production process as claimed in claim 5, characterized in that: The process of obtaining the suspected key frame is as follows: The average of the mutation values ​​of adjacent frames of all connected domains in each frame image is recorded as the mutation mean, and the first clusters of all frame images are obtained by the mutation mean, and the first cluster with the largest element mean is selected; Counting the maximum value of the mutation value of adjacent frames of all connected domains in each frame image, obtaining each second cluster of all frame images according to the maximum value, and extracting the second cluster with the largest element mean; The image frames corresponding to the selected results and the extracted results are taken as suspected key frames.

7. The method for high-quality rendering of video special effects in a film and television production process according to claim 6, characterized in that: The process of obtaining the key coefficients is as follows: Calculate the time interval between each suspected key frame and its adjacent previous suspected key frame, and record the average of the time intervals of all suspected key frames as the time difference average; and record the ratio of the time interval to the time difference average as the time difference ratio; The calculation formula of the key coefficient is: Where, is the key coefficient of the jth suspected key frame; is the maximum value function; Represent the values ​​of the j-th suspected key frame in the first and second clusters respectively; is the time difference ratio of the jth suspected key frame; 、 、 Respectively represent the sum of the contour difference and color histogram difference between the jth suspected key frame and its adjacent previous frame image, the next frame image, and the previous suspected key frame, Indicates the absolute value operation.

8. The method for high-quality rendering of video special effects in a film and television production process according to claim 1, wherein: The process of screening and determining each key frame and the key change connected domain of each key frame is as follows: Obtain the segmentation threshold of the key coefficients of all suspected key frames, and take each suspected key frame whose key coefficient is greater than the segmentation threshold as each key frame; take each connected domain within each key frame whose normalized value of the adjacent frame mutation value is greater than or equal to the preset threshold as each key change connected domain.

9. A high-quality rendering device for video special effects in a film and television production process, applying the high-quality rendering method for video special effects in a film and television production process as claimed in claim 1, characterized in that: The device comprises: The frame processing module is used to process the video frame to obtain each frame image; obtain each corner point and each connected domain in each frame image; A key point recognition module is used to combine each frame image and its preset neighboring images into a neighboring image set for each frame image, and to obtain the displacement fluctuation significance of each corner point by analyzing the position change of each corner point in each frame image between any two adjacent frames in the neighboring image set; The feature region recognition module is used to obtain the matching connected domains of each connected domain in each frame image in its previous frame image. By analyzing the contour differences and color differences between each connected domain and its matching connected domain, combined with the displacement fluctuation significance of all corner points in each connected domain, the adjacent frame mutation value of each connected domain is obtained; The rendering area recognition module is used to perform cluster analysis on the mutation values ​​of adjacent frames of all connected domains in all frame images to obtain each suspected key frame; Calculate the time interval between each suspected key frame and its previous suspected key frame, and obtain the key coefficient of each suspected key frame by comparing the contour differences and color differences between each suspected key frame and its adjacent frame images and the remaining suspected key frames, combined with the changes in the time interval and the distribution of the mutation values ​​of the adjacent frames of all connected domains within each suspected key frame; Through the distribution of key coefficients of all suspected key frames and the adjacent frame mutation values ​​of each connected domain of all suspected key frames, each key frame and the key change connected domain of each key frame are screened and determined.

10. A high-quality rendering system for video special effects in a film and television production process, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method for high-quality rendering of video special effects in a film and television production process as described in any one of claims 1-8 are implemented.

Images