Method, device and storage medium for removing breathing flicker from film images

Abstract

The embodiment of the present application provides a method, device and storage medium for removing breath flicker from a movie screen. The method comprises: obtaining a to-be-processed movie clip after the de-flickering process; wherein the to-be-processed movie clip after the de-flickering process is obtained after performing a correction process on an exposure abnormality frame; the exposure abnormality frame is determined from multiple movie frames of the to-be-processed movie clip based on the local spatial information capacity; the local spatial information capacity is used to characterize the regional energy distribution of the movie frame; based on the local spatial information capacity of the to-be-processed movie clip after the de-flickering process, similarity division processing is performed on the to-be-processed movie clip after the de-flickering process to obtain at least one to-be-processed sub-segment; determining a standard frame of the to-be-processed sub-segment, and optimizing the non-standard frame of the to-be-processed sub-segment based on the standard frame to obtain the to-be-processed movie clip after the breath flickering process. The method is used to achieve the effect of improving the user's movie viewing experience.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, and storage medium for removing respiratory flicker from a movie image. Background Art

[0002] After de-skewing the frames with abnormal exposure in the processed video clip to obtain the de-flickered video clip, the processed video clip may contain breathing flicker. Breathing flicker refers to the phenomenon of gradual changes in brightness or color across multiple consecutive frames in the processed video clip, which affects the user's viewing experience.

[0003] Therefore, there is an urgent need for a method to remove breathing flicker to improve the viewing experience of the film. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and storage medium for removing breathing flicker from a movie screen, so as to improve the user's movie viewing experience.

[0005] In a first aspect, an embodiment of the present application provides a method for removing breathing flicker from a video image, comprising:

[0006] Obtaining a flicker-free film segment to be processed; wherein the flicker-free film segment to be processed is a segment of a film image in the same film scene; the flicker-free film segment to be processed is obtained by performing a de-skew process on a frame with abnormal exposure; the frame with abnormal exposure is determined from multiple frames of the film segment to be processed based on a local spatial information capacity; the local spatial information capacity is used to characterize the regional energy distribution of the film frame;

[0007] performing similarity division processing on the to-be-processed video segment after the de-flickering processing according to the local spatial information capacity of the to-be-processed video segment after the de-flickering processing to obtain at least one to-be-processed sub-segment;

[0008] A standard frame of the sub-segment to be processed is determined, and non-standard frames of the sub-segment to be processed are optimized according to the standard frame to obtain a film segment to be processed after respiratory flicker removal processing.

[0009] In one possible implementation, the local spatial information capacity includes at least one local spatial information sub-capacity; wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of pixel blocks in the film frame; and performing similarity partitioning on the de-flickered film segment to be processed based on the local spatial information capacity of the de-flickered film segment to be processed to obtain at least one sub-segment to be processed, including: determining the similarity between every two film frames in the de-flickered film segment to be processed based on the at least one local spatial information sub-capacity included in the local spatial information capacity; and performing similarity partitioning on the de-flickered film segment to be processed based on the similarity between every two film frames to obtain at least one sub-segment to be processed.

[0010] In one possible implementation, determining the similarity between every two film frames in the to-be-processed film segment after the de-flickering processing based on at least one of the local spatial information sub-capacity included in the local spatial information capacity includes: performing feature extraction processing on the at least one of the local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector for each film frame;

[0011] A cosine similarity calculation process is performed on the local spatial information sub-capacity feature vectors of the film frames to obtain the similarity between each two film frames.

[0012] In one possible implementation, determining the similarity between every two film frames in the to-be-processed film segment after the de-flickering processing based on at least one of the local spatial information subcapacity included in the local spatial information capacity includes: performing feature extraction processing on at least one of the local spatial information subcapacity included in the local spatial information capacity to obtain a local spatial information subcapacity feature vector for each film frame; obtaining an information entropy feature vector for each film frame, and concatenating the local spatial information subcapacity feature vector and the information entropy feature vector to obtain an integrated information feature vector for each film frame; and performing cosine similarity calculation processing on the integrated information feature vector of each film frame to obtain the similarity between the two film frames.

[0013] In one possible implementation, performing similarity division processing on the deflickered film segment to be processed based on the similarity between each two film frames to obtain at least one sub-segment to be processed includes: determining at least one film segmentation point based on the similarity between each two film frames and a dynamic programming technique; wherein the film segmentation point is used to divide the two segments to be processed; and performing segmentation processing on the film segment to be processed based on the at least one film segmentation point to obtain the at least one sub-segment to be processed.

[0014] In one possible implementation, determining the standard frame for the sub-segment to be processed includes: obtaining at least one normalized histogram for each film frame in the sub-segment to be processed; and calculating, based on the at least one normalized histogram, a global similarity value for each film frame in the sub-segment to be processed; wherein the global similarity value is the sum of similarity values ​​between a film frame and other film frames; if the maximum value of the global similarity values ​​of the film frames corresponds to only one film frame, determining that film frame as the standard frame; and if the maximum value of the global similarity values ​​of the film frames corresponds to multiple film frames, determining a film frame that is located at an intermediate position in the time series among the multiple film frames as the standard frame.

[0015] In one possible implementation, the non-standard frames of the sub-segment to be processed are optimized according to the standard frames to obtain the film segment to be processed after the respiratory flicker removal processing, including: obtaining at least one normalized histogram of each film frame in the sub-segment to be processed; calculating the normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed according to the at least one normalized histogram; and mapping the normalized histogram cumulative distribution function value of each non-standard frame according to the normalized histogram cumulative distribution function value of the standard frame to obtain the film segment to be processed after the respiratory flicker removal processing.

[0016] In one possible embodiment, after optimizing the non-standard frames of the sub-segments to be processed according to the standard frames, the method further includes: calculating the brightness histogram cumulative distribution function value of the standard frames of each optimized sub-segment to be processed, and determining the brightness mapping relationship between adjacent optimized sub-segments to be processed based on the brightness histogram cumulative distribution function value; setting an overlapping area between adjacent optimized sub-segments to be processed, and performing brightness interpolation adjustment on each film frame in the overlapping area based on the brightness mapping relationship to obtain two groups of overlapping areas after brightness interpolation adjustment; wherein the overlapping area includes all boundary frames of the adjacent optimized sub-segments to be processed; generating a value of a transparency channel corresponding to the overlapping area, and performing weighted fusion processing on the pixel brightness values ​​of each film frame in the two groups of overlapping areas after brightness interpolation adjustment according to the value of the transparency channel to obtain the film segment to be processed after the respiratory flicker removal processing.

[0017] In a second aspect, an embodiment of the present application provides a device for removing breathing flicker from a video image, comprising:

[0018] An acquisition module is configured to acquire a flicker-free film segment to be processed; wherein the flicker-free film segment to be processed is a segment of a film image in the same film scene; the flicker-free film segment to be processed is obtained by performing a de-skew process on an exposure-abnormal frame; the exposure-abnormal frame is determined from multiple film frames of the flicker-free film segment based on a local spatial information capacity; the local spatial information capacity is used to characterize the regional energy distribution of the film frame;

[0019] a first processing module configured to perform similarity division processing on the de-flickered video segment to be processed according to the local spatial information capacity of the de-flickered video segment to be processed, to obtain at least one sub-segment to be processed;

[0020] The second processing module is configured to determine a standard frame of the sub-segment to be processed, and optimize the non-standard frame of the sub-segment to be processed according to the standard frame to obtain the video segment to be processed after the respiratory flicker removal process.

[0021] In one possible implementation, the local spatial information capacity includes at least one local spatial information sub-capacity; wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of pixel blocks in the film frame; the first processing module is specifically configured to determine the similarity between every two film frames in the de-flickered film segment to be processed based on the at least one local spatial information sub-capacity included in the local spatial information capacity; and perform similarity partitioning processing on the de-flickered film segment to be processed based on the similarity between every two film frames to obtain at least one sub-segment to be processed.

[0022] In one possible implementation, the first processing module is further configured to perform feature extraction processing on at least one of the local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector for each film frame; and perform cosine similarity calculation processing on the local spatial information sub-capacity feature vectors for each film frame to obtain the similarity between each two film frames.

[0023] In one possible embodiment, the first processing module is further specifically configured to perform feature extraction processing on at least one of the local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector for each film frame; obtain an information entropy feature vector for each film frame, and concatenate the local spatial information sub-capacity feature vector and the information entropy feature vector to obtain an integrated information feature vector for each film frame; and perform cosine similarity calculation processing on the integrated information feature vector of each film frame to obtain the similarity between each two film frames.

[0024] In one possible implementation, the first processing module is further specifically configured to determine, based on the similarity between each two video frames and a dynamic programming technique, at least one video segmentation point; wherein the video segmentation point is used to divide the two to-be-processed segments; and the to-be-processed video segment is segmented according to the at least one video segmentation point to obtain the at least one to-be-processed sub-segment.

[0025] In one possible implementation, the second processing module is specifically configured to obtain at least one normalized histogram for each film frame in the sub-segment to be processed; and calculate, based on the at least one normalized histogram, a global similarity value for each film frame in the sub-segment to be processed; wherein the global similarity value is the sum of similarity values ​​between a film frame and other film frames; if the maximum value of the global similarity values ​​of the film frames corresponds to only one film frame, then determine that film frame as the standard frame; and if the maximum value of the global similarity values ​​of the film frames corresponds to multiple film frames, then determine a film frame that is located in the middle of the multiple frames in time series as the standard frame.

[0026] In one possible embodiment, the second processing module is further specifically used to obtain at least one normalized histogram of each film frame in the sub-segment to be processed; based on the at least one normalized histogram, calculate the normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed; based on the normalized histogram cumulative distribution function value of the standard frame, map the normalized histogram cumulative distribution function value of each non-standard frame to obtain the film segment to be processed after respiratory flicker removal processing.

[0027] In one possible embodiment, the second processing module is further used to calculate the cumulative distribution function value of the brightness histogram of the standard frame of each optimized sub-segment to be processed, and determine the brightness mapping relationship between adjacent optimized sub-segments to be processed based on the brightness histogram cumulative distribution function value; set an overlapping area between adjacent optimized sub-segments to be processed, and perform brightness interpolation adjustment on each film frame in the overlapping area based on the brightness mapping relationship to obtain two groups of overlapping areas after brightness interpolation adjustment; wherein the overlapping area includes all boundary frames of the adjacent optimized sub-segments to be processed; generate a value of a transparency channel corresponding to the overlapping area, and perform weighted fusion processing on the pixel brightness values ​​of each film frame in the two groups of overlapping areas after brightness interpolation adjustment according to the value of the transparency channel to obtain the film segment to be processed after the respiratory flicker removal processing.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0029] The memory stores computer-executable instructions;

[0030] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0033] The embodiments of the present application provide a method, device, and storage medium for removing breathing flicker from a film screen. By dividing the film into sub-segments to be processed and optimizing the non-standard frames through the standard frames of the sub-segments to be processed, the breathing flicker phenomenon of the film can be efficiently and accurately removed, thereby improving the user's film viewing experience. Specifically, through the local spatial information capacity, the sub-segments to be processed can be accurately divided from the film segment to be processed after the de-flickering processing, further improving the accuracy of the film processing, thereby improving the user's film viewing experience. In summary, the film processing method provided by the present application can improve the user's film viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] Figure 1 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 1 ;

[0036] Figure 2 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 2 ;

[0037] Figure 3 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 3 ;

[0038] Figure 4 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 4 ;

[0039] Figure 5Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 5 ;

[0040] Figure 6 This is a schematic diagram of the structure of the device for removing breathing flicker from movie images provided by this application;

[0041] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application.

[0042] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0044] In existing technology, after de-skewing the frames of a video clip with abnormal exposure, resulting in a flicker-free clip, breathing flicker may occur in the processed clip. Breathing flicker refers to the gradual change in brightness or color between consecutive frames in the processed clip, which can affect the user's viewing experience. Therefore, a method for removing breathing flicker is urgently needed to improve the viewing experience.

[0045] The present application provides a method for removing breathing flicker from a film screen. By dividing the film into sub-segments to be processed and optimizing the non-standard frames using the standard frames of the sub-segments to be processed, the method can efficiently and accurately remove the breathing flicker phenomenon in the film, thereby improving the user's film viewing experience. Specifically, through the local spatial information capacity, the sub-segments to be processed can be accurately divided from the film segments to be processed after the de-flickering process, thereby further improving the accuracy of the film processing and thus improving the user's film viewing experience. In summary, the film processing method provided by the present application can improve the user's film viewing experience.

[0046] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0047] Figure 1 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 1 ,like Figure 1 As shown, the method includes:

[0048] Step S101: Obtain a flicker-free video clip to be processed.

[0049] Specifically, a flicker-free video clip to be processed can be obtained. The flicker-free video clip is a fragment of a video image from the same scene. The flicker-free video clip is obtained by performing deskew processing on frames with abnormal exposure. The abnormal exposure frames are determined from multiple frames in the video clip to be processed based on the local spatial information capacity. The local spatial information capacity is used to characterize the regional energy distribution of the video frame.

[0050] Specifically, the present application does not limit the process of obtaining a film segment to be processed after de-flickering. Optionally, a film segment to be processed can be obtained; wherein the film segment to be processed is a segment of a film screen under the same film scene; the film segment to be processed includes multiple film frames; the local spatial information capacity of the film frame is determined, and the exposure abnormality frame is determined from the multiple film frames based on the local spatial information capacity; wherein the local spatial information capacity is used to characterize the regional energy distribution of the film frame; the exposure abnormality frame is corrected according to the pixel values ​​of the adjacent frames of the exposure abnormality frame to obtain the film to be processed after de-flickering.

[0051] Specifically, during the flicker removal process, if the video to be processed is a fragment of video images from different video scenes, the problem of inaccurate identification of abnormal frames may occur due to the different video scenes to which the video frames belong, further leading to inaccurate flicker removal processing. Therefore, a video fragment to be processed can be obtained. The video fragment to be processed is a fragment of video images from the same video scene. The video fragment to be processed includes multiple video frames.

[0052] Specifically, the present application does not limit the process of obtaining the film segment to be processed. Optionally, a complete film can be obtained; wherein the complete film is a film segment that includes at least one film segment to be processed; based on deep feature extraction technology and shallow feature extraction technology, the similarity between each adjacent film frame in the complete film is determined; according to the similarity between each adjacent film frame in the complete film, the complete film is divided and processed to obtain at least one film segment to be processed.

[0053] Specifically, after obtaining the video segment to be processed, the local spatial information capacity of each video frame in the video segment to be processed may be determined.

[0054] In optical imaging systems, objects outside the depth of field will produce defocus blur. When an object moves away from the focal plane, its high-frequency spatial components, i.e., detail information, will decay nonlinearly with increasing defocus distance, manifesting as a band-limited change in spatial frequency. A more accurate description should be "Local Spatial Information Capacity" (LSIC). Specifically, for near-field areas, unit pixels correspond to smaller physical scales and have higher spatial sampling density, i.e., high local spatial information capacity. For distant areas, unit pixels cover larger physical scales, resulting in spatial aliasing, i.e., low local spatial information capacity.

[0055] The local spatial information capacity is used to characterize the regional energy distribution of a film frame. Specifically, based on the above description of local spatial information capacity, the stronger the regional energy distribution of a film frame represented by the local spatial information capacity, the higher the local spatial information capacity. Conversely, the weaker the regional energy distribution of a film frame represented by the local spatial information capacity, the lower the local spatial information capacity.

[0056] Specifically, the present application does not limit the process of determining the local spatial information capacity of a film frame. Optionally, the noise variance of the film frame can be obtained, and the spectral energy of each pixel block in the film frame can be obtained; the local spatial information capacity is determined based on the noise variance and the spectral energy of each pixel block.

[0057] Specifically, after determining the local spatial information capacity of each film frame in the to-be-processed film segment, the exposure abnormality frame may be determined from the plurality of film frames according to the local spatial information capacity.

[0058] The abnormal exposure frame is a frame to be subjected to a de-skew process among multiple frames of the video clip to be processed during the de-scintillation process. The present application does not limit the number of abnormal exposure frames to be determined.

[0059] Specifically, the present application does not limit the process of determining an exposure-abnormal frame from multiple film frames based on the local spatial information capacity. Optionally, the local spatial information capacity includes at least one local spatial information sub-capacity; wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of pixel blocks in the film frame; based on the at least one local spatial information sub-capacity included in the local spatial information capacity, the probability density of the local spatial information sub-capacity of the film frame is calculated; wherein the probability density of the local spatial information sub-capacity indicates the ratio of the number of pixel blocks corresponding to each local spatial information sub-capacity to the total number of pixel blocks in the film frame; based on the at least one local spatial information sub-capacity included in the local spatial information capacity, a thermal histogram of the film frame is constructed; and based on the ratio of the number of pixel blocks corresponding to each local spatial information sub-capacity in the film frame to the total number of pixel blocks in the film frame, and / or the thermal histogram of the film frame, it is determined whether the film frame is an exposure-abnormal frame.

[0060] Specifically, after the abnormal exposure frame is determined, a correction process may be performed on the abnormal exposure frame according to pixel values ​​of adjacent frames of the abnormal exposure frame to obtain a flicker-free film to be processed.

[0061] Specifically, the present application does not limit the process of performing deskew processing on the exposure abnormality frame according to the pixel values ​​of the adjacent frames of the exposure abnormality frame to obtain the de-flickering process for the to-be-processed film. Optionally, the pixel values ​​of the adjacent frames of the exposure abnormality frame can be subjected to regression calculation processing to obtain the target pixel value of the exposure abnormality frame; the pixel values ​​of the exposure abnormality frame can be deskewed according to the target pixel value of the exposure abnormality frame to obtain the exposure abnormality frame after deskew processing; if the structural similarity index and peak signal-to-noise ratio between the exposure abnormality frame after deskew processing and the exposure abnormality frame meet the preset conditions, the exposure abnormality frame after deskew processing is replaced by the exposure abnormality frame; otherwise, the number of adjacent frames of the exposure abnormality frame is increased, and the regression calculation processing and deskew processing processes are repeated.

[0062] Among them, through the local spatial information capacity, the exposure abnormality frame can be accurately determined from the video clip to be processed, further improving the accuracy of the video to be processed after the flicker removal process, thereby improving the user's video viewing experience.

[0063] After de-skewing the abnormally exposed frames in the processed video clips based on the process described above, resulting in the de-skewing processed video clips, breathing flicker may occur in the processed video clips. Breathing flicker refers to the gradual change in brightness or color between multiple consecutive frames in the processed video clips, which can affect the user's viewing experience.

[0064] The video clip to be processed after the flicker removal process can be expressed as: , where T represents the total number of video frames included in the video clip to be processed after de-flickering, and F t represents the t-th movie frame, where (t=1,2,⋯,T).

[0065] Step S102 : performing similarity division processing on the de-flickered video segment to be processed according to the local spatial information capacity of the de-flickered video segment to be processed, to obtain at least one sub-segment to be processed.

[0066] Specifically, if a segment of a video frame from the same scene has a high degree of similarity, the brightness or color variations within that segment should be small. Therefore, during the breathing flicker removal process, the segmented video frames from the same scene can be first divided into at least one segment with a high degree of similarity. However, during the segmentation process, if there are frames with abnormal exposure, the segment division process may be inaccurate, resulting in inaccurate breathing flicker removal. Therefore, during the breathing flicker removal process for the segmented video frames from the same scene, the frames with abnormal exposure can be first de-skewed, thereby generating the unprocessed video segments described in step S101 after the flicker removal process.

[0067] Specifically, after obtaining a flicker-free video clip to be processed, similarity division can be performed on the flicker-free video clip to be processed based on the local spatial information capacity of the flicker-free video clip to be processed, thereby obtaining at least one sub-segment to be processed. The sub-segment to be processed is a video clip with a high similarity among the flicker-free video clips to be processed.

[0068] Specifically, the present application does not limit the process of performing similarity division processing on the de-flickered film segment to be processed based on the local spatial information capacity of the de-flickered film segment to obtain at least one sub-segment to be processed. Optionally, the local spatial information capacity includes at least one local spatial information sub-capacity; wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of pixel blocks in the film frame; based on the at least one local spatial information sub-capacity included in the local spatial information capacity, the similarity between every two film frames in the de-flickered film segment to be processed can be determined; and based on the similarity between every two film frames, the de-flickered film segment to be processed is subjected to similarity division processing to obtain at least one sub-segment to be processed.

[0069] The sub-fragment to be processed can be expressed as: The video clip to be processed after the flicker removal process includes K+1 sub-segments to be processed. Represents a set of video segmentation points of the video segment to be processed after de-flickering, where: .

[0070] Step S103 : determining a standard frame of the sub-segment to be processed, and optimizing the non-standard frame of the sub-segment to be processed according to the standard frame to obtain the video segment to be processed after the respiratory flicker removal process.

[0071] Specifically, after obtaining a sub-segment to be processed, the standard frame of each sub-segment to be processed can be determined, and the non-standard frame of the corresponding sub-segment to be processed can be optimized according to the standard frame to obtain the video segment to be processed after the respiratory flicker removal process.

[0072] The standard frame is a film frame determined as an optimization standard in the optimization process among multiple film frames of the sub-segment to be processed. Specifically, the present application does not limit the process of determining the standard frame of the sub-segment to be processed. Optionally, at least one normalized histogram of each film frame in the sub-segment to be processed can be obtained; based on the at least one normalized histogram, a global similarity value of each film frame in the sub-segment to be processed is calculated; wherein the global similarity value is the sum of the similarity values ​​between the film frame and other film frames; if the maximum value of the global similarity values ​​of each film frame corresponds to one film frame, then the film frame is determined as the standard frame; if the maximum value of the global similarity values ​​of each film frame corresponds to multiple film frames, then the film frame at the middle position of the time series among the multiple film frames is determined as the standard frame. Each sub-segment to be processed includes one standard frame.

[0073] The non-standard frames are the frames to be optimized among the multiple frames of the sub-segment to be processed. All the frames other than the standard frames in the sub-segment to be processed are the non-standard frames of the sub-segment to be processed.

[0074] Specifically, the present application does not limit the process of optimizing the non-standard frames of the corresponding sub-segment to be processed according to the standard frame to obtain the film segment to be processed after the respiratory flicker is removed. Optionally, at least one normalized histogram of each film frame in the sub-segment to be processed can be obtained; based on at least one normalized histogram, the normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed is calculated; based on the normalized histogram cumulative distribution function value of the standard frame, the normalized histogram cumulative distribution function value of each non-standard frame is mapped to obtain the film segment to be processed after the respiratory flicker is removed.

[0075] The embodiment of the present application provides a method for removing breathing flicker from a film screen. By dividing the film into sub-segments to be processed and optimizing the non-standard frames using the standard frames of the sub-segments to be processed, the breathing flicker phenomenon of the film can be removed efficiently and accurately, thereby improving the user's film viewing experience. Specifically, through the local spatial information capacity, the sub-segments to be processed can be accurately divided from the film segment to be processed after the de-flickering processing, further improving the accuracy of the film processing, thereby improving the user's film viewing experience. In summary, the film processing method provided by the present application can improve the user's film viewing experience.

[0076] Figure 2 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 2 ,like Figure 2 As shown, this embodiment Figure 1 Based on the embodiment, a process of performing similarity division processing on the de-flickered video segment to be processed based on the local spatial information capacity of the de-flickered video segment to obtain at least one sub-segment to be processed is described in detail. The method includes:

[0077] Step S201: Determine the similarity between every two video frames in the to-be-processed video segment after de-flickering according to at least one local spatial information sub-capacity included in the local spatial information capacity.

[0078] The local spatial information capacity includes at least one local spatial information sub-capacity, wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of the pixel block in the film frame.

[0079] Specifically, the number of local spatial information sub-capacities included in the local spatial information capacity corresponds to the number of pixel blocks included in a film frame. Specifically, a pixel block is a pixel block centered at pixel point (i, j) in the film frame and having a preset size. This application does not limit the preset size of the pixel blocks included in the film frame; optionally, the preset size may be 8 pixels × 8 pixels.

[0080] Among them, the local spatial information capacity of the movie frame can be expressed as: , among which, LSIC T represents the local area spatial information sub-capacity, and T represents the number of local area spatial information sub-capacities included in the local area spatial information capacity.

[0081] Specifically, the present application does not limit the process of determining the similarity between each two film frames in the to-be-processed film segment after de-flickering based on at least one local spatial information sub-capacity included in the local spatial information capacity. Optionally, determining the similarity between each two film frames in the to-be-processed film segment after de-flickering based on at least one local spatial information sub-capacity included in the local spatial information capacity includes:

[0082] A feature extraction process is performed on at least one local spatial information sub-capacity included in the local spatial information capacity to obtain a feature vector of the local spatial information sub-capacity of each film frame.

[0083] The cosine similarity calculation is performed on the local spatial information sub-capacity feature vector of each movie frame to obtain the similarity between every two movie frames.

[0084] Specifically, in the process of performing feature extraction processing on at least one local spatial information sub-capacity included in the local spatial information capacity to obtain a feature vector of the local spatial information sub-capacity of each film frame:

[0085] We can first obtain the value range of the local spatial information subcapacity (i.e., the local stability index LSIC that characterizes the regional energy distribution) [S min ,S max ], divide it into N segments, each segment is (in, Secondly, for each frame of the film image, the number of pixel blocks whose local spatial information sub-capacity value (LSIC value) falls within each interval is counted and normalized to obtain the LSIC feature component ,in is the number of pixel blocks whose LSIC values ​​fall in the kth interval in the tth frame, is the total number of pixel blocks.

[0086] Specifically, when performing cosine similarity calculation on the local spatial information sub-capacity feature vectors of each movie frame, the formula for obtaining the similarity between each two movie frames is as follows:

[0087]

[0088] in, is a vector and The dot product of They are vectors and The calculation formula is .

[0089] In the process of determining the similarity between every two film frames in the to-be-processed film segment after de-flickering based on at least one local spatial information sub-capacity included in the local spatial information capacity, by performing feature extraction processing on the at least one local spatial information sub-capacity included in the local spatial information capacity and calculating the cosine similarity, the similarity between every two film frames in the to-be-processed film segment after de-flickering can be accurately and efficiently determined, thereby further improving the efficiency and accuracy of film processing and thereby enhancing the user's film viewing experience.

[0090] Optionally, determining the similarity between every two video frames in the to-be-processed video segment after the de-flickering processing based on at least one local spatial information sub-capacity included in the local spatial information capacity includes:

[0091] performing feature extraction processing on at least one local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector of each film frame;

[0092] Obtaining the information entropy feature vector of each film frame, and concatenating the local spatial information subcapacity feature vector and the information entropy feature vector to obtain the integrated information feature vector of each film frame;

[0093] The cosine similarity calculation is performed on the integrated information feature vector of each video frame to obtain the similarity between every two video frames.

[0094] Specifically, in the process of calculating the information entropy feature vector of each video frame, the global information entropy H of each video frame can be calculated first. t (a measure of the uncertainty of pixel grayscale distribution), and then normalize it to Interval, the information entropy feature vector is obtained, that is, the information entropy feature component, which is expressed as: , where H min and H max is the minimum and maximum global information entropy of each frame in the video segment, i.e., the video segment to be processed after flicker removal. Represents a constant to avoid zero denominator, for example, 10 6 .

[0095] Specifically, in the process of splicing the information entropy feature vector and the local spatial information subcapacity feature vector to obtain the integrated information features of each film frame, the local spatial information subcapacity feature vector and the information entropy feature vector can be spliced ​​to form a local spatial information subcapacity feature vector containing regional energy distribution features and grayscale distribution complexity features. .

[0096] Specifically, the process of calculating the cosine similarity of the integrated information feature vectors of each film frame to obtain the similarity between each two film frames can refer to the process described above of calculating the cosine similarity of the local spatial information sub-capacity feature vectors of each film frame to obtain the similarity between each two film frames, which will not be repeated here.

[0097] Specifically, information entropy is used to measure the uncertainty of the grayscale distribution of pixels in a film frame. This information is calculated as an independent feature component and then concatenated with the local stability index (LSIC) feature component, which characterizes regional energy distribution. This allows the local spatial information subcapacity feature vector to reflect the regional energy distribution of pixel blocks within a film frame through the LSIC feature component (for example, by counting and normalizing the number of pixels with LSIC values ​​in different intervals to form a basic feature reflecting the characteristics of regional energy distribution). It also reflects the complexity of the pixel grayscale distribution through the information entropy feature component (by normalizing the global information entropy and incorporating it into the feature vector). This creates a complete feature vector that encompasses both grayscale distribution complexity and regional energy distribution characteristics, laying a comprehensive feature foundation for accurately measuring the similarity between film frames.

[0098] In determining the similarity between each two frames in the de-flickered film segment based on at least one local spatial information subcapacity included in the local spatial information capacity, the similarity calculation simultaneously reflects the local stability of the frames (based on the LSIC feature of regional energy distribution) and the consistency of the global grayscale distribution (based on the uncertainty of grayscale distribution characterized by information entropy) by fusing information entropy features with the local spatial information subcapacity features. This approach more comprehensively and accurately characterizes the similarity between frames: the LSIC feature reflects the stability of the regional energy distribution of pixel blocks within the frame, while the information entropy reflects the complexity of the frame's global grayscale distribution. This combination of the two allows the similarity calculation result to better characterize the overall similarity between frames in terms of luminance content and grayscale distribution. This facilitates more rational segmentation (for example, when determining segmentation points through dynamic programming, it can more accurately identify subsegments with substantially consistent luminance content and stable grayscale distribution complexity), providing a more reliable basis for optimizing non-standard frames based on standard frames, thereby more efficiently and accurately removing film breathing flicker. This process improves the accuracy of video processing, enhances the naturalness of transitions between video frames, reduces the visual flicker caused by local brightness changes and grayscale distribution fluctuations, and significantly improves the user viewing experience.

[0099] Step S202 : performing similarity division processing on the to-be-processed video segment after the flicker removal processing according to the similarity between every two video frames to obtain at least one to-be-processed sub-segment.

[0100] Specifically, the present application does not limit the process of performing similarity division processing on the to-be-processed film segments after the de-flickering processing according to the similarity between each two film frames to obtain at least one to-be-processed sub-segment. Optionally, performing similarity division processing on the to-be-processed film segments after the de-flickering processing according to the similarity between each two film frames to obtain at least one to-be-processed sub-segment includes:

[0101] Based on the similarity between each two video frames, at least one video segmentation point is determined using dynamic programming technology. The video segmentation point is used to divide the two segments to be processed.

[0102] The video segment to be processed is segmented according to at least one video segmentation point to obtain at least one sub-segment to be processed.

[0103] Specifically, in the process of determining at least one video segmentation point based on the similarity between each two video frames using dynamic programming technology, an objective function can be set so that when the objective function is minimized, the similarity of each video frame in the sub-segment to be processed, which is divided by the video segmentation point, is the highest. The formula of the objective function can be:

[0104]

[0105] Among them, C(S) represents the objective function value, |V k+1 | indicates the sub-fragment V to be processed k+1 , sim(i,j) represents the similarity between the i-th and j-th video frames, and λ1 represents a weighting coefficient used to balance the similarity within the sub-segments to be processed and the number of sub-segments to be processed.

[0106] Among them, according to the similarity between every two video frames, based on the dynamic programming technology, the formula for determining at least one video segmentation point is as follows:

[0107]

[0108] Where dp[t] represents the minimum objective function value for dividing the first t frames of the video into several sub-segments to be processed, λ2 represents the trade-off coefficient, which is used to balance the similarity within the sub-segments to be processed and the number of sub-segments to be processed, and dp[1]=0.

[0109] Specifically, the process of determining the segmentation points of the film based on the above formula may be based on an initialization process, an iterative calculation process, a final result determination process, and a backtracking path process.

[0110] Among them, in the process of performing similarity division processing on the to-be-processed film segments after de-flickering processing according to the similarity between each two film frames to obtain at least one to-be-processed sub-segment, the film segmentation points can be accurately determined based on dynamic programming technology, and the accuracy of the to-be-processed sub-segment division can be improved, thereby improving the accuracy of film processing and further improving the user's film viewing experience.

[0111] An embodiment of the present application provides a process for performing similarity division processing on a de-flickered film segment to be processed based on the local spatial information capacity of the de-flickered film segment to be processed to obtain at least one sub-segment to be processed. The process determines the similarity between every two film frames in the de-flickered film segment to be processed based on at least one local spatial information sub-capacity included in the local spatial information capacity, and performs similarity division processing on the de-flickered film segment to be processed based on the similarity between every two film frames to obtain at least one sub-segment to be processed. By calculating the similarity between every two film frames in the de-flickered film segment to be processed, the sub-segments to be processed can be accurately and efficiently divided, thereby improving the efficiency and accuracy of film processing and further enhancing the user's film viewing experience.

[0112] Figure 3 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 3 ,like Figure 3 As shown, this embodiment Figure 1 or Figure 2 Based on the embodiment, a process of determining a standard frame of a sub-segment to be processed is described in detail. The method includes:

[0113] Step S301: Obtain at least one normalized histogram of each video frame in the sub-segment to be processed.

[0114] Specifically, at least one normalized histogram of each film frame in the sub-segment to be processed may be obtained.

[0115] Specifically, the present application does not limit the process of obtaining at least one normalized histogram of each film frame in the sub-segment to be processed. Optionally, each film frame in the sub-segment to be processed can be converted into the HSV color space, where the HSV color space includes a brightness color channel V, a hue color channel H, and a saturation color channel S; and then the normalized histogram of each color channel is calculated separately.

[0116] Optionally, after obtaining at least one normalized histogram of each film frame in the sub-segment to be processed, Gaussian smoothing may be performed on the histogram to reduce interference of noise on similarity calculation. Optionally, the smoothing coefficient σ=1.5.

[0117] Step S302: Calculate the global similarity value of each video frame in the sub-segment to be processed according to at least one normalized histogram.

[0118] Specifically, based on the at least one normalized histogram obtained in step S301, a global similarity value of each video frame in the sub-segment to be processed can be calculated, wherein the global similarity value is the sum of similarity values ​​between the video frame and other video frames.

[0119] Specifically, the present application does not limit the process of calculating the global similarity value of each video frame in the sub-segment to be processed based on at least one normalized histogram. Optionally, the similarity matrix between each two video frames in the sub-segment to be processed can be calculated first, and then the global similarity value of each video frame in the sub-segment to be processed can be calculated based on the similarity matrix between each two video frames in the sub-segment to be processed.

[0120] Specifically, the present application does not limit the process of calculating the similarity matrix between each two film frames in the sub-segment to be processed. Optionally, the similarity matrix between each two film frames in the sub-segment to be processed can be calculated according to the Bhattacharyya distance calculation method.

[0121] The calculation formula of the similarity matrix between each two video frames is as follows:

[0122]

[0123] Among them, d B (H i ,H j ) represents the similarity matrix between the i-th video frame and the j-th video frame, that is, the Bhattacharyya distance, where the smaller the Bhattacharyya distance, the more similar the distribution of the histograms between the two video frames. i and H j represents the normalized histogram of the i-th movie frame and the j-th movie frame, where L represents the number of pixel values ​​in the normalized histogram, for example, 256.

[0124] Specifically, the calculation formula for the global similarity value of each video frame is as follows:

[0125]

[0126] Among them, S(F i ) represents the movie frame F i The global similarity value of represents a smoothing constant, such as 10 -3 , used to avoid the denominator being zero.

[0127] The calculation of the similarity matrix and global similarity value described above is the similarity matrix and global similarity value corresponding to the brightness and hue channels.

[0128] Among them, the normalized histograms of each color channel described in step S301 are comprehensively considered. For example, in the process of obtaining the similarity matrix between each two movie frames, the similarity matrix corresponding to the normalized histogram of each color channel is obtained. For example, in the process of calculating the global similarity value of each movie frame, the similarity matrix corresponding to the normalized histogram of each color channel obtained can be comprehensively considered.

[0129] Step S303: If the maximum value among the global similarity values ​​of the video frames corresponds to only one video frame, then the video frame is determined as the standard frame.

[0130] Specifically, according to the calculation process in step S302 , if it is determined that the maximum value among the global similarity values ​​of the video frames corresponds to only one video frame, the video frame corresponding to the maximum value is determined as the standard frame.

[0131] Step S304 : If there are multiple frames corresponding to the maximum value of the global similarity values ​​of the frames, a frame at a middle position in the time sequence among the multiple frames is determined as the standard frame.

[0132] Specifically, according to the calculation process in step S302, if it is determined that there are multiple film frames corresponding to the maximum value of the global similarity values ​​of each film frame, the film frame at the middle position in the time series among the multiple film frames is determined as the standard frame to ensure temporal continuity.

[0133] When the number of the plurality of film frames is an odd number, the middle frame is taken as the standard frame; when the number of the plurality of film frames is an even number, the frame with the smaller sequence number of the two middle frames in the time series is taken as the standard frame.

[0134] The embodiment of the present application provides a process for determining a standard frame of a sub-segment to be processed, by obtaining at least one normalized histogram of each film frame in the sub-segment to be processed, and calculating the global similarity value of each film frame in the sub-segment to be processed based on the at least one normalized histogram. If the maximum value of the global similarity values ​​of each film frame corresponds to one film frame, then the film frame is determined as the standard frame. If the maximum value of the global similarity values ​​of each film frame corresponds to multiple film frames, then the film frame at the middle position of the time series of the multiple film frames is determined as the standard frame. In the process of determining the standard frame, the frame to be processed is selected by global similarity. By processing the film frame with the highest similarity to other film frames in the sub-segment as the standard frame, the optimization processing process based on the standard frame can be made more accurate, thereby improving the accuracy of film processing and further improving the user's film viewing experience. Specifically, in the process of determining the global similarity value of each film frame, based on at least one normalized histogram, the comprehensiveness and accuracy of the global similarity calculation can be improved, thereby improving the accuracy of film processing and further improving the user's film viewing experience. In summary, the process of determining the standard frame of the sub-segment to be processed provided in the embodiment of the present application can improve the user's film viewing experience.

[0135] Figure 4 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 4 ,like Figure 4 As shown, this embodiment Figure 1 or Figure 2 or Figure 3 Based on the embodiment, a process of optimizing the non-standard frames of the sub-segment to be processed according to the standard frames to obtain the video segment to be processed after the respiratory flicker removal is described in detail. The method includes:

[0136] Step S401: Obtain at least one normalized histogram of each video frame in the sub-segment to be processed.

[0137] Specifically, at least one normalized histogram of each video frame in the sub-segment to be processed may be obtained. The specific description of this step may refer to the description of step S301 and will not be repeated here.

[0138] Step S402: Calculate the normalized histogram cumulative distribution function value of each video frame in the sub-segment to be processed according to at least one normalized histogram.

[0139] Specifically, based on the at least one normalized histogram obtained in step S401, the normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed can be calculated. The formula for calculating the normalized histogram cumulative distribution function value of the standard frame in the sub-segment to be processed is as follows:

[0140]

[0141] Among them, CDF ref (C) represents the normalized histogram cumulative distribution function value of the standard frame, C represents the set of normalized histograms of the movie frames, where C∈{V: brightness, H: hue, S: saturation}, and L represents the number of pixel values ​​in the normalized histogram, for example, 256.

[0142] The formula for calculating the normalized histogram cumulative distribution function value of the non-standard frame in the sub-segment to be processed is as follows:

[0143]

[0144] Among them, CDF i (C) represents the normalized histogram cumulative distribution function value of non-standard frames.

[0145] The normalized histogram cumulative distribution function value of the non-standard frame in the sub-segment to be processed and the calculated normalized histogram cumulative distribution function value of the non-standard frame in the sub-segment to be processed described above are the cumulative distribution function values ​​corresponding to the brightness and color tone channels.

[0146] Among them, the normalized histogram of each color channel described in step S401 is comprehensively considered. For example, in the process of calculating the normalized histogram cumulative distribution function value of the non-standard frame in the sub-segment to be processed and calculating the normalized histogram cumulative distribution function value of the non-standard frame in the sub-segment to be processed, the cumulative distribution function value corresponding to the normalized histogram of each color channel is calculated respectively.

[0147] Step S403 : Mapping the normalized histogram cumulative distribution function value of each non-standard frame according to the normalized histogram cumulative distribution function value of the standard frame to obtain a to-be-processed film segment after the respiratory flicker removal process.

[0148] Specifically, based on the normalized histogram cumulative distribution function value of the standard frame obtained in step S402, the normalized histogram cumulative distribution function value of each non-standard frame can be mapped to obtain the processed video segment after the respiratory flicker removal process. The calculation formula in the mapping process is as follows:

[0149]

[0150] Among them, y is the mapping value in the mapping process. Specifically, the mapping value is the standard frame CDF for each pixel value x. ref (H) Find the closest CDF i (H) pixel value.

[0151] Optionally, in the process of calculating the mapping value, linear interpolation can be used to improve the mapping accuracy and avoid the staircase effect.

[0152] The mapping process described above corresponds to the brightness color channel. Specifically, for the hue color channel and the saturation color channel, the mapping process described above can also be referred to, and will not be repeated here.

[0153] Optionally, during the mapping process corresponding to the hue color channel, edge continuity processing may be performed first.

[0154] Specifically, after the mapping process described above, each color channel can be merged to obtain a processed HSV image, and then the HSV image is converted into RGB space to obtain a to-be-processed video clip after the breathing flicker removal process.

[0155] The embodiment of the present application provides a process for optimizing the non-standard frames of the sub-segment to be processed according to the standard frame to obtain the film segment to be processed after the respiratory flicker removal process, by obtaining at least one normalized histogram of each film frame in the sub-segment to be processed, calculating the normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed according to the at least one normalized histogram, and mapping the normalized histogram cumulative distribution function value of each non-standard frame according to the normalized histogram cumulative distribution function value of the standard frame to obtain the film segment to be processed after the respiratory flicker removal process, wherein the normalized histogram of the standard frame is the normalized histogram of the non-standard frame. Cumulative distribution function value, mapping processing is performed on the normalized histogram cumulative distribution function value of each non-standard frame, which can accurately and efficiently perform respiratory flicker removal processing, thereby improving the user's movie viewing experience. Among them, in the mapping processing process, based on at least one normalized histogram, the comprehensiveness and accuracy of the respiratory flicker removal processing can be improved, and the user's movie viewing experience can be further improved. Based on the above description, the embodiment of the present application provides a process for optimizing the non-standard frames of the sub-segment to be processed according to the standard frame to obtain the movie segment to be processed after the respiratory flicker removal processing, which can improve the user's movie viewing experience.

[0156] Figure 5 Schematic diagram of the process of removing the breathing flicker of the film screen provided by this application Figure 5 ,like Figure 5 As shown, this embodiment Figure 1 or Figure 2 or Figure 3 or Figure 4 Based on the embodiment, a method for removing breathing flicker from another video image is described in detail. The method includes:

[0157] Step S501: Obtain the video clip to be processed after the flicker removal process.

[0158] Specifically, the specific process of this step can refer to the description in step S101, which will not be repeated here.

[0159] Step S502 : performing similarity division processing on the de-flickered video segment to be processed according to the local spatial information capacity of the de-flickered video segment to be processed, to obtain at least one sub-segment to be processed.

[0160] Specifically, the specific process of this step can refer to the description in step S102, which will not be repeated here.

[0161] Step S503: Determine the standard frame of the sub-segment to be processed, and optimize the non-standard frame of the sub-segment to be processed according to the standard frame to obtain the optimized sub-segment to be processed.

[0162] Specifically, the specific process of this step can refer to the description in step S103, which will not be repeated here.

[0163] Step S504 : Calculate the luminance histogram cumulative distribution function value of the standard frame of each optimized sub-segment to be processed, and determine the luminance mapping relationship between adjacent optimized sub-segments to be processed based on the luminance histogram cumulative distribution function value.

[0164] Specifically, after completing the optimization processing of the non-standard frames of each sub-fragment to be processed according to the standard frame, the brightness histogram of the standard frame of each sub-fragment to be processed after the optimization processing can be obtained first, and then based on the brightness histogram of the standard frame of each sub-fragment to be processed after the optimization processing, the cumulative distribution function value of the brightness histogram of the standard frame of each sub-fragment to be processed after the optimization processing can be calculated.

[0165] Specifically, the formula for the brightness histogram of the standard frame of the sub-segment to be processed after optimization is as follows:

[0166]

[0167] Among them, n s (k) is the standard frame F of the sub-segment to be processed after optimization s The number of pixels with brightness value k, N s is the standard frame F of the sub-segment to be processed after optimization s The total number of pixels.

[0168] Specifically, the formula for the cumulative distribution function value of the brightness histogram of the standard frame of the sub-segment to be processed after optimization is as follows:

[0169]

[0170] Among them, the brightness histogram cumulative distribution function value C of the standard frame of the sub-segment to be processed after optimization processing is s (k) reflects the cumulative proportion of pixels with brightness values ​​less than or equal to k in the standard frame of the sub-segment to be processed after optimization, providing key data for the subsequent determination of the brightness mapping relationship between adjacent sub-segments.

[0171] Specifically, the formula for the brightness mapping relationship between every two adjacent optimized sub-segments to be processed is as follows:

[0172]

[0173] Wherein, M(k) represents the brightness mapping relationship between two adjacent optimized sub-segments to be processed, and the brightness mapping relationship is determined by minimizing the difference in the cumulative distribution function of adjacent standard frames. Indicates the S i The cumulative distribution function value of the brightness histogram of the standard frame of the sub-segment to be processed after optimization, Indicates the S i+1 The cumulative distribution function value of the brightness histogram of the standard frame of the sub-segment to be processed after optimization processing.

[0174] Specifically, for two adjacent optimized sub-segments S to be processed i and S i+1 , define the brightness mapping relationship in the following two directions: First, the forward mapping function M i→i+1 (k): The brightness value of the optimized sub-segment to be processed is mapped to the brightness space of the i+1th optimized sub-segment to be processed, and the determination process satisfies , the formula is:

[0175]

[0176] Where: C i (k) is the sub-segment S to be processed after optimization i The cumulative distribution function value of the brightness histogram of the standard frame, is the sub-segment S to be processed after optimization i+1 The inverse function of the cumulative distribution function value of the brightness histogram of the standard frame. The pixels with brightness value k in the optimized sub-segment to be processed are calculated by their cumulative distribution function C i (k) is converted into a probability value (cumulative proportion), and then the inverse function of the cumulative distribution function of the i+1th optimized sub-segment to be processed is used , map this probability value back to the brightness space of the i+1th optimized sub-segment to be processed, and get Mi→i+1 (k).

[0177] Second, the backward mapping function : Map the brightness value of the i+1th sub-segment to the The brightness space of the sub-segments is determined by , the formula is:

[0178]

[0179] The specific description of this formula can refer to the description of the forward mapping function, which will not be repeated here.

[0180] Step S505 : setting an overlapping region between adjacent optimized sub-segments to be processed, and performing brightness interpolation adjustment on each video frame in the overlapping region based on the brightness mapping relationship to obtain two sets of overlapping regions after brightness interpolation adjustment.

[0181] The overlapping area includes all boundary frames of adjacent optimized sub-segments to be processed.

[0182] Specifically, to ensure that the adjacent optimized sub-segments S to be processed i and S i+1 The regional energy distribution and grayscale complexity of the smooth transition can be set between the two. The overlapping area O contains the sub-segment S to be processed after the previous optimization process. i The latter part of the frame, that is, the rear boundary frame, and the next optimized sub-segment to be processed S i+1 The front partial frame is a front boundary frame, wherein the rear partial frame and the front partial frame include multiple film frames.

[0183] Specifically, the process of performing brightness interpolation adjustment on each film frame in the overlapping area based on the brightness mapping relationship to obtain two sets of overlapping areas after brightness interpolation adjustment can be calculated by a linear interpolation algorithm, as follows:

[0184] Specifically, for any frame in the overlapping area, two adjusted brightness values ​​are generated based on the bidirectional brightness mapping relationship.

[0185] Among them, the adjustment value y1 is based on the forward mapping.

[0186]

[0187] Specifically, for one of the overlapping regions after brightness interpolation adjustment, each video frame in the overlapping region gradually transitions to the optimized sub-segment S to be processed. i+1, where x represents the current brightness value of each frame in the overlapping region before brightness interpolation, that is, the original brightness value of the frame itself. α is the interpolation coefficient, ranging from [0, 1]. Adjusting α can control the degree of brightness adjustment and reduce the sudden brightness changes that may occur at the boundaries between adjacent sub-segments.

[0188] Among them, the adjustment value y2 is based on the backward mapping.

[0189]

[0190] Specifically, for one of the overlapping regions after brightness interpolation adjustment, each video frame in the overlapping region gradually transitions to the sub-segment S i brightness characteristics.

[0191] For each frame in the overlapping area, there are two adjusted brightness values. In each overlapping area after brightness interpolation adjustment, the adjusted brightness values ​​corresponding to the same frame are different. This is because each overlapping frame needs to retain two adjustment results at the same time, so that weighted fusion can be performed through the transparency channel later to achieve smooth transition. For example, for sub-segment S i The back boundary frame of α can be linearly increased from 0 to 1; for the sub-segment S i+1 , α can be linearly reduced from 1 to 0 to ensure that the brightness of the movie picture is more consistent.

[0192] Step S506 : Generate a transparency channel value corresponding to the overlapping area, and perform weighted fusion processing on the pixel brightness values ​​of each video frame in the overlapping area after the two sets of brightness interpolation adjustments based on the transparency channel value to obtain a video segment to be processed after the respiratory flicker removal processing.

[0193] Specifically, after setting the overlapping area O, the value T of the corresponding transparency channel can be generated for the overlapping area O. Assume that the number of frames of the overlapping area O is N O , the overlapping area O is in the sub-segment S to be processed after the previous brightness interpolation adjustment i The number of frames in , after the brightness interpolation adjustment of the sub-segment to be processed S i+1 The number of frames in ,and Among them, the value T of the transparency channel is the sub-segment S to be processed after the previous brightness interpolation adjustment. i The overlapping area of ​​​​decreases linearly from 1 to 0, and the sub-segment S to be processed after the brightness interpolation adjustment is i+1 In the overlapping area, the value increases linearly from 0 to 1. Specifically, for the nth frame (n=1,⋯,N O ), the value of the transparency channel T(n) can be calculated by the following formula:

[0194]

[0195] In addition, the value T of the transparency channel is generated based on the local spatial information sub-capacity characteristics of the sub-segment to be processed after brightness interpolation adjustment. For example, it can be adjusted according to information such as the local stability index (LSIC) value of the pixel blocks in adjacent sub-segments to be processed, to ensure a smooth transition of the regional energy distribution and grayscale complexity of the sub-segments to be processed after adjacent brightness interpolation adjustment, thereby avoiding obvious visual discontinuities.

[0196] Specifically, based on the generated transparency channel value T(n), a weighted fusion process can be performed on the pixel brightness values ​​of each video frame in the overlapping area after the two sets of brightness interpolation adjustments to obtain the processed video segment after the breathing flicker removal process. Specifically, the formula in the weighted fusion process is as follows:

[0197]

[0198] Among them, I(n) is the brightness value after fusion, I i (n) and I i+1 (n) is the pixel brightness value of the nth film frame in the overlapping area after the two sets of brightness interpolation adjustment, where I i (n) is the sub-segment S to be processed after the optimization processing of the pixels of the nth frame in the overlapping area O i According to the optimized sub-segment S to be processed i+1 The brightness value after adjustment of the standard frame, I i+1 (n) is the sub-segment S to be processed after the optimization processing of the pixels of the nth frame in the overlapping area O i+1 According to the optimized sub-segment S to be processed i Specifically, after such weighted fusion processing, a video clip to be processed after the breathing flicker removal processing is finally obtained, and the brightness and picture transition of the video clip are more natural and smooth.

[0199] The embodiments of the present application provide a further processing process after optimizing the non-standard frames of the sub-segments to be processed according to the standard frames, including calculating the brightness histogram cumulative distribution function value of the standard frames of each optimized sub-segment to be processed, determining the brightness mapping relationship between adjacent optimized sub-segments to be processed based on the brightness histogram cumulative distribution function value, performing brightness interpolation adjustment on the boundary frames of adjacent optimized sub-segments to be processed based on the brightness mapping relationship to obtain sub-segments to be processed after the brightness interpolation adjustment, setting an overlapping region between the adjacent sub-segments to be processed after the brightness interpolation adjustment, generating a transparency channel value corresponding to the overlapping region, and performing weighted fusion processing on the pixel brightness values ​​of each frame in the overlapping region based on the transparency channel value to obtain the processed video segment to be processed after the respiratory flicker removal process. The precise determination of the brightness mapping relationship and the brightness interpolation adjustment using a mathematical formula, as well as the weighted fusion using the transparency channel value, can accurately and efficiently achieve a smooth transition between adjacent sub-segments, reduce brightness abrupt changes and visual discontinuities, and further improve the effectiveness of the respiratory flicker removal process. At the same time, the transparency channel value is generated by considering the local spatial information sub-capacity characteristics, ensuring the consistency and naturalness of the film image in terms of regional energy distribution and grayscale complexity, thereby significantly improving the user's film viewing experience. In summary, the subsequent processing process provided by the embodiment of the present application can further improve the quality of film processing and provide users with a better viewing experience.

[0200] Figure 6 This is a schematic diagram of the structure of the device for removing the breathing flicker from the movie screen provided by this application. Figure 6 As shown, the breathing flicker removal processing device 60 provided in this embodiment includes:

[0201] Acquisition module 601 is configured to acquire a flicker-free video clip to be processed; the flicker-free video clip is a video clip from the same video scene; the flicker-free video clip is obtained by performing a de-skew correction on the exposure-abnormal frame; the exposure-abnormal frame is determined from multiple frames of the video clip to be processed based on the local spatial information capacity; the local spatial information capacity is used to characterize the regional energy distribution of the video frame;

[0202] A first processing module 602 is configured to perform similarity division processing on the de-flickered video segment to be processed according to the local spatial information capacity of the de-flickered video segment to be processed, to obtain at least one sub-segment to be processed;

[0203] The second processing module 603 is configured to determine a standard frame of the sub-segment to be processed, and optimize the non-standard frame of the sub-segment to be processed according to the standard frame to obtain the video segment to be processed after the respiratory flicker removal process.

[0204] In one possible embodiment, the local spatial information capacity includes at least one local spatial information sub-capacity; wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of pixel blocks in a film frame; the first processing module 602 is specifically configured to determine the similarity between every two film frames in a to-be-processed film segment after de-flickering processing based on the at least one local spatial information sub-capacity included in the local spatial information capacity; and perform similarity partitioning processing on the to-be-processed film segment after de-flickering processing based on the similarity between every two film frames to obtain at least one to-be-processed sub-segment.

[0205] In one possible embodiment, the first processing module 602 is further specifically configured to perform feature extraction processing on at least one local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector for each film frame; and perform cosine similarity calculation processing on the local spatial information sub-capacity feature vector for each film frame to obtain a similarity between every two film frames.

[0206] In one possible implementation, the first processing module 602 is further specifically configured to perform feature extraction processing on at least one local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector for each film frame; obtain an information entropy feature vector for each film frame, and concatenate the local spatial information sub-capacity feature vector and the information entropy feature vector to obtain an integrated information feature vector for each film frame; and perform cosine similarity calculation processing on the integrated information feature vector of each film frame to obtain the similarity between every two film frames.

[0207] In a possible embodiment, the first processing module 602 is further specifically configured to determine at least one film segmentation point based on the similarity between each two film frames and based on dynamic programming technology; wherein the film segmentation point is used to divide two segments to be processed; and the segmentation processing is performed on the segment to be processed according to the at least one film segmentation point to obtain at least one sub-segment to be processed.

[0208] In a possible embodiment, the second processing module 603 is specifically configured to obtain at least one normalized histogram of each film frame in the sub-segment to be processed; calculate a global similarity value for each film frame in the sub-segment to be processed based on the at least one normalized histogram; wherein the global similarity value is the sum of similarity values ​​between the film frame and other film frames; if the maximum value of the global similarity values ​​of each film frame corresponds to only one film frame, then determine that film frame as the standard frame; if the maximum value of the global similarity values ​​of each film frame corresponds to multiple film frames, then determine a film frame that is located in the middle of the time series of the multiple film frames as the standard frame.

[0209] In one possible embodiment, the second processing module 603 is further specifically used to obtain at least one normalized histogram of each film frame in the sub-segment to be processed; calculate the normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed based on the at least one normalized histogram; and map the normalized histogram cumulative distribution function value of each non-standard frame based on the normalized histogram cumulative distribution function value of the standard frame to obtain the film segment to be processed after the respiratory flicker removal processing.

[0210] In a possible embodiment, the second processing module 603 is further used to calculate the cumulative distribution function value of the brightness histogram of the standard frame of each optimized sub-segment to be processed, and determine the brightness mapping relationship between adjacent optimized sub-segments to be processed based on the brightness histogram cumulative distribution function value; set an overlapping area between adjacent optimized sub-segments to be processed, and perform brightness interpolation adjustment on each film frame in the overlapping area based on the brightness mapping relationship to obtain two groups of overlapping areas after brightness interpolation adjustment; wherein the overlapping area includes all boundary frames of the adjacent optimized sub-segments to be processed; generate a value of a transparency channel corresponding to the overlapping area, and perform weighted fusion processing on the pixel brightness values ​​of each film frame in the two groups of overlapping areas after brightness interpolation adjustment according to the value of the transparency channel to obtain the film segment to be processed after respiratory flicker removal processing.

[0211] The device for removing respiratory flicker from a film image provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0212] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the electronic device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0213] During the specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that the at least one processor 701 performs the above method.

[0214] The specific implementation process of the processor 701 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0215] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0216] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0217] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0218] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0219] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0220] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0221] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0222] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0223] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0224] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0225] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0226] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0227] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A method for removing breathing flicker from a film image, characterized in that: include: Obtain a flicker-free film segment to be processed; wherein the flicker-free film segment to be processed is a segment of a film image in the same film scene; the flicker-free film segment to be processed is obtained by performing deskew processing on an exposure-abnormal frame; the exposure-abnormal frame is determined from multiple film frames in the flicker-free film segment based on a local spatial information capacity; the local spatial information capacity is used to characterize the regional energy distribution of the film frame; the local spatial information capacity is a parameter for characterizing the regional energy distribution of the film frame, determined by obtaining a noise variance of the film frame and a spectral energy of each pixel block in the film frame, and based on the noise variance and the spectral energy of each pixel block; performing similarity division processing on the to-be-processed video segment after the de-flickering processing according to the local spatial information capacity of the to-be-processed video segment after the de-flickering processing to obtain at least one to-be-processed sub-segment; Determining a standard frame of the sub-segment to be processed; Obtaining at least one normalized histogram of each video frame in the sub-segment to be processed; Calculating a normalized histogram cumulative distribution function value of each film frame in the sub-segment to be processed according to the at least one normalized histogram; According to the normalized histogram cumulative distribution function value of the standard frame, the normalized histogram cumulative distribution function value of each non-standard frame is mapped to obtain a to-be-processed film segment after the respiratory flicker removal process.

2. The method according to claim 1, characterized in that The local spatial information capacity includes at least one local spatial information sub-capacity; wherein the local spatial information sub-capacity is used to characterize the regional energy distribution of the pixel block in the film frame; The method further comprises: performing similarity division processing on the to-be-processed video segment after the de-flickering processing according to the local spatial information capacity of the to-be-processed video segment after the de-flickering processing to obtain at least one to-be-processed sub-segment; determining, according to at least one of the local spatial information sub-capacity included in the local spatial information capacity, a similarity between every two video frames in the to-be-processed video segment after the de-flickering process; A similarity division process is performed on the to-be-processed video segment after the de-flickering process according to the similarity between each two video frames to obtain at least one to-be-processed sub-segment.

3. The method according to claim 2, characterized in that Determining, based on at least one of the local spatial information sub-capacity included in the local spatial information capacity, a similarity between every two video frames in the to-be-processed video segment after the de-flickering process, comprises: performing feature extraction processing on at least one of the local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector of each film frame; A cosine similarity calculation process is performed on the local spatial information sub-capacity feature vectors of the film frames to obtain the similarity between each two film frames.

4. The method according to claim 2, characterized in that Determining, based on at least one of the local spatial information sub-capacity included in the local spatial information capacity, a similarity between every two video frames in the to-be-processed video segment after the de-flickering process, comprises: performing feature extraction processing on at least one of the local spatial information sub-capacity included in the local spatial information capacity to obtain a local spatial information sub-capacity feature vector of each film frame; Obtaining an information entropy feature vector of each film frame, and performing splicing processing on the local spatial information sub-capacity feature vector and the information entropy feature vector to obtain an integrated information feature vector of each film frame; The cosine similarity calculation process is performed on the integrated information feature vectors of the film frames to obtain the similarity between each two film frames.

5. The method according to claim 2, characterized in that The method further comprises: performing similarity division processing on the to-be-processed video segment after the flicker removal processing according to the similarity between each two video frames to obtain at least one to-be-processed sub-segment; and Determining at least one video segmentation point based on the similarity between each two video frames and a dynamic programming technique; wherein the video segmentation point is used to divide the two to-be-processed segments; The to-be-processed video segment is segmented according to the at least one video segmentation point to obtain the at least one to-be-processed sub-segment.

6. The method according to claim 1, characterized in that Determining the standard frame of the sub-segment to be processed includes: Obtaining at least one normalized histogram of each video frame in the sub-segment to be processed; Calculating a global similarity value of each video frame in the sub-segment to be processed according to the at least one normalized histogram; wherein the global similarity value is a sum of similarity values ​​between the video frame and other video frames; If the maximum value among the global similarity values ​​of the video frames corresponds to only one video frame, then determining the video frame as the standard frame; If there are multiple film frames corresponding to the maximum value of the global similarity values ​​of the film frames, a film frame located in the middle of the time sequence among the multiple film frames is determined as the standard frame.

7. The method according to any one of claims 1 to 6, characterized in that After optimizing the non-standard frame of the sub-segment to be processed according to the standard frame, the method further includes: Calculating a luminance histogram cumulative distribution function value of a standard frame of each optimized sub-segment to be processed, and determining a luminance mapping relationship between adjacent optimized sub-segments to be processed based on the luminance histogram cumulative distribution function value; Setting an overlapping region between adjacent optimized sub-segments to be processed, and performing brightness interpolation adjustment on each film frame in the overlapping region based on the brightness mapping relationship to obtain two sets of brightness interpolated overlapping regions; wherein the overlapping region includes all boundary frames of the adjacent optimized sub-segments to be processed; Generate a transparency channel value corresponding to the overlapping area, and perform weighted fusion processing on the pixel brightness values ​​of each film frame in the overlapping area after the two groups of brightness interpolation adjustments according to the transparency channel value to obtain the film segment to be processed after the respiratory flicker removal processing.

8. A device for removing breath flicker from a video image, characterized in that: include: An acquisition module is configured to acquire a flicker-free film segment to be processed; wherein the flicker-free film segment to be processed is a segment of a film image in the same film scene; the flicker-free film segment to be processed is obtained by performing a de-skew process on an exposure-abnormal frame; the exposure-abnormal frame is determined from multiple film frames in the flicker-free film segment based on a local spatial information capacity; the local spatial information capacity is used to characterize the regional energy distribution of the film frame; the local spatial information capacity is a parameter for characterizing the regional energy distribution of the film frame, determined by obtaining a noise variance of the film frame and a spectral energy of each pixel block in the film frame, and based on the noise variance and the spectral energy of each pixel block; a first processing module configured to perform similarity division processing on the de-flickered video segment to be processed according to the local spatial information capacity of the de-flickered video segment to be processed, to obtain at least one sub-segment to be processed; The second processing module is configured to determine a standard frame for the sub-segment to be processed, obtain at least one normalized histogram for each film frame in the sub-segment to be processed, calculate a normalized histogram cumulative distribution function value for each film frame in the sub-segment to be processed based on the at least one normalized histogram, and map the normalized histogram cumulative distribution function value of each non-standard frame based on the normalized histogram cumulative distribution function value of the standard frame to obtain the film segment to be processed after respiratory flicker removal processing.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images