Video dirty point detection method and device and electronic device

Abstract

The invention provides a video dirty point detection method, video dirty point detection equipment and electronic equipment, which are used for locally detecting a dirty point area based on the similarity of adjacent image frames by utilizing various traditional operation methods in the field of image processing without depending on a deep learning framework, effectively reducing the complexity of operation and improving the flexibility of detection. The method comprises the steps that a reference frame corresponding to a current image frame in a video is acquired, the reference frame comprises a front reference frame and a rear reference frame, the front reference frame is an image frame before the current image frame in the video, and the rear reference frame is an image frame after the current image frame in the video; performing filtering processing on the current image frame and a front reference frame and a rear reference frame corresponding to the current image frame to obtain a filtered image frame; and determining whether dirty points exist in the current image frame or not according to the similarity between the filtered image frame and the current image frame and the similarity between the filtered image frame and a front reference frame and a rear reference frame corresponding to the current image frame.

Description

Method, device and electronic device for detecting dirty spots in video Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and electronic device for detecting dirty points in a video. Background Art

[0002] Dirty pixels, as a type of video damage, refer to areas in the video's image frames where image information is missing or blocked. Dirty pixels usually appear as connected areas in the image frames, destroying the original image information in the image frames. These connected areas are usually white or black block-like spots or black stripes. In addition, dirty pixels are also random.

[0003] Currently, deep learning methods can be used to remove and repair dirty spots. However, in actual applications, the configuration requirements of the equipment used are relatively high. More applications are deployed in AI server devices, which are expensive to use. Although end-to-end deep learning methods make it easier for users to get started, due to their limited hyperparameters, they are difficult to flexibly adjust when dealing with scenarios such as videos with different frame rates and resolutions, and different sizes and colors of dirty spots. Retraining or fine-tuning requires using data from similar scenarios, which leads to the need for repeated development based on scenario customization.

[0004] Summary of the Invention

[0005] The present invention provides a method, device and electronic device for video dirty point detection, which utilize various traditional computing methods in the field of image processing, do not rely on deep learning frameworks, and perform local detection of dirty point areas based on the similarity of adjacent image frames, effectively reducing the complexity of calculations and improving the flexibility of detection.

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting dirty points in a video, the method comprising:

[0007] Obtain a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a previous reference frame and a subsequent reference frame, wherein the previous reference frame is an image frame before the current image frame in the video, and the subsequent reference frame is an image frame after the current image frame in the video;

[0008] Performing filtering on the current image frame, a previous reference frame, and a subsequent reference frame corresponding to the current image frame to obtain a filtered image frame;

[0009] It is determined whether there is a dirty point in the current image frame according to the similarity between the filtered image frame and the current image frame, and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

[0010] In a second aspect, an embodiment of the present invention provides a device for detecting dirty spots in a video, including: a processor and a memory, wherein the memory is configured to store a program executable by the processor, and the processor is configured to read the program in the memory and perform the following steps:

[0011] In response to a user input operation on at least one of a time parameter, a similarity parameter, a similarity threshold, a preset window size, and a preset step size, based on the method of the first aspect, dirty points of image frames in the video are detected according to the parameters corresponding to the input operation;

[0012] The time parameter represents the number of frames between the current image frame and the reference frame of the current image frame; the similarity parameter represents the range of the local area that affects the structural similarity of pixels at the same position in the two image frames when the average structural similarity is calculated using the structural similarity of pixels at the same position in the two image frames.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory is configured to store a program executable by the processor, and the processor is configured to read the program in the memory and perform the following steps:

[0014] Obtain a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a previous reference frame and a subsequent reference frame, wherein the previous reference frame is an image frame before the current image frame in the video, and the subsequent reference frame is an image frame after the current image frame in the video;

[0015] Performing filtering on the current image frame, a previous reference frame, and a subsequent reference frame corresponding to the current image frame to obtain a filtered image frame;

[0016] It is determined whether there is a dirty point in the current image frame according to the similarity between the filtered image frame and the current image frame, and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

[0017] In a fourth aspect, an embodiment of the present invention further provides a device for detecting dirty points in a video, the device comprising:

[0018] an acquisition unit, configured to acquire a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a previous reference frame and a subsequent reference frame, wherein the previous reference frame is an image frame before the current image frame in the video, and the subsequent reference frame is an image frame after the current image frame in the video;

[0019] a filtering unit, configured to perform filtering processing on the current image frame, a previous reference frame, and a subsequent reference frame corresponding to the current image frame to obtain a filtered image frame;

[0020] The judgment unit is used to determine whether there are dirty points in the current image frame according to the similarity between the filtered image frame and the current image frame, and the previous reference frame and the next reference frame corresponding to the current image frame.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect above.

[0022] These and other aspects of the present application will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] FIG1 is a flowchart of a method for detecting dirty points in a video according to an embodiment of the present invention;

[0025] FIG2 is a schematic diagram of extracting a reference frame according to an embodiment of the present invention;

[0026] FIG3 is a schematic diagram of a median filtering algorithm provided by an embodiment of the present invention;

[0027] FIG4 is a schematic diagram of a block-cropped image frame according to an embodiment of the present invention;

[0028] 5A-5D are comparative schematic diagrams of median filtering provided by an embodiment of the present invention;

[0029] FIG6 is a flow chart of automatic dirty point location according to an embodiment of the present invention;

[0030] FIG7 is a schematic diagram of a relationship between a similarity matrix and w according to an embodiment of the present invention;

[0031] FIG8 is a schematic diagram showing a binary image changing with a v value according to an embodiment of the present invention;

[0032] FIG9 is a schematic diagram showing changes in a binary image during an erosion-dilation process according to an embodiment of the present invention;

[0033] FIG10 is a schematic diagram of a dirty spot repair process according to an embodiment of the present invention;

[0034] FIG11 is a flowchart of a video dirty point detection and repair method provided by an embodiment of the present invention;

[0035] FIG12 is a schematic diagram of a flow chart of a detection and repair algorithm provided by an embodiment of the present invention;

[0036] FIG13 is a schematic diagram of a device for detecting dirty spots in a video according to an embodiment of the present invention;

[0037] FIG14 is a schematic diagram of an electronic device provided by an embodiment of the present invention;

[0038] FIG15 is a schematic diagram of a device for detecting dirty points in a video provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0040] In embodiments of the present invention, the term "and / or" describes the association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0041] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Persons skilled in the art will appreciate that as new application scenarios emerge, the technical solutions provided by the embodiments of the present invention will also be applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0042] Dirty pixels, as a type of video impairment, refer to areas in the video's image frames where image information is missing or blocked. Dirty pixels usually appear as connected regions in the image frames, destroying the original image information. These connected regions are usually white or black block-like spots or black stripes. In addition, dirty pixels are also random, that is, they appear randomly in the video.

[0043] The formation of dirty spots can be caused by film damage or other factors. This embodiment does not impose too many restrictions on the causes of dirty spots. Taking film damage as an example, during the storage and use of the film, due to the effect of static electricity, dust, hair and other fibers in the air may fall on and be adsorbed on the film. If it is not cleaned in time during use, such dirty spots will become part of the film copy. During the projection process, the dirty spots on the film are usually displayed as white or black block spots in the video. In addition, if the film itself has scratches, bad tapes and other damage, the scratches and bad tapes will also cause dirty spots to appear in the video display during projection. Dirty spots generally have the following three characteristics:

[0044] Feature 1: The grayscale variance of pixels inside a dirty spot is very small, and the grayscale within the block is basically consistent. Visually, it usually appears as an irregular black or white block.

[0045] Feature 2: The grayscale discontinuity of dirty spots in both the temporal and spatial domains. Because this damage is randomly distributed within a frame, the probability of a dirty spot reappearing in the same location between two adjacent frames is very low. Therefore, it appears as an impulsive damage in the temporal domain, i.e., it flickers during video playback. Within a frame, the grayscale of the dirty spot area is generally significantly different from the surrounding background grayscale, making it perceptible to the human eye.

[0046] Feature 3: Dirty points have spatial proximity. That is, if a pixel is within a dirty point, then the pixels around it are likely to belong to the dirty point area as well. In other words, a dirty point is not a single pixel, but an area.

[0047] Currently, deep learning methods can be used to remove and repair dirty spots. However, in actual applications, the configuration requirements of the equipment used are relatively high. More applications are deployed in AI server devices, which are expensive to use. Although end-to-end deep learning methods make it easier for users to get started, due to their limited hyperparameters, they are difficult to flexibly adjust when dealing with scenarios such as videos with different frame rates and resolutions, and different sizes and colors of dirty spots. Retraining or fine-tuning requires using data from similar scenarios, which leads to the need for repeated development based on scenario customization.

[0048] This embodiment provides a method for detecting dirty spots in a video. It uses various traditional computing methods in the field of image processing and does not rely on deep learning frameworks. It can be deployed on CPU computer devices without GPU computing resources, or can be run and used on mobile terminals or display terminals with better performance. This embodiment detects dirty spots based on their characteristics in the video. Since dirty spots are discontinuous in the time domain, and the contents of two adjacent frames in the same scene in the video are usually continuous, that is, the pixel values ​​at the same position in the two adjacent frames are usually similar, this application uses the adjacent frames before and after in the video frame sequence as reference frames to automatically detect and repair dirty spots in the current frame.

[0049] It should be noted that the method for detecting video dirty spots provided in this embodiment is used to detect dirty spots in the video and repair them. The dirty spots in this embodiment are used to represent video damage with the above three characteristics of dirty spots. Other video damage that is not called dirty spots but has the above three characteristics of dirty spots can also be detected and repaired using the method of this application, and also fall within the scope of protection of this application.

[0050] As shown in FIG1 , the implementation process of a method for detecting dirty spots in a video provided by this embodiment is as follows:

[0051] Step 100: Obtain a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a previous reference frame and a subsequent reference frame. The previous reference frame is an image frame before the current image frame in the video, and the subsequent reference frame is an image frame after the current image frame in the video.

[0052] Optionally, the front reference frame in this embodiment is one or more image frames before the current image frame in the video, and the rear reference frame is one or more image frames after the current image frame in the video; when the current reference frame includes multiple image frames, the multiple image frames are continuous in the video, and when the rear reference frame includes multiple image frames, the multiple image frames are also continuous in the video.

[0053] During implementation, except for the first frame in the video, each time an image frame is obtained, the previous reference frame and the next reference frame corresponding to the image frame are determined. When the first frame in the video is obtained, no processing is required, and detection can be started after the second frame is obtained.

[0054] In some embodiments, the previous reference frame and the current image frame in this embodiment are separated by n image frames, where n is determined based on the number of frames in which the dirty points appear continuously in consecutive image frames, and n is an integer greater than or equal to 0; and / or, the subsequent reference frame and the current image frame are separated by m image frames, where m is determined based on the number of frames in which the dirty points appear continuously in consecutive image frames, and m is an integer greater than or equal to 0.

[0055] Optionally, n and m in this embodiment are generally set to the same value, or different values ​​may be set based on the characteristics of actual dirty points. This embodiment does not impose any further limitations on this. In this embodiment, n is generally set to 0. When a dirty point appears continuously in multiple image frames, the value of n may be appropriately increased based on the number of consecutive frames in which the dirty point appears. Similarly, in this embodiment, m is generally set to 0, but the value of m may be appropriately increased based on the number of consecutive frames in which the dirty point appears.

[0056] In implementation, since dirty spots are discontinuous in the time domain, and the contents of two adjacent frames of a video in the same scene are usually continuous, it is possible to determine whether there are dirty spots in the current image based on the similarity between the current image frame and the previous and next adjacent frames. That is, at this time, it can be defined that the previous reference frame and the current image frame are separated by 0 image frames, that is, the previous reference frame and the current image frame are adjacent, and the subsequent reference frame and the current image frame are separated by 0 image frames, that is, the subsequent reference frame and the current image frame are adjacent, and the previous reference frame and the subsequent reference frame are two adjacent frames before and after the current image frame; in the time domain, at this time, the previous reference frame and the current image frame differ by 1 time interval, and similarly, the subsequent reference frame and the current image frame differ by 1 time interval, where the time interval represents the time difference between two adjacent frames.

[0057] If an abnormal dirty point appears in the same position in multiple frames, the value of n can be increased according to the number of frames in which the abnormal dirty point appears, until the reference frame no longer contains the dirty point in the same position. As shown in Figure 2, this embodiment provides a schematic diagram of extracting a reference frame, where I t Indicates the current image frame, I t-1 Indicates the previous adjacent image frame, i.e., the image frame with a time interval difference from the current image frame, I t-2 Indicates the previous image frame that is 2 time intervals away from the current image frame, I t+1 Indicates the next adjacent image frame, I t+2 Indicates the next image frame that is 2 time intervals away from the current image frame. When a dirty point appears in consecutive adjacent frames, that is, in I t-1 , I t , I t+1 When both appear, n can be set to 1, the previous reference frame and the current image frame are separated by 1 image frame, and the next reference frame and the current image frame are also separated by 1 image frame, that is, extract I t-2 , I t+2 As the previous reference frame and the next reference frame of the current image frame, I t-2 , I t+2 There are no dirty points, and the two reference frames can be used to detect dirty points in the current image frame.

[0058] In the implementation, for continuous image frames in the video, when processing the dirty points in the current frame, it is necessary to refer to the pixel values ​​of the same position in the adjacent frames for filling and repair. This processing link introduces interval n and interval m as time parameters, and the image frame I corresponding to the time node t is t For example, when extracting this image frame, it is also necessary to extract the reference frame I with n and m image frames before and after. t-(n+1) , I t+(m+1) ; The values ​​of the intervals n and m are related to the regularity of the appearance of dirty points in the video and can be set by the user after observation. As described in the conventional characteristics of dirty points above, most dirty points generally do not appear at the same position in a continuous sequence of video frames. Therefore, n and m are usually set to 0 to meet the requirements. If abnormal dirty points appear at the same position in multiple frames, the values ​​of n and m can be increased as appropriate until the reference frame no longer contains dirty points at the same position.

[0059] Step 101: Filter the current image frame, the previous reference frame, and the subsequent reference frame corresponding to the current image frame to obtain a filtered image frame;

[0060] Step 102: Determine whether there are dirty points in the current image frame based on the similarities between the filtered image frame and the current image frame, and the previous reference frame and the next reference frame corresponding to the current image frame.

[0061] In practice, the dirty point detection method provided in this embodiment performs dirty point detection on an image frame basis. After extracting the preceding and succeeding reference frames of the current image frame, each extracted image frame can be filtered to obtain a filtered image frame. Based on the magnitude relationship between the similarity between the filtered image frame and the extracted image frames, a determination is made as to whether dirty points exist in the current image frame.

[0062] In some embodiments, a filtering algorithm is used to perform filtering processing on the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain a filtered image frame.

[0063] Optionally, the filtering algorithm in this embodiment includes but is not limited to a temporal median filtering algorithm or a temporal majority filtering algorithm, wherein the temporal median filtering algorithm is used to obtain the median of the pixel values ​​at the same position in multiple image frames, that is, to obtain the median of the pixel values ​​at the same position of the current image frame, the previous reference frame, and the subsequent reference frame; the temporal majority filtering algorithm is used to obtain the largest number of identical pixel values ​​among the pixel values ​​at the same position in multiple image frames, that is, to obtain the majority of the pixel values ​​at the same position of the current image frame, the previous reference frame, and the subsequent reference frame, that is, the largest number of identical pixel values ​​among the pixel values ​​at the same position.

[0064] In implementation, a median filtering algorithm may be used to calculate the filtered image frames of the current image frame, the previous reference frame, and the subsequent reference frame corresponding to the current image frame; the calculation process is shown in the following formula: mid (i,j)=f median (I t-(n+1) (i,j),I t (i,j),I t+(m+1) (i,j)) Formula (1a);

[0065] I in formula (1a) mid (i, j) represents the filtered image frame, I t-(n+1) (i, j) represents the previous reference frame, I t (i, j) represents the current image frame, I t+(m+1) (i, j) represents the next reference frame, n represents the number of frames between the current image frame and the previous reference frame, m represents the number of frames between the current image frame and the next reference frame, (i, j) represents the coordinates of the pixels in the image block, f median (x, y, z) represents the calculation of the median of x, y, and z. That is, formula (1a) is used to calculate the median of the pixel values ​​at each identical position in the previous reference frame, the current reference frame, and the next reference frame.

[0066] As shown in Figure 3, this embodiment provides a schematic diagram of a median filtering algorithm, wherein box A indicates that there are no dirty points in both the current image frame and the reference frame, and there are still no dirty points after median filtering. Since the scene in box A is relatively uniform, the original image structure is not destroyed after median filtering. Box B indicates that there are no dirty points in both the current image frame and the reference frame, and there are still no dirty points after median filtering. The scene in box B is in motion, and the structure of the current image is changed after median filtering. Box C indicates that there are dirty points in the current image frame, there are no dirty points in the reference frame, and there are no dirty points after median filtering.

[0067] In some embodiments, the following steps are used to determine whether there are dirty points in the current image frame based on the similarity between the current image frame and the reference frame:

[0068] Step 1) determining a maximum similarity from the similarities between the filtered image frame and the previous reference frame and the next reference frame;

[0069] Optionally, the similarity in this embodiment is used to characterize the correlation between pixels of different image frames, and the similarity includes but is not limited to structural similarity and / or average structural similarity.

[0070] In implementation, the structural similarity is calculated using the following formula:

[0071] in,

[0072] In formula (2a), SSIM(x, y) represents the structural similarity of image frame x and image frame y; μ(x) and μ(y) represent the mean values ​​of image frame x and image frame y, respectively; σ(x) and σ(y) represent the standard deviations of image frame x and image frame y, respectively; σ(x, y) represents the covariance of image frame x and image frame y, c1=6.5025, c2=58.5225, H, W, and C represent the height, width, and number of channels of image frame x or image frame y, respectively (usually, the image is an RGB image, and the value is 3). In this embodiment, the height, width, and number of channels of the two image frames for which the structural similarity is calculated are the same.

[0073] Step 2) Determine whether there are dirty points in the current image frame based on a comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame.

[0074] In the implementation, the filtered image frame I mid With the previous reference frame I t-(n+1) The structural similarity of the filtered image block I is recorded as S1; mid With the post-reference block I t+(m+1) The structural similarity of the filtered image block I is recorded as S2; mid With the current image block I t The structural similarity of is denoted as S0;

[0075] In some embodiments, if the similarity between the filtered image frame and the current image frame is less than the maximum similarity, it is determined that there are dirty points in the current image frame;

[0076] If the similarity between the filtered image frame and the current image frame is greater than or equal to the maximum similarity, it is determined that no dirty points exist in the current image frame.

[0077] In the implementation, if S0 < max(S1, S2), it is determined that there are dirty points in the current image frame;

[0078] If S0≥max(S1, S2), it is determined that there is no dirty point in the current image frame.

[0079] In some embodiments, when the reference frame is a plurality of consecutive image frames, this embodiment may further perform any one or more of the following:

[0080] (1) If the previous reference frame is a plurality of consecutive image frames before the current image frame in the video, the maximum similarity is determined from the similarities between the filtered image frame and each of the previous reference frames and the subsequent reference frame; and based on the comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame, it is determined whether there is a dirty point in the current image frame.

[0081] (2) If the subsequent reference frame is a plurality of consecutive image frames following the current image frame in the video, a maximum similarity is determined from the similarities between the filtered image frame and the previous reference frame and each subsequent reference frame; and based on a comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame, it is determined whether there is a dirty point in the current image frame.

[0082] (3) If the previous reference frame is a plurality of consecutive image frames before the current image frame in the video, and the subsequent reference frame is a plurality of consecutive image frames after the current image frame in the video, then the maximum similarity is determined from the similarities between the filtered image frame and each previous reference frame and each subsequent reference frame respectively; based on the comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame, it is determined whether there is a dirty point in the current image frame.

[0083] In some embodiments, in order to roughly locate the dirty point area, the dirty point determination range is limited to a smaller image block to avoid inaccurate determination caused by the influence of the motion artifact area being greater than the dirty point area when calculating the structural similarity. This embodiment also provides a method for performing dirty point detection in blocks, that is, performing dirty point detection in units of image blocks.

[0084] Optionally, before using a filtering algorithm to filter the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain a filtered image frame, the current image frame, the previous reference frame and the subsequent reference frame are respectively cut into multiple current image blocks, multiple previous reference blocks and multiple subsequent reference blocks; for each current image block, the current image block, the previous reference block and the subsequent reference block corresponding to the current image block are filtered using a filtering algorithm to obtain a filtered image block; and based on the similarity between the filtered image block and the current image block, the previous reference block and the subsequent reference block, it is determined whether there is a dirty point in the current image block.

[0085] During implementation, this embodiment can obtain the current image frame and the corresponding reference frame, and then divide the current image frame and the reference frame into blocks respectively. Then, a filtering algorithm such as a median filtering algorithm is used to calculate a filtered image block based on the current image block, the corresponding previous reference block, and the subsequent reference block. Based on the similarity between the filtered image block and the current image block, the previous reference block, and the subsequent reference block, it is determined whether there are dirty points in the current image block.

[0086] Optionally, after filtering the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame using a filtering algorithm to obtain a filtered image frame, the current image frame, the previous reference frame, the subsequent reference frame and the filtered image frame are respectively cut into multiple current image blocks, multiple previous reference blocks, multiple subsequent reference blocks and multiple filtered image blocks; and based on the similarity between the filtered image block and the current image block, the previous reference block and the subsequent reference block corresponding to the filtered image block, it is determined whether there are dirty points in the current image block.

[0087] It should be noted that, when calculating the similarity, the positions of the filtered image block, the current image block, the previous reference block and the next reference block in the respective image frames are the same.

[0088] In some embodiments, in order to ensure that each image block obtained after segmentation does not lose information of the original image frame, the image block further has the following characteristics:

[0089] Partial areas of the two current image blocks that are closest to each other in a preset direction overlap, partial areas of the two previous reference blocks that are closest to each other in a preset direction overlap, and partial areas of the two subsequent reference blocks that are closest to each other in a preset direction overlap; the preset direction includes a horizontal axis direction or a vertical axis direction.

[0090] In some embodiments, the image frame may be cropped to obtain image blocks by the following steps:

[0091] traversing the current image frame, the previous reference frame, and the next reference frame respectively according to a preset step size using a preset window, and cutting out a plurality of current image blocks, a plurality of previous reference blocks, and a plurality of next reference blocks having the same size as the preset window;

[0092] The preset step size includes a horizontal step size in the horizontal direction and a vertical step size in the vertical direction, the horizontal step size is less than or equal to the width of the preset window, and the vertical step size is less than or equal to the height of the preset window.

[0093] In practice, the preset window can be a square or a rectangle. In order to reduce the amount of operation parameters, the preset window can be set to a square. In this case, a preset step size can be set. For example, for the three extracted image frames (previous reference frame I t-1 , current image frame I t , back reference frame I t+1 ), cut and divide it into blocks according to the spatial step length s and the window side length (square width / height) k, and I t-1 , I t , I t+1Image blocks corresponding to the same position are grouped together to obtain several groups of image blocks for dirty point detection. As shown in FIG4 , this embodiment provides a schematic diagram of a block-by-block cropping image frame, where the window side length is k, the traversal step length is s, and s≤k.

[0094] The purpose of block cropping is to roughly locate dirty areas and limit the scope of dirty point determination to a smaller image block. This prevents the motion artifact area from having a greater influence than the dirty area when calculating structural similarity, leading to inaccurate determination. Parameters k and s control the size and spacing of the image blocks, with the fixed constraint of s ≤ k. Generally speaking, parameter k should be slightly larger than the size of the dirty area. If k is too small, the cropped image block will not cover the dirty area, and excessive block cutting will result in excessive computations, affecting algorithm efficiency. If k is too large, the structural differences in the dirty area will be too small relative to the entire image block, affecting the accuracy of dirty point determination.

[0095] After the current image frame and the reference frame are divided into blocks, the following methods are used to determine whether there are dirty points in the current image block, including:

[0096] First, a front reference block having the same position as each current image block is determined from a front reference frame as the front reference block corresponding to the current image block, and a rear reference block having the same position as each current image block is determined from a rear reference frame as the rear reference block corresponding to the current image block. Each current reference block, the corresponding front reference block, and the rear reference block can be regarded as a group of image blocks, and then a corresponding filtered image block is calculated for each group of image blocks. In implementation, for each current image block, a filtering algorithm is used to perform filtering processing on the current image block, the front reference block corresponding to the current image block, and the rear reference block corresponding to the current image block to obtain a filtered image block.

[0097] Optionally, a median filtering algorithm may be used to calculate the filtered image blocks of the current image block, the front reference block, and the back reference block corresponding to the current image block; the calculation process is shown in the following formula: mid (i,j)=f median (P t-(n+1) (i,j),P t (i,j),P t+(m+1) (i,j)) Formula (1b);

[0098] P in formula (1b) mid (i, j) represents the filtered image block, P t-(n+1) (i, j) represents the previous reference block, P t (i, j) represents the current image block, P t+(m+1) (i, j) represents the back reference block, n and m represent the number of frames between each block, (i, j) represents the coordinates of the pixel in the image block, and f median (x,y,z) means calculating the median of x, y, and z.

[0099] In practice, the filtered image blocks obtained by formula (1b) can be divided into the following cases:

[0100] Case a, as shown in FIG5A, is a comparative schematic diagram of a median filter provided by this embodiment; the current image block P t Contains dirty points, front and back reference blocks P t-(n+1) 、P t+(m+1) There is no dirty point, because the pixel value of the dirty point area is quite different from the surrounding pixel value, and in general, the pixel value of the same area in consecutive frames of the same scene does not change much, so the obtained filtered image block P mid No dirty spots;

[0101] Case b, as shown in FIG5B, is a comparative schematic diagram of a median filter provided by this embodiment; the current image block P t Contains dirty points, front and back reference blocks P t-(n+1) 、P t+(m+1) There is no dirty point, but the moving object in the previous and next reference frames is exactly located at the dirty point position and the pixel value is close to the dirty point. At this time, the filtered image block P mid There are dirty spots;

[0102] Case c, as shown in FIG5C, is a comparative schematic diagram of a median filter provided by this embodiment; the current image block P t No dirty points, front and back reference blocks P t-(n+1) 、P t+(m+1) Any one of the filtered image blocks P contains a dirty point mid No dirty spots;

[0103] Case d, as shown in FIG5D, is a comparative schematic diagram of a median filter provided by this embodiment; the current image block P t No dirty points, front and back reference blocks P t-(n+1) 、P t+(m+1) No dirty points, filtered image block P mid No dirty spots.

[0104] Secondly, a maximum similarity is determined from the similarities between the filtered image block and the previous reference block and the next reference block, respectively; wherein the similarity is used to characterize the correlation between pixels of image blocks at the same position in different image frames; the similarity includes structural similarity and / or average structural similarity.

[0105] In the implementation, for each group containing the current image block P t , the previous reference block P t-(n+1) , the back reference block P t+(m+1) The structural similarity between the filtered image block and the previous reference block and the subsequent reference block can be calculated:

[0106] (1) Calculate the filtered image block P mid With the previous reference block P t-(n+1) The structural similarity of , denoted as S1′;

[0107] (2) Calculate the filtered image block P mid With the post-reference block P t+(m+1) The structural similarity of , denoted as S2′;

[0108] (3) Calculate the filtered image block P mid With the current image block P t The structural similarity of , denoted as S0′;

[0109] In implementation, the structural similarity is calculated using the following formula:

[0110] in,

[0111] In formula (2b), SSIM(x, y) represents the structural similarity of image blocks x and y; μ(x) and μ(y) represent the mean values ​​of image blocks x and y, respectively; σ(x) and σ(y) represent the standard deviations of image blocks x and y, respectively; σ(x, y) represents the covariance of image blocks x and y, c1=6.5025, c2=58.5225, H, W, and C represent the height, width, and number of channels of image block x or image block y, respectively (usually 3 for RGB images). In this embodiment, the height, width, and number of channels of the two image blocks for which the structural similarity is calculated are the same.

[0112] Finally, it is determined whether there is a dirty point in the current image block according to a comparison result of the maximum similarity and the similarity between the filtered image block and the current image block.

[0113] Optionally, if the similarity between the filtered image block and the current image block is less than the maximum similarity, it is determined that there is a dirty point in the current image frame;

[0114] If the similarity between the filtered image block and the current image block is greater than or equal to the maximum similarity, it is determined that no dirty points exist in the current image block.

[0115] During implementation, there are several scenarios for determining whether there are dirty points, depending on the actual situation, as shown below:

[0116] In scenario a, if the current set of image blocks is as shown in FIG5A, because the previous and next reference blocks P t-(n+1) 、P t+(m+1) and Pmid There are no dirty points, and the current image block P t There are dirty points. According to formula (2), P mid With P t The structural similarity is lower than P mid The structural similarity calculated with the previous and next reference blocks is S0′<S1′ or S0′<S2′. In this case, the judgment result is that there are dirty points in the current image block.

[0117] In scenario b, if the current set of image blocks is as shown in FIG5B, although the previous and next reference blocks P t-(n+1) 、P t+(m+1) There are no dirty spots in the t There are dirty points, but because there is a large inter-frame motion at the location of the dirty points, the pixel values ​​of the dirty point area are different from P t-(n+1) The values ​​of normal pixels in the image are close, so the position in the median filtered image is the pixel value of the dirty point area. Therefore, there may be a situation where S0′≥max(S1′,S2′) or a situation where S0′<max(S1′,S2′). At this time, if S0′<max(S1′,S2′), it is determined that there is a dirty point and the subsequent process is processed normally. If S0′≥max(S1′,S2′), it is mistakenly determined that there is no dirty point. Because there are similar pixel values ​​in the dirty point area in the adjacent frames, the visual sensitivity to dirty points is much weaker during video playback. Therefore, no processing will have little impact on the overall viewing effect.

[0118] In scenario c, if the current group of image blocks is as shown in Figure 5C, P t-(n+1) There are two possibilities: if S0′≥S2′, it is directly judged that there is no dirty point; if S0′<S2′, it is judged that there is a dirty point and needs to be filled. However, due to P t There is no dirty point. When filling the defect, the reference block P corresponding to S2′ with higher structural similarity will be used. t+(m+1) As a filling material, it will not be affected by the previous reference block P t-(n+1) The impact of the dirty point.

[0119] In scenario d, if the current group of image blocks is as shown in Figure 5D, P mid 、P t-(n+1) 、P t+(m+1) and P t There are no dirty points. According to formula (1) and formula (2), S0′>S1′ and S0′>S2′; it is determined that there are no dirty points.

[0120] In summary, the image blocks containing dirty points can be filtered out by judging the condition S0′<max(S1′, S2′).

[0121] In some embodiments, after determining that there are dirty spots in the current image frame / current image block using the above method, it is necessary to further determine the specific position and shape range of the dirty spots in the image frame / image block, so as to use pixels in the same position and range in the previous and next reference frames / reference blocks for filling and repair.

[0122] As shown in FIG6 , this embodiment provides a process for automatically locating dirty spots, wherein a current image block containing a dirty spot and corresponding front reference blocks and back reference blocks are subjected to time-domain median filtering to obtain a corresponding filtered image block, and structural similarities are calculated between the filtered image block and the current image block, the front reference block, and the back reference block, respectively. By comparing the structural similarities between the filtered image block and the current image block with the maximum structural similarity, it is determined whether the current image block contains a dirty spot, wherein the maximum structural similarity is the maximum value of the structural similarities between the filtered image block and the front reference block and the back reference block, respectively.

[0123] Taking the image frame as the detection unit as an example, this embodiment can also repair the image frame containing dirty spots through the following steps:

[0124] Step 1) obtaining a binary image based on a comparison result of the similarity between the filtered image frame and the current image frame and a similarity threshold, wherein the binary image is used to represent a mask image of a dirty point in the current image frame;

[0125] In the implementation, the current image frame I is selected t and the filtered image frame I after median filtering mid , calculate the average structural similarity (MSSIM) between the two, and obtain the structural similarity matrix M of each pixel in the corresponding image frame. The calculation formula of the structural similarity matrix is ​​shown in formula (3a). The similarity parameter w represents the range of the local area that affects the structural similarity of the pixels at the same position in the two image frames when the structural similarity of the pixels at the same position in the two image frames is used to calculate the average structural similarity. That is, w is used to control the range of the local area that affects the structural similarity value of the current position during the calculation of the average structural similarity. The smaller w is, the smaller the local range is, and the structural difference is more reflected in the details. The larger w is, the larger the local range is, and the structural difference change is closer to the overall.

[0126] As shown in FIG7 , this embodiment provides a schematic diagram of the relationship between the similarity matrix and the change of w. For the same set of image frames / image blocks, as w changes, the range of the low-similarity region (the black area in the figure) in MSSIM becomes larger. According to experience, the value of w is approximately 0.2 times the preset window size.

[0127] In formula (3a), M(i, j) represents the structural similarity (matrix) of different image frames, and the similarity parameter w represents the range of the local area that affects the structural similarity of pixels at the same position in two image frames when the average structural similarity is calculated using the structural similarity of pixels at the same position in two image frames; x represents one image frame, y represents another image frame, and (i, j) represents the coordinates of the pixel points in the image. w is an odd number greater than 1, represents a sub-block with a side length of w and a pixel x(i,j) as the center in the image frame x. It represents the sub-block with side length w and centered at pixel x(i,j) in image frame y. The value of M(i,j) is the structural similarity value of the corresponding sub-blocks x and y.

[0128] The structural similarity matrix M is equal to the width and height of the image frame. Each value in M ​​is the structural similarity value of the pixel at the corresponding coordinate of the image frame. The smaller the difference between the two pixels, the closer the similarity value is to 1, and the larger the difference between the two pixels, the closer the similarity value is to 0. The value is controlled by the similarity threshold v. Pixels greater than v in M ​​are reassigned to 0, and pixels less than v are reassigned to 1, thereby obtaining a binary image mask of the dirty point:

[0129] In formula (4a), M(i, j) represents the structural similarity matrix of different image frames, and v represents the similarity threshold. Usually, v is set between 0.85 and 0.95. As shown in FIG8 , this embodiment provides a schematic diagram of a binary image changing with the v value. The smaller the v value, the fewer scattered error points in the mask. However, if the v value is too small, the dirty point area will not be fully delineated. The larger the v value, the more complete the dirty point area, but more erroneous scattered points will be introduced.

[0130] In some embodiments, after obtaining the binary image, the following processing may be performed:

[0131] The binary image is subjected to at least one denoising process to obtain a processed binary image, which is used to repair dirty points in the current image frame based on the processed binary image and a reference frame corresponding to the current image frame; the denoising process includes corrosion and / or dilation processing.

[0132] During implementation, multiple erosion-dilation processes can be performed. The specific number of times the erosion-dilation processes are performed is related to the size of the dirty points. The method and number of denoising processes can be customized.

[0133] After obtaining the binary image, it is necessary to further perform two consecutive erosion-dilation processes on the binary image of the dirty points to remove the scattered white point noise on the binary image as much as possible, and obtain a relatively complete large connected area, that is, a more accurate dirty point area range. As shown in Figure 9, this embodiment provides a schematic diagram of the changes in the binary image during the erosion-dilation process.

[0134] Step 2) Repairing dirty points in the current image frame based on the binary image and a reference frame corresponding to the current image frame.

[0135] During implementation, when performing dirty point repair, any frame among the reference frames of the current image frame can be selected for repair, such as the previous reference frame or the next reference frame, to repair the dirty points in the current image frame. To ensure the repair effect, the reference frame corresponding to the maximum structural similarity with the filtered image frame can be selected as the reference frame for dirty point repair.

[0136] In practice, the restored image frame is expressed as follows:

[0137] In formula (5a), I′ t (i, j) represents the restored image frame, I t (i, j) represents the current image frame, I t-a (i, j) represents the reference frame corresponding to the maximum similarity, and mask (i, j) represents the binary image.

[0138] In some embodiments, during the process of calculating the restored image frame, matrix operations may be used to speed up the processing. The specific restoration steps are as follows:

[0139] Step 2a) performing inversion processing on the binary image to obtain an inverted image;

[0140] In practice, the inverse image mask′ of the binary image mask is obtained by the following formula: mask′(i,j)=1-mask(i,j) Formula (6a);

[0141] In formula (6a), mask(i,j) represents a binary image, mask′(i,j) represents an inverted image, and (i,j) represents the coordinates of the pixel.

[0142] Step 2b) determining a first image matrix based on the product of the reference frame corresponding to the current image frame and the binary image, and determining a second image matrix based on the product of the current image frame and the inverted image;

[0143] Optionally, the reference frame corresponding to the current image frame includes a reference frame corresponding to a maximum similarity among similarities between the filtered image frame and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

[0144] Step 2c) Determine an image frame after repairing the dirty points in the current image frame according to the sum of the first image matrix and the second image matrix.

[0145] In practice, the restored image frame is obtained by the following formula: I′ t (i,j)=I t-a (i,j)*mask(i,j)+I t (i,j)*mask′(i,j) Formula (7a);

[0146] In formula (7a), I′ t (i, j) represents the restored image frame, I t (i, j) represents the current image frame, I t-a (i, j) represents the reference frame corresponding to the maximum similarity, mask (i, j) represents the binary image, and mask ′ (i, j) represents the inverted image.

[0147] As shown in Figure 10, this embodiment also provides a flow chart of dirty point repair. When detecting with the image frame as the basic unit, the average structural similarity between the filtered image frame and the current image frame is first calculated, and a binary image is obtained by comparing with the similarity threshold; then, the binary image is eroded and expanded, and the inverse image of the processed binary image is calculated. Finally, a first image matrix is ​​determined based on the product of the reference frame corresponding to the current image frame and the binary image, and a second image matrix is ​​determined based on the product of the current image frame and the inverse image; and based on the sum of the first image matrix and the second image matrix, the image frame after the dirty points in the current image frame are repaired is determined. When performing detection using image blocks as the basic unit, the average structural similarity between the filtered image block and the current image block is first calculated, and a binary image block is obtained by comparing it with a similarity threshold. Then, after performing erosion-dilation processing on the binary image, the inverse image of the processed binary image block is calculated. Finally, a first image matrix is ​​determined based on the product of the reference block corresponding to the current image block and the binary image block, and a second image matrix is ​​determined based on the product of the current image block and the inverse image of the binary image block. Based on the sum of the first image matrix and the second image matrix, an image frame / image block after repairing the dirty points in the current image block is determined.

[0148] In some embodiments, taking an image block as a detection unit, if it is determined that a dirty point exists in the current image block, this embodiment may further perform the following steps to repair the image block containing the dirty point:

[0149] Step 1) obtaining a binary image block based on a comparison result of the similarity between the filtered image block and the current image block and a similarity threshold, wherein the binary image block is used to represent a mask image of a dirty point in the current image block;

[0150] In the implementation, the current image block P is selected t and the filtered image frame P after median filteringmid , calculate the average structural similarity (MSSIM) between the two, and obtain the structural similarity matrix M of each pixel in the corresponding image block. The calculation formula of the structural similarity matrix is ​​shown in formula (3b). The similarity parameter w represents the range of the local area that affects the structural similarity of the pixels at the same position in the two image blocks when the structural similarity of the pixels at the same position in the two image blocks is used to calculate the average structural similarity. That is, w is used to control the range of the local area that affects the structural similarity value of the current position during the calculation of the average structural similarity. The smaller w is, the smaller the local range is, and the structural difference is more reflected in the details. The larger w is, the larger the local range is, and the structural difference change is closer to the overall.

[0151] In formula (3a), M(i, j) represents the structural similarity (matrix) of different image blocks, and the similarity parameter w represents the range of the local area that affects the structural similarity of pixels at the same position in two image blocks when the average structural similarity is calculated using the structural similarity of pixels at the same position in two image blocks; x represents one image block, y represents another image block, and (i, j) represents the coordinates of the pixel points in the image block. w is an odd number greater than 1, represents the sub-block with pixel x(i,j) as the center and side length w in the image block x, It represents the sub-block with side length w and centered at pixel x(i,j) in image block y. The value of M(i,j) is the structural similarity value of the corresponding sub-blocks x and y.

[0152] The structural similarity matrix M is equal to the width and height of the image block. Each value in M ​​is the structural similarity value of the pixel at the corresponding coordinate of the image block. The smaller the difference between the two pixels, the closer the similarity value is to 1, and the larger the difference between the two pixels, the closer the similarity value is to 0. The value is controlled by the similarity threshold v. Pixels in M ​​greater than v are reassigned to 0, and pixels less than v are reassigned to 1, thereby obtaining a binary image mask of the dirty point:

[0153] In formula (4b), M(i, j) represents the structural similarity matrix of different image blocks, and v represents the similarity threshold, which is usually set between 0.85 and 0.95. The smaller the v value, the fewer scattered error points in the mask, but too small a value will result in incomplete delineation of the dirty point area. The larger the v value, the more complete the dirty point area, but more erroneous scattered points will be introduced.

[0154] In some embodiments, after obtaining the binary image, the following processing may be performed:

[0155] The binary image is subjected to at least one denoising process to obtain a processed binary image, which is used to repair dirty points in the current image block based on the processed binary image and a reference frame corresponding to the current image block; the denoising process includes erosion and / or dilation processing.

[0156] During implementation, multiple erosion-dilation processes can be performed. The specific number of times the erosion-dilation processes are performed is related to the size of the dirty points. The method and number of denoising processes can be customized.

[0157] After obtaining the binary image, it is necessary to further perform two consecutive erosion-dilation processes on the binary image of the dirty points to remove the scattered white point noise on the binary image as much as possible and obtain a relatively complete large connected area, which is a more accurate dirty point area range.

[0158] Step 2) repairing dirty points in the current image block based on the binary image block and a reference block corresponding to the current image block to obtain a repaired image block, wherein the reference block is determined based on an image block in a reference frame corresponding to the current image frame that has the same position as the current image block;

[0159] During implementation, when performing dirty point repair, any reference block of the current image block can be selected for repair, such as a previous reference block or a subsequent reference block, to repair the dirty points in the current image block. To ensure the repair effect, the reference block corresponding to the maximum structural similarity of the filtered image block can be selected as the reference block for dirty point repair.

[0160] In practice, the restored image block is expressed as follows:

[0161] In formula (5b), P′ t (i, j) represents the restored image block, P t (i, j) represents the current image block, P t-a (i, j) represents the reference block corresponding to the maximum similarity, and mask(i, j) represents the binary image.

[0162] In some embodiments, during the calculation of the restored image block, matrix operations may be used to speed up the processing. The specific restoration steps are as follows:

[0163] Step 2a) performing inversion processing on the binary image block to obtain an inverted image block;

[0164] In practice, the inverse image mask′ of the binary image mask is obtained by the following formula: mask′(i,j)=1-mask(i,j) Formula (6b);

[0165] In formula (6b), mask(i,j) represents a binary image block, mask′(i,j) represents an inverted image block, and (i,j) represents the coordinates of the pixel.

[0166] Step 2b) determining a first image block matrix based on the product of a reference block corresponding to the current image block and the binary image block, and determining a second image block matrix based on the product of the current image block and the inverted image block;

[0167] Optionally, the reference block corresponding to the current image block includes a reference block corresponding to a maximum similarity among similarities between the filtered image block and a previous reference block and a subsequent reference block corresponding to the current image block.

[0168] Step 2c) Determine an image block after repairing the dirty points in the current image block according to the sum of the first image block matrix and the second image block matrix.

[0169] In practice, the restored image block is obtained by the following formula: P′ t (i,j)=P t-a (i,j)*mask(i,j)+P t (i,j)*mask′(i,j) Formula (7b);

[0170] In formula (7b), P′ t (i, j) represents the restored image block, P t (i, j) represents the current image block, P t-a (i, j) represents the reference block corresponding to the maximum similarity, mask(i, j) represents the binary image block, and mask′(i, j) represents the inverted image block.

[0171] Step 3) Replace the current image block with the repaired image block.

[0172] During implementation, the repaired image blocks can be reattached to the current image blocks at their corresponding positions. Repeat the above steps of detecting and repairing dirty spots on each image block until all dirty spots are repaired. Finally, extract the next frame in the image sequence and repeat the above steps until the entire video is completely repaired.

[0173] As shown in FIG11 , this embodiment provides a process for detecting and repairing dirty spots in a video. The specific implementation steps of the process are as follows:

[0174] Step 1100: Obtain a previous reference frame and a subsequent reference frame corresponding to a current image frame in a video;

[0175] Step 1101: Cut the current image frame, the previous reference frame, and the next reference frame into a plurality of current image blocks, a plurality of previous reference blocks, and a plurality of next reference blocks respectively;

[0176] Step 1102: For each current image block, perform median filtering on the current image block, the previous reference block, and the next reference block to obtain a filtered image block.

[0177] Step 1103: Determine the maximum similarity from the similarities between the filtered image block and the previous reference block and the next reference block;

[0178] Step 1104: Determine whether there is a dirty point in the current image block based on the comparison result of the maximum similarity, the similarity between the filtered image block and the current image block;

[0179] Step 1105: If there are dirty points in the current image block, a binary image block is obtained based on a comparison result of the similarity between the filtered image block and the current image block and a similarity threshold;

[0180] Step 1106: Repair dirty points in the current image block based on the binary image block and the reference block corresponding to the current image block to obtain a repaired image block.

[0181] Step 1107: Replace the current image block with the repaired image block.

[0182] As shown in FIG12 , this embodiment also provides a flow chart of a detection and restoration algorithm. Assume that the resolution of the video to be processed is H×W, where H is the number of rows of pixels and W is the number of columns of pixels. Assume that the image frame at the current moment is I t After decoding a video into a continuous sequence of image frames, for each image frame at time node t, the required reference frame is extracted based on the hyperparameters "interval n" and "interval m." The extracted current image frame and reference frame are then divided into blocks according to the hyperparameters "preset window size" and "preset step size." The algorithm processes each group of image blocks in sequence, where each group of image blocks includes the image block of the current image frame and the reference block at the same position as the image block in the reference frame. First, defect localization is performed on the current group of image blocks. This involves calculating and comparing structural similarities based on preset criteria to determine whether the current image block contains dirty points, thus achieving automatic detection of dirty points. A similarity matrix M is then calculated, and the hyperparameter "similarity threshold" is used to determine the precise location of the dirty points. Finally, pixel values ​​at the same position in the reference frame are extracted to fill in the content obscured by the dirty points.

[0183] During implementation, a three-frame image sequence with an interval of n can be extracted from the video, and the sequence is divided into blocks according to the parameters of a preset window size k and a preset step size s. For each group of three image blocks, the corresponding pixels are subjected to median filtering. The structural similarity of the filtered image block is calculated with the three image blocks in turn. The formula S0 < max(S1, S2) is used to determine whether there are dirty spots in the image blocks of the current group. For the image block group with dirty spots, the average structural similarity between the current image block and the filtered image block is calculated. According to the similarity threshold v, the specific area of ​​the dirty spots in the picture is obtained, and the dirty spots are filled and repaired from the adjacent image blocks with higher structural similarity, thereby realizing the detection and repair of dirty spots in the video picture.

[0184] Based on the same inventive concept, an embodiment of the present invention also provides a device for video dirty point detection. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0185] As shown in FIG13 , the device includes a processor 1300 and a memory 1301 . The memory 1301 is used to store a program executable by the processor 1300 . The processor 1300 is used to read the program in the memory 1301 and perform the following steps:

[0186] In response to a user input operation on at least one of a time parameter, a similarity parameter, a similarity threshold, a preset window size, and a preset step size, based on any of the above-mentioned video dirty point detection methods, dirty points in image frames in the video are detected according to the parameters corresponding to the input operation;

[0187] The time parameter represents the number of frames between the current image frame and the reference frame of the current image frame; the similarity parameter represents the range of the local area that affects the structural similarity of pixels at the same position in the two image frames when the average structural similarity is calculated using the structural similarity of pixels at the same position in the two image frames.

[0188] In some embodiments, the time parameters include interval n and interval m, where n represents the number of frames between the previous reference frame and the current image frame, m represents the number of frames between the subsequent reference frame and the current image frame, n is an integer greater than or equal to 0, and m is an integer greater than or equal to 0.

[0189] In some embodiments, the similarity parameter includes w, which represents the range of the local area that affects the structural similarity of pixels at the same position in two image frames when calculating the average structural similarity.

[0190] In some embodiments, the similarity threshold comprises v; for obtaining a binary image block based on a comparison result of the similarity between the filtered image block and the current image block and the similarity threshold;

[0191] In some embodiments, a preset window size and a preset step size are used to traverse the current image frame, the previous reference frame and the subsequent reference frame respectively using the preset window at the preset step size, and cut out multiple current image blocks, multiple previous reference blocks and multiple subsequent reference blocks of the same size as the preset window; wherein the preset step size includes a horizontal step size in the horizontal direction and a vertical step size in the vertical direction, the horizontal step size is less than or equal to the width of the preset window, and the vertical step size is less than or equal to the height of the preset window.

[0192] In implementation, the above-mentioned method for detecting dirty spots in video can be designed as a video processing plug-in and applied to the device. It can be used as a toolkit in video editing and processing software such as Da Vinci editing software and Adobe Premiere, and used when editing video files. The plug-in tool can expose multiple parameters such as interval n, interval m, preset window size, preset step size, similarity parameter, similarity threshold, etc. for users to freely set to match video files with different resolutions, different qualities, and different scenes.

[0193] The video dirty point detection method provided in this embodiment can be flexibly adjusted, and the intermediate process is controllable, making it suitable for solving dirty point videos with different problems. It does not rely on deep learning application frameworks and can be flexibly deployed and used on different platforms. It does not rely on AI computing resources such as GPUs and NPUs, and has fewer restrictions on hardware devices. In addition, the method of local detection of dirty point areas based on the structural similarity of adjacent image frames improves the accuracy of detection and reduces the probability of introducing artifacts compared to global detection. By calculating the average structural similarity, the outline and shape of the dirty point are further accurately located, and the area is repaired and filled with reference to the pixels of adjacent frames. The video dirty point detection method provided in this application can also be combined with the deep learning method described in the background technology to provide more hyperparameters for the deep learning method, thereby expanding the applicability of the deep learning solution.

[0194] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device. Since the electronic device is the electronic device in the method in the embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0195] As shown in FIG14 , the electronic device includes a processor 1400 and a memory 1401 . The memory 1401 is used to store programs executable by the processor 1400 . The processor 1400 is used to read the programs in the memory 1401 and perform the following steps:

[0196] Obtain a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a previous reference frame and a subsequent reference frame, wherein the previous reference frame is an image frame before the current image frame in the video, and the subsequent reference frame is an image frame after the current image frame in the video;

[0197] Performing filtering on the current image frame, a previous reference frame, and a subsequent reference frame corresponding to the current image frame to obtain a filtered image frame;

[0198] It is determined whether there is a dirty point in the current image frame according to the similarity between the filtered image frame and the current image frame, and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

[0199] As an optional implementation,

[0200] The previous reference frame and the current image frame are separated by n image frames, where n is determined based on the number of frames in which dirty points appear continuously in consecutive image frames, and n is an integer greater than or equal to 0; and / or, the subsequent reference frame and the current image frame are separated by m image frames, where m is determined based on the number of frames in which dirty points appear continuously in consecutive image frames, and m is an integer greater than or equal to 0.

[0201] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0202] Determining a maximum similarity from the similarities between the filtered image frame and the previous reference frame and the next reference frame respectively;

[0203] It is determined whether there are dirty points in the current image frame according to a comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame.

[0204] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0205] If the similarity between the filtered image frame and the current image frame is less than the maximum similarity, it is determined that there are dirty points in the current image frame; or,

[0206] If the similarity between the filtered image frame and the current image frame is greater than or equal to the maximum similarity, it is determined that no dirty points exist in the current image frame.

[0207] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0208] If the previous reference frame is a plurality of consecutive image frames preceding the current image frame in the video, then determining the maximum similarity from the similarities between the filtered image frame and each of the previous reference frames and the subsequent reference frame; or

[0209] If the subsequent reference frame is a plurality of consecutive image frames following the current image frame in the video, determining the maximum similarity from the similarities between the filtered image frame and the previous reference frame and each subsequent reference frame; or

[0210] If the previous reference frame is a plurality of consecutive image frames before the current image frame in the video, and the subsequent reference frame is a plurality of consecutive image frames after the current image frame in the video, then the maximum similarity is determined from the similarities between the filtered image frame and each previous reference frame and each subsequent reference frame respectively.

[0211] As an optional implementation manner, if it is determined that there are dirty points in the current image frame, the processor 1400 is further configured to execute:

[0212] Obtaining a binary image based on a comparison result of a similarity between the filtered image frame and the current image frame and a similarity threshold, wherein the binary image is used to represent a mask image of a dirty point in the current image frame;

[0213] Repairing dirty points in the current image frame according to the binary image and a reference frame corresponding to the current image frame.

[0214] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0215] Performing inversion processing on the binary image to obtain an inverted image;

[0216] Determining a first image matrix according to a product of a reference frame corresponding to the current image frame and the binary image, and determining a second image matrix according to a product of the current image frame and the inverted image;

[0217] An image frame after repairing the dirty points in the current image frame is determined according to the sum of the first image matrix and the second image matrix.

[0218] As an optional implementation manner, the reference frame corresponding to the current image frame includes a reference frame corresponding to the maximum similarity between the filtered image frame and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

[0219] As an optional implementation manner, after obtaining the binary image based on the comparison result of the similarity between the filtered image frame and the current image frame and the similarity threshold, the processor 1400 is further configured to execute:

[0220] The binary image is subjected to at least one denoising process to obtain a processed binary image, which is used to repair dirty points in the current image frame based on the processed binary image and a reference frame corresponding to the current image frame; the denoising process includes corrosion and / or dilation processing.

[0221] As an optional implementation manner, before filtering the current image frame and the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain the filtered image frame, the processor 1400 is further configured to execute:

[0222] Cutting the current image frame, the previous reference frame and the next reference frame into a plurality of current image blocks, a plurality of previous reference blocks and a plurality of next reference blocks respectively;

[0223] The processor 1400 is specifically configured to execute:

[0224] For each current image block, use a filtering algorithm to filter the current image block, the previous reference block, and the next reference block corresponding to the current image block to obtain a filtered image block;

[0225] It is determined whether there is a dirty point in the current image block according to the similarities between the filtered image block and the current image block, the previous reference block and the next reference block respectively.

[0226] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0227] Determining a maximum similarity from the similarities between the filtered image block and the previous reference block and the next reference block respectively;

[0228] It is determined whether there is a dirty point in the current image block according to a comparison result of the maximum similarity and the similarity between the filtered image block and the current image block.

[0229] As an optional implementation,

[0230] Partial areas of the two current image blocks that are closest to each other in a preset direction overlap, partial areas of the two previous reference blocks that are closest to each other in a preset direction overlap, and partial areas of the two subsequent reference blocks that are closest to each other in a preset direction overlap; the preset direction includes a horizontal axis direction or a vertical axis direction.

[0231] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0232] traversing the current image frame, the previous reference frame, and the next reference frame respectively according to a preset step size using a preset window, and cutting out a plurality of current image blocks, a plurality of previous reference blocks, and a plurality of next reference blocks having the same size as the preset window;

[0233] The preset step size includes a horizontal step size in the horizontal direction and a vertical step size in the vertical direction, the horizontal step size is less than or equal to the width of the preset window, and the vertical step size is less than or equal to the height of the preset window.

[0234] As an optional implementation manner, if it is determined that there are dirty points in the current image block, the processor 1400 is further configured to execute:

[0235] Obtaining a binary image block according to a comparison result of a similarity between the filtered image block and the current image block and a similarity threshold, wherein the binary image block is used to represent a mask image of a dirty point in the current image block;

[0236] Repairing dirty points in the current image block according to the binary image block and a reference block corresponding to the current image block to obtain a repaired image block, wherein the reference block is determined based on an image block in a reference frame corresponding to the current image frame and having the same position as the current image block;

[0237] The current image block is replaced by the restored image block.

[0238] As an optional implementation manner, the processor 1400 is specifically configured to execute:

[0239] Performing filtering processing on the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame using a filtering algorithm to obtain a filtered image frame;

[0240] The filtering algorithm includes a temporal median filtering algorithm, which is used to obtain the median of pixel values ​​at the same position in multiple frames of images; or

[0241] The filtering algorithm includes a time majority filtering algorithm, which is used to obtain the largest number of identical pixel values ​​among pixel values ​​at the same position in multiple frames of images.

[0242] As an optional implementation, the similarity is used to characterize the correlation between pixels of different image frames, or the correlation between pixels of image blocks at the same position in different image frames; the similarity includes structural similarity and / or average structural similarity.

[0243] Based on the same inventive concept, an embodiment of the present invention also provides a device for detecting video dirty spots. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0244] As shown in FIG15 , the device includes:

[0245] An acquisition unit 1500 is configured to acquire a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a previous reference frame and a subsequent reference frame. The previous reference frame is an image frame before the current image frame in the video, and the subsequent reference frame is an image frame after the current image frame in the video.

[0246] The filtering unit 1501 is configured to perform filtering processing on the current image frame, the previous reference frame, and the next reference frame corresponding to the current image frame to obtain a filtered image frame;

[0247] The judgment unit 1502 is configured to determine whether there are dirty points in the current image frame according to the similarities between the filtered image frame and the current image frame, and the previous reference frame and the next reference frame corresponding to the current image frame.

[0248] As an optional implementation,

[0249] The previous reference frame and the current image frame are separated by n image frames, where n is determined based on the number of frames in which dirty points appear continuously in consecutive image frames, and n is an integer greater than or equal to 0; and / or, the subsequent reference frame and the current image frame are separated by m image frames, where m is determined based on the number of frames in which dirty points appear continuously in consecutive image frames, and m is an integer greater than or equal to 0.

[0250] As an optional implementation manner, the determining unit 1502 is specifically configured to:

[0251] Determining a maximum similarity from the similarities between the filtered image frame and the previous reference frame and the next reference frame respectively;

[0252] It is determined whether there are dirty points in the current image frame according to a comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame.

[0253] As an optional implementation manner, the determining unit 1502 is specifically configured to:

[0254] If the similarity between the filtered image frame and the current image frame is less than the maximum similarity, it is determined that there are dirty points in the current image frame; or,

[0255] If the similarity between the filtered image frame and the current image frame is greater than or equal to the maximum similarity, it is determined that no dirty points exist in the current image frame.

[0256] As an optional implementation,

[0257] If the previous reference frame is a plurality of consecutive image frames preceding the current image frame in the video, then determining the maximum similarity from the similarities between the filtered image frame and each of the previous reference frames and the subsequent reference frame; or

[0258] If the subsequent reference frame is a plurality of consecutive image frames following the current image frame in the video, determining the maximum similarity from the similarities between the filtered image frame and the previous reference frame and each subsequent reference frame; or

[0259] If the previous reference frame is a plurality of consecutive image frames before the current image frame in the video, and the subsequent reference frame is a plurality of consecutive image frames after the current image frame in the video, then the maximum similarity is determined from the similarities between the filtered image frame and each previous reference frame and each subsequent reference frame respectively.

[0260] As an optional implementation manner, if it is determined that there are dirty points in the current image frame, a repair unit is further included to specifically:

[0261] Obtaining a binary image based on a comparison result of a similarity between the filtered image frame and the current image frame and a similarity threshold, wherein the binary image is used to represent a mask image of a dirty point in the current image frame;

[0262] Repairing dirty points in the current image frame according to the binary image and a reference frame corresponding to the current image frame.

[0263] As an optional implementation manner, the repair unit is specifically used to:

[0264] Performing inversion processing on the binary image to obtain an inverted image;

[0265] Determining a first image matrix according to a product of a reference frame corresponding to the current image frame and the binary image, and determining a second image matrix according to a product of the current image frame and the inverted image;

[0266] An image frame after repairing the dirty points in the current image frame is determined according to the sum of the first image matrix and the second image matrix.

[0267] As an optional implementation manner, the reference frame corresponding to the current image frame includes a reference frame corresponding to the maximum similarity between the filtered image frame and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

[0268] As an optional implementation manner, after obtaining the binary image based on the comparison result of the similarity between the filtered image frame and the current image frame and the similarity threshold, the restoration unit is further configured to:

[0269] The binary image is subjected to at least one denoising process to obtain a processed binary image, which is used to repair dirty points in the current image frame based on the processed binary image and a reference frame corresponding to the current image frame; the denoising process includes corrosion and / or dilation processing.

[0270] As an optional implementation manner, before filtering the current image frame and the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain the filtered image frame, a blocking unit is further included to:

[0271] Cutting the current image frame, the previous reference frame and the next reference frame into a plurality of current image blocks, a plurality of previous reference blocks and a plurality of next reference blocks respectively;

[0272] For each current image block, use a filtering algorithm to filter the current image block, the previous reference block, and the next reference block corresponding to the current image block to obtain a filtered image block;

[0273] It is determined whether there is a dirty point in the current image block according to the similarities between the filtered image block and the current image block, the previous reference block and the next reference block respectively.

[0274] As an optional implementation manner, the determining unit 1502 is specifically configured to:

[0275] Determining a maximum similarity from the similarities between the filtered image block and the previous reference block and the next reference block respectively;

[0276] It is determined whether there is a dirty point in the current image block according to a comparison result of the maximum similarity and the similarity between the filtered image block and the current image block.

[0277] As an optional implementation,

[0278] Partial areas of the two current image blocks that are closest to each other in a preset direction overlap, partial areas of the two previous reference blocks that are closest to each other in a preset direction overlap, and partial areas of the two subsequent reference blocks that are closest to each other in a preset direction overlap; the preset direction includes a horizontal axis direction or a vertical axis direction.

[0279] As an optional implementation manner, the block division unit is specifically configured to:

[0280] traversing the current image frame, the previous reference frame, and the next reference frame respectively according to a preset step size using a preset window, and cutting out a plurality of current image blocks, a plurality of previous reference blocks, and a plurality of next reference blocks having the same size as the preset window;

[0281] The preset step size includes a horizontal step size in the horizontal direction and a vertical step size in the vertical direction, the horizontal step size is less than or equal to the width of the preset window, and the vertical step size is less than or equal to the height of the preset window.

[0282] As an optional implementation manner, if it is determined that there are dirty points in the current image block, the repair unit is further configured to:

[0283] Obtaining a binary image block according to a comparison result of a similarity between the filtered image block and the current image block and a similarity threshold, wherein the binary image block is used to represent a mask image of a dirty point in the current image block;

[0284] Repairing dirty points in the current image block according to the binary image block and a reference block corresponding to the current image block to obtain a repaired image block, wherein the reference block is determined based on an image block in a reference frame corresponding to the current image frame and having the same position as the current image block;

[0285] The current image block is replaced by the restored image block.

[0286] As an optional implementation manner, the filtering unit 1501 is specifically configured to:

[0287] Performing filtering processing on the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame using a filtering algorithm to obtain a filtered image frame;

[0288] The filtering algorithm includes a temporal median filtering algorithm, which is used to obtain the median of pixel values ​​at the same position in multiple frames of images; or

[0289] The filtering algorithm includes a time majority filtering algorithm, which is used to obtain the largest number of identical pixel values ​​among pixel values ​​at the same position in multiple frames of images.

[0290] As an optional implementation, the similarity is used to characterize the correlation between pixels of different image frames, or the correlation between pixels of image blocks at the same position in different image frames; the similarity includes structural similarity and / or average structural similarity.

[0291] Based on the same inventive concept, embodiments of the present disclosure provide a computer storage medium comprising computer program code. When executed on a computer, the computer program code causes the computer to perform any of the methods for detecting dirty video points discussed above. Because the principles underlying the problems solved by the computer storage medium are similar to those of the methods for detecting dirty video points, the implementation of the computer storage medium can be referenced to the implementation of the methods, and any repetitions will be omitted.

[0292] In a specific implementation process, computer storage media may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0293] Based on the same inventive concept, embodiments of the present disclosure further provide a computer program product comprising computer program code that, when executed on a computer, causes the computer to perform any of the methods for detecting dirty video points discussed above. Because the principles underlying the problems solved by the aforementioned computer program products are similar to those of the methods for detecting dirty video points, the implementation of the aforementioned computer program products can be referenced to the implementation of the methods, and any repetitive details will not be repeated.

[0294] The computer program product can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0295] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0296] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0297] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0298] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0299] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for detecting dirty spots in a video, wherein: The method includes: Obtain a reference frame corresponding to a current image frame in a video, wherein the reference frame includes a front reference frame and a back reference frame, wherein the front reference frame is an image frame before the current image frame in the video, and the back reference frame is an image frame after the current image frame in the video; Performing filtering processing on the current image frame, a previous reference frame and a subsequent reference frame corresponding to the current image frame to obtain a filtered image frame; It is determined whether there are dirty points in the current image frame according to the similarities between the filtered image frame and the current image frame, and the previous reference frame and the subsequent reference frame corresponding to the current image frame.

2. The method according to claim 1, wherein: The previous reference frame and the current image frame are separated by n image frames, where n is determined based on the number of frames in which dirty points appear continuously in consecutive image frames, and n is an integer greater than or equal to 0; and / or, the subsequent reference frame and the current image frame are separated by m image frames, where m is determined based on the number of frames in which dirty points appear continuously in consecutive image frames, and m is an integer greater than or equal to 0.

3. The method according to claim 1, wherein: The determining whether there is a dirty point in the current image frame according to the similarity between the filtered image frame and the current image frame, and the previous reference frame and the subsequent reference frame corresponding to the current image frame, respectively, comprises: Determining a maximum similarity from the similarities between the filtered image frame and the previous reference frame and the next reference frame respectively; It is determined whether there are dirty points in the current image frame according to a comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame.

4. The method according to claim 3, wherein: The determining whether there is a dirty point in the current image frame according to the comparison result of the maximum similarity and the similarity between the filtered image frame and the current image frame comprises: If the similarity between the filtered image frame and the current image frame is less than the maximum similarity, it is determined that there are dirty points in the current image frame; or, If the similarity between the filtered image frame and the current image frame is greater than or equal to the maximum similarity, it is determined that there is no dirty point in the current image frame.

5. The method according to claim 3, wherein: If the previous reference frame is a plurality of consecutive image frames before the current image frame in the video, determining the maximum similarity from the similarities between the filtered image frame and each previous reference frame and the subsequent reference frame respectively; or, If the rear reference frame is a plurality of consecutive image frames after the current image frame in the video, determining a maximum similarity from the similarities between the filtered image frame and the front reference frame and each rear reference frame respectively; or, If the previous reference frame is a plurality of consecutive image frames before the current image frame in the video, and the subsequent reference frame is a plurality of consecutive image frames after the current image frame in the video, then the maximum similarity is determined from the similarities between the filtered image frame and each previous reference frame and each subsequent reference frame.

6. The method according to any one of claims 1 to 5, wherein: If it is determined that there are dirty points in the current image frame, the method further includes: According to the comparison result of the similarity between the filtered image frame and the current image frame and the similarity threshold, a binary image is obtained, wherein the binary image is used to represent the mask image of the dirty points in the current image frame; The dirty points in the current image frame are repaired according to the binary image and a reference frame corresponding to the current image frame.

7. The method according to claim 6, wherein: The repairing of dirty points in the current image frame according to the binary image and a reference frame corresponding to the current image frame includes: Performing inversion processing on the binary image to obtain an inverted image; Determine a first image matrix according to the product of a reference frame corresponding to the current image frame and the binary image, and determine a second image matrix according to the product of the current image frame and the inverted image; An image frame after the dirty points in the current image frame are repaired is determined according to the sum of the first image matrix and the second image matrix.

8. The method according to claim 6, wherein: The reference frame corresponding to the current image frame includes a reference frame corresponding to the maximum similarity among similarities between the filtered image frame and a previous reference frame and a subsequent reference frame corresponding to the current image frame.

9. The method according to claim 6, wherein: After obtaining the binary image according to the comparison result of the similarity between the filtered image frame and the current image frame and the similarity threshold, the method further includes: The binary image is subjected to at least one denoising process to obtain a processed binary image, which is used to repair dirty spots in the current image frame based on the processed binary image and a reference frame corresponding to the current image frame; the denoising process includes corrosion and / or dilation processing.

10. The method according to any one of claims 1 to 5, wherein: Before filtering the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain a filtered image frame, the method further includes: Cutting the current image frame, the front reference frame and the back reference frame into a plurality of current image blocks, a plurality of front reference blocks and a plurality of back reference blocks respectively; The filtering process is performed on the current image frame and the previous reference frame and the next reference frame corresponding to the current image frame to obtain a filtered image frame; and according to the similarity between the filtered image frame and the current image frame and the previous reference frame and the next reference frame corresponding to the current image frame, respectively, determining whether there is a dirty point in the current image frame, including: For each current image block, use a filtering algorithm to filter the current image block, a front reference block and a back reference block corresponding to the current image block to obtain a filtered image block; It is determined whether there is a dirty point in the current image block according to the similarities between the filtered image block and the current image block, the previous reference block and the next reference block respectively.

11. The method according to claim 10, wherein: The determining whether there is a dirty point in the current image block according to the similarities between the filtered image block and the current image block, the previous reference block and the next reference block respectively comprises: Determining a maximum similarity from the similarities between the filtered image block and the previous reference block and the next reference block respectively; It is determined whether there is a dirty point in the current image block according to a comparison result of the maximum similarity and the similarity between the filtered image block and the current image block.

12. The method according to claim 10, wherein: Partial areas of two current image blocks that are closest in position in a preset direction overlap, partial areas of two previous reference blocks that are closest in position in a preset direction overlap, and partial areas of two subsequent reference blocks that are closest in position in a preset direction overlap; The preset direction includes a horizontal axis direction or a vertical axis direction.

13. The method according to claim 10, wherein: The step of cutting the current image frame, the front reference frame and the back reference frame into a plurality of current image blocks, a plurality of front reference blocks and a plurality of back reference blocks respectively comprises: Using a preset window, traversing the current image frame, the previous reference frame and the next reference frame respectively according to a preset step size, cutting out a plurality of current image blocks, a plurality of previous reference blocks and a plurality of next reference blocks having the same size as the preset window; The preset step size includes a horizontal step size in the horizontal direction and a vertical step size in the vertical direction, the horizontal step size is less than or equal to the width of the preset window, and the vertical step size is less than or equal to the height of the preset window.

14. The method according to claim 10, wherein: If it is determined that there are dirty points in the current image block, the method further includes: According to the comparison result of the similarity between the filtered image block and the current image block and the similarity threshold, a binary image block is obtained, wherein the binary image block is used to represent the mask image of the dirty point in the current image block; Repairing dirty points in the current image block according to the binary image block and a reference block corresponding to the current image block to obtain a repaired image block, wherein the reference block is determined according to an image block in a reference frame corresponding to the current image frame and having the same position as the current image block; The current image block is replaced by the repaired image block.

15. The method according to claim 1, wherein: The filtering process is performed on the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain a filtered image frame, including: Using a filtering algorithm to filter the current image frame, the previous reference frame and the subsequent reference frame corresponding to the current image frame to obtain a filtered image frame; The filtering algorithm includes a temporal median filtering algorithm, and the temporal median filtering algorithm is used to obtain the median of pixel values ​​at the same position in multiple frames of images; or, The filtering algorithm comprises a time majority filtering algorithm, and the time majority filtering algorithm is used to obtain the largest number of identical pixel values ​​among the pixel values ​​at the same position in multiple frames of images.

16. The method according to any one of claims 1 to 5, 8 to 9, 11 to 14, wherein: The similarity is used to characterize the correlation between pixels of different image frames, or the correlation between pixels of image blocks at the same position in different image frames; The similarity includes structural similarity and / or average structural similarity.

17. A device for detecting dirty spots in a video, wherein: The device includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps: In response to a user input operation on at least one of a time parameter, a similarity parameter, a similarity threshold, a preset window size, and a preset step size, based on the method according to any one of claims 1 to 16, dirty points of an image frame in a video are detected according to the parameter corresponding to the input operation; The time parameter represents the number of frames between the current image frame and the reference frame of the current image frame; the similarity parameter represents the range of the local area that affects the structural similarity of pixels at the same position in the two image frames when the average structural similarity is calculated using the structural similarity of pixels at the same position in the two image frames.

18. An electronic device, wherein: The electronic device comprises a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods of claims 1 to 16.

19. A computer storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images