A motion detection method, device, storage medium and electronic device

Abstract

Embodiments of the present application provide a motion detection method, device, storage medium and electronic equipment. The motion detection method comprises: obtaining an i-th image pair according to a video sequence; dividing two images included in the i-th image pair into a plurality of image blocks respectively; obtaining target motion weights of all pixel points on each image block to obtain a motion mask of the i-th image pair, wherein the target motion weight of any pixel point is obtained by the following strategy: obtaining the target motion weight of the any pixel point according to the difference between the pixel value of the any pixel point and the characteristic value of the image block in which the any pixel point is located; repeating the above process to obtain motion masks corresponding to a plurality of image pairs, and fusing all motion masks to obtain a target motion mask. Some embodiments of the present application only calculate the motion mask by comparing the pixel value, and the calculation speed is relatively fast.

Description

Technical Field

[0001] This application relates to the field of motion detection based on image frames, and more specifically, embodiments of this application relate to a motion detection method, apparatus, storage medium, and electronic device. Background Technology

[0002] Motion detection algorithms are software algorithms that process two or more frames of images to detect moving objects or regions containing moving pixels. Currently, image motion detection algorithms can be broadly categorized into two types: traditional methods and deep learning methods.

[0003] Traditional methods distinguish moving and stationary regions in an image by finding features. Deep learning methods, on the other hand, utilize various deep learning techniques based on neural network models.

[0004] The above two types of methods have at least the following drawbacks: First, both traditional and deep learning methods are easily affected by noise. Second, for the same scene, the brightness of the same area in different frames under different exposures often varies significantly, which can greatly affect motion detection. This is because algorithms in traditional methods that simply rely on brightness differences or local statistical information can easily detect changes in brightness as motion. Deep learning methods face two challenges: First, obtaining labeled data is difficult, and motion labeling is time-consuming. Second, deep learning models are typically large and difficult to meet the time requirements of algorithms on mobile terminal platforms.

[0005] Therefore, how to improve the technical effectiveness of motion detection methods has become an urgent technical problem to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a motion detection method, apparatus, storage medium, and electronic device. Some embodiments of this application calculate the motion mask simply by comparing pixel values, which is relatively fast. Some embodiments of this application use a block matching method, making the method insensitive to brightness differences between different frames. Some embodiments of this application also comprehensively consider different block sizes, as well as median and center values, making the method more robust and effectively reducing noise interference.

[0007] In a first aspect, some embodiments of this application provide a motion detection method, the motion detection method comprising: obtaining an i-th image pair according to a video sequence, wherein the i-th image pair includes an i-th frame image and a reference image, the reference image including an (i+1)-th frame image adjacent to the i-th frame image or a fused image of several subsequent frames adjacent to the i-th frame image, where i is an integer greater than or equal to 1; dividing the two images included in the i-th image pair into multiple image blocks respectively; obtaining the target motion weights of all pixels in each image block to obtain a motion mask for the i-th image pair, wherein the target motion weight of any pixel is obtained by the following strategy: obtaining the target motion weight of any pixel based on the difference between the pixel value of any pixel and the feature value of the image block in which the pixel is located; repeating the above process to obtain motion masks corresponding to multiple image pairs, and fusing all motion masks to obtain a target motion mask.

[0008] In some embodiments of this application, the target weight coefficient is determined by comparing the difference between the pixel value of any pixel point in each image block and the feature value (e.g., center value and / or median value) of that image block in two or more adjacent frames (i.e., multiple frames of images are fused). Because the embodiments of this application calculate the motion mask by comparing the size of pixel values, the motion detection method can be directly executed on mobile terminals such as mobile phones.

[0009] In some embodiments, obtaining the target motion weights of all pixels on each image block includes: confirming that any pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels included in the image block where the pixel is located; obtaining the block weight coefficient of the image block where the pixel is located based on the total number of moving pixels and the size of the image block where the pixel is located; and obtaining the target weight coefficient of the pixel based on the block weight coefficient.

[0010] Some embodiments of this application can improve the accuracy of motion detection by obtaining block weight coefficients to obtain motion masks.

[0011] In some implementations, the characteristic value includes at least one of the median and the center value.

[0012] Some embodiments of this application improve the objectivity and accuracy of the obtained differences by selecting at least one of the center value and median value of each image block to compare with any pixel point.

[0013] In some embodiments of this application, dividing the two images of the i-th image pair into multiple image blocks includes: dividing the two images of the i-th image pair into the multiple image blocks according to a size, wherein any pixel is a first pixel, and the first pixel is assigned to the j-th image block, where j is an integer greater than or equal to 1; confirming that any pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels in the image block containing any pixel, includes: confirming that the first pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels in the j-th image block; obtaining the block weight coefficient of the image block containing any pixel based on the total number of moving pixels and the size of the image block containing any pixel, includes: calculating the ratio of the total number of moving pixels in the j-th image block to the size of the j-th image block to obtain the block weight coefficient; obtaining the target weight coefficient of any pixel based on the block weight coefficient includes: using the ratio as the target weight coefficient of the first pixel.

[0014] Some embodiments of this application divide adjacent two-frame video images or multiple adjacent video images (i.e., the second image is a fusion of multiple frames) by a certain size, and then obtain the motion mask by comparing the pixel values ​​within the block with the feature values ​​of the image block. This improves the speed of data processing (by comparing the pixel values ​​with the median to measure the difference between different frames) and effectively avoids the technical problem of poor accuracy when motion detection is performed solely based on the brightness difference between two frames.

[0015] In some embodiments, dividing the two images in the i-th image pair into multiple image blocks includes: dividing the two images in the i-th image pair into blocks according to N different sizes to obtain the multiple image blocks, where N is an integer greater than 1, any pixel is a second pixel, and the second pixel is divided into N target image blocks, each target image block being obtained by dividing the two images using a different size; confirming that any pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels included in the image block containing any pixel, includes: confirming that the second pixel belongs to a moving pixel based on the difference. The process involves: 1) determining the total number of moving pixels in each of the N types of target image blocks; 2) obtaining the block weight coefficient of the image block containing any pixel based on the total number of moving pixels and the size of the image block containing any pixel, which includes: solving for the block weight coefficients of various target image blocks based on the total number of moving pixels in each of the N types of target image blocks and the size of the N types of target image blocks, to obtain N block weight coefficients; 3) obtaining the target weight coefficient of any pixel based on the block weight coefficients, which includes: determining the target weight coefficient of the second pixel based on the N block weight coefficients and the size of each type of target image block in the N types of target image blocks.

[0016] Some embodiments of this application divide the first image and the second image into multiple scales (i.e., each scale corresponds to a group of multiple image blocks), which can avoid noise interference and fully consider the impact of the motion of objects of different sizes (image blocks of different sizes cover different ranges) on motion detection. This makes the motion detection method of this application more robust and can effectively reduce noise interference, thereby significantly improving the accuracy of motion weight coefficients on the motion mask.

[0017] In some embodiments, determining the target weight coefficient of the second pixel based on the N block weight coefficients and the size of each of the N target image blocks includes: weighting the N block weight coefficients to obtain the target motion weight of any pixel, wherein the larger the size of the image block, the larger the weight coefficient.

[0018] In some embodiments, if the two images are divided into blocks using three different sizes, Q, W, and E, then any pixel at coordinates r and c is divided into three target image blocks of sizes Q*Q, W*W, and E*E, corresponding to three block weight coefficients; wherein, the calculation formula for the target weight coefficient of any pixel is as follows:

[0019]

[0020] Among them, Weight Q，(r，c) Weight w，(r，c) and Weight E，(r，c) The three block weight coefficients represent any given pixel.

[0021] Some embodiments of this application provide a formula for calculating the target motion weight of a pixel based on three block weight coefficients, so that the target motion weight can be quantified and compared.

[0022] In some embodiments, confirming that any pixel belongs to a moving pixel based on the difference includes: on the i-th frame image, obtaining the feature value of any image block where the pixel is located to obtain the i-th feature value, and calculating the difference between the pixel value of the pixel and the i-th feature value to obtain the i-th difference, wherein the i-th feature value includes the i-th median and / or the i-th center value; on the reference image, obtaining the feature value of any image block to obtain the (i+1)-th feature value, and calculating the difference between the pixel value of the pixel and the (i+1)-th feature value to obtain the (i+1)-th difference, wherein the (i+1)-th feature value includes the (i+1)-th median and / or the (i+1)-th center value; when it is confirmed by the i-th difference and the (i+1)-th difference that the change trend of the pixel is inconsistent in the two images, then the pixel is confirmed to be a moving pixel.

[0023] Some embodiments of this application determine whether a pixel belongs to a moving pixel by determining the relationship between the differences of the same image block where any pixel is located in two images, and then determine the block weight coefficient of the same image block.

[0024] In some embodiments of this application, the difference is represented by a difference or ratio.

[0025] Some embodiments of this application provide differences or ratios to quantify the difference values, thereby improving the objectivity and comparability of the difference values.

[0026] In some embodiments of this application, the i-th difference is obtained by calculating the difference between the pixel value of any pixel and the i-th feature value; the (i+1)-th difference is obtained by calculating the difference between the pixel value of any pixel and the (i+1)-th feature value; wherein, the step of confirming that any pixel is a moving pixel when the change trend of any pixel in the two images is inconsistent through the i-th difference and the (i+1)-th difference includes: if the positive and negative characteristics of the i-th difference and the (i+1)-th difference are opposite, then the pixel is confirmed to be a moving pixel.

[0027] Some embodiments of this application determine whether a pixel can be counted as a moving pixel by calculating whether the sign of the difference between each pixel and the median and / or center value is the same in different images (i.e., the i-th image pair includes the i-th frame image and the reference image). This improves the estimation speed of the number of moving pixels in each image block and improves the accuracy of detection on the entire image.

[0028] In some embodiments of this application, the i-th difference is obtained by calculating the ratio of the pixel value of any pixel to the i-th feature value; the (i+1)-th difference is obtained by calculating the ratio of the pixel value of any pixel to the (i+1)-th feature value; wherein, the step of confirming that any pixel is a moving pixel when the change trend of any pixel in the two images is inconsistent through the i-th difference and the (i+1)-th difference includes: if the absolute value of the difference between the i-th ratio and the (i+1)-th ratio is less than a set threshold, then the pixel is confirmed as a moving pixel.

[0029] Some embodiments of this application quantify the difference by using a ratio, and then estimate whether the trend of change is consistent based on whether the difference between the ratios of the same image block in two images exceeds a set threshold, and determine whether the pixel can be counted as a moving pixel. This improves the estimation speed of the number of moving pixels in each image block and improves the accuracy of detection on the entire image.

[0030] In some embodiments, the block weight coefficient is determined by at least one of the block median weight coefficient and the block center value weight coefficient, wherein the block median weight coefficient is obtained by determining the difference between the pixel value of any pixel and the median of the image block in which the pixel is located, and the block center value weight coefficient is obtained by determining the difference between the pixel value of any pixel and the center value of the image block in which the pixel is located.

[0031] Some embodiments of this application determine the block weight coefficient of an image block based on at least one of the block median weight coefficient and the block center value weight coefficient, making the motion detection method of this application more robust and effectively reducing noise interference.

[0032] In some embodiments, the formula for calculating the block median weighting coefficient is as follows:

[0033] Weight Median，k =m*count1 / Q*Q

[0034] or,

[0035] The formula for calculating the weighting coefficient of the block center value is:

[0036] Weight Middle，k = m * count2 / Q * Q

[0037] Where k represents the k-th image block, count1 represents the total number of moving pixels on the k-th image block as determined by the median of the k-th image block, count2 represents the total number of moving pixels on the k-th image block as determined by the center value of the k-th image block, Q*Q represents the total number of pixels included in the k-th image block, m represents the normalization parameter, and the value of k is greater than or equal to 1 and less than or equal to the total number of multiple image blocks obtained by dividing the image according to size Q.

[0038] Some embodiments of this application use the above formula to obtain the block median weight coefficient and the block center value weight coefficient. The formula needs to be multiplied by the parameter m (for example, m can be equal to 255) to normalize the weight coefficient to the range of [0, m], which facilitates subsequent operations.

[0039] In some embodiments of this application, the block weight coefficient is the average of the block median weight coefficient and the block center value weight coefficient.

[0040] Some embodiments of this application obtain the block median weight coefficient and the block center value weight coefficient by using the median and center value respectively, and then jointly determine the block weight coefficient based on the block median weight coefficient and the block center value weight coefficient, making the method more robust and effectively reducing noise interference.

[0041] In some embodiments, after fusing all motion masks to obtain a target motion mask, the method further includes: obtaining a binary mask based on the target motion mask, wherein the binary mask uses a first value to represent moving pixels and a second value to represent stationary pixels.

[0042] Some embodiments of this application can directly output the detection results of whether each pixel belongs to a moving pixel by using a binary mask.

[0043] Some embodiments of this application eliminate the influence of noise and distortion of fine textures on the subsequent motion detection process by performing guided filtering preprocessing on the i-th image pair.

[0044] In some embodiments, dividing each image in the i-th image pair into multiple image blocks includes: obtaining the multiple image blocks by traversing each image in the i-th image pair using one size; or obtaining the multiple image blocks by traversing each image in the i-th image pair one by one using each of a variety of sizes.

[0045] Some embodiments of this application provide a method for dividing image pairs into blocks based on their respective sizes.

[0046] Secondly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the methods described in any of the embodiments of the first aspect above.

[0047] Thirdly, some embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any of the embodiments of the first aspect above.

[0048] Fourthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the motion detection method as described in any embodiment of the first aspect. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the composition of the motion detection system provided in the embodiments of this application;

[0051] Figure 2 A schematic diagram of the second image included in any set of image pairs provided in the embodiments of this application;

[0052] Figure 3 This is one of the flowcharts for the motion detection method provided in the embodiments of this application;

[0053] Figure 4 This is the second flowchart of the motion detection method provided in the embodiments of this application;

[0054] Figure 5 A schematic diagram illustrating the process of dividing an image into blocks using images traversed by the same size, as provided in an embodiment of this application;

[0055] Figure 6 This is a block diagram of the motion detection device provided in the embodiments of this application;

[0056] Figure 7 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0058] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the composition of a motion detection system 100 provided in some embodiments of this application.

[0060] Figure 1 The provided motion detection system 100 includes a mobile phone 10 for capturing multi-frame images and a server 20 for performing motion detection based on the multi-frame images.

[0061] Figure 1 The mobile phone 10 continuously captures a video sequence consisting of multiple frames. Specifically, in some embodiments of this application, the mobile phone 10 captures N frames, i.e. Figure 1 The first frame image 101, the second frame image 102, the third frame image 103, the fourth frame image 104, ..., up to the Nth frame image, these images can be obtained based on different exposure parameters or based on the same exposure. Since some embodiments of this application use a pairwise detection method for motion pixel detection, in some embodiments of this application, the mobile phone 10 (or the server 20 performs this process) also needs to divide these N frames into multiple image pairs. For example, in... Figure 1 In this process, N frames of images are grouped into pairs. The first image pair includes the first frame image 101 and the second frame image 102, the second image pair includes the second frame image 102 and the third frame image 103, the third image pair includes the third frame image 103 and the fourth frame image 104, and so on. The N frames of images can be divided into N-1 image pairs. Then, motion pixel detection is performed on each image pair to obtain the motion mask corresponding to each image pair. Finally, the target motion mask is obtained based on the N-1 motion masks corresponding to the N-1 image pairs.

[0062] It should be noted that, in some embodiments of this application, the images included in each image pair may be different. Figure 1 In other words, in some embodiments of this application, the image pair on which pairwise detection is based may not be... Figure 1 The image is not a pair of two consecutive frames, but a fused image composed of the preceding frame and several subsequent frames adjacent to it. For example... Figure 2 As shown, for Figure 1 The N frames of images are paired, where, Figure 2 The first image pair includes a first frame image 101 and a fused image 112, and the fused image is obtained by fusing the pixel values ​​of three frames (i.e., the second frame image 102, the third frame image 103, and the fourth frame image 104) following the first frame image 101. It can be understood that, based on... Figure 2 The first image pair shown indicates that the second image pair includes the second frame image 102 and a fused image obtained by fusing (e.g., averaging) the three frames following the second frame image 102 (as a specific example of the second image included in each image pair).

[0063] In some embodiments of this application, Figure 1 The mobile phone 10 directly sends the acquired multi-frame images to the server 20, which then performs motion detection based on the multi-frame images. Finally, the server 20 returns the obtained target motion mask to the mobile phone 10. In some embodiments of this application, Figure 1 The mobile phone 10 directly divides the acquired multi-frame images into multiple image pairs and sends the resulting multiple image pairs to the server 20, which then performs motion detection based on these image pairs.

[0064] It should be noted that, Figure 1 The motion detection method executed by server 20 can also be executed by mobile phone 10. That is, in some embodiments of this application, a microprocessor is integrated in mobile phone 10 or other mobile terminals, which can perform motion detection based on multiple frames of images. Figure 1 The mobile phone 10 is a type of mobile terminal. For example, in some embodiments of this application, the mobile phone 10 can also be a mobile device such as a PAD.

[0065] The following is combined with Figure 3 An exemplary motion detection method executed by mobile phone 10 or server 20 is described.

[0066] like Figure 3According to some embodiments of this application, a motion detection method is provided, which includes: S101, obtaining an i-th image pair based on a video sequence, wherein the i-th image pair includes an i-th frame image and a reference image, the reference image including an (i-1)-th frame image adjacent to the i-th frame image or a fused image of several preceding frames adjacent to the i-th frame image, where i is an integer greater than or equal to 2. S102, dividing the two images included in the i-th image pair into multiple image blocks respectively; S203, obtaining the target motion weights of all pixels in each image block to obtain a motion mask for the i-th image pair, wherein the target motion weight of any pixel is obtained by the following strategy: obtaining the target motion weight of any pixel based on the difference between the pixel value of any pixel and the feature value of the image block in which the pixel is located; S204, repeating the above process to obtain motion masks corresponding to multiple image pairs respectively, and fusing all motion masks (i.e., all motion masks corresponding to the video sequence) to obtain a target motion mask.

[0067] It should be noted that, in some embodiments of this application, at least to eliminate the influence of noise and distortion of fine textures on motion detection, guided filtering is required on the i-th image pair input in S101 before executing S102. This also allows the edges of the image to be better preserved and details mixed with noise to be discarded. In some embodiments of this application, in order to directly output the result of whether each pixel belongs to a moving pixel, the target motion mask needs to be converted into a binary mask. That is, after executing S104, the target motion mask needs to be converted into a binary mask based on empirical values ​​or medians (i.e., the global median of the target motion mask). For example, in some embodiments of this application, a pixel value of 255 is used to represent that the corresponding pixel belongs to a moving pixel, and 0 is used to represent that the corresponding pixel belongs to a stationary pixel; in other embodiments of this application, 0 and 1 can also be used to represent whether each pixel belongs to a moving pixel. It should be noted that, in some embodiments of this application, the value range of each pixel on the target motion mask can be any integer value in the range [0, 255]. In some embodiments of this application, the pixel value of each pixel on the target motion mask can be a decimal in the range [0,1], and its value represents the motion confidence of that pixel (in other words, how likely it is to be moving). It is understood that the target motion mask is non-binary and can be converted into a binary mask by comparing it with a set threshold (for example, the threshold is the global median of the target motion mask).

[0068] The steps described above are illustrated below.

[0069] In some embodiments of this application, the i-th image pair obtained in S101 may be two adjacent frames selected from multiple captured frames. That is, in these embodiments, the i-th image pair includes the i-th frame image and the (i+1)-th frame image.

[0070] In some other embodiments of this application, the i-th image pair involved in S101 includes an image arbitrarily selected from multiple captured images (i.e., the i-th frame image) and a fused image of multiple frames following that frame image. For example, in some embodiments of this application, the i-th image pair includes the r-th frame image, and includes a fused image obtained by fusing the (r+1)-th and (r+2)-th frames. In some embodiments of this application, the process of obtaining the fused image includes, for example, direct average fusion or weighted fusion. Taking weighted fusion as an example, when fusing images using weighted fusion, the closer a pixel's value is to 128, the higher its weight. Assuming the pixel values ​​at the same location in two images are denoted as A and B, their weights are w1 = e^(abs(A-128)) and w2 = e^(abs(B-128)). Normalization gives the weight of pixel A as w1 / (w1+w2) and the weight of pixel B as w2 / (w1+w2). This pattern continues for multi-frame image fusion, where abs represents the absolute value operation. It should be noted that the embodiments of this application do not limit the specific number of frames fused to obtain the reference image in each image pair.

[0071] In other words, in some embodiments of this application, the i-th image pair includes two adjacent frames, while in other embodiments of this application, the i-th image pair includes a fused image of a single frame and multiple adjacent frames.

[0072] Before executing S102, some embodiments of this application further include a guided filtering process for the i-th image pair. Guided filtering explicitly uses a reference image (guidance image) to calculate the output image, where the guidance image can be the input image itself or another image. In some embodiments of this application, the i-th image pair includes a first image and a second image, respectively. Guided filtering for the first image can use the first image itself as the guidance image, and guided filtering for the second image can use the second image itself as the guidance image. However, embodiments of this application are not limited to using the image itself as the guidance image.

[0073] In some embodiments of this application, before dividing the two images included in the i-th image pair into multiple image blocks, the motion detection method further includes: obtaining an original i-th frame image and an original i+1-th frame image from the video sequence; performing guided filtering on the original i-th frame image to obtain the i-th frame image, and performing guided filtering on the original i+1-th frame image to obtain the i+1-th frame image. For example, guided filtering is performed on the original i-th frame image based on the original i-th frame image itself to obtain the i-th frame image, and guided filtering is performed on the original i+1-th frame image based on the original i+1-th frame image to obtain the i+1-th frame image.

[0074] In some embodiments of this application, before dividing the two images included in the i-th image pair into multiple image blocks, the motion detection method further includes: obtaining an original i-th frame image and several subsequent original frames of the original i-th frame image from the video sequence; obtaining an original fused image of the several subsequent original frames of the original i-th frame image; performing guided filtering on the original i-th frame image to obtain the i-th frame image, and performing guided filtering on the original fused image to obtain the fused image. For example, guided filtering is performed on the original i-th frame image based on the original i-th frame image itself to obtain the i-th frame image, and guided filtering is performed on the original fused image based on the original fused image itself to obtain the fused image.

[0075] In other words, in some embodiments of this application, before S102, the motion detection method further includes: performing guided filtering on the i-th image pair acquired in S101, including the i-th frame image and the reference image, and then performing S102 and the subsequent steps on the two guided-filtered images respectively.

[0076] In some embodiments of this application, S102 includes: dividing the two images included in the i-th image pair into multiple image blocks according to a size.

[0077] For example, in some embodiments of this application, S102 includes: obtaining the plurality of image blocks by traversing each image in the i-th image pair by a size.

[0078] For example, in some embodiments of this application, S102 includes: dividing each image in the i-th image pair into blocks according to each of multiple sizes (e.g., N sizes) to obtain the plurality of image blocks, wherein the plurality of image blocks are divided into multiple groups according to different partitioning sizes. For example, in some embodiments of this application, S102 can obtain multiple sets of image blocks by traversing each image in the i-th image pair one by one according to each of the multiple sizes, wherein multiple image blocks in the same group are obtained by dividing the images according to the same size, and multiple image blocks in different groups are obtained by dividing the images according to different sizes.

[0079] It should be noted that when S102 is a set of multiple image blocks obtained by dividing the image into blocks one by one using multiple sizes (each set includes at least two images), then in some embodiments of this application, S103 includes: obtaining the block weight coefficient of each image block in each set according to at least the number of moving pixels, to obtain multiple block weight coefficients corresponding to any pixel, wherein the total number of block weight coefficients corresponding to the pixel is the same as the number of sets (i.e., the number of sets of multiple image blocks); obtaining the target motion weight of the pixel according to the multiple block weight coefficients corresponding to the pixel; and obtaining the motion mask of the i-th image pair according to the target motion weight. For example, in some embodiments of this application, the step of obtaining the target motion weight of the pixel according to the multiple block weight coefficients corresponding to the pixel in S103 includes: obtaining the target motion weight according to the multiple block weight coefficients and the total number of pixels in each size image block in the multiple size image blocks where the pixel is located.

[0080] It is understandable that when the image is divided into blocks using the same size traversal, the resulting image blocks will not necessarily be the same size (see reference for details). Figure 5 ).

[0081] To obtain the target weight coefficient of any pixel, in some embodiments of this application, it is necessary to first obtain the block weight coefficient of the image block to which the pixel belongs. For example, in some embodiments of this application, obtaining the target motion weight of all pixels on each image block in S103 includes: confirming that the pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels included in the image block to which the pixel belongs; obtaining the block weight coefficient of the image block to which the pixel belongs based on the total number of moving pixels and the size of the image block to which the pixel belongs; and obtaining the target weight coefficient of the pixel based on the block weight coefficient.

[0082] It should be noted that in some embodiments of this application, the feature value of the image block involved in S103 is the median of the image block. In other embodiments of this application, the feature value of the image block involved in S103 is the center value of the image block. In still other embodiments of this application, the feature value of the image block involved in S103 is the mean of the image block. That is, the feature value of the image block involved in S103 is a pixel reference value obtained from the image block. This pixel reference value is used to characterize the pixel value features of the image block. In some examples, the pixel reference value is the pixel value of the pixel located at the center of the image block (i.e., the center value of the image block). In some examples, the pixel reference value is the median of the pixel values ​​of all pixels in the image block (i.e., sorting the pixel values ​​of all pixels by size and using the pixel value at the center of the sort as the pixel reference value). In some embodiments, the pixel reference value is the average of the pixel values ​​of all pixels in the image block. The embodiments of this application do not limit the specific type of the pixel reference value.

[0083] The following describes the implementation process of motion detection methods in some embodiments of this application when dividing two images into blocks of one size and when dividing two images into blocks of multiple sizes.

[0084] When S102 uses only one size to divide the two images into blocks, the motion detection method in some embodiments of this application includes: S102 includes: dividing the two images included in the i-th image pair into the plurality of image blocks according to one size, wherein any pixel is a first pixel, and the first pixel is assigned to the j-th image block, where j is an integer greater than or equal to 1. Correspondingly, S103 involves the process of confirming that any pixel belongs to a moving pixel based on the difference and counting the total number of moving pixels included in the image block containing any pixel, including: confirming that the first pixel belongs to a moving pixel based on the difference and counting the total number of moving pixels included in the j-th image block. S103 involves the process of obtaining the block weight coefficient of the image block containing any pixel based on the total number of moving pixels and the size of the image block containing any pixel, including: calculating the ratio of the total number of moving pixels included in the j-th image block to the size of the j-th image block to obtain the block weight coefficient. S103 involves obtaining the target weight coefficient of any pixel based on the block weight coefficient, which includes: using the ratio as the target weight coefficient of the first pixel.

[0085] When two images are divided into blocks using multiple different sizes, some pixels may simultaneously occupy multiple image blocks of different sizes. Obtaining the target motion weight for each pixel using multiple image blocks of different sizes can effectively overcome significant noise interference in some image blocks. That is, in some embodiments of this application, S102 includes: dividing the two images included in the i-th image pair into blocks according to N different sizes to obtain the multiple image blocks, where N is an integer greater than 1, and any pixel is a second pixel, which is divided into N target image blocks. Each target image block is obtained by dividing the two images using a different size. Correspondingly, S103, which involves confirming that any pixel belongs to a moving pixel based on the difference and counting the total number of moving pixels included in the image block containing any pixel, includes, for example: confirming that the second pixel belongs to a moving pixel based on the difference and counting the total number of moving pixels included in each of the N target image blocks. S103 involves obtaining the block weight coefficient of the image block containing any pixel based on the total number of moving pixels and the size of the image block containing any pixel. The process includes, for example, solving for the block weight coefficients of various target image blocks based on the total number of moving pixels in each of the N target image blocks and the size of the N target image blocks, thus obtaining N block weight coefficients. S103 also involves obtaining the target weight coefficient of any pixel based on the block weight coefficients, including determining the target weight coefficient of the second pixel based on the N block weight coefficients and the size of each type of image block in the N target image blocks.

[0086] In some embodiments of this application, when a pixel is located on N target image blocks obtained by dividing an image into N different sizes, the pixel also corresponds to N block weight coefficients. Therefore, the target motion weight of the pixel can be comprehensively solved based on these N block weight coefficients. For example, in some embodiments of this application, the process of determining the target weight coefficient of the second pixel based on the N block weight coefficients and the size of each of the N target image blocks in S103 includes, for example, the weighted average of the N block weight coefficients to obtain the target motion weight of any pixel, wherein the larger the size of the image block, the larger the weighting coefficient. That is, a weighting coefficient value is assigned to each of the N block weight coefficients, and the magnitude of the weighting coefficient value is proportional to the size of the image block from which the block weight coefficient is obtained. Then, the sum of the products of the N block weight coefficients and their respective weighting coefficient values ​​is calculated, and then the ratio of this sum to the sum of the sizes of the N target image blocks corresponding to the N block weight coefficients is calculated. This yields the weighted average of the N block weight coefficients, which in turn yields the target motion weight of the pixel. It should be noted that in some embodiments of this application, the value of the weighting coefficient can be adjusted according to the specific application scenario. For example, the N weighting coefficient values ​​can be set to the same size.

[0087] In other words, some embodiments of this application divide the images in each image pair into blocks (an image pair includes two adjacent frames or a fused image of one frame and multiple adjacent frames). By comparing the difference between the pixel value of any pixel in each image block and the feature value (e.g., center value and / or median value) of the image block, whether the trend of change is consistent in the adjacent two or more frames (i.e., multiple frames are fused), it is preliminarily determined whether each pixel belongs to a moving pixel. Based on the number of moving pixels obtained from the preliminary determination, the block weight coefficient of each image block is obtained. Then, based on the multiple block weight coefficients of the same pixel in various image blocks (i.e., the image block in which the pixel is located is obtained by dividing the image into blocks of one size), the target weight coefficient of the pixel is obtained.

[0088] To improve the accuracy of motion detection results, in some embodiments of this application, to fuse N block weight coefficients to obtain the target motion weight, a larger weight coefficient is set for the larger the image block size. For example, in some embodiments of this application, if the two images are divided into blocks of three different sizes, Q, W, and E, then any pixel with coordinates r and c is divided into three target image blocks of sizes Q*Q, W*W, and E*E, corresponding to three block weight coefficients; wherein, the calculation formula for the target weight coefficient of any pixel is as follows:

[0089]

[0090] Among them, Weight Q，(r，c) Weight W，(r，c) and Weight E，(r，c) These represent the three block weight coefficients corresponding to any given pixel. It can be understood that the weighting coefficient for an image block of size Q*Q is Q*Q, the weighting coefficient for an image block of size W*W is W*W, and the weighting coefficient for an image block of size E*E is E*E. Therefore, it can be seen that the larger the size, the larger the corresponding block weight coefficient.

[0091] It should be noted that larger image patches are better able to account for larger-scale motion (such as large-area solid color motion, which is more prone to misjudgment at smaller scales). Setting different scales when dividing an image into patches can take into account motion at different scales. Giving larger-scale image patches greater weight can reduce the impact of misjudgment at smaller scales (larger scales consider more pixels, making them less prone to misjudgment).

[0092] To more accurately determine the motion of each pixel based on the differences involved in S103 and thus obtain the block weight coefficients of each image block, in some embodiments of this application, the process of confirming that any pixel belongs to a moving pixel based on the differences involved in S103 includes, for example,: on the i-th frame image included in the i-th image pair, obtaining the feature value of any image block where the pixel is located to obtain the i-th feature value, and calculating the difference between the pixel value of the pixel and the i-th feature value to obtain the i-th difference, wherein the i-th feature value includes the i-th median and / or the i-th center value; on the reference image included in the i-th image pair, obtaining the feature value of any image block to obtain the (i+1)-th feature value, and calculating the difference between the pixel value of the pixel and the (i+1)-th feature value to obtain the (i+1)-th difference, wherein the (i+1)-th feature value includes the (i+1)-th median and / or the (i+1)-th center value; and, when it is confirmed by the i-th difference and the (i+1)-th difference that the change trend of the pixel in the two images (i.e., the i-th frame image and the reference image) is inconsistent, Then any pixel is confirmed to be a moving pixel.

[0093] It should be noted that the difference between the pixel value of the pixel involved in S103 and the feature value of the image block to which the pixel belongs can be characterized by methods such as difference or ratio. That is, to better quantify the difference involved in S103, in some embodiments of this application, the difference involved in S103 is characterized by the difference between the pixel value of any pixel and the median and / or center value of any image block to which the pixel belongs. In other embodiments of this application, the difference involved in S103 is characterized by the ratio of the pixel value of any pixel to the median and / or center value of any image block to which the pixel belongs.

[0094] The following example illustrates how to initially determine whether any pixel belongs to a moving pixel when using difference and ratio to characterize differences.

[0095] When using difference to represent differences, in some embodiments of this application, the i-th difference involved in S103 is obtained by calculating the difference between the pixel value of any pixel and the i-th feature value; the (i+1)-th difference involved in S103 is obtained by calculating the difference between the pixel value of any pixel and the (i+1)-th feature value. Correspondingly, the process of confirming that any pixel is a moving pixel when the change trends of the i-th difference and the (i+1)-th difference are inconsistent in the two images, exemplarily includes: if the positive and negative characteristics of the i-th difference and the (i+1)-th difference are opposite, then the pixel is confirmed as a moving pixel.

[0096] When using ratios to represent differences, in some embodiments of this application, the i-th difference involved in S103 is obtained by calculating the ratio of the pixel value of any pixel to the i-th feature value; the (i+1)-th difference involved in S103 is obtained by calculating the (i+1)-th ratio of the pixel value of any pixel to the (i+1)-th feature value. Correspondingly, the process of confirming that any pixel is a moving pixel when the change trends of the i-th difference and the (i+1)-th difference are inconsistent in the two images, exemplarily includes: if the absolute value of the difference between the i-th ratio and the (i+1)-th ratio is less than a set threshold, then the pixel is confirmed as a moving pixel. It should be noted that in some embodiments of this application, it is also possible to initially determine whether any pixel belongs to a moving pixel by solving the ratio of the i-th ratio to the (i+1)-th ratio. It is understood that in some embodiments of this application, this method is only used to initially determine whether any pixel belongs to a moving pixel. Then, based on this determination result, the data of moving pixels on the image block are statistically analyzed to determine the block weight coefficient of the image block. Finally, the probability of each pixel belonging to a moving pixel is characterized by the target motion weight of each pixel.

[0097] In some embodiments of this application, the process of determining the number of moving pixels on each image block based on differences in step S103 includes, for example, confirming that any pixel is stationary if the differences conform to the same trend, otherwise confirming that any pixel is moving. It should be noted that directly comparing the pixel value of each pixel with the median and center value to determine whether it belongs to a moving pixel is only a preliminary judgment. This is because image-based motion detection only detects motion; therefore, noise, lighting changes, moiré patterns, etc., can interfere with motion detection. Motion masks or determining whether a pixel belongs to a moving pixel based on differences are probabilistic values. However, it is understood that the embodiments of this application... Figure 3 The method of determining the number of moving pixels and block weight coefficients on an image block before determining the target motion mask can effectively improve the accuracy of motion detection.

[0098] For example, in order to measure whether the changing trends of each pixel in different images are consistent through the difference, in some embodiments of this application, the process of obtaining the block weight coefficient of each image block in multiple image blocks based at least on the number of moving pixels in S103 includes: obtaining the k-th median and / or the k-th center value of the k-th image block, wherein the k-th image block belongs to any one of the multiple image blocks, the value of k is greater than or equal to 1 and the value of k is less than or equal to the total number of the multiple image blocks; determining the number of moving pixels in the k-th image block based on the sign of the difference between each pixel in the k-th image block and the k-th median and / or the k-th center value; and obtaining the block weight coefficient of the k-th image block based on the number of moving pixels in the k-th image block and the total number of pixels in the k-th image block. For example, in some embodiments of this application, S103, determining the number of moving pixels in the k-th image block based on the sign of the difference between each pixel in the k-th image block and the k-th median and / or the k-th center value, includes: obtaining the number of moving pixels by counting all pixels with opposite signs of the difference in the different images.

[0099] For example, in order to measure whether the changing trends of each pixel in different images are consistent through ratios, in some embodiments of this application, the step S103 of obtaining the block weight coefficient of each image block in a plurality of image blocks based at least on the number of moving pixels includes: obtaining the k-th median and / or the k-th center value of the k-th image block, wherein the k-th image block belongs to any one of the plurality of j-th image blocks, and the value of k is greater than or equal to 1 and the value of k is less than or equal to the total number of the plurality of j-th image blocks; determining the number of moving pixels included in the k-th image block based on the ratio of each pixel included in the k-th image block to the k-th median and / or the k-th center value; and obtaining the block weight coefficient of the k-th image block based on the number of moving pixels included in the k-th image block and the total number of pixels included in the k-th image block. For example, in some embodiments of this application, S103 determines the number of moving pixels in the k-th image block based on the ratio of each pixel in the k-th image block to the k-th median and / or the k-th center value, including: obtaining the number of moving pixels by counting all pixels in different images whose absolute value of the difference of the ratios is greater than or equal to a set threshold.

[0100] It should be noted that, when obtaining the block weight coefficient of an image block based on a ratio or difference, in some embodiments of this application, S103, which describes obtaining the block weight coefficient of the k-th image block based on the number of moving pixels included in the k-th image block and the total number of pixels included in the k-th image block, includes: calculating the ratio of the number of moving pixels included in the k-th image block to the total number of pixels included in the k-th image block to obtain the block weight coefficient of the k-th image block.

[0101] To improve data processing speed, in some embodiments of this application, S103 includes: obtaining the block weight coefficient of each image block in a plurality of image blocks based at least on the number of moving pixels; and obtaining the motion mask of the i-th image pair based on the block weight coefficient.

[0102] It should be noted that when an image pair is divided into images using a certain size, the block weight coefficient of each image block is obtained, which is equivalent to a weight coefficient for each pixel. The motion mask for the i-th image pair is then derived from this block weight coefficient corresponding to each pixel. In other embodiments of this application, when multiple image block sets are obtained by successively dividing the image using multiple block sizes (each set includes image blocks obtained by dividing the first and second images according to a certain size), multiple block weight coefficients corresponding to a pixel are also obtained. In this case, the target motion weight of the pixel needs to be obtained based on these multiple block weight coefficients, and then the motion mask for the i-th image pair is obtained based on the combined weight coefficient.

[0103] To further improve the accuracy of the obtained block weight coefficients, some embodiments of this application provide a method for quantifying block weight coefficients, namely, determining the block weight coefficients based on the number of moving pixels and the size of the image blocks. For example, in some embodiments of this application, the process of obtaining the block weight coefficients of each image block in a plurality of image blocks at least based on the number of moving pixels includes: obtaining the block weight coefficients based on the number of moving pixels and the size of each image block in the plurality of image blocks.

[0104] In some embodiments of this application, the block weight coefficients of each image block involved in S103 are determined by at least one of the block median weight coefficient and the block center value weight coefficient. The block median weight coefficient is obtained by determining the difference between the pixel value of any pixel and the median of the image block in which the pixel is located, and the block center value weight coefficient is obtained by determining the difference between the pixel value of any pixel and the center value of the image block in which the pixel is located.

[0105] It is understandable that, in order to minimize the technical problem of poor accuracy in counting moving pixels in each image block based solely on the median or center value due to high noise near the median or center value in certain image blocks, in some embodiments of this application, the block weight coefficient of each image block involved in S103 is determined by both the block median weight coefficient and the block center value weight coefficient. That is, in some embodiments of this application, the block weight coefficient is related to both the block median weight coefficient and the block center value weight coefficient of the image block. For example, in some embodiments, the block weight coefficient is equal to the weighted average of the block median weight coefficient and the block center value weight coefficient of the image block, such as the mean of the two.

[0106] To quantify the block median weight coefficient or block center weight coefficient based on the number of moving pixels in each image block, in some embodiments of this application, the calculation formula for the block median weight coefficient is as follows:

[0107] Weight Median，k =m*count1 / Q*Q

[0108] or,

[0109] The formula for calculating the weighting coefficient of the block center value is:

[0110] Weight Middle，k = m * count2 / Q * Q

[0111] Where k represents the k-th image block, count1 represents the total number of moving pixels on the k-th image block as determined by the median of the k-th image block, count2 represents the total number of moving pixels on the k-th image block as determined by the center value of the k-th image block, Q*Q represents the total number of pixels included in the k-th image block, m represents the normalization parameter, and the value of k is greater than or equal to 1 and less than or equal to the total number of multiple image blocks obtained by dividing the image according to size Q.

[0112] In some embodiments of this application, the target motion mask involved in S104 is used to characterize the final probability that each pixel belongs to a moving pixel. Accordingly, the target motion weight coefficient of any pixel in the target motion mask involved in S104 can be expressed as the following formula:

[0113]

[0114] Among them, M i，(r，c) Let be the motion mask corresponding to the i-th image pair, n be the total number of image pairs, and r and c represent the coordinates of any pixel.

[0115] In order to directly obtain the motion detection results of the video frame sequence, in some embodiments of this application, after S104, the motion detection method further includes: obtaining a binary mask according to the target motion mask, wherein the binary mask uses a first value to represent moving pixels and a second value to represent stationary pixels.

[0116] It should be noted that, in some embodiments of this application, obtaining a binary mask based on the target motion mask includes: obtaining the binary mask based on the median or center value of the target motion mask. That is, embodiments of this application obtain the binary mask based on the median of the target motion mask (i.e., comparing the pixel values ​​of each pixel on the target motion mask with the median; if the median is greater, a first numerical representation is used; if it is less than or equal to the median, a second numerical representation is used). In other embodiments of this application, the binary mask can also be obtained based on empirical values ​​(i.e., comparing the pixel values ​​of each pixel on the target motion mask with empirical values; if the median is greater, a first numerical representation is used; if it is less than or equal to the median, a second numerical representation is used).

[0117] It should be noted that in some embodiments of this application, when the first image block among the multiple image blocks obtained in S102 includes an odd number of pixels, the center value of the first image block is the pixel value of the pixel located at the center position of the first image block; or when the second image block among the multiple image blocks obtained in S102 includes an even number of pixels, the center value of the second image block is determined by the pixel values ​​of multiple pixels located in the central region of the second image block. For example, if the size of each divided image block is 4*4, then the average of the 2*2 pixel values ​​at the center position of the image block can be taken as the center value of the 4*4 image block. For example, if the size of each divided image block is 3*3, then the pixel value at the center position of the 3*3 image block can be taken as the center value of the image block. In some embodiments of this application, the method of obtaining the median of each image block includes sorting the pixel values ​​of each pixel in any image block from largest to smallest or smallest to largest, taking the pixel value at the middle position of the sorted queue as the median, and if there are two pixel values ​​at the middle position, the average of these two pixel values ​​can be calculated as the median.

[0118] The following uses two adjacent frames as an image pair to illustrate some embodiments of the motion detection method provided in this application.

[0119] In some embodiments of this application, the server 20 or the mobile phone 10 performs motion detection on multiple frames of images (i.e., obtains a target motion mask) through the following process: The server 20 performs motion detection on each pair of adjacent video frame images (i.e., as an image pair) to obtain multiple motion masks, and then combines the multiple motion masks to obtain the target motion mask. A motion mask is obtained by performing motion detection on two images (e.g., on two adjacent frames). This process includes: calculating the median and center value of each image block; calculating the difference between the pixel value of each pixel within the corresponding image block and the median / center value of the image block; if the difference conforms to the same trend in different images (e.g., the difference sign is the same in different images or the difference in the ratio is less than a set threshold), then the pixel is considered stationary; otherwise, the pixel is moving. Based on this, the number of all moving pixels in the image block can be determined, thus obtaining a motion mask.

[0120] In other words, in some embodiments of this application, the motion detection process performed by the server 20 or the mobile phone 10 includes:

[0121] The user inputs n consecutive images {L1, L2, ..., L...} n} can be grouped in pairs as {L1, L2}, {L2, L3}, ..., {L n-1 L n There are n-1 image pairs in total. Motion detection is performed on each image pair. Here, we take the i-th image pair as an example.

[0122] Step 1: For the i-th image pair {L i L i+1}, for the i-th frame image L i Using the i-th frame image L i Guided filtering is performed on the guide image, for the (i+1)th frame image L i+1 Using the (i+1)th frame image L i+1 Guided filtering is applied to the guide graph.

[0123] It should be noted that in some embodiments of this application, other images may also be used as guide images for the i-th frame or the (i+1)-th frame. In some embodiments of this application, the second step may be performed directly on the i-th image instead of performing guide filtering on the i-th image pair.

[0124] The second step is to set multiple dimensions (e.g., three different dimensions Q, W, and E). It can be understood that by setting three different partitioning dimensions, the i-th image pair can be divided three times, resulting in three sets of image blocks.

[0125] In some embodiments of this application, W = 2 * Q - 1, E = 2 * W - 1, and Q, W, and E are all odd numbers. This ensures that each image block obtained by dividing the image using these dimensions includes an odd number of pixels. Then, the center value of each image block can be directly determined as the pixel value of the pixel located at the center of the image block.

[0126] It should be noted that the embodiments of this application do not limit the side length corresponding to the set size to an odd number. When the image block obtained according to the set size includes an even number of pixels, the center value of the pixel block can be determined by using the maximum or average value of multiple pixel values ​​located in the central region as the center value of the image block.

[0127] The third step involves dividing the two images in the i-th image pair into blocks according to different sizes, resulting in image block sets corresponding to each size. Specifically, dividing the i-th frame image into blocks according to a first size Q yields multiple first image blocks, and dividing the (i+1)-th frame image into blocks according to the same first size Q also yields multiple first image blocks. It can be understood that the maximum size of a first image block is the total number of pixels within the area defined by the first size Q. Similarly, dividing the i-th frame image into blocks according to a second size W (a different size from the first size) yields multiple second image blocks, and dividing the (i+1)-th frame image into blocks according to the same second size W also yields multiple second image blocks. It can be understood that the maximum size of a second image block is the total number of pixels within the area defined by the second size. Finally, dividing the i-th frame image into blocks according to a third size E (a different size from both the first and second sizes) yields multiple third image blocks, and dividing the (i+1)-th frame image into blocks according to the same third size E also yields multiple third image blocks. It can be understood that the maximum size of a third image block is the total number of pixels within the area defined by the third size E.

[0128] Calculate the median and center value of all image blocks obtained from the block division, including each first image block and each second image block.

[0129] The following section uses a specific image patch as an example to further illustrate the further processing in this third step.

[0130] Whether a pixel is a moving pixel is determined by whether the pixel value of any pixel within an image block in different frames (i.e., the i-th frame and the (i+1)-th frame in each image pair belong to different frames) is consistent with the median / center value of that image block. Then, the target motion mask is obtained based on the total number of moving pixels in that image block. Because some embodiments of this application comprehensively consider the influence of multiple block sizes, medians, and center values ​​on motion determination, the obtained target mask can more accurately reflect whether each pixel belongs to a moving pixel.

[0131] It is understandable that by performing the third step, the motion mask M corresponding to each image pair can be obtained. i .

[0132] The fourth step is to apply multiple motion masks {M1, M2, ... M} n-1 The final motion mask M is obtained by combining these elements.

[0133] It is understood that some embodiments of this application calculate the motion mask simply by comparing pixel values ​​(i.e., comparing the pixel values ​​of each pixel in each image block with the median and / or center value), thus achieving a faster calculation speed. Because some embodiments of this application use block matching (i.e., fusing the same image block based on the comparison results of the first and second images), the methods of some embodiments of this application are insensitive to brightness differences between different frames. Furthermore, because some embodiments of this application comprehensively consider different image block sizes as well as the median and center values ​​of each image block when obtaining the motion mask for each image pair, the methods of some embodiments of this application are more robust and effectively reduce noise interference.

[0134] The following is combined with Figure 4 The motion detection method performed by a mobile phone in some embodiments of this application is illustrated by taking two adjacent frames as an image pair and dividing the images included in each image pair into blocks of three sizes.

[0135] S201, Input n frames of images {L1, L2, ..., L...} n The multi-frame image {L1, L2, ... L} n} represents multiple frames of images taken by the user through a mobile phone with different exposures, where n is a natural number greater than 2.

[0136] S202, divided into n-1 image pairs {L1, L2}, {L2, L3}, ..., {L... n-1 L n}

[0137] In other words, the n frames of images input from S101 are grouped pairwise as {L1, L2}, {L2, L3}, ..., {L... n-1 L n There are a total of n-1 image pairs.

[0138] S203, three different block side lengths Q, W, and E are set in advance. These block side lengths are used to divide each image pair into blocks.

[0139] In order to quickly determine the center value of each image block (i.e., to ensure that the image block includes an odd number of pixels so that the pixel value of the pixel located at the center can be directly used as the center value of the image block), the size relationship of Q, W, and E is: W = 2 * Q - 1, E = 2 * W - 1, and Q, W, and E are all odd numbers.

[0140] The following uses the i-th image to represent {L} i L i+1 For example, how to calculate the motion mask M for the i-th image pair? i .

[0141] S204, for the input i-th image pair {L i L i+1}, for the i-th frame image L i Using the i-th frame image L i To perform guided filtering on the guide image, for the (i+1)th frame image L i+1 Using the (i+1)th frame image L i+1 To perform guided filtering on the guiding image, the i-th image pair after guided filtering is represented as {L}. in L ref This approach allows for better preservation of image edges and the discarding of details mixed with noise. It also helps to eliminate the impact of noise and distortion of fine textures on motion detection.

[0142] The following example illustrates how to determine the block weight coefficients based on the median and center value of each image block.

[0143] S211, starting from point (0, 0), for the guided filter processed image pair {L in L ref The image in the i-th image pair is divided into two frames, i-th and i+1-th, according to a square with side length Q, resulting in multiple image blocks. For each image block Pj, the median MIDin and median MIDref of the image block Pj can be calculated on the first guide map Lin and the second guide map Lref, respectively. For each image block Pj, the difference between any pixel within the block and the median MIDin and median MIDref is calculated on the first guide map Lin and the second guide map Lref, respectively. It should be noted that the subscript j is only for convenience in indicating which image block it belongs to, where j∈[0,X], and X represents the number of blocks, the specific value of X depends on the scale and image size.

[0144] S212, count the number of pixels whose product of differences is negative in the image blocks at the same position in the two frames (i.e., count the number of initially determined moving pixels), divide the number by the total number of pixels included in the image block Pj, and use it as the median weight of the image block Pj (i.e., obtain the block median weight coefficient).

[0145] S213: Following the same steps as the median of a block, S211 and S212 are performed on the block center value to obtain the block center value weight corresponding to image block Pj. Then, the weight of the block (i.e., the block center value weight coefficient) is obtained based on the block center value weight (i.e., the block center value weight coefficient) and the block median weight (i.e., the block median weight coefficient). For example, the average of the block median weight coefficient and the block center value weight coefficient is calculated as the block weight coefficient of image block Pj.

[0146] In other words, taking size Q as an example, based on size Q, {L} can be... in L ref The image is divided into multiple blocks with a side length of Q. The first image block obtained from the division is used as an example to illustrate the processing steps of S211-S213 above.

[0147] 1) In the first guide diagram L in The first image block P in，1 The median of the image patch can be calculated from the above. in，1 and the central value Middle in，1 .

[0148] 2) In the second guide diagram L ref The first image block P ref，1 The median of the image patch can be calculated from the above. ref，1 and the middle value ref，1 .

[0149] It is understandable that the first image block P in，1 and P ref，1 The elements are at the same position and have the same size on the divided image, except that P in，1 It is the first image patch obtained by dividing from point (0.0) on the first guide map, and P ref，1 It is the first image block obtained by dividing from point (0.0) on the second guide map.

[0150] 3) In the first image block P in，1 Above, calculate the median of each pixel q relative to the median of the image patch. in，1 Difference Diff0 in，q，1 And the middle value of the pixel and the center of the image patch. in，1 Difference Diff1in，n，1 .

[0151] 4) In the first image block P ref，1 Above, calculate the median of each pixel q relative to the median of the image patch. ref，1 Difference Diff0 ref，q，1 And the value of the pixel relative to the center of the block (Middle). ref，1 Difference Diff1 ref，q，1 .

[0152] 5) The process of obtaining the block median weight coefficient is as follows.

[0153] Initialize the first moving pixel count parameter count1 to 0. If Diff0 in，q，1 *Diff0 ref，q，1 If Diff < 0, then the pixel is considered to be a moving pixel, and the value of the first moving pixel count parameter is incremented, i.e., count1 += 1. If Diff0 in，q，1 *Diff0 ref，q，1 If the value is greater than or equal to 0, then the pixel is considered to be stationary.

[0154] Let the number of points in the block (i.e., the first image block) be Q*Q, then the block midpoint weight coefficient of the entire image block can be expressed as Weight. Median，1 = 255 * count1 / Q * Q, which means the weight coefficient of the block median of the first image block is Weight. Median，1 =255*count / Q*Q.

[0155] 6) The process of obtaining the weight coefficient of the block center value is as follows.

[0156] Initialize the second moving pixel count parameter count2 to 0. If Diff1 in，q，1 *Diff1 ref，q，1 If the value is less than 0, the pixel is considered to be a moving pixel, and therefore the value of the second moving pixel count parameter is incremented, i.e., the second moving pixel count value count2 += 1. If Diff1 in，q，1 *Diff1 ref，q，1 If the value is ≥0, then the pixel is considered to be stationary.

[0157] Let the number of points within the block (i.e., the first image block) be Q*Q, then the center value weight of the entire block can be expressed as Weight. Middle，1 = 255 * count2 / Q * Q, that is, the center value weight of the first block is Weight. Middle，1 =255*count / Q*Q.

[0158] 7) Combining the results of the block median weight coefficient and the block center value weight coefficient of the first image block, the first weight coefficient of the first image block is obtained. The specific process is as follows.

[0159] In some embodiments of this application, the first weight coefficient of the first image patch with scale Q is represented by the formula: Weight Q，1 =(Weiggt Midde，1 +Weight Midsan，1 ) / 2, that is, the first weight coefficient of any pixel (r, c) within the block corresponding to the first image patch at scale Q is Weight Q，(r，c) =Weight Q，1 .

[0160] The first weight coefficients of each image block in the multiple image blocks were obtained through S211-S213 above. It is understandable that... Figure 4 S221-S223 are also shown to obtain the second weighting coefficients for each image block in the second group of multiple image blocks. Figure 4 S231-S233 are also shown to obtain the third weight coefficients for each image block in the third group of multiple image blocks. To avoid repetition, S221, S222, S223, S231, S232 and S233 will not be explained in detail here.

[0161] After performing all the above steps, three weight coefficients (i.e., the first weight coefficient, the second weight coefficient, and the third weight coefficient) are obtained, and then S205 is executed.

[0162] When the scale is W, the block weight coefficients of each image patch are calculated in the same way as when the scale is Q. For each pixel, there are weights at three scales. Q，(r，c) Weight W，(r，c) Weight E，(r，c) The final motion weight of a pixel is obtained by taking a weighted average of the three weights (the larger the scale, the greater the weight). (r，c) (That is, the comprehensive weighting coefficient is obtained). The motion weights of the entire image can be obtained, i.e., through {L i L i+1} Obtain the motion mask M i .

[0163] S205, each pixel corresponds to three block weights (i.e., three weight coefficients, because each pixel is located in three image blocks of different sizes, and each image block corresponds to one weight coefficient). The comprehensive weight coefficient of the corresponding pixel is obtained based on these three weight coefficients. For example, the comprehensive weight coefficient is obtained by weighting the three weight coefficients (the larger the size, the higher the weight).

[0164] For example, in some embodiments of this application, the formula for calculating the comprehensive weight coefficient of any pixel (r, c) in the first block is as follows:

[0165]

[0166] The specific meanings of each parameter in the formula can be found in the description above.

[0167] The comprehensive weight coefficients are used as the pixel values ​​of each pixel on the motion mask corresponding to the i-th image pair to obtain the mask corresponding to the i-th image pair.

[0168] S206, Median filtering removes outliers, that is, median filtering is performed on the pixel values ​​on the motion mask corresponding to the i-th image pair to remove outliers.

[0169] S207. Merge n-1 motion weight maps (i.e., motion masks corresponding to each image pair) to form a single motion weight map. Calculate the global median of the entire weight map and use the global median as a threshold. Pixels with values ​​greater than the global median are considered to be in motion regions.

[0170] In other words, in some embodiments of this application, n-1 motion masks can be obtained from n-1 image pairs, and then the union of these n-1 motion masks is taken to obtain the final motion mask (i.e., the target motion mask). Let Mi, (r, c) be the weight of the i-th motion mask corresponding to the i-th image pair at position (r, c), then the target weight coefficient of the target motion mask at position (r, c) is:

[0171]

[0172] The weights of the entire image are normalized to the range [0, 255].

[0173] Find the global median of the target motion mask, then iterate through the motion weight map corresponding to the target motion mask. If the target weight coefficient at a certain position on the target motion mask is greater than the global median, then set the weight value of that position to 255; otherwise, set it to 0. This converts the target motion mask into a binary image. In this binary image, the pixels with a pixel value of 255 are the finally detected moving pixels, and the pixels with a pixel value of 0 are the finally detected falling stationary pixels.

[0174] It should be noted that the size of each image block obtained by dividing an image into blocks of a certain size is not necessarily the same. The following diagram illustrates the process of dividing an image into blocks by traversing the image using image blocks of a set size.

[0175] like Figure 5As described above, let the initial point of the first image in the i-th image pair be (0, 0), and let the height of the first image be height, the width be width, and the side length of the block be Q. The coordinates of the four vertices of the first image block are {(0, 0), (0, Q-1), (Q-1, 0), (Q-1, Q-1)}, as shown below. Figure 5 As shown, the vertex coordinates of the second image block, which is the block immediately to the right of the first block, are {(0, Q), (0, 2Q-1), (Q-1, Q), (Q-1, 2Q-1)}. Figure 5 The coordinates of the four vertices are not shown. The vertex coordinates of the block below the first block, i.e., the third image block, are {(Q, 0), (Q, Q-1), (2Q-1, 0), (2Q-1, Q-1)}. Figure 5 The coordinates of two vertices are shown. These are located at the right boundary, bottom boundary, and bottom right corner of the first image, as shown below. Figure 5 The fourth image block doesn't need to be a perfect square or exceed the bounds. Following this method, all image blocks can be traversed to obtain multiple image blocks corresponding to the first image. Then, the number of moving pixels in each image block needs to be calculated. As described above, determining the number of pixels in each image block also requires considering the image blocks obtained by dividing the second image according to the Q-size. The total number of moving pixels in the same image block is determined by comparing the differences between the median or center value of each pixel in the first and second images and the corresponding image block.

[0176] It is understandable that even if an image is divided into blocks using the same dimensions, the resulting image blocks may not be the same size. The size of an image block located at the upper right, lower right, or lower left corner of the image may be smaller than the size of other image blocks. In some embodiments of this application, the image may also be divided into blocks starting from the upper right corner, but the division strategy for each image pair must be the same in order to enable the combination operation of motion masks.

[0177] Please refer to Figure 6 , Figure 6 The present application illustrates a motion detection device according to an embodiment of the present application. It should be understood that this device is similar to the one described above. Figure 3 or Figure 4 Corresponding to the method embodiments, it can execute the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in the memory or embedded in the device's operating system in the form of software or firmware. The motion detection device includes: an image pair acquisition module 201, an image block division module 202, an image pair motion mask acquisition module 203, and a target motion mask acquisition module 204.

[0178] Figure 6 The image pair acquisition module 201 is configured to acquire the i-th image pair from the video sequence, wherein the i-th image pair includes a reference image of the i-th frame, and the reference image includes the (i+1)-th frame image adjacent to the i-th frame image or a fused image of the subsequent multiple frames adjacent to the i-th frame image, and the value of i is an integer greater than or equal to 1.

[0179] Figure 6 The image block division module 202 is configured to divide the two images included in the i-th image pair into multiple image blocks respectively.

[0180] Figure 6 The image pair motion mask acquisition module 203 is configured to acquire the target motion weights of all pixels on each image block to obtain the motion mask of the i-th image pair. The target motion weight of any pixel is acquired by the following strategy: the target motion weight of any pixel is obtained based on the difference between the pixel value of any pixel and the feature value of the image block in which the pixel is located.

[0181] Figure 6 The target motion mask acquisition module 204 is configured to fuse multiple motion masks to obtain a target motion mask.

[0182] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned motion detection method, and will not be elaborated further here.

[0183] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the methods described in any of the above embodiments.

[0184] like Figure 7 As shown, some embodiments of this application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads and executes the program via a bus 530, it can implement the methods described in any of the above embodiments.

[0185] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0186] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of the embodiments of this disclosure can be used to execute the instructions in the memory 510 to implement… Figure 3 The method shown. Memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0187] Some embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the motion detection method described in the above embodiments.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0189] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0190] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0193] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A motion detection method, characterized in that, The motion detection method includes: The i-th image pair is obtained from the video sequence, wherein the i-th image pair includes the i-th frame image and a reference image, and the reference image includes the (i+1)-th frame image adjacent to the i-th frame image or a fused image of the subsequent multiple frames adjacent to the i-th frame image, where i is an integer greater than or equal to 1. The two images in the i-th image pair are divided into multiple image blocks respectively; The target motion weights of all pixels in each image block are obtained to obtain the motion mask of the i-th image pair. The target motion weight of any pixel is obtained by the difference between the pixel value of any pixel and the feature value of the image block in which the pixel is located. The feature value includes at least one of the median and the center value. The difference is represented by a difference or a ratio. Repeat the above process of obtaining the i-th image pair until obtaining the motion mask of the i-th image pair to obtain motion masks corresponding to multiple image pairs respectively, and fuse all motion masks to obtain the target motion mask; The step of obtaining the target motion weights of all pixels in each image block includes: Based on the difference, it is confirmed that any pixel belongs to a moving pixel, and the total number of moving pixels included in the image block to which any pixel belongs is counted. The block weight coefficient of the image block to which any pixel is located is obtained based on the total number of moving pixels and the size of the image block to which any pixel is located; The target weight coefficient of any pixel is obtained based on the block weight coefficient; Wherein, confirming that any pixel belongs to a moving pixel based on the difference includes: In the i-th frame image, the feature value of any image block where any pixel point is located is obtained to obtain the i-th feature value, and the difference between the pixel value of any pixel point and the i-th feature value is calculated to obtain the i-th difference, wherein the i-th feature value includes the i-th median value and / or the i-th center value; On the reference image, the feature value of any image block is obtained to obtain the (i+1)th feature value, and the difference between the pixel value of any pixel and the (i+1)th feature value is calculated to obtain the (i+1)th difference, wherein the (i+1)th feature value includes the (i+1)th median value and / or the (i+1)th center value. If the change trend of any pixel point in the two images is not consistent based on the i-th difference and the (i+1)-th difference, then the pixel point is confirmed to be a moving pixel point.

2. The motion detection method as described in claim 1, characterized in that, The step of dividing the two images in the i-th image pair into multiple image blocks includes: The two images in the i-th image pair are divided into the plurality of image blocks according to a certain size, wherein any pixel is a first pixel and the first pixel is divided into the j-th image block, where j is an integer greater than or equal to 1; The step of confirming that any pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels in the image block containing any pixel, includes: Based on the difference, the first pixel is confirmed to be a moving pixel, and the total number of moving pixels included in the j-th image block is counted. The step of obtaining the block weight coefficient of the image block containing any given pixel based on the total number of moving pixels and the size of the image block containing any given pixel includes: The block weight coefficient is obtained by calculating the ratio of the total number of moving pixels in the j-th image block to the size of the j-th image block. The step of obtaining the target weight coefficient of any pixel based on the block weight coefficient includes: The ratio is used as the target weight coefficient for the first pixel.

3. The motion detection method as described in claim 1, characterized in that, The step of dividing the two images in the i-th image pair into multiple image blocks includes: The i-th image pair is divided into blocks according to each of the N sizes, resulting in the plurality of image blocks. Here, N is an integer greater than 1, and any pixel is a second pixel. The second pixel is divided into N target image blocks, and each target image block is obtained by dividing the two images using a size. The step of confirming that any pixel belongs to a moving pixel based on the difference, and counting the total number of moving pixels in the image block containing any pixel, includes: Based on the difference, it is confirmed that the second pixel belongs to a moving pixel, and the total number of moving pixels included in each image block in the N types of target image blocks is counted; The step of obtaining the block weight coefficient of the image block containing any given pixel based on the total number of moving pixels and the size of the image block containing any given pixel includes: Based on the total number of moving pixels in each of the N types of target image blocks and the size of the N types of target image blocks, the block weight coefficients of each type of target image block are solved to obtain N block weight coefficients; The step of obtaining the target weight coefficient of any pixel based on the block weight coefficient includes: The target weight coefficient of the second pixel is determined based on the N block weight coefficients and the size of each target image block among the N types of target image blocks.

4. The motion detection method as described in claim 3, characterized in that, The step of determining the target weight coefficient of the second pixel based on the N block weight coefficients and the size of each target image block among the N types of target image blocks includes: weighting the N block weight coefficients to obtain the target motion weight of any pixel, wherein the larger the size of the image block, the larger the weight coefficient.

5. The motion detection method as described in claim 4, characterized in that, If the two images are divided into blocks using different sizes Q, W and E respectively, then any pixel with coordinates r and c is divided into three target image blocks of sizes Q*Q, W*W and E*E respectively, corresponding to three block weight coefficients. The formula for calculating the target weight coefficient of any pixel is as follows: in, , and The three block weight coefficients represent any given pixel.

6. The motion detection method as described in claim 1, characterized in that, The i-th difference is obtained by calculating the difference between the pixel value of any pixel and the i-th feature value. The (i+1)th difference is obtained by calculating the difference between the pixel value of any pixel and the (i+1)th feature value; where... The step of confirming that any pixel is a moving pixel when the change trend of any pixel in the two images is inconsistent through the i-th difference and the (i+1)-th difference includes: if the positive and negative characteristics of the i-th difference and the (i+1)-th difference are opposite, then the pixel is confirmed as a moving pixel.

7. The motion detection method as described in claim 1, characterized in that, The i-th difference is obtained by calculating the ratio of the pixel value of any pixel point to the i-th feature value; The (i+1)th difference is obtained by calculating the ratio of the pixel value of any pixel to the (i+1)th feature value; where... The step of confirming that any pixel is a moving pixel when the change trend of any pixel in the two images is inconsistent through the i-th difference and the (i+1)-th difference includes: if the absolute value of the difference between the i-th ratio and the (i+1)-th ratio is less than a set threshold, then the pixel is confirmed as a moving pixel.

8. The motion detection method as described in claim 1, characterized in that, The block weight coefficient is determined by at least one of the block median weight coefficient and the block center value weight coefficient, wherein the block median weight coefficient is obtained by determining the difference between the pixel value of any pixel and the median of the image block in which the pixel is located, and the block center value weight coefficient is obtained by determining the difference between the pixel value of any pixel and the center value of the image block in which the pixel is located.

9. The motion detection method as described in claim 8, characterized in that, The two images are divided using size Q; wherein, The formula for calculating the block median weight coefficient is as follows: or, The formula for calculating the weighting coefficient of the block center value is: Where k represents the k-th image block, count1 represents the total number of moving pixels on the k-th image block as determined by the median of the k-th image block, and count2 represents the total number of moving pixels on the k-th image block as determined by the center value of the k-th image block. The value of k represents the total number of pixels in the k-th image block, m represents the normalization parameter, and k takes the value greater than or equal to 1 and less than or equal to the total number of image blocks obtained by dividing the image according to size Q.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it can implement the method described in any one of claims 1-9.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it can implement the method described in any one of claims 1-9.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the motion detection method according to any one of claims 1-9.

Images