Motion blur generation method, apparatus, device, storage medium, and program product

Abstract

The present application relates to a motion blur generation method, an apparatus, a device, a storage medium, and a program product. The method comprises: acquiring multiple image frames, and on the basis of the multiple image frames, determining an optical flow of an image frame to be processed; respectively performing down-sampling of different scales on said image frame and the optical flow of said image frame multiple times to obtain a plurality of feature maps and sampling optical flows corresponding to the feature maps; and performing vector blur on pixel points in the feature maps on the basis of the sampling optical flows to obtain a target motion blur image of said image frame. The use of the method can improve the motion blur image generation efficiency.

Description

Dynamic fuzzy generation method, device, equipment, storage medium and program product Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a dynamic blur generation method, apparatus, device, storage medium and program product. Background Art

[0002] With the development of science and computer technology, more and more users are demanding motion blur images with motion blur effects. Motion blur is actually image blur caused by the relative motion between the camera equipment and the photographed scene. Motion blur images cannot be captured directly and require processing of the captured images.

[0003] In traditional technology, the dynamic fuzzy map is obtained by calculating the pin insertion method. However, this method has a large amount of calculation and results in a slow calculation speed. Therefore, the traditional technology has the problem of low efficiency in generating dynamic fuzzy maps.

[0004] Summary of the Invention

[0005] Based on this, it is necessary to provide a dynamic blur generation method, device, equipment, storage medium and program product that can improve the generation efficiency of dynamic blur images in response to the above technical problems.

[0006] In the first aspect, the present application provides a dynamic blur generation method, which includes: acquiring multiple frames of images, determining the optical flow of the image frame to be processed based on the multiple frames of images; performing multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; performing vector blurring on the pixel points in each feature map according to the sampled optical flow to obtain a target dynamic blur map of the image frame.

[0007] In one embodiment, vector blurring is performed on pixels in each feature map according to the sampled optical flow to obtain a target dynamic blur map of the image frame, including: determining pixels to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map, and vector blurring the pixels to be blurred in the L-th layer feature map to obtain an L-th layer dynamic blur map; the L-th layer feature map has the smallest size; and according to the L-th layer dynamic blur map, vector blurring is performed on pixels in the feature maps of other layers in turn to obtain a target dynamic blur map.

[0008] In one embodiment, according to the L-th layer dynamic blur map, vector blur is performed on the pixel points in the feature maps of other layers in sequence to obtain a target dynamic blur map, including: performing blur operations on the feature maps of each layer in sequence to obtain a target dynamic blur map; the blur operation includes: performing a first preprocessing on the L-th layer dynamic blur map, and upsampling the preprocessed L-th layer dynamic blur map to obtain a sampled dynamic blur map of the L-1 layer; according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer, vector blur is performed on the feature map of the L-1 layer to obtain an initial dynamic blur map of the L-1 layer; according to the sampled dynamic blur map of the L-1 layer and the initial dynamic blur map of the L-1 layer, the dynamic blur map of the L-1 layer is determined.

[0009] In one embodiment, the first preprocessing is performed on the L-th layer dynamic fuzzy map, including: corroding the label values ​​corresponding to the L-th layer dynamic fuzzy map; filtering the corroded label values ​​to obtain the preprocessed L-th layer dynamic fuzzy map.

[0010] In one embodiment, according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer, the feature map of the L-1 layer is vector blurred to obtain the initial dynamic blur map of the L-1 layer, including: determining the first pixel point to be blurred in the feature map of the L-1 layer according to the label value corresponding to the sampled dynamic blur map of the L-1 layer; determining the pixel point in the feature map of the L-1 layer whose sampled optical flow is greater than the first sampling threshold as the second pixel point to be blurred in the feature map of the L-1 layer; and performing vector blur on the first pixel point and the second pixel point to obtain the initial dynamic blur map of the L-1 layer.

[0011] In one embodiment, the pixel points to be blurred in the L-th layer feature map are determined based on the sampled optical flow of the L-th layer feature map, including: determining the pixel points in the L-th layer feature map whose sampled optical flow is greater than a second sampling threshold as the pixel points to be blurred in the L-th layer feature map.

[0012] In one embodiment, vector blur is performed on the pixel points to be blurred in the L-th layer feature map to obtain the L-th layer dynamic blur map, including: determining the initial pixel value of each pixel point to be blurred in the L-th layer feature map in each direction according to the sampled optical flow in each direction of the L-th layer feature map; determining the pixel value of each pixel point to be blurred in the L-th layer feature map according to the initial pixel value of each pixel point to be blurred in the L-th layer feature map in each direction and a preset number of iterations to obtain the L-th layer dynamic blur map.

[0013] In one embodiment, the pixel value of each pixel to be blurred in the L-th layer feature map is determined according to the initial pixel value of each pixel to be blurred in each direction in the L-th layer feature map and the preset number of iterations, and the L-th layer dynamic blur map is obtained, including: according to the weight of each direction, the initial pixel value of each pixel to be blurred in the L-th layer feature map in each direction and the preset number of iterations, the pixel value of each pixel to be blurred in the L-th layer feature map is determined, and the L-th layer dynamic blur map is obtained.

[0014] In one embodiment, determining the optical flow of an image frame to be processed based on multiple frames of images includes: determining the initial optical flow of each pixel in the image frame based on the multiple frames of images; performing a second preprocessing on the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame.

[0015] In one embodiment, the second pre-processing includes at least one of median filtering, weighting processing, threshold processing, mask processing, and intensity adjustment processing.

[0016] In one embodiment, the second pre-processing further includes spatial domain smoothing.

[0017] In one embodiment, threshold processing includes: for each pixel in the image frame, setting the initial optical flow of the pixel whose absolute value is less than a first preset threshold to zero to obtain the processed optical flow of the pixel; performing an erosion operation on the processed optical flow of each pixel; and obtaining the optical flow of each pixel in the image frame based on the processed optical flow of each pixel after erosion and a second preset threshold.

[0018] In one embodiment, the mask processing includes: setting the initial optical flow of each pixel in the region of interest in the image frame to zero to obtain the optical flow of each pixel in the image frame.

[0019] In one embodiment, the initial pixel value of each pixel to be blurred in the L-th layer feature map in each direction is determined based on the sampled optical flow in each direction of the L-th layer feature map, including: determining multiple candidate pixel values ​​of each pixel to be blurred in each direction based on a preset number of iterations and the sampled optical flow in each direction of each pixel to be blurred in the L-th layer feature map; determining the maximum candidate pixel value in each direction as the initial pixel value of the third pixel in each direction; the third pixel is a pixel to be blurred whose pixel value is greater than a saturation threshold; determining the average value of the candidate pixel values ​​in each direction as the initial pixel value of the fourth pixel in each direction; the fourth pixel is a pixel to be blurred whose pixel value is not greater than a saturation threshold.

[0020] In one embodiment, the method further includes: detecting a fifth pixel point in the image frame, the fifth pixel point being a pixel point having a pixel value greater than a saturation threshold; performing vector blurring on the fifth pixel point according to the optical flow of the fifth pixel point to obtain a pixel value of the fifth pixel point; and superimposing the pixel value of the fifth pixel point and the pixel value of the pixel point corresponding to the fifth pixel point in the target dynamic blur map to obtain a dynamic blur map of the image frame.

[0021] In the second aspect, the present application provides a dynamic blur generation method, which includes: dividing the image frame into blocks to obtain multiple image blocks; blurring the pixel points in each image block according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame.

[0022] In one embodiment, blurring is performed on the pixels in each image block according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame, including: for the same image block, using the same optical flow to perform vector blurring on each pixel in the image block to obtain a target dynamic blur map.

[0023] In one embodiment, pixel points in each image block are blurred according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame, including: determining a target convolution kernel corresponding to each image block according to the optical flow of each image block; and convolving each pixel point in the corresponding image block according to each target convolution kernel to obtain a target dynamic blur map.

[0024] In a third aspect, the present application provides a method for generating a dynamic blur image of a panoramic image, the method comprising: expanding the panoramic image to obtain an expanded image; acquiring the optical flow of each pixel point of the expanded image, and obtaining the optical flow of the panoramic image based on the optical flow of each pixel point of the expanded image; blurring the panoramic image based on the optical flow of the panoramic image to obtain a target dynamic blur image of the panoramic image.

[0025] In one embodiment, a panoramic image is expanded to obtain an expanded image, including: determining a first image area in the panoramic image along a first direction, and determining a second image area in the panoramic image along a second direction; the first direction and the second direction are opposite; and splicing the first image area to an edge of the second image area, and splicing the second image area to an edge of the first image area.

[0026] In one embodiment, the optical flow of each pixel of the expanded image is obtained, and the optical flow of the panoramic image is obtained based on the optical flow of each pixel of the expanded image, including: determining the initial optical flow of each pixel in the first image area based on the optical flow of each pixel in the first image area and the optical flow of each pixel in the second image area spliced ​​to the first image area; determining the initial optical flow of each pixel in the second image area based on the optical flow of each pixel in the second image area and the optical flow of each pixel in the first image area spliced ​​to the second image area; determining the optical flow of the panoramic image based on the initial optical flow of each pixel in the first image area, the initial optical flow of each pixel in the second image area, and the optical flow of each pixel in a third image area in the panoramic image; the third image area is an image area between the first image area and the second image area.

[0027] In one embodiment, a panoramic image is blurred according to the optical flow of the panoramic image to obtain a target dynamic blurred map of the panoramic image, including: downsampling the panoramic image and the optical flow of the panoramic image multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; and vector blurring the pixels in each feature map according to the sampled optical flows to obtain a target dynamic blurred map.

[0028] In one embodiment, vector blurring is performed on pixels in each feature map according to the sampled optical flow to obtain a target dynamic blur map, including: vector blurring is performed on each pixel in the feature map according to the sampled optical flow in sequence according to a preset pixel order; the preset pixel order includes: increasing the width of the panoramic image along the first direction to the horizontal coordinate of the current pixel to determine the next pixel; or, subtracting the width of the panoramic image along the second direction to the horizontal coordinate of the current pixel to determine the next pixel; the first direction and the second direction are opposite.

[0029] In one embodiment, a panoramic image is blurred according to the optical flow of each pixel in the panoramic image to obtain a target dynamic blurred map of the panoramic image, including: dividing the panoramic image into blocks to obtain multiple image blocks; and blurring the pixels in each image block according to the optical flow of the panoramic image to obtain a target dynamic blurred map.

[0030] In a fourth aspect, the present application provides a method for generating a dynamic blur map of a panoramic image, the method comprising: acquiring multiple frames of images, determining the optical flow of the image frame to be processed based on the multiple frames of images, the image frame being an image selected by the user in the panoramic image after the camera moves, and the multiple frames of images being images having the same perspective as the image frame; blurring the image frame according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame.

[0031] In one embodiment, determining the optical flow of an image frame to be processed based on multiple image frames includes: determining the initial optical flow of each pixel in the image frame based on the multiple image frames; and preprocessing the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame.

[0032] In one embodiment, the preprocessing includes at least one of median filtering, weighting processing, threshold processing, mask processing, and intensity adjustment processing.

[0033] In one embodiment, the pre-processing further includes spatial domain smoothing.

[0034] In one embodiment, the image frame is blurred according to the optical flow of the image frame to obtain a target dynamic blurred map of the image frame, including: downsampling the image frame and the optical flow of the image frame multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; and vector blurring the pixels in each feature map according to the sampled optical flow to obtain the target dynamic blurred map.

[0035] In one embodiment, blurring the image frame according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame includes: dividing the image frame into blocks to obtain multiple image blocks; blurring each image block according to the optical flow of the image frame to obtain a target dynamic blur map.

[0036] In the fifth aspect, the present application provides an image processing method for simulating a slow shutter speed shooting effect, which is applied to a shooting device, and the method includes: switching to an electronic neutral density filter mode of the shooting device based on a user's interactive instruction; in the electronic neutral density filter mode, blurring the image frame taken by the shooting device so that the output image frame is an image simulating a slow shutter speed shooting effect.

[0037] In one embodiment, the photographing device stores a mapping table, which includes the correspondence between different electronic neutral density filter gears and algorithm parameters. Before blurring the image frames captured by the photographing device, the method also includes: obtaining the target electronic neutral density filter gear selected by the user, querying the mapping table according to the target electronic neutral density filter gear, and obtaining the target algorithm parameters; accordingly, blurring the image frames captured by the photographing device includes: blurring the image frames according to the target algorithm parameters.

[0038] In one embodiment, before processing the image frames captured by the shooting device, the method further includes: obtaining each image frame during the process of recording using the shooting device; accordingly, blurring the image frames captured by the shooting device, including: blurring each image frame.

[0039] In one embodiment, blurring is performed on image frames captured by a capturing device, including: acquiring an image frame sequence, the image frame sequence including multiple image frames captured in chronological order; blurring at least one image frame except the first and last two frames in the image frame sequence to obtain at least one blurred image frame, and superimposing the at least one blurred image frame.

[0040] In one embodiment, blurring is performed on at least one image frame in an image frame sequence except for the first and last two frames to obtain at least one blurred image frame, including: using the same optical flow, blurring is performed on at least one image frame in an image frame sequence except for the first and last two frames to obtain at least one blurred image frame.

[0041] In one embodiment, blurring the image frame captured by the capturing device includes blurring the pixels of non-interested areas in the image frame.

[0042] In one embodiment, image frames captured by a camera are blurred so that the output image frames are images that simulate slow shutter speed shooting effects, including: downsampling the image frames and the optical flows of the image frames multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; performing vector blurring on the pixels in each feature map according to the sampled optical flows to output a target dynamic blurred map of the image frame.

[0043] In one embodiment, vector blurring is performed on pixels in each feature map according to the sampled optical flow to output a target dynamic blur map of the image frame, including: determining pixels to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map, performing vector blurring on the pixels to be blurred in the L-th layer feature map to obtain an L-th layer dynamic blur map; the L-th layer feature map has the smallest size; and according to the L-th layer dynamic blur map, vector blurring is performed on pixels in the feature maps of other layers in turn to output a target dynamic blur map.

[0044] In one embodiment, according to the L-th layer dynamic blur map, vector blur is performed on the pixel points in the feature maps of other layers in sequence to output a target dynamic blur map, including: performing blur operation on the feature maps of each layer in sequence to output a target dynamic blur map; the blur operation includes: performing a first preprocessing on the L-th layer dynamic blur map, and upsampling the preprocessed L-th layer dynamic blur map to obtain a sampled dynamic blur map of the L-1 layer; according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer, vector blur is performed on the feature map of the L-1 layer to obtain an initial dynamic blur map of the L-1 layer; according to the sampled dynamic blur map of the L-1 layer and the initial dynamic blur map of the L-1 layer, the dynamic blur map of the L-1 layer is determined.

[0045] In one embodiment, blurring is performed on image frames captured by a shooting device so that the output image frames are images that simulate slow shutter speed shooting effects, including: dividing the image frames into blocks to obtain multiple image blocks; blurring the pixels in each image block according to the optical flow of the image frame, and outputting a target dynamic blur map of the image frame.

[0046] In one embodiment, blurring is performed on pixels in each image block according to the optical flow of the image frame, and a target dynamic blur map of the image frame is output, including: for the same image block, using the same optical flow to perform vector blurring on each pixel in the image block, and outputting the target dynamic blur map.

[0047] In one embodiment, pixel points in each image block are blurred according to the optical flow of the image frame, and a target dynamic blur map of the image frame is output, including: determining a target convolution kernel corresponding to each image block according to the optical flow of each image block; convolving each pixel point in the corresponding image block according to each target convolution kernel, and outputting the target dynamic blur map.

[0048] In the sixth aspect, the present application also provides a dynamic blur generation device, which includes: a determination module for acquiring multiple frames of images and determining the optical flow of the image frame to be processed based on the multiple frames of images; a sampling module for down-sampling the image frame and the optical flow of the image frame multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; a processing module for performing vector blur on the pixel points in each feature map according to the sampled optical flow to obtain a target dynamic blur map of the image frame.

[0049] In the seventh aspect, the present application also provides a dynamic blur generation device, which includes: a first processing module for performing block processing on the image frame to obtain multiple image blocks; a second processing module for blurring the pixel points in each image block according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame.

[0050] In the eighth aspect, the present application also provides a device for generating a dynamic blur image of a panoramic image, the device comprising: an expansion module for expanding the panoramic image to obtain the expanded image; a determination module for acquiring the optical flow of each pixel point of the expanded image, and obtaining the optical flow of the panoramic image based on the optical flow of each pixel point of the expanded image; a processing module for blurring the panoramic image according to the optical flow of the panoramic image to obtain a target dynamic blur image of the panoramic image.

[0051] In the ninth aspect, the present application also provides a device for generating a dynamic blur map of a panoramic image, the device comprising: a determination module for acquiring multiple frames of images, determining the optical flow of the image frame to be processed based on the multiple frames of images, the image frame being the image selected by the user in the panoramic image after the camera moves, and the multiple frames of images being the images with the same perspective as the image frame; a processing module for blurring the image frame according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame.

[0052] In the tenth aspect, the present application provides an image processing device that simulates a slow shutter shooting effect, which is applied to a shooting device, and the device includes: a switching module for switching to an electronic neutral density filter mode of the shooting device based on a user's interactive instructions; an output module for blurring the image frames taken by the shooting device in the electronic neutral density filter mode, so that the output image frames are images that simulate a slow shutter shooting effect.

[0053] In the eleventh aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect are implemented.

[0054] In the twelfth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect are implemented.

[0055] In the thirteenth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect, the second aspect, the third aspect, the fourth aspect, or the fifth aspect.

[0056] The above-mentioned dynamic blur generation method, device, equipment, storage medium and program product obtain multiple frames of images, determine the optical flow of the image frame to be processed based on the multiple frames of images, and then perform multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map. Then, according to the sampled optical flow, vector blur is performed on the pixel points in each feature map to obtain the target dynamic blur map of the image frame. In this way, vector blurring of each feature map has less computational complexity than the pin insertion method, thereby improving the generation efficiency of the dynamic blur map. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] FIG1 is a diagram illustrating an application environment of a dynamic fuzzy generation method according to an embodiment;

[0059] FIG2 is a schematic flow chart of a dynamic fuzzy generation method according to an embodiment;

[0060] FIG3 is a schematic diagram of a flow chart of a fuzzy operation in one embodiment;

[0061] FIG4 is a schematic diagram showing a principle of generating a dynamic fuzzy image in one embodiment;

[0062] FIG5 is a comparison diagram of a target dynamic fuzzy image obtained with and without threshold processing in one embodiment;

[0063] FIG6 is a schematic flow chart of another dynamic fuzzy generation method according to one embodiment;

[0064] FIG7 is a schematic diagram of 32 types of 17×17 convolution kernels in one embodiment;

[0065] FIG8 is a schematic diagram of a spherical panoramic image according to an embodiment;

[0066] FIG9 is a blurred image generated by a camera movement in one embodiment;

[0067] FIG10 is a fuzzy image of a fault at a connection between two ends in one embodiment;

[0068] FIG11 is a schematic flow chart of a method for generating a dynamic blur image of a panoramic image according to an embodiment;

[0069] FIG12 is a flow chart of another method for generating a dynamic blur image of a panoramic image according to an embodiment;

[0070] FIG13 is a schematic diagram showing a principle for generating a dynamic blur image of a panoramic image according to an embodiment;

[0071] FIG14 is a schematic flow chart of an image processing method for simulating a slow shutter speed shooting effect according to an embodiment;

[0072] FIG15 is a schematic diagram of images of three slow shutter speed shooting effects according to an embodiment;

[0073] FIG16 is a structural block diagram of a dynamic fuzzy generation device according to an embodiment;

[0074] FIG17 is a structural block diagram of another dynamic fuzzy generating device according to one embodiment;

[0075] FIG18 is a structural block diagram of a device for generating a dynamic blur image of a panoramic image according to an embodiment;

[0076] FIG19 is a structural block diagram of another apparatus for generating a dynamic blur image of a panoramic image according to an embodiment;

[0077] FIG20 is a structural block diagram of an image processing device for simulating slow shutter speed shooting effects according to an embodiment;

[0078] FIG21 is a diagram showing the internal structure of a computer device according to an embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0080] Currently, there are three methods to generate dynamic blur images:

[0081] The first method is the interpolation method, which inserts a pin between two video frames and then fuses these frames to generate a motion blur image. However, this method is computationally intensive and slow, resulting in low efficiency in generating motion blur images.

[0082] The second method is to blur the image frame with a line segment as the blur kernel based on the calculated motion vector to generate a dynamic blur map. This method is faster than the pin insertion method, but it is difficult to run in real time on a low-performance platform (such as an embedded board).

[0083] The third method is to use a neutral density filter on the lens and extend the exposure time to create a motion blur effect, resulting in a motion blur image. However, this requires a high level of expertise and the use of specialized equipment. Furthermore, when the shaking is severe, without physical image stabilization, a long shutter speed can cause unexpected shaking in all directions.

[0084] Based on this, it is necessary to propose effective technical means to solve the above problems. The following specific embodiments are used to describe in detail the technical solution of this application and how the technical solution of this application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0085] The dynamic fuzzy generation method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 via the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and cameras. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0086] As shown in Figure 2, a flow chart of a dynamic blur generation method is provided. This embodiment uses the method applied to the terminal in Figure 1 as an example. It is understood that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. The method includes the following steps 201, 202, and 203:

[0087] Step 201 : Acquire multiple frames of images, and determine the optical flow of the image frame to be processed based on the multiple frames of images.

[0088] The optical flow of the image frame includes the optical flow of each pixel in the image frame, and the optical flow of each pixel includes forward optical flow and backward optical flow. The forward optical flow and backward optical flow can be determined based on the images of the two frames before and after the image frame.

[0089] Optical flow is the motion vector of a pixel. For example, the optical flow of a pixel (x, y) can be expressed as flow(x, y) = (u, v), where flow(x, y) is the optical flow of the pixel (x, y), u is the speed in the x direction, and v is the speed in the y direction.

[0090] Optionally, two frames of images adjacent to the image frame are obtained, and the optical flow of the image frame to be processed is determined based on the two frames of images.

[0091] In step 202 , the image frame and the optical flow of the image frame are downsampled multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map.

[0092] Optionally, the image frame is downsampled multiple times at different scales, with the sampling factor of each downsampling being greater than the sampling factor of the previous downsampling, to obtain a feature map sequence, where the feature map sequence includes multiple feature maps arranged in descending order of size. Similarly, the optical flow of the image frame is downsampled multiple times at different scales, with the sampling factor of each downsampling being greater than the sampling factor of the previous downsampling, to obtain a sampled optical flow corresponding to each feature map, where the sampling factor is a preset value.

[0093] For example, the image frames and their optical flows are downsampled multiple times at different scales to obtain feature maps of the 0th layer, the 1st layer, ..., the L-1th layer, and the Lth layer, as well as the sampled optical flows corresponding to the feature maps of each layer, where the 0th layer is the image frame with the largest size, and the feature map of the Lth layer has the smallest size.

[0094] Step 203 , performing vector blurring on the pixels in each feature map according to the sampled optical flow to obtain a target dynamic blur map of the image frame.

[0095] Optionally, for the feature maps of each layer, the pixels whose sampled optical flow in the feature map of this layer is greater than the sampling threshold of this layer are used as the pixels to be blurred in this layer, and vector blur is performed on the pixels to be blurred in this layer to obtain the initial dynamic blur map of this layer. Starting from the L-1 layer, the initial dynamic blur map of the previous layer is upsampled and fused with the initial dynamic blur map of this layer to obtain the dynamic blur map of this layer. This step is repeated until the dynamic blur map of layer 0 is obtained, that is, the target dynamic blur map of the image frame is obtained.

[0096] For example, the initial dynamic fuzzy maps of the 0th, 1st, 2nd and 3rd layers are A0, A1, A2 and A3 respectively, then the dynamic fuzzy map of the 2nd layer is A2+A3=B2, the dynamic fuzzy map of the 1st layer is B2+A1=B1, and the dynamic fuzzy map of the 0th layer is B1+A0=B0.

[0097] In another optional embodiment, based on the sampled optical flow of the L-th layer feature map, the pixel points to be blurred in the L-th layer feature map are determined, and then vector blur is performed on the pixel points to be blurred in the L-th layer feature map to obtain the L-th layer dynamic blur map, and all the pixel points in the L-th layer feature map are marked to obtain the marking value corresponding to the L-th layer dynamic blur map.

[0098] The L-th layer dynamic blur map is upsampled to the resolution of the L-1 layer feature map (that is, upsampled to the same size as the L-1 layer feature map) to obtain the L-1 layer sampled dynamic blur map, and the label value corresponding to the L-1 layer dynamic blur map is upsampled to the resolution of the L-1 layer feature map to obtain the label value corresponding to the L-1 layer sampled dynamic blur map. According to the label value of the L-1 layer sampled dynamic blur map and the sampled optical flow corresponding to the L-1 layer feature map, the pixel points to be blurred in the L-1 layer feature map are determined, and vector blur is performed on the pixel points to be blurred in the L-1 layer feature map to obtain the L-1 layer initial dynamic blur map. The L-1 layer sampled dynamic blur map and the L-1 layer initial dynamic blur map are fused to obtain the L-1 layer dynamic blur map.

[0099] Then upsample the L-1th layer motion blur map to the resolution of the L-2th layer feature map, and perform similar steps to the previous paragraph adjacent to this paragraph to obtain the L-2th layer motion blur map. Similarly, the target motion blur map of the image frame can be obtained.

[0100] The above-mentioned dynamic blur generation method obtains multiple frames of images, determines the optical flow of the image frame to be processed based on the multiple frames of images, and then performs multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map. Then, vector blur is performed on the pixel points in each feature map according to the sampled optical flow to obtain the target dynamic blur map of the image frame. In this way, vector blurring of each feature map has less computational complexity than the pin insertion method, thereby improving the generation efficiency of the dynamic blur map.

[0101] In one embodiment, the above step 202, performing vector blurring on the pixels in each feature map according to the sampled optical flow to obtain a target dynamic blur map of the image frame, includes the following steps 2021, 2022, and 2023:

[0102] Step 2021: Determine the pixel points to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map.

[0103] Among them, the size of the L-th layer feature map is the smallest. The sampling optical flow of the L-th layer feature map includes the sampling optical flow of each pixel in the L-th layer feature map, and the sampling optical flow of each pixel includes the forward sampling optical flow and the backward sampling optical flow.

[0104] Optionally, step 2021 includes: determining the pixel points in the L-th layer feature map whose sampled optical flow is greater than the second sampling threshold as the pixel points to be blurred in the L-th layer feature map.

[0105] The implementation method may be that, for each pixel point in the L-th layer feature map, if the absolute value of the forward sampled optical flow or the absolute value of the backward sampled optical flow of the pixel point is greater than the second sampling threshold, then the pixel point is the pixel point to be blurred in the L-th layer feature map. Alternatively, for each pixel point in the L-th layer feature map, the absolute value of the forward sampled optical flow and the absolute value of the backward sampled optical flow of the pixel point are added to obtain the total sampled optical flow of the pixel point. If the total sampled optical flow of the pixel point is greater than the second sampling threshold, then the pixel point is the pixel point to be blurred in the L-th layer feature map.

[0106] Step 2022: Perform vector blur on the pixel points to be blurred in the L-th layer feature map to obtain the L-th layer dynamic blur map.

[0107] Optionally, step 2022 includes the following steps 20221 and 20222:

[0108] Step 20221: Determine the initial pixel value of each pixel to be blurred in each direction in the L-th layer feature map based on the sampled optical flow in each direction of the L-th layer feature map.

[0109] Step 20222: Determine the pixel value of each pixel to be blurred in the Lth layer feature map according to the initial pixel value of each pixel to be blurred in each direction in the Lth layer feature map and the preset number of iterations to obtain the Lth layer dynamic blur map.

[0110] Since the optical flow of the pixel point includes forward optical flow and backward optical flow, the pixel point (x, y) to be blurred has two directions of sampling optical flow, namely forward sampling optical flow f =(u f ,v f ) and backward sampling optical flow b =(u b ,v b ), where u f and v f are the speed in the forward x direction and the speed in the forward y direction, u b and v b are the speed in the backward x direction and the speed in the backward y direction respectively.

[0111] The preset number of iterations includes the first number of iterations N f and the second iteration number N b , where N f =min(max(|u f |,|v f |),N max ), N b =min(max(|u b |,|v b |),N max ), N max It is the preset maximum number of iterations, which is used to prevent the algorithm from being too computationally intensive and causing a sharp increase in time consumption when the motion amplitude is too large.

[0112] There are two ways to implement step 20221:

[0113] The first implementation method: According to formula (1), the initial pixel value of the pixel point (x, y) to be blurred in the forward direction is obtained According to formula (2), the initial pixel value of the pixel point (x, y) to be blurred in the backward direction is obtained (Δx f ,Δy f )=(|u f | / Nf ,|v f | / N f ), (Δx b ,Δy b )=(|u b | / N b ,|v b | / N b ).

[0114] The second implementation method is to determine multiple candidate pixel values ​​of each pixel to be blurred in each direction according to a preset number of iterations and the sampled optical flow of each pixel to be blurred in each direction in the L-th layer feature map; determine the maximum candidate pixel value in each direction as the initial pixel value of the third pixel in each direction; the third pixel is the pixel to be blurred whose pixel value is greater than the saturation threshold; determine the average value of the candidate pixel values ​​in each direction as the initial pixel value of the fourth pixel in each direction; the fourth pixel is the pixel to be blurred whose pixel value is not greater than the saturation threshold.

[0115] That is, according to formula (3), multiple candidate pixel values ​​of the pixel point (x, y) to be blurred in the forward direction are obtained; according to formula (4), multiple candidate pixel values ​​of the pixel point (x, y) to be blurred in the backward direction are obtained; {I(x+n*Δx f ,y+n*Δy f )|n=0,…,N f} (3) {I(x+n*Δx b ,y+n*Δy b )|n=0,…,N b} (4)

[0116] The meanings of the various quantities in formula (3) and formula (4) have been explained above and will not be repeated here.

[0117] Then, according to whether the pixel value of each pixel to be blurred in the L-th layer feature map is greater than the saturation threshold, all the pixels to be blurred in the L-th layer feature map are divided into two categories, one category is the third pixel point whose pixel value is greater than the saturation threshold, and the other category is the fourth pixel point whose pixel value is not greater than the saturation threshold.

[0118] For the third pixel, the initial pixel value in the forward direction is N in formula (3). f +1 candidate pixel value; the initial pixel value in the backward direction is N in formula (4) b +1 maximum candidate pixel value among the candidate pixel values;

[0119] For the fourth pixel point, the initial pixel value in the forward direction is the initial pixel value calculated according to formula (1); the initial pixel value in the backward direction is the initial pixel value calculated according to formula (2).

[0120] In real motion blur, saturated highlights also produce saturated lines. However, when using vector blur, the highlights are averaged with the surrounding area, resulting in non-saturated lines. Therefore, the second implementation method mentioned above can ensure that the lines produced by near-saturated highlights remain near saturated values.

[0121] There are two ways to implement step 20222:

[0122] The first implementation method is to calculate the pixel value of the pixel point (x, y) to be blurred according to formula (5) Alternatively, according to formula (6), the pixel value of the pixel point (x, y) to be blurred is calculated as

[0123] and According to formula (1) and formula (2), the meanings of other quantities have been explained above and will not be repeated here.

[0124] The second implementation method is to determine the pixel value of each pixel to be blurred in the L-th layer feature map according to the weight of each direction, the initial pixel value of each pixel to be blurred in each direction in the L-th layer feature map and the preset number of iterations to obtain the L-th layer dynamic blur map.

[0125] That is, according to formula (7), the pixel value of the pixel point (x, y) to be blurred is calculated

[0126] k f (n) is the weight of the pixel value of the pixel to be blurred in the forward direction at the nth iteration; k b (n) is the weight of the pixel value of the pixel to be blurred in the backward direction at the nth iteration.

[0127] Then, the L-th layer dynamic blur map is obtained based on the pixel values ​​of each pixel to be blurred obtained according to the above formula (5) or formula (6) or formula (7), as well as the pixel values ​​of the pixel points other than all the pixel points to be blurred.

[0128] Step 2023: Based on the Lth layer dynamic blur map, vector blur is performed on the pixel points in the feature maps of other layers in turn to obtain the target dynamic blur map of the image frame.

[0129] Optionally, step 2023 includes: performing a blur operation on the feature maps of each layer in sequence to obtain a target dynamic blur map of the image frame. As shown in FIG3 , a flow chart of a blur operation is provided, which includes steps 301, 302, and 303:

[0130] Step 301 : perform a first preprocessing on the Lth layer of dynamic fuzzy image, and upsample the preprocessed Lth layer of dynamic fuzzy image to obtain a sampled dynamic fuzzy image of the L-1th layer.

[0131] Optionally, step 301 includes step 3011, step 3012, and step 3013:

[0132] Step 3011, performing corrosion processing on the label value corresponding to the L-th layer dynamic fuzzy map.

[0133] Among them, the method for realizing the marking value corresponding to the L-th layer dynamic blur map can be to set the marking value mask of the pixel points in the L-th layer feature map whose sampling optical flow is greater than the second sampling threshold to the first threshold (for example, 1), and set the marking value mask of the pixel points in the L-th layer feature map whose sampling optical flow is less than or equal to the second sampling threshold to the second threshold (for example, 0), thereby obtaining the marking value corresponding to the L-th layer dynamic blur map.

[0134] Step 3012: filter the eroded mark value to obtain the Lth layer dynamic fuzzy image after preprocessing.

[0135] The filtering process includes mean filtering or Gaussian filtering.

[0136] Optionally, the eroded mark values ​​are subjected to mean / Gaussian filtering with the same radius, where the radius is a preset transition zone size.

[0137] The above steps 3011 and 3012 can ensure that the pixel points whose mark value mask is greater than the second threshold (for example, 0) are all pixel points that have been vector blurred in the Lth layer.

[0138] In order to further improve the performance, for steps 3011 and 3012, the mark values ​​corresponding to the L-th layer dynamic blur map can be downsampled to a preset resolution (the preset resolution can be very small), and then the downsampled mark values ​​are corroded, and then the corroded mark values ​​are filtered and then upsampled.

[0139] Step 3013 , up-sample the pre-processed L-th layer dynamic fuzzy image to obtain an L-1-th layer sampled dynamic fuzzy image.

[0140] Optionally, the preprocessed Lth layer dynamic fuzzy map is upsampled to the resolution of the feature map of the L-1 layer to obtain the sampled dynamic fuzzy map of the L-1 layer. The sampled dynamic fuzzy map of the L-1 layer includes two layers of information. The first layer of information is the pixel value after upsampling of each pixel point in the L-layer dynamic fuzzy map, that is, the pixel value corresponding to the sampled dynamic fuzzy map of the L-1 layer; the second layer of information is the label value after upsampling of each pixel point in the L-layer dynamic fuzzy map, that is, the label value corresponding to the sampled dynamic fuzzy map of the L-1 layer.

[0141] Step 302: Perform vector blur on the feature map of the L-1 layer according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer to obtain an initial dynamic blur map of the L-1 layer.

[0142] Optionally, step 302 includes step 3021, step 3022, and step 3023:

[0143] Step 3021: Determine the first pixel to be blurred in the feature map of the L-1 layer according to the label value corresponding to the sampled dynamic blur map of the L-1 layer.

[0144] Optionally, the sampling dynamic blur map of the L-1 layer has the same size as the feature map of the L-1 layer. For the pixel points in the feature map of the L-1 layer that correspond to the pixel points whose marking value is the first threshold (for example, 1) in the sampling dynamic blur map of the L-1 layer, vector blurring is not performed, and such pixel points are determined as pixels not to be blurred.

[0145] For the pixel points in the feature map of the L-1 layer corresponding to the pixel points whose label values ​​are not the first threshold (for example, 1) in the sampled dynamic blur map of the L-1 layer, for example, the pixel points whose label values ​​are between 0 and 1, vector blur is performed, and such pixel points are determined as the first pixel points to be blurred.

[0146] Step 3022: Determine the pixel point in the feature map of the L-1 layer whose sampling optical flow is greater than the first sampling threshold as the second pixel point to be blurred in the feature map of the L-1 layer.

[0147] Step 3023: Perform vector blur on the first pixel and the second pixel to obtain an initial dynamic blur image of the L-1th layer.

[0148] Optionally, vector blur is performed on the first pixel to obtain the pixel value of the blurred first pixel, vector blur is performed on the second pixel to obtain the pixel value of the blurred second pixel, and then the initial dynamic blur image of the L-1 layer is obtained based on the pixel value of the blurred first pixel, the pixel value of the blurred second pixel, and the pixel values ​​of all pixels except the first pixel and the second pixel in the feature map of the L-1 layer.

[0149] Among them, the implementation methods of vector blurring the first pixel point and vector blurring the second pixel point can refer to the implementation method of vector blurring the pixel point to be blurred in the Lth layer feature map in the above step 2022.

[0150] Step 303 : Determine the L-1th layer dynamic fuzzy map according to the L-1th layer sampled dynamic fuzzy map and the L-1th layer initial dynamic fuzzy map.

[0151] Optionally, the L-1 layer dynamic fuzzy map is determined based on the mark value of each pixel point in the L-1 layer sampled dynamic fuzzy map, the pixel value of each pixel point in the L-1 layer sampled dynamic fuzzy map and the pixel value of each pixel point in the L-1 layer initial dynamic fuzzy map.

[0152] That is, according to formula (8), the pixel value of the pixel point (x, y) in the L-1 layer dynamic blurred image is obtained, blured′ L-1 (x,y)=mask L (x,y)*blured L (x,y)+[1-mask L (x,y)]*blured L-1 (x,y) (8)

[0153] Among them, blurred L-1 (x, y) is the pixel value of the pixel point (x, y) in the L-1 layer dynamic blurred image, blurred L (x, y) is the pixel value of the pixel point (x, y) in the sampled dynamic blurred image of the L-1 layer, blurred L-1 (x, y) is the pixel value of the pixel point (x, y) in the initial dynamic blur image of the L-1 layer; mask L (x,y) is the label value of the pixel point (x,y) in the sampled dynamic blur image of the L-1th layer.

[0154] For the L-1th layer motion blur map and the L-2th layer feature map, similar to the L-1th layer motion blur map and the L-1th layer feature map, execute the above steps 301, 302 and 303 respectively to obtain the L-2th layer motion blur map, and so on until the 0th layer motion blur map is obtained, and the target motion blur map of the image frame can be obtained.

[0155] The principle behind generating the target dynamic blur map of an image frame is to generate blur on a multi-scale pyramid and then fuse it back into the original image. For ease of understanding, this application provides a dynamic blur map generation principle diagram, as shown in Figure 4. The specific implementation process has been detailed above and will not be repeated here.

[0156] In addition, in real motion blur, the lines drawn by saturated highlights are also saturated, while when using vector blur processing, the highlights are averaged with the surrounding area, and the effect produced is not a saturated line. Therefore, according to the method of calculating the pixel value of the pixel to be blurred according to formula (5) in step 20222, the target motion blur map of the image frame finally obtained needs further processing, that is,

[0157] The method also includes: detecting a fifth pixel point in the image frame, where the fifth pixel point is a pixel point whose pixel value is greater than a saturation threshold; performing vector blur on the fifth pixel point according to the optical flow of the fifth pixel point to obtain a pixel value of the fifth pixel point; and superimposing the pixel value of the fifth pixel point with the pixel value of the pixel point corresponding to the fifth pixel point in the target dynamic blur map to obtain a dynamic blur map of the image frame.

[0158] Optionally, the pixel value of the fifth pixel point and the pixel value of the pixel point corresponding to the fifth pixel point in the target dynamic blur map are superimposed, including: multiplying the pixel value of the fifth pixel point by a preset weight to obtain the initial pixel value of the fifth pixel point, and superimposing the initial pixel value of the fifth pixel point and the pixel value of the pixel point corresponding to the fifth pixel point in the target dynamic blur map.

[0159] In one embodiment, the above-mentioned step 201, determining the optical flow of the image frame to be processed based on multiple frames of images, includes: determining the initial optical flow of each pixel point in the image frame based on the multiple frames of images; performing a second preprocessing on the initial optical flow of each pixel point in the image frame to obtain the optical flow of each pixel point in the image frame.

[0160] Optionally, two frames of images adjacent to the image frame are obtained; and an initial optical flow of each pixel in the image frame is determined based on the two frames of images.

[0161] Among them, the image frame is the current frame image, and I t Indicates that the images of the two adjacent frames before and after the image frame are represented by I t-1 and I t+1 The initial optical flow of each pixel includes the initial optical flow in the forward direction and the initial optical flow in the backward direction, respectively represented by flow t→t-1 and flow t→t+1 express.

[0162] In order to improve the speed of calculating the initial optical flow, the image frame I t And the images of the two frames before and after I t-1 and I t+1 Reduce them separately, and then use the reduced image frame I t And the images of the two frames before and after the reduction I t-1 and I t+1, calculate the initial optical flow in the forward direction of each pixel in the image frame (i.e., the forward initial optical flow) flow t→t-1 , and the initial optical flow in the backward direction (ie, the backward initial optical flow) flow t→t+1 .

[0163] In order to further improve the speed of calculating the initial optical flow, when the image frame I t When it is not the first frame image, calculate flow according to formula (9) and formula (10) respectively t→t-1 and flow t→t+1 , flow t→t-1 =flow(t-1)+1→(t-1) (9) flow t→t+1 =-flow t+1→t (10)

[0164] Among them, flow(t-1)+1→(t-1) is the optical flow at time t0=t-1 Therefore, the initial optical flow in the forward direction of each pixel in the image frame is calculated t→t-1 , and the initial optical flow in the backward direction t→t+1 , you can only calculate the optical flow t+1→t .

[0165] Optionally, the second preprocessing includes at least one of median filtering, weighted processing, threshold processing, mask processing, and intensity adjustment processing; the second preprocessing also includes spatial domain smoothing processing.

[0166] In this embodiment, performing a second preprocessing on each initial optical flow can achieve a better motion blur effect. Generally, two situations can cause the added motion blur to appear jerky: first, the video's speed is uneven, sometimes fast and sometimes slow; second, unstable optical flow, with significant discrepancies in the initial optical flow calculation results in the front-to-back direction, and sometimes errors. Therefore, two temporal smoothing methods, median filtering and weighted processing, are proposed.

[0167] (1) Perform median filtering on the initial optical flow, that is, the optical flow of each pixel in each direction is the median of the initial optical flow in each direction from time t-Δt to time t+Δt.

[0168] (2) Perform weighted processing on the initial optical flow, that is, according to formula (11) and formula (2), calculate the optical flow in the forward direction of each pixel (i.e., forward optical flow) and the optical flow in the backward direction (i.e. backward optical flow)

[0169] Here, α is a preset optical flow weight, which can be 0.6, for example. Represents the backward optical flow calculated based on the frame image at time t-1 and the frame image at time t-2, flow t→t-1 represents the forward initial optical flow calculated based on the frame image at time t (i.e., image frame) and the frame image at time t-1, flow t+1→t It represents the forward initial optical flow calculated based on the frame image at time t+1 and the frame image at time t (i.e., image frame).

[0170] (3) In order to prevent the central area or the focused area of ​​the picture from being blurred and reducing the image quality, the initial optical flow can be thresholded, that is, threshold truncation can be performed. That is, for each pixel in the image frame, the initial optical flow of the pixel whose absolute value is less than the first preset threshold is set to zero to obtain the processed optical flow of the pixel; the processed optical flow of each pixel is eroded; and the optical flow of each pixel in the image frame is obtained based on the processed optical flow of each pixel after erosion and the second preset threshold.

[0171] Optionally, for each pixel's forward initial optical flow, the forward initial optical flow with an absolute value less than a preset threshold is set to 0, and then the initial optical flow is eroded so that the forward initial optical flow of each pixel is the optical flow with the smallest absolute value around it. This erosion operation can further ensure the clarity of the focused part of the subject. In order to prevent the dynamic blur mutation at the cutoff point, causing a sense of fault, the threshold coefficient α is obtained according to formula (13) thr , and then multiply the forward initial optical flow of each pixel after the corrosion operation by the threshold coefficient α thr , and obtain the forward optical flow of each pixel. For each pixel's backward initial optical flow, perform similar operations as for the forward initial optical flow of each pixel in this paragraph to obtain the backward optical flow of each pixel. The forward optical flow and backward optical flow of each pixel are the optical flow of each pixel.

[0172] Among them, α thr is the threshold coefficient, thr is the preset optical flow threshold, s is the preset transition band multiple, and flow is the forward initial optical flow of the pixel after the corrosion operation.

[0173] As shown in Figure 5, a comparison diagram of the target motion blur image obtained with and without threshold processing is provided, where 5(a) is the target motion blur image corresponding to the target motion blur image without threshold processing, and 5(b) is the target motion blur image corresponding to the target motion blur image with threshold processing. Comparing the positions of the arrows in 5(a) and 5(b), as well as the positions of the rectangular boxes on the right side of the camera in 5(a) and 5(b), it can be clearly seen that 5(b) is clearer than 5(a), that is, after setting the threshold, the visual focus area with less movement is clearer. Therefore, it can be confirmed that threshold processing of the initial optical flow can prevent the central area of ​​the picture or the focused area from being blurred and improve the image quality.

[0174] (4) Sometimes users have specific targets that they do not want to add blur. For example, when the camera is shooting around an object, dynamic blur is added only to the background (or a certain area) without adding blur to the main object (or another area). This can be achieved by masking the initial optical flow, that is, setting the initial optical flow of each pixel point in the area of ​​interest in the image frame to zero, and obtaining the optical flow of each pixel point in the image frame.

[0175] Optionally, a mask is used to cover the main object, and the initial optical flow (motion information) of the mask area is set to zero, wherein the mask can be obtained by manual selection or algorithm segmentation.

[0176] (5) In order to achieve the effect of adjusting the intensity of dynamic blur according to the application scenario or user preference, the intensity of the initial optical flow can be adjusted, that is, the initial optical flow is multiplied by a preset intensity multiple to obtain the optical flow, thereby achieving the effect of adjusting the intensity of dynamic blur.

[0177] (6) In order to prevent local mutations or distortions in dynamic blur (the main reason for local mutations or distortions is errors in optical flow calculation), and to improve the smoothness of the optical flow mutation boundary, after the initial optical flow is subjected to at least one of the above-mentioned median filtering, weighting processing, threshold processing, mask processing, and intensity adjustment processing, the optical flow is then subjected to spatial domain smoothing processing, wherein the spatial domain smoothing processing includes mean blur processing, Gaussian blur processing, or edge-preserving blur processing.

[0178] In order to enable the step of generating dynamic blur to be processed at high speed on the embedded board, as shown in FIG6 , the present application also provides a flow chart of another method for generating dynamic blur, which includes the following steps 601 and 602:

[0179] Step 601: perform block processing on the image frame to obtain multiple image blocks.

[0180] Optionally, the image frame is divided into a plurality of regions of the same size to obtain a plurality of image blocks of the same size, or the image frame is divided into a plurality of regions of different sizes to obtain a plurality of image blocks of different sizes, wherein the size of each image block can be expressed as n x ×n y .

[0181] Step 602 , blurring is performed on the pixels in each image block according to the optical flow of the image frame to obtain a target dynamic blur image of the image frame.

[0182] In an optional embodiment, step 602 includes: for the same image block, using the same optical flow to perform vector blur on each pixel in the image block to obtain a target dynamic blur map of the image frame.

[0183] The optical flow of each image block can be preset, or can refer to the above-mentioned embodiment of "determining the initial optical flow of each pixel in the image frame based on multiple frames of images; performing a second preprocessing on the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame" to obtain the optical flow of each pixel in the image frame. Then, for each image block, the optical flows of multiple pixels in the image block are averaged to obtain an average optical flow, which is used as the optical flow of the image block. Then, vector blur is performed on each pixel in the image block based on the average optical flow to obtain a dynamic blur map of the image block. The dynamic blur maps of multiple image blocks are spliced ​​to obtain a target dynamic blur map.

[0184] Performing vector blurring on each pixel in the image block based on the average optical flow includes: performing vector blurring on a sixth pixel in the image block based on the average optical flow to obtain a pixel value of the sixth pixel; and determining the pixel value of the sixth pixel as the pixel value of each seventh pixel in the image block. The implementation of vector blurring on the sixth pixel in the image block based on the average optical flow can refer to step 20222 above.

[0185] In this embodiment, by using the same optical flow to perform vector blur on each pixel in the same image block, the pixels in the same image block are processed in parallel, and the accelerated parallel processing of the media acceleration unit NEON inside the CPU (Central Processing Unit) of the embedded board is realized, so that the step of generating dynamic blur can be processed at high speed on the embedded board.

[0186] In another optional embodiment, step 602 includes: determining the target convolution kernel corresponding to each image block according to the optical flow of each image block; and convolving each pixel point in the corresponding image block according to each target convolution kernel to obtain a target dynamic blur map of the image frame.

[0187] The target convolution kernel corresponding to each image block is determined based on the optical flow of each image block. This includes matching the magnitude and direction of the optical flow of each image block with multiple pre-stored convolution kernels to obtain the target convolution kernel corresponding to each image block. Figure 7 shows a schematic diagram of 32 17×17 convolution kernels. The upper left corner of the 17x17 kernel in the figure is at (-8,-8), and the lower right corner is at (8,8).

[0188] For example, taking the center of the convolution kernel (i.e., point 0,0) as point (x0,y0), and considering the optical flow in both directions, the forward optical flow is (-10,-10) and the backward optical flow is (10,10). After adding them to point (0,0), the line connecting the two points is most similar in length and direction to convolution kernel 1_1 in Figure 7. For the forward optical flow of (5,-3) and the backward optical flow of (-3,1), after adding them to point (0,0), the line connecting the two points is most similar in length and direction to convolution kernel 7_11 in the figure.

[0189] Convolving each pixel point in the corresponding image block according to each target convolution kernel to obtain a target dynamic blur map of the image frame, including: for each image block, convolving the image block at least once with the target convolution kernel corresponding to the image block to obtain a dynamic blur map of the image block; splicing the dynamic blur maps of multiple image blocks to obtain the target dynamic blur map of the image frame.

[0190] In this embodiment, the image frame is blurred by using a convolution method, which is convenient for processing by the VPU (Video Processing Unit) on the embedded board, so that the step of generating dynamic blur can be processed at high speed on the embedded board.

[0191] The aforementioned dynamic blur generation method divides the image frame into blocks, generating multiple image blocks. Then, the pixels within each block are blurred based on the optical flow of the image frame to generate the target dynamic blur map. This block-based processing method can run in real time on low-performance platforms (such as embedded boards). Furthermore, the block-based processing method is computationally less complex than the pin-insertion method, thereby improving the efficiency of dynamic blur map generation.

[0192] A 360° spherical panoramic video is a 2:1 panoramic image. Its characteristics are: the top row represents the same point in the world, the bottom row represents the same point in the world, and the leftmost and rightmost ends are connected in the world. Panoramic videos are generally played for viewing, meaning that during the export process, users can freely select angles and move the camera as needed. As shown in Figure 8, a schematic diagram of a spherical panorama is provided. The spherical panorama is shown in the left figure of Figure 8. Unfolding the spherical panorama yields an image with an aspect ratio of 2:1, as shown in the right figure of Figure 8. According to the longitude-latitude unfolding method, the image width is the latitude 0-2π, and the image height is the longitude 0-π. Therefore, a spherical panorama can record all information for 360° horizontally and 180° vertically.

[0193] If motion blur is added to the exported 2D video, the camera movement will also introduce blur. This means that when the user pans the camera, the entire image will be severely blurred. This effect is sometimes desirable, but sometimes, users prefer to preserve the blur naturally introduced by the movement of objects in the world. Figure 9 shows a blur map generated by camera movement.

[0194] If you generate motion blur directly on a 2:1 spherical panorama, there will be issues such as discontinuity at the top, bottom, and sides. When playing back at these angles, there will be discontinuities. Figure 10 shows a blur image of the discontinuity at the two ends.

[0195] Therefore, the present application proposes the following method for generating motion blur on a 2:1 spherical panorama, that is, a method for generating a motion blur map of a panoramic image in a panoramic mode, as shown in FIG11 . The method includes the following steps 1101, 1102, and 1103:

[0196] Step 1101: Expand the panoramic image to obtain an expanded image.

[0197] The panoramic image is an image obtained by expanding the spherical panoramic image.

[0198] Optionally, a first image area is determined in the panoramic image along a first direction, and a second image area is determined in the panoramic image along a second direction; the first direction and the second direction are opposite; the first image area is spliced ​​to the edge of the second image area, and the second image area is spliced ​​to the edge of the first image area.

[0199] For example, a panoramic image includes areas A and B on both sides, and area C between A and B. A is spliced ​​to B, and B is spliced ​​to A. A' represents the area spliced ​​to B, and B' represents the area spliced ​​to A. The expanded image then includes areas B', A, C, B, and A'. Furthermore, for ease of understanding, the first direction may be from left to right in the image, and the second direction may be from right to left in the image.

[0200] Step 1102 : Obtain the optical flow of each pixel of the expanded image, and obtain the optical flow of the panoramic image based on the optical flow of each pixel of the expanded image.

[0201] Optionally, referring to the above-mentioned embodiment of "determining the initial optical flow of each pixel in the image frame based on multiple frames of images; performing a second preprocessing on the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame", the optical flow of each pixel in the panoramic image is obtained, thereby obtaining the optical flow of each pixel in the first image area, the optical flow of each pixel in the second image area, and the optical flow of each pixel in the third image area between the first image area and the second image area.

[0202] Obtaining the optical flow of the panoramic image according to the optical flow of each pixel of the expanded image includes the following steps 11021, 11022, and 11023:

[0203] Step 11021 : determining an initial optical flow of each pixel in the first image region according to the optical flow of each pixel in the first image region and the optical flow of each pixel in the second image region spliced ​​to the first image region.

[0204] This can be achieved by weightedly fusing the optical flow of each pixel in the first image area and the optical flow of each pixel in the second image area spliced ​​to the first image area to obtain the initial optical flow of each pixel in the first image area.

[0205] For example, the first image area and the second image area spliced ​​to the first image area are represented by A and B′ respectively. The column in A close to the edge of B′ is regarded as the first column of A, and so on along the first direction (from left to right), it is the second column, the third column, etc. of A; the column in B′ close to the edge of A is regarded as the first column of B′, and so on along the second direction (from right to left), it is the second column, the third column, etc. of B′.

[0206] The optical flow of each pixel in the first column of A is averaged with the optical flow of each pixel in the first column of B′ to obtain the initial optical flow of each pixel in the first column of A.

[0207] The initial optical flow of each pixel in the second column of A is obtained by multiplying the optical flow of each pixel in the second column of A by 0.6 + the optical flow of each pixel in the second column of B′ by 0.4. In other words, the optical flow weight of each pixel in each column of A increases as it moves along the first direction (to the right).

[0208] By analogy, we know that we get the initial optical flow of each pixel in the last column of A. Then, according to the initial optical flow of each pixel in each second column of A, we get the initial optical flow of each pixel in A.

[0209] Step 11022 : determining an initial optical flow of each pixel in the second image region according to the optical flow of each pixel in the second image region and the optical flow of each pixel in the first image region spliced ​​to the second image region.

[0210] This can be achieved by weightedly fusing the optical flow of each pixel in the second image area with the optical flow of each pixel in the first image area spliced ​​to the second image area to obtain the initial optical flow of each pixel in the second image area.

[0211] For example, the second image area and the first image area spliced ​​to the second image area are represented by B and A′ respectively. The column of B close to the edge of A′ is regarded as the first column of B, and so on along the second direction (from right to left), which are the second column, third column, etc. of B; the column of A′ close to the edge of B is regarded as the first column of A′, and so on along the first direction (from left to right), which are the second column, third column, etc. of A′.

[0212] The optical flow of each pixel in the first column of B is averaged with the optical flow of each pixel in the first column of A′ to obtain the initial optical flow of each pixel in the first column of B.

[0213] The initial optical flow of each pixel in the second column of B is obtained by multiplying the optical flow of each pixel in the second column of B by 0.6 + the optical flow of each pixel in the second column of A′ by 0.4. In other words, the optical flow weight of each pixel in each column of B increases as the direction moves to the left.

[0214] By analogy, we know that we get the initial optical flow of each pixel in the last column of B. Then, based on the initial optical flow of each pixel in each second column of B, we get the initial optical flow of each pixel in B.

[0215] Step 11023, determining the optical flow of the panoramic image based on the initial optical flow of each pixel point in the first image area, the initial optical flow of each pixel point in the second image area, and the optical flow of each pixel point in the third image area of ​​the panoramic image; the third image area is the image area between the first image area and the second image area.

[0216] In this embodiment, through the above steps 11021, 11022 and 11023, the optical flows at both ends of the scene image can be smoothly connected.

[0217] Step 1103 , blurring the panoramic image according to the optical flow of the panoramic image to obtain a target dynamic blur map of the panoramic image.

[0218] In one of the optional embodiments, step 1103 includes: downsampling the panoramic image and the optical flow of the panoramic image multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; performing vector blurring on the pixel points in each feature map according to the sampled optical flow to obtain a target dynamic blur map.

[0219] The implementation method can refer to the implementation method of the above steps 202 and 203.

[0220] In another optional embodiment, step 1103 includes: performing block processing on the panoramic image to obtain multiple image blocks; and performing blur processing on the pixels in each image block according to the optical flow of the panoramic image to obtain a target dynamic blur image.

[0221] The implementation method may refer to the implementation method of the above-mentioned step 601 and step 602.

[0222] The above-mentioned method for generating a target dynamic blur map for a panoramic image expands the panoramic image to obtain an expanded image, then obtains the optical flow of each pixel of the expanded image, and obtains the optical flow of the panoramic image based on the optical flow of each pixel of the expanded image. Then, the panoramic image is blurred based on the optical flow of the panoramic image to obtain the target dynamic blur map of the panoramic image. In this way, the optical flow of the panoramic image is obtained based on the optical flow of each pixel of the expanded image, which allows the optical flows at both ends of the panoramic image to be smoothly connected, thereby solving the problem of discontinuity in the panoramic image. Most importantly, this method can achieve a scene with no dynamic blur due to camera movement, retaining only the dynamic blur naturally generated by the movement of objects in the world.

[0223] It's also worth noting that for non-panoramic modes, the x-axis coordinates of the pixel endpoints are limited to 0 to width, where width is the width of the panoramic image. For panoramic mode, when x reaches -1 or width, x is increased by +width or -width. In other words, when x reaches a negative coordinate, it continues selecting points from the rightmost edge of the image to the left. For example, if the panoramic image width is 1000, when the -1 point is selected, the coordinate is changed to point 999, and when the -2 point is selected, the coordinate is changed to point 998.

[0224] Therefore, in panoramic mode, vector blur is performed on the pixels in each feature map according to the sampled optical flow to obtain a target dynamic blur map, including: vector blurring is performed on each pixel in the feature map according to the sampled optical flow in sequence according to the preset pixel order; the preset pixel order includes: increasing the width of the panoramic image by the horizontal coordinate of the current pixel along the first direction to determine the next pixel; or subtracting the width of the panoramic image by the horizontal coordinate of the current pixel along the second direction to determine the next pixel; the first direction and the second direction are opposite.

[0225] Furthermore, if motion blur is generated directly on a 2:1 spherical panorama, in addition to the aforementioned discontinuity issues, the following problems will also arise: ① Near the top and bottom, the panoramic unfolded image is severely distorted, making it easy to make errors in calculating optical flow; ② After the user finally selects the camera movement image, the finished film only uses a small portion of the 2:1 panorama, resulting in a lot of extra calculations.

[0226] Therefore, the present application proposes a method for generating a dynamic blur image after camera movement, that is, another method for generating a dynamic blur image of a panoramic image as shown in FIG12 . The method includes the following steps 1201, 1202, and 1203:

[0227] Step 1201 , obtaining multiple frames of images, where the multiple frames of images are images with the same viewing angle as the image frame, and the image frame is an image selected by the user in the panoramic image after the camera moves.

[0228] As shown in Figure 13, a schematic diagram of the principle of generating a dynamic blur image of a panoramic image is provided. The panoramic video is from time t0 to time t3, and the picture selected by the user moves from the right to the left, that is, the image frame F is selected at time t1. t1 , image frame G is selected at time t2 t2 .

[0229] If directly in F t1 , G t2 To generate dynamic blur that is not affected by the camera movement, it is necessary to render the image frame F at the same time at time t1. t1 and image frame F t1 The image F of the two adjacent frames t0 and F t2 , then F t1 、F t0 and F t2 Execute steps 1202 and 1203 to obtain the image frame F t1 Add motion blur to get image frame F t1 Similarly, it is necessary to render G at time t2. t1 , G t2 , G t3 , then for G t1, G t2 and G t3 Execute steps 1202 and 1203 to obtain the image frame G. t2 Add motion blur to get image frame G t2 Target motion blur map.

[0230] Step 1202 : determining the optical flow of the image frame to be processed based on the multiple image frames.

[0231] Optionally, step 1002 includes: determining an initial optical flow for each pixel in the image frame based on the multiple image frames; and preprocessing the initial optical flow for each pixel in the image frame to obtain an optical flow for each pixel in the image frame. The preprocessing includes at least one of median filtering, weighting, thresholding, masking, and intensity adjustment. The preprocessing also includes spatial smoothing.

[0232] The implementation method can refer to the above-mentioned embodiment of "determining the initial optical flow of each pixel in the image frame based on multiple frames of images; performing a second preprocessing on the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame".

[0233] Step 1203 , blurring the image frame according to the optical flow in the image frame to obtain a target dynamic blur map of the image frame.

[0234] In one of the optional embodiments, step 1203 includes: performing multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; performing vector blurring on the pixel points in each feature map according to the sampled optical flow to obtain a target dynamic blur map.

[0235] The implementation method can refer to the implementation method of the above steps 202 and 203.

[0236] In another optional embodiment, step 1203 includes: performing block processing on the image frame to obtain multiple image blocks; and performing blur processing on each image block according to the optical flow of the image frame to obtain a target dynamic blur map.

[0237] The implementation method may refer to the implementation method of the above-mentioned step 601 and step 602.

[0238] The above-mentioned method for generating a dynamic blur map of a panoramic image obtains multiple image frames and determines the optical flow of the image frame to be processed based on the multiple images. The image frame is the image selected by the user in the panoramic image after the camera movement, and the multiple images are images with the same perspective as the image frame. Then, the image frame is blurred based on the optical flow of the image frame to obtain the target dynamic blur map of the image frame. In this way, only the image frame selected by the user in the panoramic image after the camera movement is blurred. Compared with blurring the entire panoramic image, the computational complexity is smaller and the problem of large distortion of the panoramic unfolded image due to positions near the top and bottom, which can easily cause errors in optical flow calculation, is avoided. Most importantly, this method can achieve an image that does not produce dynamic blur due to camera movement, retaining only the dynamic blur naturally generated by the movement of objects in the world.

[0239] In response to the aforementioned problem of needing to wear neutral density filters and extend exposure time to capture motion blur effects, this application proposes an image processing method that simulates slow shutter speed photography effects. As shown in FIG14 , this method is applied to a photographing device and includes the following steps 1401 and 1402:

[0240] Step 1401: Based on the user's interactive instruction, switch to the electronic neutral density filter mode of the shooting device.

[0241] Optionally, when the shooting device detects a trigger operation for the electronic neutral density filter mode option, it switches to the electronic neutral density filter mode of the shooting device.

[0242] Step 1402 : In the electronic neutral density filter mode, blurring is performed on the image frames captured by the camera, so that the output image frames are images simulating a slow shutter speed shooting effect.

[0243] The scene of the image frame captured by the shooting device includes a dynamic object moving relative to the shooting device.

[0244] Optionally, the camera device integrates a motion blur image generation algorithm. When in electronic neutral density filter mode, the camera device can process the captured image frames based on the motion blur image generation algorithm. When the image frame includes a dynamic object, the dynamic object can be rendered smoother, resulting in the output processed image frame having a slow shutter speed effect. As shown in Figure 15, schematic diagrams of three images with slow shutter speed effects are provided. In Figure 15(a), the dynamic object is a tree, in Figure 15(b), it is a cloud, and in Figure 15(c), it is water.

[0245] In the above-mentioned image processing method for simulating the slow shutter speed shooting effect, the shooting device switches to the electronic neutral density filter mode of the shooting device based on the user's interactive instructions. In the electronic neutral density filter mode, the image frames shot by the shooting device are blurred so that the output image frames are images simulating the slow shutter speed shooting effect. In this way, the user only needs to send corresponding instructions to the shooting device, and the shooting device will output images with the simulated slow shutter speed shooting effect. Compared with a series of professional operations such as wearing a neutral density filter and extending the exposure time to achieve the effect of motion blur, the operation is simple, does not require high professional knowledge, and does not require the wearing of additional equipment. At the same time, it can also avoid unexpected jitter caused by a long shutter speed.

[0246] In one embodiment, the photographing device stores a mapping table, which includes the correspondence between different electronic neutral density filter gears and algorithm parameters. Before blurring the image frames captured by the photographing device, the method also includes: obtaining the target electronic neutral density filter gear selected by the user, querying the mapping table according to the target electronic neutral density filter gear, and obtaining the target algorithm parameters; accordingly, blurring the image frames captured by the photographing device includes: blurring the image frames according to the target algorithm parameters.

[0247] Optionally, the mapping table can be obtained through this experiment, that is, using two identical shooting devices to shoot side by side, where the first shooting device is equipped with a neutral density filter and reduces the exposure of the corresponding gear; the second shooting device is not equipped with a neutral density filter and uses normal exposure. The neutral density filter of different gears of the first shooting device is replaced multiple times to shoot multiple sets of videos. For each video shot with a neutral density filter of each gear, the corresponding video without a neutral density filter is sent to the dynamic blur image generation algorithm, and the algorithm parameters of the dynamic blur image generation algorithm (such as blur intensity, etc.) are adjusted to simulate the shooting effect of the neutral density filter of the gear. The algorithm parameters of the neutral density filters of multiple gears are obtained, and the algorithm parameters are fitted to the data to obtain the corresponding relationship between different gears and algorithm parameters, where the gear can be called the electronic neutral density filter gear.

[0248] Different electronic neutral density filter levels are displayed on the shooting device. When the user clicks on one of the electronic neutral density filter levels, the shooting device can obtain the target electronic neutral density filter level selected by the user, and then query the mapping table based on the target electronic neutral density filter level to obtain the target algorithm parameters. Therefore, when using the shooting device to record or take pictures, the shooting device can blur the image frame according to the target algorithm parameters and the dynamic blur map generation algorithm.

[0249] In this embodiment, by obtaining the target electronic neutral density filter position selected by the user, querying the mapping table according to the target electronic neutral density filter position, obtaining the target algorithm parameters, and blurring the image frame according to the target algorithm parameters, the effect of the neutral density filter can be further realistically simulated.

[0250] In one embodiment, before processing the image frames captured by the shooting device, the method further includes: obtaining each image frame during the process of recording using the shooting device; and correspondingly, blurring the image frames captured by the shooting device, including: blurring each image frame.

[0251] Optionally, the camera is integrated with a motion blur image generation algorithm. During the video recording process, if the camera detects a trigger operation for the electronic neutral density filter mode option, the camera will switch to the electronic neutral density filter mode of the camera. In the electronic neutral density filter mode, the camera will perform ISP (Image Signal Processing) and anti-shake processing on each image frame captured by the camera, and then blur the processed image frame based on the motion blur image generation algorithm to obtain a target motion blur image of the image frame and save it. Therefore, when the user finishes recording, the camera can directly output the target motion blur image of each image frame, thereby achieving a result equivalent to shooting with a neutral density filter without wearing a neutral density filter.

[0252] In this embodiment, during the process of recording video using a shooting device, each image frame is obtained and vector blurred. This achieves the effect of processing short-exposure video into a video with long exposure characteristics during the shooting process inside the shooting device, so that the target dynamic blur map of each image frame output by the shooting device has an effect similar to that of a video shot with a neutral density filter.

[0253] Furthermore, existing methods for synthesizing long-exposure images require the camera to capture a large number of photos in a short period of time, save them, and then send them to an external device for synthesis. Furthermore, to improve consistency, the external device still needs to insert frames into the photos. Finally, these photos are superimposed and synthesized to create a motion blurred image, which is then sent back to the camera. However, this synthesis method is time-consuming and cumbersome.

[0254] Therefore, this application proposes to integrate a motion blur image generation algorithm into the camera device to add motion blur to short-exposure photos, thereby simulating long-exposure photos taken with neutral density filters. The implementation method is as follows:

[0255] In one embodiment, blurring is performed on image frames captured by a capturing device, including: acquiring an image frame sequence, the image frame sequence including multiple image frames captured in chronological order; blurring at least one image frame except the first and last two frames in the image frame sequence to obtain at least one blurred image frame, and superimposing the at least one blurred image frame.

[0256] For example, three pictures are taken within the selected exposure time, and the order of these three pictures corresponds to the exposure start time, exposure center time, and exposure end time, respectively. The shooting device blurs the picture corresponding to the exposure center time based on the picture corresponding to the exposure start time and the picture corresponding to the exposure end time.

[0257] If you need to obtain higher quality photos, you can also take more pictures within the selected simulated exposure time, such as taking 5 pictures in the middle in addition to the first and last two pictures, for a total of 7 pictures; for each of the 5 pictures taken in the middle, the shooting device blurs the picture according to the two pictures before and after it, so as to obtain 5 blurred pictures, and then superimpose and fuse these 5 blurred pictures to obtain a target dynamic blur map, and output the target dynamic blur map, which has a slow shutter shooting effect.

[0258] In addition, if the performance of the shooting device's cache frame is high enough, the stacking fusion method can also be used to generate a target motion blur map to simulate the neutral density filter effect, that is, obtain an image frame sequence, superimpose multiple image frames in the image frame sequence, and obtain the target motion blur map.

[0259] The method superimposes multiple image frames in an image frame sequence to obtain a target dynamic fuzzy map, including superimposing multiple image frames according to weights corresponding to each image frame to obtain the target dynamic fuzzy map.

[0260] For example, if the camera caches at 360fps, you can generate a 30fps video by grouping 12 frames in the camera, or you can combine the 360 ​​frames cached within 1 second into a single photo in the camera and save it. The formula for combining a video frame or a photo is:

[0261] Among them, I result is a frame or a picture of the synthesized video, I n is the nth image frame involved in the synthesis, and w(n) is the weight of the nth image frame.

[0262] In one embodiment, blurring is performed on at least one image frame except the first and last two frames in an image sequence to obtain at least one blurred image frame, including: blurring is performed on at least one image frame except the first and last two frames in an image sequence using the same optical flow to obtain at least one blurred image frame.

[0263] For example, excluding the first and last two pictures, 5 pictures are taken in the middle, for a total of 7 pictures; in the process of blurring the 5 pictures taken in the middle, the optical flows of the 5 pictures are the same, and then the 5 blurred pictures are superimposed to obtain the target dynamic blur map.

[0264] In this embodiment, by using the same optical flow, relatively weak motion blur can be generated, and then superposition can be performed to avoid fish-scale faults.

[0265] In addition, after the video is encoded, for example, when it is uploaded to a video website or encoded and saved after shooting, the image quality generally degrades, especially for panoramic videos or ultra-wide-angle videos. Since the original data contains a large FOV (Field of View) image, the number of pixels allocated to the area of ​​interest to the user is small, resulting in lower image quality in the area of ​​interest. Therefore, this application proposes to add motion blur or other blur to the image to improve the image quality of the focused area / area of ​​interest after encoding. The implementation method is as follows:

[0266] In one embodiment, blurring the image frame captured by the capturing device includes blurring the pixels of non-interested areas in the image frame.

[0267] Optionally, when capturing an image, or before uploading a video to a video website, the user selects an uninterested area for the video or image, and after identifying the uninterested area, the pixels in the uninterested area are blurred.

[0268] After blurring, the details of the area are reduced and the blurred area is converted into a low-frequency area. Then, at the same target bit rate, since most of the details exist in the area without blurring, that is, the area of ​​interest, these areas of interest will be able to obtain higher encoding quality.

[0269] For example, during the video recording process, when the shooting device moves forward, the user selects the side of the moving direction as an area of ​​no interest. After blurring the pixels of the area of ​​no interest, dynamic blur will be added to the side of the moving direction, so that the picture in the moving direction will have higher image quality.

[0270] In one embodiment, image frames captured by a camera are blurred so that the output image frames are images that simulate slow shutter speed shooting effects, including: downsampling the image frames and the optical flows of the image frames multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; performing vector blurring on the pixels in each feature map according to the sampled optical flows to output a target dynamic blurred map of the image frame.

[0271] Optionally, vector blur is performed on the pixels in each feature map according to the sampled optical flow, and a target dynamic blur map of the image frame is output, including: determining the pixels to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map, and vector blurring the pixels to be blurred in the L-th layer feature map to obtain the L-th layer dynamic blur map; the size of the L-th layer feature map is the smallest; according to the L-th layer dynamic blur map, vector blurring is performed on the pixels in the feature maps of other layers in turn to output the target dynamic blur map.

[0272] Among them, according to the L-th layer dynamic fuzzy map, vector blur is performed on the pixel points in the feature maps of other layers in turn, and a target dynamic fuzzy map is output, including: performing a blur operation on the feature maps of each layer in turn, and outputting a target dynamic fuzzy map; the blur operation includes: performing a first preprocessing on the L-th layer dynamic fuzzy map, and upsampling the preprocessed L-th layer dynamic fuzzy map to obtain a sampled dynamic fuzzy map of the L-1 layer; according to the label value corresponding to the sampled dynamic fuzzy map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer, vector blur is performed on the feature map of the L-1 layer to obtain an initial dynamic fuzzy map of the L-1 layer; according to the sampled dynamic fuzzy map of the L-1 layer and the initial dynamic fuzzy map of the L-1 layer, the dynamic fuzzy map of the L-1 layer is determined.

[0273] The implementation of this embodiment refers to the implementation of the above steps 201, 202 and 203.

[0274] In one embodiment, blurring is performed on image frames captured by a shooting device so that the output image frames are images that simulate slow shutter speed shooting effects, including: dividing the image frames into blocks to obtain multiple image blocks; blurring the pixels in each image block according to the optical flow of the image frame, and outputting a target dynamic blur map of the image frame.

[0275] Optionally, the pixels in each image block are blurred according to the optical flow of the image frame, and a target dynamic blurred map of the image frame is output, including: for the same image block, using the same optical flow to perform vector blur on each pixel in the image block, and outputting a target dynamic blurred map.

[0276] Optionally, the pixel points in each image block are blurred according to the optical flow of the image frame, and a target dynamic blur map of the image frame is output, including: determining the target convolution kernel corresponding to each image block according to the optical flow of each image block; convolving each pixel point in the corresponding image block according to each target convolution kernel, and outputting the target dynamic blur map.

[0277] The implementation of this embodiment refers to the implementation of the above steps 601 and 602.

[0278] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0279] Based on the same inventive concept, embodiments of the present application also provide a dynamic blur generation device for implementing the aforementioned dynamic blur generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the dynamic blur generation device provided below can be found in the above-mentioned limitations of the dynamic blur generation method and will not be further elaborated here.

[0280] In one embodiment, as shown in FIG16 , a dynamic fuzzy generation device is provided. The dynamic fuzzy generation device 1600 includes: a determination module 1601 , a sampling module 1602 , and a processing module 1603 , wherein:

[0281] The determination module 1601 is configured to acquire multiple frames of images and determine the optical flow of the image frame to be processed based on the multiple frames of images.

[0282] The sampling module 1602 is configured to perform multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map.

[0283] The processing module 1603 is configured to perform vector blurring on the pixels in each feature map according to the sampled optical flow to obtain a target dynamic blur map of the image frame.

[0284] In one embodiment, the processing module 1603 is specifically used to determine the pixel points to be blurred in the L-th layer feature map based on the sampled optical flow of the L-th layer feature map, perform vector blur on the pixel points to be blurred in the L-th layer feature map, and obtain the L-th layer dynamic blur map; the size of the L-th layer feature map is the smallest; according to the L-th layer dynamic blur map, vector blur is performed on the pixel points in the feature maps of other layers in turn to obtain the target dynamic blur map.

[0285] In one embodiment, the processing module 1603 is specifically used to perform blur operations on the feature maps of each layer in sequence to obtain a target dynamic blur map; the processing module 1603 is specifically used to perform a first preprocessing on the dynamic blur map of the L-th layer, and upsample the preprocessed dynamic blur map of the L-th layer to obtain a sampled dynamic blur map of the L-1 layer; according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer, the feature map of the L-1 layer is vector blurred to obtain an initial dynamic blur map of the L-1 layer; based on the sampled dynamic blur map of the L-1 layer and the initial dynamic blur map of the L-1 layer, the dynamic blur map of the L-1 layer is determined.

[0286] In one embodiment, the processing module 1603 is specifically configured to perform an erosion process on the label values ​​corresponding to the L-th layer dynamic fuzzy image; and perform a filtering process on the eroded label values ​​to obtain a pre-processed L-th layer dynamic fuzzy image.

[0287] In one embodiment, the processing module 1603 is specifically used to determine the first pixel point to be blurred in the feature map of the L-1 layer based on the label value corresponding to the sampled dynamic blur map of the L-1 layer; determine the pixel point in the feature map of the L-1 layer whose sampled optical flow is greater than the first sampling threshold as the second pixel point to be blurred in the feature map of the L-1 layer; and perform vector blur on the first pixel point and the second pixel point to obtain the initial dynamic blur map of the L-1 layer.

[0288] In one embodiment, the processing module 1603 is specifically configured to determine the pixel points in the L-th layer feature map whose sampling optical flow is greater than the second sampling threshold as the pixel points to be blurred in the L-th layer feature map.

[0289] In one embodiment, the processing module 1603 is specifically used to determine the initial pixel value of each pixel to be blurred in the L-th layer feature map in each direction based on the sampled optical flow in each direction of the L-th layer feature map; determine the pixel value of each pixel to be blurred in the L-th layer feature map based on the initial pixel value of each pixel to be blurred in the L-th layer feature map in each direction and a preset number of iterations to obtain the L-th layer dynamic blur map.

[0290] In one embodiment, the processing module 1603 is specifically used to determine the pixel value of each pixel to be blurred in the Lth layer feature map based on the weights of each direction, the initial pixel value of each pixel to be blurred in each direction in the Lth layer feature map and the preset number of iterations, so as to obtain the Lth layer dynamic blur map.

[0291] In one embodiment, the determination module 1601 is specifically configured to determine the initial optical flow of each pixel in the image frame based on multiple frames of images; and perform a second preprocessing on the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame.

[0292] In one embodiment, the second pre-processing includes at least one of median filtering, weighting processing, threshold processing, mask processing, and intensity adjustment processing.

[0293] In one embodiment, the second pre-processing further includes spatial domain smoothing.

[0294] In one embodiment, the determination module 1601 is specifically used to set the initial optical flow of each pixel point in the image frame whose absolute value is less than a first preset threshold to zero to obtain the processed optical flow of the pixel point; perform an erosion operation on the processed optical flow of each pixel point; and obtain the optical flow of each pixel point in the image frame based on the processed optical flow of each pixel point after erosion and the second preset threshold.

[0295] In one embodiment, the determination module 1601 is specifically configured to set the initial optical flow of each pixel in the region of interest in the image frame to zero, to obtain the optical flow of each pixel in the image frame.

[0296] In one embodiment, the processing module 1603 determines multiple candidate pixel values ​​of each pixel to be blurred in each direction based on a preset number of iterations and the sampled optical flow of each pixel to be blurred in each direction in the L-th layer feature map; determines the maximum candidate pixel value in each direction as the initial pixel value of the third pixel in each direction; the third pixel is a pixel to be blurred whose pixel value is greater than a saturation threshold; determines the average value of the candidate pixel values ​​in each direction as the initial pixel value of the fourth pixel in each direction; and the fourth pixel is a pixel to be blurred whose pixel value is not greater than a saturation threshold.

[0297] In one embodiment, the processing module 1603 is further used to detect the fifth pixel point in the image frame, where the fifth pixel point is a pixel point whose pixel value is greater than a saturation threshold; perform vector blur on the fifth pixel point according to the optical flow of the fifth pixel point to obtain the pixel value of the fifth pixel point; and superimpose the pixel value of the fifth pixel point and the pixel value of the pixel point corresponding to the fifth pixel point in the target dynamic blur map to obtain a dynamic blur map of the image frame.

[0298] In one embodiment, as shown in FIG17 , another dynamic fuzzy generation device is provided. The dynamic fuzzy generation device 1700 includes: a first processing module 1701 and a second processing module 1702 , wherein:

[0299] The first processing module 1701 is used to perform block processing on the image frame to obtain multiple image blocks;

[0300] The second processing module 1702 is configured to perform blur processing on the pixels in each image block according to the optical flow of the image frame to obtain a target dynamic blur image of the image frame.

[0301] In one embodiment, the second processing module 1702 is specifically configured to perform vector blurring on each pixel in the same image block using the same optical flow to obtain a target dynamic blur image.

[0302] In one embodiment, the second processing module 1702 is specifically used to determine the target convolution kernel corresponding to each image block according to the optical flow of each image block; and convolve each pixel point in the corresponding image block according to each target convolution kernel to obtain a target dynamic blur map.

[0303] Based on the same inventive concept, embodiments of the present application also provide a device for generating a dynamic blurred image of a panoramic image, which is used to implement the aforementioned method for generating a dynamic blurred image of a panoramic image. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more embodiments of the device for generating a dynamic blurred image of a panoramic image provided below can be found in the aforementioned limitations of the method for generating a dynamic blurred image of a panoramic image, and will not be further elaborated here.

[0304] In one embodiment, as shown in FIG18 , a device for generating a dynamic blur image of a panoramic image is provided. The device 1800 for generating a dynamic blur image of a panoramic image includes: an expansion module 1801 , a determination module 1802 , and a processing module 1803 , wherein:

[0305] The expansion module 1801 is used to expand the panoramic image to obtain an expanded image.

[0306] The determination module 1802 is configured to obtain the optical flow of each pixel of the expanded image, and obtain the optical flow of the panoramic image based on the optical flow of each pixel of the expanded image.

[0307] The processing module 1803 is configured to perform blur processing on the panoramic image according to the optical flow of the panoramic image to obtain a target dynamic blur map of the panoramic image.

[0308] In one embodiment, the expansion module 1801 is specifically used to determine a first image area in the panoramic image along a first direction, and to determine a second image area in the panoramic image along a second direction; the first direction and the second direction are opposite; and the first image area is spliced ​​to the edge of the second image area, and the second image area is spliced ​​to the edge of the first image area.

[0309] In one embodiment, the determination module 1802 is specifically used to determine the initial optical flow of each pixel in the first image area based on the optical flow of each pixel in the first image area and the optical flow of each pixel in the second image area spliced ​​to the first image area; determine the initial optical flow of each pixel in the second image area based on the optical flow of each pixel in the second image area and the optical flow of each pixel in the first image area spliced ​​to the second image area; determine the optical flow of the panoramic image based on the initial optical flow of each pixel in the first image area, the initial optical flow of each pixel in the second image area, and the optical flow of each pixel in the third image area in the panoramic image; the third image area is the image area between the first image area and the second image area.

[0310] In one embodiment, the processing module 1803 is specifically used to perform multiple downsampling of the panoramic image and the optical flow of the panoramic image at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; and perform vector blurring on the pixel points in each feature map according to the sampled optical flows to obtain a target dynamic blur map.

[0311] In one embodiment, the processing module 1803 is specifically used to perform vector blur on each pixel in the feature map according to the sampled optical flow in the order of preset pixel points; the order of preset pixel points includes: increasing the width of the panoramic image by the horizontal coordinate of the current pixel point along the first direction to determine the next pixel point; or subtracting the width of the panoramic image from the horizontal coordinate of the current pixel point along the second direction to determine the next pixel point; the first direction and the second direction are opposite.

[0312] In one embodiment, the processing module 1803 is specifically configured to perform block processing on the panoramic image to obtain a plurality of image blocks; and perform blur processing on the pixels in each image block according to the optical flow of the panoramic image to obtain a target dynamic blur image.

[0313] In one embodiment, as shown in FIG19 , a device for generating a dynamic blur image of a panoramic image is provided. The device 1900 for generating a dynamic blur image of a panoramic image includes: a determination module 1901 and a processing module 1902 , wherein:

[0314] The determination module 1901 is used to obtain multiple frames of images and determine the optical flow of the image frame to be processed based on the multiple frames of images. The image frame is the image selected by the user in the panoramic image after the camera moves, and the multiple frames of images are images with the same viewing angle as the image frame.

[0315] The processing module 1902 is configured to perform blur processing on the image frame according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame.

[0316] In one embodiment, the determination module 1901 is specifically configured to determine the initial optical flow of each pixel in the image frame based on multiple frames of images; and preprocess the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame.

[0317] In one embodiment, the preprocessing includes at least one of median filtering, weighting processing, threshold processing, mask processing, and intensity adjustment processing.

[0318] In one embodiment, the pre-processing further includes spatial domain smoothing.

[0319] In one embodiment, the processing module 1902 is specifically used to perform multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; and perform vector blurring on the pixel points in each feature map according to the sampled optical flows to obtain a target dynamic blur map.

[0320] In one embodiment, the processing module 1902 is specifically configured to perform block processing on the image frame to obtain a plurality of image blocks; and perform blur processing on each image block according to the optical flow of the image frame to obtain a target dynamic blur image.

[0321] Based on the same inventive concept, embodiments of the present application further provide an image processing device for simulating slow shutter speed photography, for implementing the aforementioned image processing method for simulating slow shutter speed photography. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the image processing device for simulating slow shutter speed photography provided below can be found in the aforementioned limitations of the image processing method for simulating slow shutter speed photography, and will not be further elaborated here.

[0322] In one embodiment, as shown in FIG20 , an image processing device for simulating a slow shutter speed shooting effect is provided. The image processing device 2000 for simulating a slow shutter speed shooting effect includes: a switching module 2001 and an output module 2002. Both the switching module 2001 and the output module 2002 are provided in a shooting device, wherein:

[0323] The switching module 2001 is configured to switch to the electronic neutral density filter mode of the photographing device based on a user's interactive instruction.

[0324] The output module 2002 is configured to perform blur processing on the image frames captured by the camera in the electronic neutral density filter mode, so that the output image frames are images simulating a slow shutter speed shooting effect.

[0325] In one embodiment, the photographing device stores a mapping table including correspondences between different electronic neutral density filter positions and algorithm parameters. The image processing device simulating a slow shutter speed photographing effect further includes a determination module for obtaining a target electronic neutral density filter position selected by a user, querying the mapping table based on the target electronic neutral density filter position, and obtaining target algorithm parameters. Accordingly, the output module 2002 is specifically configured to perform blur processing on the image frame based on the target algorithm parameters.

[0326] In one embodiment, the image processing device for simulating slow shutter shooting effects further includes an acquisition module for acquiring each image frame during the process of recording using a shooting device; correspondingly, the output module 2002 is specifically used to blur each image frame.

[0327] In one embodiment, the output module 2002 is specifically used to obtain an image frame sequence, which includes multiple image frames obtained by continuous shooting and arranged in chronological order; blurring at least one image frame except the first and last two frames in the image frame sequence to obtain at least one blurred image frame, and superimposing the at least one blurred image frame.

[0328] In one embodiment, the output module 2002 is specifically configured to perform blur processing on at least one image frame except the first and last two frames in the image frame sequence using the same optical flow to obtain at least one blurred image frame.

[0329] In one embodiment, the output module 2002 is specifically configured to perform blur processing on pixels in non-interest areas in the image frame.

[0330] In one embodiment, the output module 2002 is specifically used to perform multiple downsampling of the image frame and the optical flow of the image frame at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; perform vector blurring on the pixel points in each feature map according to the sampled optical flow, and output a target dynamic blur map of the image frame.

[0331] In one embodiment, the output module 2002 is specifically used to determine the pixel points to be blurred in the L-th layer feature map based on the sampled optical flow of the L-th layer feature map, perform vector blur on the pixel points to be blurred in the L-th layer feature map, and obtain the L-th layer dynamic blur map; the size of the L-th layer feature map is the smallest; according to the L-th layer dynamic blur map, vector blur is performed on the pixel points in the feature maps of other layers in turn, and the target dynamic blur map is output.

[0332] In one embodiment, the output module 2002 is specifically used to perform blur operations on the feature maps of each layer in sequence to output a target dynamic blur map; the output module 2002 is specifically used to perform a first preprocessing on the dynamic blur map of the L-th layer, and upsample the preprocessed dynamic blur map of the L-th layer to obtain a sampled dynamic blur map of the L-1 layer; according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer, the feature map of the L-1 layer is vector blurred to obtain an initial dynamic blur map of the L-1 layer; according to the sampled dynamic blur map of the L-1 layer and the initial dynamic blur map of the L-1 layer, the dynamic blur map of the L-1 layer is determined.

[0333] In one embodiment, the output module 2002 is specifically configured to perform block processing on the image frame to obtain multiple image blocks; blur the pixels in each image block according to the optical flow of the image frame, and output a target dynamic blur map of the image frame.

[0334] In one embodiment, the output module 2002 is specifically configured to perform vector blur on each pixel in the same image block using the same optical flow, and output a target dynamic blur image.

[0335] In one embodiment, the output module 2002 is specifically used to determine the target convolution kernel corresponding to each image block according to the optical flow of each image block; convolve each pixel point in the corresponding image block according to each target convolution kernel, and output a target dynamic blur map.

[0336] Each module in each of the above-mentioned devices may be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0337] In one embodiment, a computer device is provided, which may be a terminal. A diagram of its internal structure may be shown in FIG21 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and an external device. The communication interface of the computer device is configured to communicate with an external terminal via wired or wireless communication, where the wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements the method of any of the above-described embodiments. The display unit of the computer device is configured to produce a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0338] Those skilled in the art will understand that the structure shown in Figure 21 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0339] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of any of the above method embodiments when executing the computer program.

[0340] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0341] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of any of the above method embodiments when executed by a processor.

[0342] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0343] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0344] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A dynamic fuzzy generation method, It is characterized in that The method comprises: Acquire multiple frames of images, and determine the optical flow of the image frame to be processed according to the multiple frames of images; Downsampling the image frame and the optical flow of the image frame multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; Vector blur is performed on the pixel points in each of the feature maps according to the sampled optical flow to obtain a target dynamic blur map of the image frame.

2. The method according to claim 1, It is characterized in that The step of performing vector blurring on the pixel points in each of the feature maps according to the sampled optical flow to obtain a target dynamic blur map of the image frame includes: Determine the pixel points to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map, perform vector blur on the pixel points to be blurred in the L-th layer feature map, and obtain the L-th layer dynamic blur map; the size of the L-th layer feature map is the smallest; According to the Lth layer dynamic fuzzy map, vector blur is performed on the pixel points in the feature maps of other layers in turn to obtain the target dynamic fuzzy map.

3. The method according to claim 2, It is characterized in that According to the L-th layer dynamic fuzzy map, vector blurring is performed on the pixel points in the feature maps of other layers in sequence to obtain the target dynamic fuzzy map, including: Performing fuzzy operation on the feature map of each layer in turn to obtain the target dynamic fuzzy map; The fuzzy operation includes: Performing a first preprocessing on the L-th layer of dynamic fuzzy image, and upsampling the preprocessed L-th layer of dynamic fuzzy image to obtain a sampled dynamic fuzzy image of the L-1th layer; According to the label value corresponding to the sampled dynamic fuzzy image of the L-1th layer and the sampled optical flow of the feature image of the L-1th layer, the feature image of the L-1th layer is vector-blurred to obtain an initial dynamic fuzzy image of the L-1th layer; The L-1th layer dynamic fuzzy map is determined according to the L-1th layer sampled dynamic fuzzy map and the L-1th layer initial dynamic fuzzy map.

4. The method according to claim 3, It is characterized in that The first preprocessing of the L-th layer dynamic fuzzy image includes: Performing corrosion processing on the label value corresponding to the L-th layer dynamic fuzzy image; The eroded mark value is filtered to obtain the L-th layer dynamic fuzzy image after the preprocessing.

5. The method according to claim 3, It is characterized in that The step of performing vector blurring on the feature map of the L-1 layer according to the label value corresponding to the sampled dynamic blur map of the L-1 layer and the sampled optical flow of the feature map of the L-1 layer to obtain the initial dynamic blur map of the L-1 layer includes: Determine the first pixel to be blurred in the feature map of the L-1 layer according to the label value corresponding to the sampled dynamic blur map of the L-1 layer point; Determine a pixel point in the feature map of the L-1th layer whose sampling optical flow is greater than a first sampling threshold as a second pixel point to be blurred in the feature map of the L-1th layer; Vector blur is performed on the first pixel point and the second pixel point to obtain an initial dynamic blur image of the L-1th layer.

6. The method according to claim 2, It is characterized in that The step of determining the pixel points to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map comprises: The pixel points whose sampling optical flow in the L-th layer feature map is greater than the second sampling threshold are determined as the pixel points to be blurred in the L-th layer feature map.

7. The method according to any one of claims 2 to 6, It is characterized in that The step of performing vector blurring on the pixel points to be blurred in the L-th layer feature map to obtain the L-th layer dynamic blur map comprises: Determine the initial pixel value of each pixel to be blurred in the L-th layer feature map in each direction according to the sampled optical flows in each direction of the L-th layer feature map; According to the initial pixel values ​​of each pixel to be blurred in the L-th layer feature map in each direction and the preset number of iterations, the pixel value of each pixel to be blurred in the L-th layer feature map is determined to obtain the L-th layer dynamic blur map.

8. The method according to claim 7, It is characterized in that The method of determining the pixel value of each pixel to be blurred in the L-th layer feature map according to the initial pixel value of each pixel to be blurred in each direction in the L-th layer feature map and the preset number of iterations to obtain the L-th layer dynamic blur map comprises: According to the weights of each direction, the initial pixel values ​​of each pixel to be blurred in the L-th layer feature map in each direction and the preset number of iterations, the pixel values ​​of each pixel to be blurred in the L-th layer feature map are determined to obtain the L-th layer dynamic blur map.

9. The method according to claim 1, It is characterized in that The step of determining the optical flow of the image frame to be processed according to the multiple image frames comprises: Determine the initial optical flow of each pixel in the image frame according to the multiple frames of images; A second preprocessing is performed on the initial optical flow of each pixel in the image frame to obtain the optical flow of each pixel in the image frame.

10. The method according to claim 9, It is characterized in that The second preprocessing includes at least one of median filtering, weighted processing, threshold processing, mask processing, and intensity adjustment processing.

11. The method according to claim 9, It is characterized in that The second preprocessing further includes spatial domain smoothing processing.

12. The method according to claim 10, It is characterized in that The threshold processing includes: For each pixel in the image frame, the initial optical flow of the pixel whose absolute value is less than the first preset threshold is set to zero, and the Processing optical flow of pixels; Performing an erosion operation on the processed optical flow of each of the pixel points; The optical flow of each pixel in the image frame is obtained according to the processed optical flow of each pixel after corrosion and the second preset threshold.

13. The method according to claim 10, It is characterized in that The mask processing includes: The initial optical flow of each pixel point in the region of interest in the image frame is set to zero to obtain the optical flow of each pixel point in the image frame.

14. The method according to claim 7, It is characterized in that Determining the initial pixel value of each pixel to be blurred in the L-th layer feature map in each direction according to the sampled optical flows in each direction of the L-th layer feature map includes: Determine a plurality of candidate pixel values ​​of each pixel to be blurred in each direction according to the preset number of iterations and the sampled optical flows of each pixel to be blurred in the L-th layer feature map in each direction; The maximum candidate pixel value in each direction is determined as the initial pixel value of the third pixel point in each direction; the third pixel point is a pixel point to be blurred whose pixel value is greater than a saturation threshold; The average value of the candidate pixel values ​​in each direction is determined as the initial pixel value of the fourth pixel point in each direction; the fourth pixel point is a pixel point to be blurred whose pixel value is not greater than a saturation threshold.

15. The method according to claim 1, It is characterized in that The method further comprises: Detecting a fifth pixel point in the image frame, where the fifth pixel point is a pixel point whose pixel value is greater than a saturation threshold; Performing vector blur on the fifth pixel according to the optical flow of the fifth pixel to obtain a pixel value of the fifth pixel; The pixel value of the fifth pixel point and the pixel value of the pixel point corresponding to the fifth pixel point in the target dynamic blur map are superimposed to obtain the dynamic blur map of the image frame.

16. A dynamic fuzzy generation method, It is characterized in that The method comprises: The image frame is processed into blocks to obtain a plurality of image blocks; The pixel points in each of the image blocks are blurred according to the optical flow of the image frame to obtain a target dynamic blurred image of the image frame.

17. The method according to claim 16, It is characterized in that The blurring of the pixels in each of the image blocks according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame includes: For the same image block, the same optical flow is used to perform vector blur on each pixel in the image block to obtain the target dynamic blur image.

18. The method according to claim 16, It is characterized in that The blurring of the pixels in each of the image blocks according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame includes: Determine the target convolution kernel corresponding to each image block according to the optical flow of each image block; Convolution is performed on each pixel point in the corresponding image block according to each target convolution kernel to obtain the target dynamic blur map.

19. A method for generating a dynamic blur image of a panoramic image. It is characterized in that The method comprises: Expanding the panoramic image to obtain an expanded image; Acquire the optical flow of each pixel of the expanded image, and obtain the optical flow of the panoramic image according to the optical flow of each pixel of the expanded image; The panoramic image is blurred according to the optical flow of the panoramic image to obtain a target dynamic blurred image of the panoramic image.

20. The method according to claim 19, It is characterized in that The step of expanding the panoramic image to obtain an expanded image includes: Determining a first image area in the panoramic image along a first direction, and determining a second image area in the panoramic image along a second direction; the first direction and the second direction are opposite; The first image area is spliced ​​to the edge of the second image area, and the second image area is spliced ​​to the edge of the first image area.

21. The method according to claim 20, It is characterized in that The step of acquiring the optical flow of each pixel of the expanded image and obtaining the optical flow of the panoramic image according to the optical flow of each pixel of the expanded image includes: determining an initial optical flow of each pixel in the first image area according to the optical flow of each pixel in the first image area and the optical flow of each pixel in the second image area spliced ​​to the first image area; determining an initial optical flow of each pixel in the second image area according to the optical flow of each pixel in the second image area and the optical flow of each pixel in the first image area spliced ​​to the second image area; The optical flow of the panoramic image is determined according to the initial optical flow of each pixel in the first image area, the initial optical flow of each pixel in the second image area, and the optical flow of each pixel in the third image area in the panoramic image; the third image area is an image area between the first image area and the second image area.

22. The method according to claim 19, It is characterized in that The step of blurring the panoramic image according to the optical flow of the panoramic image to obtain a target dynamic blur map of the panoramic image includes: Downsampling the panoramic image and the optical flow of the panoramic image at different scales for multiple times to obtain multiple feature maps and sampled optical flows corresponding to the feature maps; Vector blur is performed on the pixel points in each of the feature maps according to the sampled optical flow to obtain the target dynamic blur map.

23. The method according to claim 22, It is characterized in that The step of performing vector blurring on the pixel points in each of the feature maps according to the sampled optical flow to obtain the target dynamic blur map comprises: According to the order of preset pixel points, vector blur is performed on each pixel point in the feature map according to the sampled optical flow in sequence; The sequence of the preset pixel points includes: Increasing the width of the panoramic image along the first direction as the horizontal coordinate of the current pixel point to determine the next pixel point; or, The next pixel point is determined by subtracting the width of the panoramic image from the horizontal coordinate of the current pixel point along the second direction; the first direction is opposite to the second direction.

24. The method according to claim 19, It is characterized in that The blurring the panoramic image according to the optical flow of each pixel in the panoramic image to obtain a target dynamic blur map of the panoramic image includes: Performing block processing on the panoramic image to obtain multiple image blocks; The pixel points in each of the image blocks are blurred according to the optical flow of the panoramic image to obtain the target dynamic blur image.

25. A method for generating a dynamic blur image of a panoramic image. It is characterized in that The method comprises: Acquire multiple frames of images, and determine the optical flow of image frames to be processed according to the multiple frames of images, wherein the image frames are images selected by the user in the panoramic image after the camera moves, and the multiple frames of images are images with the same viewing angle as the image frames; The image frame is blurred according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame.

26. The method according to claim 25, It is characterized in that The step of determining the optical flow of the image frame to be processed according to the multiple image frames comprises: Determine the initial optical flow of each pixel in the image frame according to the multiple frames of images; The initial optical flow of each pixel in the image frame is preprocessed to obtain the optical flow of each pixel in the image frame.

27. The method according to claim 26, It is characterized in that The preprocessing includes at least one of median filtering, weighted processing, threshold processing, mask processing, and intensity adjustment processing.

28. The method according to claim 26, It is characterized in that The preprocessing also includes spatial domain smoothing.

29. The method according to claim 25, It is characterized in that The blurring of the image frame according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame includes: Downsampling the image frame and the optical flow of the image frame multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; Vector blur is performed on the pixel points in each of the feature maps according to the sampled optical flow to obtain the target dynamic blur map.

30. The method according to claim 25, It is characterized in that The blurring of the image frame according to the optical flow of the image frame to obtain a target dynamic blur map of the image frame includes: Performing block processing on the image frame to obtain multiple image blocks; Each of the image blocks is blurred according to the optical flow of the image frame to obtain the target dynamic blur map.

31. An image processing method for simulating slow shutter shooting effect. It is characterized in that Applied to a photographing device, the method comprises: Switching to the electronic neutral density filter mode of the photographing device based on the user's interactive instruction; In the electronic neutral density filter mode, blur processing is performed on the image frames captured by the shooting device so that the output image frames are images simulating a slow shutter speed shooting effect.

32. The method according to claim 31, It is characterized in that The photographing device stores a mapping table, the mapping table including correspondences between different electronic neutral density filter gears and algorithm parameters. Before blurring the image frames photographed by the photographing device, the method further includes: Obtaining a target electronic neutral density filter position selected by a user, querying the mapping table according to the target electronic neutral density filter position, and obtaining a target algorithm parameter; Accordingly, the blurring of the image frame captured by the capturing device includes: The image frame is blurred according to the target algorithm parameters.

33. The method according to claim 31, It is characterized in that Before processing the image frames captured by the shooting device, the method further includes: In the process of recording video using the shooting device, obtaining each image frame; Accordingly, the blurring of the image frame captured by the capturing device includes: Blur the image frame for each frame.

34. The method according to claim 31, It is characterized in that The blurring of the image frame captured by the shooting device includes: Acquire an image frame sequence, wherein the image frame sequence includes the image frames obtained by continuous shooting of multiple frames arranged in time sequence; Performing blur processing on at least one of the image frames except the first and last two frames in the image frame sequence to obtain at least one of the image frames after blur processing, At least one of the blurred image frames is superimposed.

35. The method according to claim 34, It is characterized in that The step of performing blur processing on at least one of the image frames except the first and last two frames in the image frame sequence to obtain at least one blurred image frame comprises: Using the same optical flow, blur processing is performed on at least one of the image frames except the first and last two frames in the image frame sequence to obtain at least one blurred image frame.

36. The method according to claim 31, It is characterized in that The blurring of the image frame captured by the shooting device includes: Blurring is performed on the pixels of the non-interested area in the image frame.

37. The method according to claim 31, It is characterized in that The blurring of the image frame captured by the shooting device so that the output image frame is an image simulating a slow shutter shooting effect includes: Downsampling the image frame and the optical flow of the image frame multiple times at different scales to obtain multiple feature maps and sampled optical flows corresponding to each feature map; Vector blur is performed on the pixel points in each of the feature maps according to the sampled optical flow, and a target dynamic blur map of the image frame is output.

38. The method according to claim 37, It is characterized in that The step of performing vector blurring on the pixel points in each of the feature maps according to the sampled optical flow and outputting a target dynamic blur map of the image frame comprises: Determine the pixel points to be blurred in the L-th layer feature map according to the sampled optical flow of the L-th layer feature map, perform vector blur on the pixel points to be blurred in the L-th layer feature map, and obtain the L-th layer dynamic blur map; the size of the L-th layer feature map is the smallest; According to the Lth layer dynamic blur map, vector blur is performed on the pixel points in the feature maps of other layers in turn, and the target dynamic blur map is output.

39. The method according to claim 38, It is characterized in that The step of performing vector blurring on the pixels in the feature maps of other layers in sequence according to the Lth layer dynamic blur map, and outputting the target dynamic blur map, comprises: Performing fuzzy operation on the feature map of each layer in turn, and outputting the target dynamic fuzzy map; The fuzzy operation includes: Performing a first preprocessing on the L-th layer of dynamic fuzzy image, and upsampling the preprocessed L-th layer of dynamic fuzzy image to obtain a sampled dynamic fuzzy image of the L-1th layer; According to the label value corresponding to the sampled dynamic fuzzy image of the L-1th layer and the sampled optical flow of the feature image of the L-1th layer, the feature image of the L-1th layer is vector-blurred to obtain an initial dynamic fuzzy image of the L-1th layer; The L-1th layer dynamic fuzzy map is determined according to the L-1th layer sampled dynamic fuzzy map and the L-1th layer initial dynamic fuzzy map.

40. The method according to claim 31, It is characterized in that The blurring of the image frame captured by the shooting device so that the output image frame is an image simulating a slow shutter shooting effect includes: Performing block processing on the image frame to obtain multiple image blocks; Blurring is performed on the pixels in each of the image blocks according to the optical flow of the image frame, and a target dynamic blur map of the image frame is output.

41. The method according to claim 40, It is characterized in that The blurring of the pixels in each of the image blocks according to the optical flow of the image frame to output a target dynamic blur map of the image frame includes: For the same image block, the same optical flow is used to perform vector blur on each pixel in the image block, and the target dynamic blur map is output.

42. The method according to claim 40, It is characterized in that The blurring of the pixels in each of the image blocks according to the optical flow of the image frame to output a target dynamic blur map of the image frame includes: Determine the target convolution kernel corresponding to each image block according to the optical flow of each image block; Convolution is performed on each pixel point in the corresponding image block according to each target convolution kernel, and the target dynamic blur map is output.

43. An image processing device for simulating slow shutter shooting effect, It is characterized in that Applied to a photographing device, the device comprises: A switching module, used for switching to the electronic neutral density filter mode of the photographing device based on a user's interactive instruction; The output module is used to process the image frames captured by the shooting device in the electronic neutral density filter mode so that the output image frames are images simulating the slow shutter shooting effect.

44. A computer device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor executes the computer program, the steps of the method of any one of claims 1 to 15, or 16 to 18, or 19 to 24, or 25 to 30, or 31 to 42 are implemented.

45. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method of any one of claims 1 to 15, or 16 to 18, or 19 to 24, or 25 to 30, or 31 to 42 are implemented.

46. ​​A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 15, or 16 to 18, or 19 to 24, or 25 to 30, or 31 to 42 are implemented.

Images