Image processing method and computer program product

Abstract

Embodiments of the present application provide an image processing method and computer program product, comprising: obtaining a k-th frame light spot weight map based on M historical frame light spot detection maps and a k-th frame light spot detection map, wherein the M historical frame light spot detection maps are obtained by performing light spot detection on M frames of original images, and the k-th frame light spot detection map is obtained by performing light spot detection on the k-th frame of original image; and obtaining a k-th frame target object light spot blur rendering image based on the k-th frame light spot weight map and the k-th frame of original image. The technical solution of the embodiments of the present application can solve the problems of poor image light spot blur effect and light spot flickering during video blur, making the video blur light spot effect more stable.

Description

Technical Field

[0001] The present application relates to the field of video image blurring processing. Specifically, embodiments of the present application relate to an image processing method and a computer program product. Background Art

[0002] With the continuous update of electronic devices (e.g., mobile devices, including cell phones), users are increasingly demanding and demanding better camera functions. For example, neon spot video blurring technology based on mobile devices (e.g., cell phones) uses computer vision algorithms to generate a DSLR-like effect. These technologies primarily include portrait segmentation and simulated depth of field rendering. The current mainstream approach for portrait segmentation is based on deep learning models, while simulated depth of field blurring primarily uses methods similar to filter blurring for simulation.

[0003] Currently, there are many problems with the image blurring effect (for example, the blurring effect in portrait shooting mode) or video blurring effect (for example, the blurring effect in portrait videos) on electronic devices, such as blurred and unclear light spots on the image after image blurring, or severe flickering of background light spots generated in the video, which seriously affects the user experience.

[0004] Therefore, how to improve the problems existing in image blur or video blur has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and a computer program product. The technical solution provided by the embodiments of the present application can solve the problem of blurred light spots on the image after image blurring or solve the problem of flickering light spots when video blurring, making the video blurring effect more natural and the blurred light spot effect more stable.

[0006] In a first aspect, some embodiments of the present application provide a method for image processing, the method comprising: obtaining a k-th frame light spot weight map based on M historical frame light spot detection maps and a k-th frame light spot detection map, wherein the M historical frame light spot detection maps are obtained by performing light spot detection on M frames of original images respectively, and the k-th frame light spot detection map is obtained by performing light spot detection on the k-th frame original image, and the M frames of original images are one frame or multiple consecutive frames of images located before the k-th frame original image and adjacent to the k-th frame original image, and M is an integer greater than or equal to 1; obtaining a k-th frame target object light spot blurred rendering map based on the k-th frame light spot weight map and the k-th frame original image, wherein the k-th frame target object light spot blurred rendering map is the acquired target image.

[0007] Since the embodiment of the present application takes into account the light spot detection map of the historical frame before the frame (i.e., the kth frame of the currently processed frame) when obtaining the light spot weight map of each frame, it can solve the problem of severe flickering of light spots generated by electronic devices in video blur mode.

[0008] In some embodiments, before obtaining the kth frame light speckle weight map based on M historical frame light speckle detection maps and the kth frame light speckle detection map, the method further includes: performing image segmentation on any one of the M frames of original images and the kth frame of original images to obtain a background segmentation map; obtaining a single-channel map based on the maximum value of multiple channels of each pixel point in the background segmentation map; and obtaining a light speckle detection map corresponding to any one frame of original image based on at least the value of each pixel point on the single-channel map.

[0009] Some embodiments of the present application detect light spots by acquiring a single-channel image corresponding to a background segmentation image and thereby obtaining a light spot detection image, which can improve the accuracy of the detected light spots.

[0010] In some embodiments, obtaining a light spot detection image corresponding to any frame of the original image at least based on the value of each pixel point on the single-channel image includes: blurring the single-channel image to obtain a blurred single-channel image; if the value of any pixel point on the single-channel image is greater than a first threshold, and it is confirmed that the value of any pixel point on the blurred single-channel image is less than or equal to a second threshold, then confirming that the any pixel point belongs to a light spot, wherein the second threshold is less than the first threshold.

[0011] Some embodiments of the present application utilize the properties of the light spot (i.e., the pixel value of the light spot located at the center of the light spot is large, while the pixel values ​​of the adjacent pixels located at the center of the light spot are smaller) to comprehensively judge whether each pixel point belongs to the light spot by constructing a fuzzy single-channel image and a single-channel image, thereby improving the accuracy of the detected light spots.

[0012] In some embodiments, M is an integer greater than 1, wherein obtaining the k-th frame light spot weight map based on the M historical frame light spot detection maps and the k-th frame light spot detection map includes: selecting X historical frame light spot detection maps from the M historical frame light spot detection maps, wherein the X historical frame light spot detection maps are adjacent to the k-th frame light spot detection map, and X is an integer greater than or equal to 1 and less than or equal to M; fusing the X historical frame light spot detection maps and the k-th frame light spot detection map to obtain the k-th frame light spot fusion map; and obtaining the k-th frame light spot weight map based on the k-th frame light spot fusion map.

[0013] Some embodiments of the present application dynamically select at least some historical frame light spot detection images from multiple historical frame light spot detection images adjacent to the currently processed frame (i.e., the kth frame), and fuse the selected at least some historical frame light spot detection images with the current frame light spot detection image, thereby improving the light spot flickering phenomenon between the current frame original image and the adjacent frame original image in the video to be blurred.

[0014] In some embodiments, selecting X historical frame light spot detection images from the M historical frame light spot detection images includes: selecting the X historical frame light spot detection images from the M historical frame light spot detection images based on distribution information of light spots on different frames of original images.

[0015] Some embodiments of the present application determine the number of historical frames required for optimizing flicker through an inter-frame difference detection mechanism of light spots, so that the light spot flicker problem of adjacent frames can be solved using as few historical frames as possible.

[0016] In some embodiments, selecting X historical frame light spot detection images from the M historical frame light spot detection images includes: if the difference value between the k-th frame light spot detection image and the j-th frame light spot detection image satisfies a difference threshold, then using the light spot detection images from the j-th frame to the (k-1)-th frame as the X historical frame light spot detection images, wherein the j-th frame light spot detection image belongs to a frame image among the M historical frame light spot detection images.

[0017] Some embodiments of the present application determine the differences in distribution of light spots on different frames based on the light spot detection image of the current frame and the light spot detection images of the historical frames, thereby improving the accuracy of the obtained difference results.

[0018] In some embodiments, if the value of j is i is an integer greater than or equal to 0, wherein the confirmation that the difference value between the k-th frame light spot detection image and the j-th historical frame light spot detection image meets the difference threshold includes: i starting from 0 and repeating the following process until the difference value obtained by the following method meets the difference threshold: calculating the difference between the k-th frame light spot detection image and the j-th historical frame light spot detection image The absolute value of the difference between the pixel values ​​corresponding to each pixel point on the frame light spot detection image is obtained to obtain the i-th difference; calculate the sum of the k-th frame light spot detection image and the k-th frame light spot detection image. The absolute value of the difference between the pixel values ​​corresponding to each pixel point on the frame light spot detection image is obtained to obtain the i+1th difference; the difference between the i-th difference and the i+1th difference is calculated, and the sum of the i-th difference and the i+1th difference is calculated; the difference value is obtained according to the ratio of the difference to the sum; wherein the "[]" operator represents a rounding operation.

[0019] Some embodiments of the present application select the light spot detection image of the historical frame farthest from the current frame, i.e., the kth frame, or select the intermediate frame included in the historical frame to obtain the difference value, thereby reducing the amount of calculation to obtain the value, improving the speed of the difference value calculation, and improving the accuracy of the obtained difference value.

[0020] In some embodiments, the selecting X historical frame light spot detection images from the M historical frame light spot detection images further includes: confirming that the difference values ​​between the k-th frame light spot detection image and the j-th frame historical frame light spot detection image do not meet the difference threshold, then using the k-1-th frame light spot detection image as the X historical frame light spot detection images, wherein the j-th frame light spot detection image belongs to a frame image among the M historical frame light spot detection images.

[0021] In some embodiments of the present application, if it is confirmed that the differences between the stored historical frame light spot detection map and the current frame light spot detection map (i.e., the kth frame) are not large, then a historical frame light spot detection map closest to the current frame (i.e., the kth frame) is selected to adjust the kth frame light spot detection map to obtain the kth frame light spot weight map. This improves the light spot flickering problem and minimizes the data processing amount.

[0022] In some embodiments, obtaining the k-th frame light spot weight map based on the k-th frame light spot fusion map includes: performing curve stretching on the k-th frame light spot fusion map to obtain the k-th frame light spot fusion stretched map; obtaining the k-th frame light spot weight map based on the k-th frame light spot fusion stretched map.

[0023] Some embodiments of the present application enhance the brightness of the entire image by stretching and expand the brightness levels, so that the final convolution spot has a better sense of layering.

[0024] In some embodiments, obtaining the k-th frame light spot weight map based on the k-th frame light spot fusion map includes: smoothing the k-th frame light spot fusion map or the k-th frame light spot fusion stretched map obtained by stretching the k-th frame light spot fusion map to obtain the k-th frame light spot weight map.

[0025] Some embodiments of the present application further improve the light spot display effect of the obtained video through smoothing processing.

[0026] In some embodiments, the smoothing of the k-th frame light spot fusion image to obtain the k-th frame light spot weight map includes: performing weighted averaging of the values ​​of corresponding pixels on the k-1-th frame light spot fusion image and the k-th frame light spot fusion image in different proportions to obtain a weighted average of each pixel, and using the weighted average of each pixel as the weight value of the corresponding pixel on the k-th frame light spot weight map; performing weighted averaging of the values ​​of the same pixel on the k-1-th frame light spot fusion stretched image and the k-th frame light spot fusion image in different proportions to obtain a weighted average of each pixel, and using the weighted average of each pixel as the weight value of the corresponding pixel on the k-th frame light spot weight map; or The k-th frame light spot fusion stretched image obtained by stretching the k-frame light spot fusion image is smoothed to obtain the k-th frame light spot weight map, including: performing weighted averaging of the values ​​of the same pixel points on the k-1-th frame light spot fusion stretched image and the k-th frame light spot fusion stretched image in different proportions to obtain a weighted average value of each pixel point, and using the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-th frame light spot weight map; or performing weighted averaging of the values ​​of the same pixel points on the k-1-th frame light spot fusion stretched image and the k-th frame light spot fusion stretched image to obtain a weighted average value of each pixel point, and using the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-th frame light spot weight map.

[0027] Some embodiments of the present application introduce weighted averaging algorithms of different proportions to smooth the light spot detection image of the current frame based on the light spot detection images of historical frames, thereby further improving the light spot display effect in the synthesized video and reducing the light spot flickering phenomenon between adjacent frames in the synthesized target video.

[0028] In some embodiments, the method further includes: repeating the process of acquiring the target image to obtain a target video, wherein the target video includes multiple target images.

[0029] Some embodiments of the present application can obtain video files with significantly improved light spot flickering effects.

[0030] In a second aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0031] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect above.

[0032] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the method described in any embodiment of the first aspect above can be implemented.

[0033] In a fifth aspect, some embodiments of the present application provide a device for processing light spot flickering in a video blur mode, the device comprising: a k-th frame light spot weight map acquisition module, configured to obtain a k-th frame light spot weight map based on M historical frame light spot detection maps and a k-th frame light spot detection map, wherein the M historical frame light spot detection maps are obtained by performing light spot detection on M frames of original images respectively, and the k-th frame light spot detection map is obtained by performing light spot detection on the k-th frame original image, and the M frames of original images are one frame or multiple consecutive frames of images located before the k-th frame original image and adjacent to the k-th frame original image, M is an integer greater than or equal to 1, wherein the k-th frame target object light spot blur rendering image is the acquired target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A schematic diagram of the composition of an image processing system provided in an embodiment of the present application;

[0036] Figure 2 One of the flowcharts of the image processing method provided in an embodiment of the present application;

[0037] Figure 3 The second flowchart of the image processing method provided in the embodiment of the present application;

[0038] Figure 4 A block diagram of the image processing apparatus provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0041] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0042] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0043] In the related technologies, the video blurring process based on mobile terminals (for example, mobile phones and other electronic devices) requires blurring the target objects in each frame image included in the video (for example, portrait video blurring technology), and the target object blurring technology (for example, portrait video blurring technology) mainly includes target object segmentation technology (for example, portrait segmentation technology) and simulated depth of field rendering technology. The current mainstream method of target object segmentation technology is based on deep learning models, and the simulated depth of field blurring rendering technology is mainly simulated through methods similar to Gaussian blur.

[0044] In order to complete the video blurring process, it is necessary to blur each frame of the video separately to obtain a blurred image of each frame. To obtain the blurred image of each frame, it is necessary to first obtain the light spot weight map of the corresponding frame image, and then render the background segmentation image based on the light spot weight map to obtain a background light spot blurred rendering map. The background light spot blurred rendering map is fused with the target object segmentation image (for example, a portrait segmentation image) to obtain a blurred image corresponding to the frame (that is, a target object light spot blurred rendering map). The inventors of this application found in their research that because the light spot weight map of each frame is currently obtained on a mobile terminal (for example, a mobile phone), only the light spot distribution of the frame is considered (that is, only the light spot detection map of the frame is used), which leads to many problems with the video blurring effect (for example, the portrait blurring effect), such as the video blurring effect is not beautiful, and there is a serious light spot flickering phenomenon between different frames.

[0045] Some embodiments of the present application use timing information to optimize the light spot detection map of each frame, that is, use the historical frame light spot detection map to optimize the light spot detection map of the current processing frame (that is, the k-th frame light spot detection map), and obtain the current processing frame light spot weight map corresponding to the current image to be blurred that integrates the light spot distribution characteristics of the historical frames (that is, obtains the k-th frame light spot weight map), which can effectively improve the stability and flickering degree of the sequence light spot, at least solve the problem of severe flickering of the video light spot, and make the video light spot generated by the algorithm more stable and closer to the real light spot effect.

[0046] Please see Figure 1 , Figure 1 An image processing system 100 is provided for some embodiments of the present application. The image processing system 100 includes: a mobile phone terminal 10 and a server 20 .

[0047] The mobile phone terminal 10 has an image or video capture unit 11. In some embodiments of the present application, the image capture unit 11 can be used to capture a video 22 to be blurred (for example, a portrait video to be blurred), and then the mobile phone terminal 10 sends the video 22 to be blurred to the server 20. In some embodiments of the present application, the image capture unit 11 can be used to capture an image to be blurred (not shown in the figure), and then the image to be blurred is sent to the server 20.

[0048] The server 20 can perform background blurring operation on each frame of the received video 22 to be blurred, obtain a target object spot blurring rendering image with good spot blurring effect in each frame, and feed back the target object spot blurring rendering images into a target video to the mobile phone terminal 10. At this point, the mobile phone terminal 10 obtains a target video with good spot display effect (for example, the spot flickering phenomenon between different frames in the target video is improved).

[0049] It should be noted that the video 22 to be blurred includes multiple frames of original images, and the target video includes a blurred light spot rendering of the target object after blurring each frame of the original image. It is understandable that, because in some embodiments of the present application, when obtaining the blurred light spot rendering of the target object in the currently processed frame, the light spot weight map used is determined by combining the light spot detection map of the currently processed frame and the light spot detection maps of the previous frame or multiple historical frames adjacent to the currently processed frame, the flickering defect of the blurred light spot rendering of the target object in adjacent frames in the target video is effectively improved.

[0050] The following uses a specific frame as an example to illustrate the process of obtaining the target object light spot blur rendering image of each frame and obtaining the target video provided by some embodiments of the present application. For example, in order to perform target object light spot rendering on the qth frame original image (belonging to any frame original image in the video to be blurred), it is necessary to obtain the qth frame light spot weight map corresponding to the qth frame original image, and then render the light spot according to the qth frame light spot weight map to obtain the background light spot blur rendering image of the frame, and then fuse the background light spot blur rendering image of the frame with the target object segmentation map of the frame to obtain the target object light spot blur rendering image of the frame, and repeat this process to obtain the target object light spot blur rendering image of each frame in the video to be blurred to obtain the target video. It should be noted that in the related art, when obtaining the light spot weight map of the qth frame, only the light spot detection map of the qth frame is considered, while in some embodiments of the present application, in order to obtain the light spot weight map of the qth frame, it is also necessary to consider the influence of the light spot detection map of the frame before the qth frame on the light spot weight map of the qth frame. Therefore, the target video obtained by using these embodiments of the present application can improve the light spot flickering phenomenon existing in adjacent frames in the video.

[0051] It is understood that the camera unit 11 of the mobile phone terminal 10 is located on the back of the mobile phone. In other embodiments, the camera may also be located on the front of the mobile phone terminal 10 or other locations. The embodiments of the present application do not limit the specific location of the image capture unit 11 on the mobile phone terminal 10. The mobile phone terminal 10 can be any type of electronic device with an image capture function. For example, the mobile phone terminal 10 can also be a laptop computer with an image capture function, etc. The embodiments of the present application do not limit this.

[0052] The server 20 and the mobile phone terminal 10 are connected via a wired network (not shown in the figure) or a wireless network, and the image to be blurred and the blurred rendering image of the target object light spot are transmitted through the connection.

[0053] It should be noted that in some embodiments of the present application, the mobile phone terminal 10 can also use its own processor to execute the image processing method provided in some embodiments of the present application, and then obtain the target video or obtain the target image. It is understandable that in these embodiments, the mobile phone terminal 10 does not output the video to be blurred to the user, but adjusts the light spot detection map corresponding to each frame image in the video to be blurred based on the historical frame light spot detection map through the processor of the mobile phone terminal 10 (that is, executes the method for processing light spot flickering in the video blur mode provided in some embodiments of the present application) and finally outputs the target video or target image to the user. It is not difficult to understand that at this time, there is no need to set up a separate server 20 to execute the method for processing light spot flickering in the video blur mode of the embodiment of the present application.

[0054] The following combination Figure 2 The method for processing light spot flicker in the video blur mode executed by the server 20 or a terminal device such as the mobile terminal 10 is exemplified.

[0055] like Figure 2 As shown, some embodiments of the present application provide an image processing method, which includes: S101, obtaining a k-th frame light spot weight map based on M historical frame light spot detection maps and the k-th frame light spot detection map; S102, obtaining a k-th frame target object light spot blur rendering map based on the k-th frame light spot weight map and the k-th frame original image. It can be understood that in order to obtain a target video with better blurring effect, the image processing method of the embodiment of the present application also includes: S103, repeating the corresponding processes of S101 and S102 above to obtain multiple frames of target object light spot blur rendering maps, and obtaining a target video based on the multiple frames of target object light spot blur rendering maps. That is, a target video including multiple target images is obtained by repeatedly obtaining the target image process.

[0056] It is not difficult to understand that since the embodiment of the present application takes into account the light spot detection map located before the frame when obtaining the light spot weight map of each frame, it can solve the problem of severe flickering of light spots generated by electronic devices in video blur mode.

[0057] It can be understood that before executing S101 , it is also necessary to obtain light spot detection images corresponding to each of the M frames of original images and the k-th frame of original image.

[0058] The following is an illustrative description of the steps involved in the above process.

[0059] First, the process of obtaining the light spot detection map corresponding to each frame of the original image is exemplarily described. It should be noted that in some embodiments of the present application, the process of obtaining the light spot detection map of each frame is performed by the mobile terminal.

[0060] That is, in some embodiments of the present application, before S101, the method for processing light spot flickering in the video blur mode further includes:

[0061] In the first step, image segmentation is performed on any one of the M frames of original image and the k-th frame of original image to obtain a background segmentation map. For example, the first step includes the following steps A) to C). It should be noted that in some embodiments of the present application, the light spot detection map corresponding to each frame of original image can also be obtained using the method disclosed in the relevant art. In other embodiments of the present application, the light spot detection map corresponding to each frame of original image and the k-th frame of original image is obtained using the method of the first step.

[0062] A),The image is captured by the mobile phone camera.

[0063] The user focuses on the portrait and clicks the video record button. The mobile device obtains a frame of original image, that is, a frame of RGB format image to be blurred. image , that is, the image to be blurred I is obtained image .

[0064] B) Treating the blurred image I according to the portrait segmentation model image The foreground and background are separated to obtain a binary portrait image, that is, the blurred image is segmented into the portrait (i.e., foreground) and the background according to the image segmentation model of the related technology (i.e., deep learning model) to obtain a portrait segmentation mask.

[0065] The deep learning model of related technology (for example, networks such as unet or resnet) treats the blurred image I image Perform portrait segmentation to obtain a single-channel portrait binary image I mask , where the values ​​of all pixels in the main area (i.e., the area corresponding to the portrait) on the single-channel portrait binary image are 1, and the values ​​of all pixels in the background area except the portrait are 0.

[0066] C), using I image ×(1-I mask ) to the background segmentation map I except the portrait bg , the area corresponding to the portrait on the background segmentation map needs to be filled with 0.

[0067] In the second step, a single-channel image is obtained based on the maximum value among the multiple channels of each pixel in the background segmentation image. In other words, for each pixel, the maximum value is selected from the multiple channels of the background segmentation image to obtain the single-channel image. For example, this second step includes the following step D).

[0068] D) Based on background segmentation map Ibg The R, G and B channels are traversed pixel by pixel to find the maximum value of the three channels, that is, the single channel image I is obtained. maxch_bg .

[0069] In a third step, a light spot detection image corresponding to the original image frame is obtained based on at least the values ​​of each pixel point on the single-channel image. It should be noted that this third step may exemplarily include the following step E), or may exemplarily include the following step E) and step F), or may exemplarily include step E), step F), and step G).

[0070] That is to say, some embodiments of the present application detect light spots by obtaining a single-channel image corresponding to the background segmentation image and then obtain a light spot detection image, which can improve the accuracy of the detected light spots. Specifically, in some embodiments of the present application, the third step exemplarily includes: blurring the single-channel image to obtain a blurred single-channel image; for any pixel point, confirming that the value of any pixel point on the single-channel image is greater than a first threshold, and confirming that the value of any pixel point on the blurred single-channel image is less than or equal to a second threshold, then confirming that the any pixel point belongs to a light spot, wherein the second threshold value is less than the first threshold value. It should be noted that the value of each pixel point can be a pixel value or a weight value, wherein the weight value is the value of each pixel point obtained by normalizing the pixel values ​​of all pixels on the entire image. That is to say, some embodiments of the present application utilize the properties of the light spot (i.e., the pixel value at the center of the light spot is large and the pixel value of the adjacent pixel point at the center of the light spot is small) to comprehensively judge whether each pixel point belongs to a light spot by constructing a blurred single-channel image and a single-channel image, thereby improving the accuracy of the detected light spot.

[0071] E) The single channel image I obtained in step D) above maxch_bg Perform light spot (or light spot) detection to obtain the light spot detection image of the frame. The detailed process is as follows:

[0072] i. For single channel graph I maxch_bg Perform N*N (for example, 3*3) Gaussian blur to obtain a blurred single-channel image I smooth_maxch_bg .

[0073] ii. For single channel graph I maxch_bg and fuzzy single channel image I smooth_maxch_bg Traverse, if the value of the pixel at the (x, y) position meets the following conditions, then the pixel is considered to be a candidate light spot (that is, it belongs to a light spot), and obtain the light spot detection image I composed of all candidate light spots init_spots :

[0074]

[0075] The above if statement is a conditional judgment statement in C language. The function of this judgment statement is: when it is judged that two inequalities are true at the same time, the assignment statement after the semicolon is executed. The above value operation represents the normalization operation of the parameters in the brackets. Specifically, value(I maxch_bg )(x,y) represents the value of the pixel point (x, y) on the normalized image after normalizing the values ​​of all pixels on the single-channel image (or called the normalized pixel value, or also called the weight value). It is not difficult to understand that this value is a value greater than or equal to 0 and less than or equal to 1. value(I maxch_smooth_bg )(x, y) represents the value of the pixel point (x, y) on the blurred single-channel image after the values ​​of all pixels on the image are normalized (or called the normalized pixel value, also called the weight value). It is not difficult to understand that this value is a value greater than or equal to 0 and less than or equal to 1. The above if statement needs to judge whether the normalized pixel value of the pixel point (x, y) on the single-channel image is greater than the set first threshold bokeh_thresh1, and the normalized pixel value of the pixel point on the blurred single-channel image is less than the set second threshold bokeh_thresh2. In combination with the above statements, it is not difficult to find that these two thresholds belong to two numerical ranges less than 1, and the maximum value included in the second threshold is less than the minimum value included in the first threshold. In addition, it can be seen from the above statements that when the pixel point (x, y) satisfies the two conditions of the judgment statement (i.e., the if statement) at the same time, it is determined that the pixel point belongs to a light spot, and for this light spot, the value of the pixel point (x, y) on the single-channel image needs to be assigned to the light spot detection image I init_spots Same pixel on top.

[0076] It should be noted that all pixels that do not meet the two conditions of the above if statement are determined not to be light spots, and the values ​​of these pixels on the light spot detection map are 0. It is not difficult to understand that a map consisting of all the determined light spots can be determined through the light spot detection map.

[0077] F) Light spot detection image I init_spots Or light spot detection image I init_spots The included dense area is thinned out to obtain a sparse light spot detection map.

[0078] For example, based on the image scaling operation resize operation provided by the open source library opencv (ie Open Source Computer Vision Library), the light spot detection image I init_spotsFirst downsample by 4 times (a specific example of the target sampling rate) and then upsample by 4 times. In this way, you can simply and directly use the bilinear interpolation provided by resize to do the screening points, and thin out the areas where the light spots are densely concentrated, so as to prevent the subsequent rendering of large areas of light spots from being overexposed and lacking a sense of layering.

[0079] The following is an example of the process of obtaining a dense area.

[0080] Judgment I init_spots The density of light spots in the image includes: dividing the image to be blurred into 4 small images of the same size according to half of its width and height; solving the centroid coordinates of each small image, that is, the center of the pixel concentration in the region, and the specific operation is to solve the zero-order moment M of each small image. 00 and the first-order moment M 10 、M 01 ; The center of mass coordinates are obtained based on the following formula: Calculate the number of non-zero values ​​sum_spots in the range of 100*100 with the centroid coordinate point of each point as the center.

[0081] Find the dense area sampling threshold thresh of the four small images = sum_spots / 10000. If at least one of them has a thresh greater than 10%, it is judged as I init_spots There are dense areas, which will cause the light spots to cluster and be overexposed. init_spots Or the dense area included in the graph is thinned out.

[0082] G) performing point filling processing on the discrete light spots to update the light spot detection image obtained in step F).

[0083] Since the operation of step F) on the dense points will reduce the number of the original sparse discrete points and the weight value will also become smaller, it is necessary to perform a point filling operation on these light spots. The point filling operation is performed on the sparse light spot detection image I' obtained in step d init_spots Proceed as follows:

[0084] iii. Light spot detection Figure I init_spots Perform pixel traversal. If the number of light spots (or light spots) in the neighborhood of the traversed pixel point (x, y) with a radius of M (for example, M is 3) pixels does not exceed L (for example, L is 8), then the pixel point is considered to be a discrete sparse point and needs to be detected in the sparse light spot detection map I' init_spots Fill in the points around the position (x, y) and the pixel. It should be noted that filling in the points means increasing the values ​​of these pixels on the sparse light spot detection image. The specific steps are as follows.

[0085] iv. Update sparse light spot detection map I'init_spots The value of the discrete sparse point (x, y) on the image (the type of the value includes a weight value or a pixel value), for example, the light spot detection image I init_spots The value of the pixel corresponding to the discrete sparse point is assigned to the same pixel on the sparse light spot detection image, in order to perform weight gain on the sparse discrete points.

[0086] v. Update sparse light spot detection map I' init_spots The values ​​of the points around the discrete sparse point (x, y) on the image are assigned the same weight within a neighborhood with a radius of P (for example, the value of P is 1) from the discrete sparse point. That is, the weight values ​​of multiple (for example, 8 when the value of P is 1) pixel points around the pixel point (x, y) corresponding to the discrete sparse point on the sparse light spot detection map are increased. The purpose is to make the discrete light spot brighter.

[0087] Based on the above steps, the light spot detection image I of the corresponding frame can be obtained. spots The purpose of obtaining this image is to enhance the weight factor of the light spot area when performing convolution blur, which is different from the ordinary blur effect of non-light spot.

[0088] It should be noted that in some embodiments of the present application, the I obtained in step E) can be init_spots As the light spot detection images for each frame required by S101. In other embodiments of the present application, the sparse light spot detection images obtained in step F) can also be used as the light spot detection images for each frame required by S101. In still other embodiments of the present application, after performing point filling processing on the discrete light spots, the image obtained in step G) can be used as the light spot detection image required by S101. The above steps A), B), C), D), E), F), and G) are steps that can be performed sequentially, and the same symbols in these steps represent the same image.

[0089] That is to say, in some embodiments of the present application, the I obtained after performing steps A) to E) can be init_spots As the light spot detection map required for S101, in other embodiments of the present application, the sparse light spot detection map obtained after executing steps A) to F)) can be used as the light spot detection map of each frame required for S101. In still other embodiments of the present application, the image obtained after executing steps A) to G) can be used as the light spot detection map described in S101.

[0090] Next, the implementation process of S101 is described as an example.

[0091] The M historical frame light speckle detection images recorded in S101 are obtained by performing light speckle detection on M frames of original images (for example, using the above-mentioned steps A) to E) and other embodiments), and the k-th frame light speckle detection image is obtained by performing light speckle detection on the k-th frame of original image (for example, using the above-mentioned steps A) to E) and other embodiments), and the M frames of original image and the k-th frame of original image are located in the same video to be blurred (for example, Figure 1 The video to be blurred 22) and the M-frame original image are one frame or multiple consecutive frames of images that are located before the k-th frame original image and adjacent to the k-th frame original image, and M is an integer greater than or equal to 1. It should be noted that adjacent includes original images that are adjacent to or closer to the k-th frame original image. The k-th frame original image is any frame original image in the video to be blurred, and the value of k is related to the frame numbering. For example, in some embodiments, the value of k is an integer greater than or equal to 0 (it can be understood that when the value of k is 0, it belongs to the starting frame of the video and therefore there is no corresponding historical frame. In some embodiments of the present application, the k-th frame is the remaining frames after the starting frame). It can be understood that the starting frame of the video to be blurred can also be defined as the first frame. For example, if k=5 and M=2, that is, the currently processed frame is the 5th frame original image, and S101 is executed, that is, the currently processed frame light spot detection map (that is, the 5th frame light spot detection map) is adjusted according to the two historical frame light spot detection maps to obtain the 5th frame light spot weight map, wherein the 5th frame light spot detection map is obtained by performing light spot detection on the 5th frame original image in the video 22 to be blurred, the 4th frame light spot detection map is obtained by performing light spot detection on the 4th frame original image in the video to be blurred, and the 3rd frame light spot detection map is obtained by performing light spot detection on the 3rd frame original image in the video to be blurred.

[0092] It should be noted that in some embodiments, the frame number corresponding to the kth original image is the absolute frame number of the original image in the video to be defocused. For example, if the video to be defocused includes 100 original image frames and the currently processed frame is the 20th frame, then the kth original image is the 20th original image. In other embodiments of the present application, the frame number corresponding to the kth original image is the cached frame number of the speckle detection map corresponding to the frame in the cache queue. For example, if the video sequence to be defocused includes 100 frames, the length of the queue for caching speckle detection maps is 8, and the frames in the queue are numbered starting from 0 according to their storage time, then it is not difficult to understand that the frame number of the currently processed frame is always k=7, i.e., the frame number of the currently processed frame is 7. In other words, the kth speckle detection map referred to in S101 is the 7th speckle detection map. It is understandable that the absolute frame number of the original image corresponding to the 7th speckle detection map in the video to be defocused may not be 7. In this case, the original frames are also numbered using the frame number in the cache queue.

[0093] In some embodiments of the present application, when the value of k is different, different numbers of partial frames are selected from the M historical frame light speckle detection maps to correct the light speckle detection map of the current frame. In other embodiments of the present application, the same number of historical frame light speckle detection maps are selected to correct the light speckle detection map of the current frame regardless of the value of k. That is, in some embodiments of the present application, S101 exemplarily includes: directly fusing the M historical frame light speckle detection maps with the k-th frame light speckle detection map to obtain the k-th frame light speckle fusion detection map, and then obtaining the k-th frame light speckle weight map based on the k-th frame light speckle fusion detection. In other embodiments of the present application, S101 exemplarily includes: first selecting X historical frame light speckle detection maps from the M historical frame light speckle detection maps, fusing the selected X historical frame light speckle detection maps with the k-th frame light speckle detection map to obtain the k-th frame light speckle fusion detection map, and then obtaining the k-th frame light speckle weight map based on the k-th frame light speckle fusion detection.

[0094] The following exemplifies the process of dynamically selecting the number of historical light speckle detection images required for the current processing frame (i.e., the kth frame) from M historical light speckle detection images. For example, some embodiments of the present application dynamically determine the number of historical frames required for flicker optimization through an inter-frame light speckle difference detection mechanism, allowing the use of as few historical frames as possible to address the light speckle flicker problem between adjacent frames.

[0095] Specifically, in some embodiments of the present application, S101 may include:

[0096] The first step is to select X historical frame light speckle detection images from the M historical frame light speckle detection images, wherein the X historical frame light speckle detection images are adjacent to the k-th frame light speckle detection image, where X is an integer greater than or equal to 1 and less than or equal to M, and the meaning of adjacent is the same as described above. For example, in some embodiments of the present application, the first step example includes: selecting the at least some historical frame light speckle detection images from the M historical frame light speckle detection images based on the difference in the distribution of light spots between different frames of original images. Specifically, in some examples of the present application, the first step exemplarily includes: confirming that the difference between the k-th frame light speckle detection image (i.e., the light speckle detection image corresponding to the current processing frame) and the j-th historical frame light speckle detection image described in S101 meets a difference threshold, then selecting the light speckle detection images from the j-th frame to the (k-1)-th frame as the X historical frame light speckle detection images, wherein the j-th frame light speckle detection image belongs to a frame of the M frame light speckle detection images. For example, the value of j is greater than or equal to kM and less than or equal to k-1.

[0097] It will be appreciated that in some embodiments of the present application, a difference value between the kth light speckle detection image and each of the M historical light speckle detection images may be obtained until a difference value that satisfies a difference threshold is found. In other embodiments of the present application, a difference value between the kth light speckle detection image and a portion of the M historical light speckle detection images may be obtained until a difference value that satisfies a difference threshold is found among the portion of the historical light speckle detection images. That is, in some embodiments of the present application, confirming that the difference value between the kth light speckle detection image and the jth historical light speckle detection image satisfies the difference threshold exemplarily includes: confirming, one by one, whether the difference value between the kth light speckle detection image and each of the M historical light speckle detection images satisfies the difference threshold. In some embodiments of the present application, confirming that the difference value between the kth light speckle detection image and the jth historical light speckle detection image satisfies the difference threshold exemplarily includes: confirming whether the difference value between the kth light speckle detection image and the portion of the M historical light speckle detection images satisfies the difference threshold.

[0098] The following takes the example of calculating the difference value between the light spot detection map of the currently processed frame (i.e., the light spot detection map of the kth frame) and the light spot detection map of some historical frames until the difference value that meets the difference threshold is found, and illustrates the process of obtaining at least part of the historical frame light spot detection map from M historical frame light spot detection maps.

[0099] For example, in some embodiments of the present application, the value of j is i is an integer greater than or equal to 0. For example, if M=8 and i=0, the difference between the k-th frame light spot detection image and the k-8-th frame light spot detection image of the historical frame is determined. If M=8 and i=1, the difference between the k-th frame light spot detection image and the k-4-th frame light spot detection image of the historical frame is determined. It should be noted that is a number greater than or equal to 0, so the maximum value of i can be determined.

[0100] The following example illustrates the case where the value of j is How to obtain the quantified difference value between the k-th light spot detection image and the j-th historical frame light spot detection image.

[0101] The confirmation in step S101 that the difference between the k-th light spot detection image and the j-th historical frame light spot detection image satisfies the difference threshold may include: i starting from 0 and repeating the following process until the difference value obtained by the following method satisfies the difference threshold: calculating the difference between the k-th light spot detection image and the j-th historical frame light spot detection image; The absolute value of the difference between the pixel values ​​corresponding to each pixel point on the frame light spot detection image is obtained to obtain the i-th difference; calculate the sum of the k-th frame light spot detection image and the k-th frame light spot detection image. The absolute value of the difference between the pixel values ​​corresponding to each pixel point on the frame light speckle detection image is obtained to obtain the i+1th difference; the difference between the i-th difference and the i+1th difference is calculated, and the sum of the i-th difference and the i+1th difference is calculated; the difference value is obtained according to the ratio of the difference to the sum; wherein the "[]" operator represents a rounding operation. It should be noted that the above-mentioned rounding operation includes rounding up or rounding down, and these formulas are also included in the following concept provided above in this application: that is, by obtaining the difference in light speckle distribution on different frames of original images, at least some historical frame light speckle detection images are selected from M historical frame light speckle detection images, so as to fuse these historical frame light speckle detection images with the k-th frame light speckle detection image to obtain the k-th frame light speckle weight map.

[0102] For example, if M=8, k=16 and i=0, the zeroth difference value is first obtained as follows: the absolute value of the difference between the 16th frame light spot detection image and each pixel on the 8th frame light spot detection image corresponding to the 8th frame original image is calculated to obtain the i-th difference value; the absolute value of the difference between the 16th frame light spot detection image and each pixel on the 12th frame light spot detection image corresponding to the 12th frame original image is calculated to obtain the i+1-th difference value; the difference between the i-th difference value and the i+1-th difference value is calculated, and the sum of the i-th difference value and the i+1-th difference value is calculated; the zeroth difference value is obtained according to the ratio of the difference to the sum, and it is confirmed that the zeroth difference value does not meet the difference threshold. It is further necessary to use the following method to calculate the corresponding first difference value when i=1: calculate the absolute value of the difference between the 16th frame light spot detection image and the 12th frame light spot detection image corresponding to the 12th frame original image to obtain the i-th difference value; calculate the absolute value of the difference between the 16th frame light spot detection image and the 14th frame light spot detection image corresponding to the 14th frame original image to obtain the i+1-th difference value; calculate the difference between the i-th difference value and the i+1-th difference value, and calculate the sum of the i-th difference value and the i+1-th difference value; obtain the first difference value according to the ratio of the difference to the sum, confirm that the first difference value meets the difference threshold, and then use the 3 historical frame light spot detection images from the 12th frame to the 15th frame as the M historical frame light spot detection images involved in S101.

[0103] It is understandable that if, after multiple cycles, it is finally confirmed that the difference values ​​between the k-th frame light speckle detection image and all the j-th frame historical frame light speckle detection images do not meet the difference threshold, then the historical frame light speckle detection image closest to the k-th frame can be selected as the historical frame light speckle detection image selected from the M historical frame light speckle detection images. For example, in some embodiments of the present application, the method for processing light speckle flicker in the video blur mode further includes: confirming that the difference values ​​between the k-th frame light speckle detection image and all the j-th frame historical frame light speckle detection images do not meet the difference threshold, then using the k-1-th frame light speckle detection image as the X historical frame light speckle detection images, wherein the j-th frame light speckle detection image belongs to a frame image in the M frame light speckle detection images. That is to say, in some embodiments of the present application, if it is confirmed that the differences between the stored historical frame light spot detection map and the current frame light spot detection map are not large, then a historical frame light spot detection map closest to the current frame (i.e., the kth frame) is selected to adjust the kth frame light spot detection map to obtain the kth frame light spot weight map. This improves the light spot flickering problem and minimizes the data processing amount.

[0104] It should be noted that in some embodiments of the present application, the kth light speckle detection image obtained in the above steps and the M historical light speckle detection images can also be placed in a queue of length M+1. A portion of the historical light speckle detection images can then be selected from the queue as reference frames for adjusting the kth light speckle detection image (the reference frames being at least a portion of the historical light speckle detection images selected from the M historical light speckle detection images). In this case, the difference value can be calculated based on the order of the historical light speckle detection images in the queue. This process will be explained below with reference to examples, and to avoid repetition, it will not be elaborated on here.

[0105] In the second step, the X historical frame light spot detection images and the k-th frame light spot detection image are fused to obtain the k-th frame light spot fusion image.

[0106] For example, a weighted average is performed on the values ​​of the same pixel points in the at least a portion (i.e., X) of the historical light speckle detection images obtained in the first step and the light speckle detection image of the kth frame to obtain a weighted average value of each pixel point, and then the light speckle fusion image of the kth frame is obtained based on the weighted average value of each pixel point. It will be understood that those skilled in the art may also use other fusion methods.

[0107] In the third step, the k-th frame light spot weight map is obtained based on the k-th frame light spot fusion map. For example, in some embodiments of the present application, the k-th frame light spot fusion map is directly subjected to pixel value normalization processing to obtain the k-th frame light spot weight map. In other embodiments of the present application, the k-th frame light spot fusion map is pixel value normalized and stretched, and the stretched map is used as the k-th frame light spot weight map. In still other embodiments of the present application, the k-th frame light spot fusion map is pixel value normalized, stretched, and the stretched map is smoothed according to the historical frame processing results, and the smoothed map is used as the k-th frame light spot weight map.

[0108] The following exemplifies how S101 adjusts the k-th frame light speckle detection image according to the selected X historical frame light speckle detection images to obtain the k-th frame light speckle weight map.

[0109] It should be noted that some embodiments of the present application obtain the light spot weight map of the current frame by fusing the pixel values ​​on the light spot detection map of the current frame (i.e., the kth frame) with the pixel values ​​on at least part of the image screened out from the light spot detection maps of M historical frames, thereby effectively improving the light spot flickering phenomenon between the previous and next frames and improving the video light spot display effect.

[0110] For example, in some embodiments of the present application, S101 exemplarily includes:

[0111] The first step is to fuse the X historical frame light speckle detection images with the k-th frame light speckle detection image to obtain the k-th frame light speckle fusion image. For example, in some embodiments of the present application, this first step exemplarily includes: performing weighted averaging processing on the values ​​of corresponding pixels on the M historical frame light speckle detection images and the k-th frame light speckle detection image to obtain a weighted average value of each pixel, and then obtaining the k-th frame light speckle fusion image based on the weighted average value of each pixel. In other words, some embodiments of the present application fuse the k-th frame light speckle detection image with the M historical frame light speckle detection images through a weighted averaging algorithm, so that the light spot distribution characteristics of the obtained k-th frame light speckle fusion image are as close as possible to the light spot distribution characteristics of the previous historical frames, effectively improving the light spot flickering phenomenon existing in the synthesized video.

[0112] The second step is to obtain the k-th frame light spot weight map according to the k-th frame light spot fusion map.

[0113] For example, in some embodiments of the present application, obtaining the k-th frame light spot weight map based on the k-th frame light spot fusion map includes: smoothing the k-th frame light spot fusion map or the k-th frame light spot fusion stretched map obtained by stretching the k-th frame light spot fusion map to obtain the k-th frame light spot weight map.

[0114] In some embodiments of the present application, the above-mentioned process of smoothing the k-th frame light spot fusion image to obtain the k-th frame light spot weight map exemplarily includes: performing weighted averaging of the values ​​of corresponding pixel points on the k-1-th frame light spot fusion image and the k-th frame light spot fusion image in different proportions to obtain a weighted average value of each pixel point, and using the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-th frame light spot weight map; or, performing weighted averaging of the values ​​of the same pixel points on the k-1-th frame light spot fusion stretching image and the k-th frame light spot fusion image in different proportions to obtain a weighted average value of each pixel point, and using the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-th frame light spot weight map.

[0115] In some embodiments of the present application, the above-mentioned process of smoothing the k-th frame light spot fusion stretched image obtained by stretching the k-th frame light spot fusion image to obtain the k-th frame light spot weight map exemplarily includes: performing weighted averaging of the values ​​of the same pixel points on the k-1-th frame light spot fusion stretched image and the k-th frame light spot fusion stretched image in different proportions to obtain the weighted average value of each pixel point, and using the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-th frame light spot weight map; or, performing weighted averaging of the values ​​of the same pixel points on the k-1-th frame light spot fusion stretched image and the k-th frame light spot fusion stretched image in different proportions to obtain the weighted average value of each pixel point, and using the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-th frame light spot weight map.

[0116] That is to say, some embodiments of the present application introduce weighted averaging algorithms with different proportions to smooth the current frame light spot detection image based on the historical frame light spot detection image, thereby further improving the light spot display effect in the synthesized video and reducing the light spot flickering phenomenon between adjacent frames in the synthesized target video.

[0117] For example, in some embodiments of the present application, the light spot weight map of the kth frame is obtained by the following formula:

[0118]

[0119] in, Characterize the k-1th frame of light spot fusion stretching image, Characterize the k-th frame light spot fusion stretching image, I result_spot Characterizing the light spot weight map of the kth frame, the value range of n is greater than or equal to 2 and less than or equal to 5. Some embodiments of the present application further improve the light spot flickering phenomenon between adjacent frames in the target video by introducing a smoothing formula.

[0120] The following exemplary S102 is an implementation process of obtaining the kth frame target object light spot blurred rendering image according to the kth frame light spot weight map and the kth frame original image. It can be understood that the kth frame light spot weight map has been obtained after executing S101.

[0121] H) Perform saturation enhancement on the k-th frame original image (i.e., the image to be blurred) to obtain a saturation enhancement image I′ image The purpose of obtaining this image is to increase the gorgeousness of the blurred background portrait and the diversity of the light spot color.

[0122] I) Using the saturation enhancement map and formula I' image ×(1-I mask ) obtains the saturation enhanced background segmentation map I' except the target object (eg, portrait) bg , the portrait area in this image is filled with 0.

[0123] J) Saturation enhanced background segmentation image I' bg Perform gamma transformation (for example, gamma = 1.6) to nonlinearly stretch the background brightness to obtain the stretched background segmentation image I' gamma_bg .

[0124] K) Perform the following spot area brightness gain operation on the background:

[0125] vi. The original image I image Convert to YUV format image, where the single channel Y image is named I gray .

[0126] vii. will I gray Normalized to [0,1], we get I light_weight , then the whole is +1, and the value range is [1, 2].

[0127] viii. According to I light_weight *I' gamma_bg Get the image after brightness gain I image4render , that is, the background image I to be rendered for the kth frame is obtained image4render .

[0128] L) at least according to the k-th frame light spot weight map and the k-th frame background image I to be rendered image4render Execute the spot rendering method to obtain the background spot blurred rendering image of the kth frame.

[0129] It should be noted that in order to execute the spot rendering method, it is also necessary to obtain a depth of field image. The process exemplarily includes the following operations: image Perform depth estimation and use deep learning methods (for example, mono depth, midas models, etc.) to obtain the depth map I depthWhen the mobile phone camera is fixed in focus, it will return the coordinate position (x0, y0) of the portrait in the image. The depth of field image I is obtained by the following formula coc ;

[0130]

[0131] Among them, coc represents the depth of field, that is, the size of the spot radius required for rendering.

[0132] For example, the k-th frame spot weight map, I image4render and I coc As input, it is sent to the convolution-based spot rendering method and gamma transformation is performed again (for example, gamma = 1 / 1.6), and finally the background spot blurred rendering image of the kth frame is obtained.

[0133] O) Fusing the target object (eg, a portrait) after saturation gain with the k-th frame of background light spot blurred rendering image to obtain the k-th frame of target object light spot blurred rendering image.

[0134] It is not difficult to understand that by repeatedly executing the above process, a blurred rendering image of the target object light spot corresponding to each frame in the video to be blurred can be obtained, and the target video is obtained.

[0135] The following combination Figure 3 The image processing method performed by the mobile phone terminal is exemplified.

[0136] Some embodiments of the present application can solve the problem of severe flickering of light spots generated in the mobile phone video blur mode, and simulate the light spot effect close to the shooting effect of SLR. For example, in some embodiments of the present application, a deep learning model (for example, unet, resnet frame segmentation model) is first used to perform portrait segmentation on each frame of the original image taken, and a depth estimation model is used to obtain a disparity estimation map coc distribution map, and then a single image is blurred and rendered by convolution. In some embodiments of the present application, in order to obtain the light spot weight map corresponding to the original image of the current processing frame, it is necessary to fuse the light spot detection map of the current processing frame with the light spot detection map of the historical frame. Since the light spot weight map of the current processing frame obtained in this way fuses the light spot detection map of the historical frame, the problem of video light spot flickering can be solved, making the video portrait blur effect more natural and the blurred light spot effect more stable. The following is combined with Figure 3 The method for processing light spot flicker in the video blur mode executed by a mobile phone is exemplified.

[0137] S201, the mobile phone camera captures the original image of the current processing frame (ie, the kth original image).

[0138] It can be understood that after the user points to the portrait and clicks video recording in the current processing frame original image, the mobile device background obtains a frame original image (i.e., the kth frame original image) in a video.

[0139] Assume that the mobile device obtains the RGB format image I to be blurred at the current moment image , the RGB format image to be blurred is the k-th frame original image.

[0140] S202: Obtain a light spot detection map of the current processing frame according to the original image of the current processing frame. It can be understood that the light spot detection point map of each frame is obtained by a light spot detection method.

[0141] The following exemplifies the process of obtaining the light speckle detection image of the current processing frame. It can be understood that the following example is also applicable to the process of obtaining the light speckle detection images of other historical frames.

[0142] 1. For the current processing frame original image I image Perform portrait segmentation and use deep learning models (such as unet, resnet and other frameworks) to obtain a single-channel binary image I mask , where the pixel value of the subject (i.e., portrait) corresponding to the pixel on the single-channel binary image is 1 and the background is 0.

[0143] 2. Use I image ×(1-I mask ) Get the background image I except the portrait bg , the portrait area is filled with 0;

[0144] 3. Based on I bg The R, G and B channels are traversed pixel by pixel to find the maximum value of the three channels, that is, the single channel image I maxch_bg ;

[0145] 4. Based on I maxch_bg Perform light spot detection to obtain the corresponding frame light spot detection image. The detailed process is as follows:

[0146] 4-1 pair single channel I maxch_bg Perform blurring (for example, using 3*3 convolution and Gaussian blurring on a single-channel image) to obtain a blurred single-channel image I smooth_maxch_bg ;

[0147] 4-2 versus I maxch_bg and I smooth_maxch_bg Traverse, if the pixel at the (x, y) position meets the following conditions, it is considered to be a candidate light spot (i.e. light spot), and the current frame light spot detection image I is obtained init_spots

[0148]

[0149] It should be noted that the current frame light spot detection map obtained from 4-2 can be further processed to obtain a more optimized current frame light spot detection map (for example, performing sparsification processing and dot filling operations on the current processed frame light spot detection map. For specific examples, refer to the above examples of obtaining the light spot detection map. To avoid repetition, no further elaboration will be provided here). Accordingly, the light spot detection map required for S101 belongs to a more optimized current processed frame light spot detection map.

[0150] S203. Obtain the historical frame light spot detection maps. It can be understood that the obtaining methods of the historical frame light spot detection maps can refer to the above step S202. To avoid repetition, no further elaboration will be provided here.

[0151] S204. Create a sequence frame buffer queue Q.

[0152] For example, when Q is set to 10, store the historical frame light spot detections (9 frames) and the current processed frame light spot detection map in the fixed-length queue container Q.

[0153] That is to say, create a global buffer queue Q to cache the generated light spot detection maps. The length of Q is 10, and the head address always stores the light spot detection map of the current processed frame, i.e., highlight spots. By analogy, the last frame of Q stores the historical frame light spot detection map that is the farthest from the current frame.

[0154] S205. Select at least a part (corresponding to the above X frames) of the historical frame light spot detection maps from the cache queue Q according to the inter-frame difference.

[0155] That is to say, determine the number of frames m (m < len(Q)) required for optimizing the flicker through the inter-frame difference detection mechanism. len(Q) represents the length of the cache queue Q, and Q = M.

[0156] The number of frames required for each frame of flicker optimization processing is uncertain and needs to be adaptively selected through the difference information of consecutive sequences in Q. It specifically includes the following two steps:

[0157] The first step: With Q having a length of 10, calculate multiple difference maps through the following four calculations. Specifically:

[0158] For the first time, take the absolute value of the difference between the head and tail frames pre-stored in Q to generate a new mat map The following formula:

[0159]

[0160] That is, the maximum difference light spot distribution map is obtained based on the difference between the value of the same position point (or the same pixel point) on the light spot detection map of the current processing frame and the light spot detection map of the earliest historical frame in the cache queue. The maximum difference light spot distribution diagram The value of each pixel on is a positive number or 0.

[0161] Then, the absolute value difference between the middle frame in Q and the current frame is taken for the second time to generate a new mat graph. As shown in the following formula:

[0162]

[0163] where N cur It is represented by the sequence number of the spot detection point of the current frame in the queue (that is, the sequence number stored in the cache queue, and the spot detection images of each historical frame are stored in the order of generation), and N0 represents the sequence number of the object to be subtracted when the absolute difference is first performed. It can be simplified into the following formula:

[0164]

[0165] And so on.

[0166] The third absolute difference simplification is:

[0167]

[0168] The fourth absolute difference simplification is:

[0169]

[0170] In the second step, set the fixed threshold parameter thresh_diff to 0.35 (as a specific example of a difference threshold, the value range of the difference threshold is greater than or equal to 0.2 or less than or equal to 0.45), which must satisfy the following inequality:

[0171]

[0172] Among them, s∈{0,1,2,3} and are executed in sequence. If s=0, the information of the ten frames numbered 0 to 9 needs to be provided to the flicker optimization module (that is, all are selected from the 9 frames of historical frame light spot detection maps, and the ten frames of 0 to 9 light spot detection maps are fused to obtain the light spot weight map of the 9th frame of the current processing frame), that is, the subsequent flicker optimization module is at least configured to obtain the light spot weight map corresponding to the 9th frame by fusing the light spot detection maps of these 10 frames. If s=1 is established, the information of the six frames numbered 4 to 9 needs to be provided to the flicker optimization module (i.e., 5 frames of the most recent historical frame images are selected from the 9 frames of historical frame light spot detection maps, and the six frames of light spot detection maps from 4 to 9 are fused to obtain the light spot weight map of the 9th frame of the current processing frame), that is, the subsequent flicker optimization module is at least configured to obtain the light spot weight map corresponding to the 9th frame by fusing the light spot detection maps of these 6 frames; if s=2 is established, the information of the four frames numbered 6 to 9 needs to be provided to the flicker optimization module (i.e., 3 frames of the most recent historical frame images are selected from the 9 frames of historical frame light spot detection maps, and the four frames of light spot detection maps from 6 to 9 are fused to obtain the light spot weight map of the 9th frame of the current processing frame), that is, the subsequent flicker optimization module is at least configured to obtain the light spot weight map corresponding to the 9th frame by fusing the light spot detection maps of these 4 frames. If s=3, then the information of the three frames numbered 7 to 9 needs to be provided to the flicker optimization module (i.e., the three most recent historical frame images are selected from the 9 historical frame light speckle detection maps, and the three light speckle detection maps of frames 7 to 9 are fused to obtain the light speckle weight map of the current processing frame, frame 9). The subsequent flicker optimization module is at least configured to obtain the light speckle weight map corresponding to frame 9 by fusing the light speckle detection maps of these three frames. If it is still not satisfied, only the information of frames 8 and 9 needs to be provided to the flicker optimization module (i.e., the most recent historical frame image is selected from the 9 historical frame light speckle detection maps, and the two light speckle detection maps of frames 8 to 9 are fused to obtain the light speckle weight map of the current processing frame, frame 9). The subsequent flicker optimization module is at least configured to obtain the light speckle weight map corresponding to frame 9 by fusing the light speckle detection maps of these two frames. It can be understood that in these examples, frame 9 is the current processing frame, and the frame number of the light speckle detection map of frame 9 is the number in the cache queue.

[0173] It should be noted that in some embodiments of the present application, when it is determined that s=0, the difference between the light spot detection image of the current processing frame and the light spot detection image of the 0th frame meets the difference threshold (the frame number in the cache queue, not the absolute frame number in the video to be blurred), then there is no need to calculate and And all the light spot detection images corresponding to the 0th frame to the current processing frame are fused to obtain the light spot fusion image of the current processing frame. The value is to calculate the k-th frame light spot detection image and the The absolute value of the difference between the corresponding pixel values ​​of each pixel on the frame light spot detection image is used to obtain the i-th difference. The value is to calculate the k-th frame light spot detection image and the The absolute value of the difference between the pixel values ​​corresponding to each pixel point on the frame light spot detection image is obtained to obtain the i+1th difference value.

[0174] S206 , performing weighted averaging of at least a portion of the historical frame light speckle detection images and the current frame light speckle detection image to generate a current frame light speckle fusion detection image.

[0175] For example, according to S205, light spots of continuous m+1 frames including the current light spot detection map can be obtained, and the light spot detection map of the m+1 frame is used as a basis to obtain a flicker optimization weight map, that is, all the continuous m+1 frames are weighted averaged to generate a light spot fusion detection map of the current processing frame containing the information of the m+1 frame.

[0176] S207 , performing curve stretching enhancement on the light spot fusion detection image of the current processing frame to obtain the light spot fusion stretching image of the current processing frame.

[0177] That is, the light spot fusion detection map obtained in S206 is stretched to increase the overall value, and the obtained light spot fusion stretching map of the current processing frame is recorded as

[0178] S208: Acquire the previous frame of the light spot fusion stretching image (i.e., the previous frame adjacent to the current processing frame). It is understood that the process of acquiring the previous frame of the light spot fusion stretching image is the same as the process of acquiring the fusion stretching image of the current processing frame. Please refer to the above for details, and we will not elaborate on it here to avoid repetition.

[0179] S209 , performing weighted averaging processing of different proportions on the light spot fusion stretching map of the current processing frame and the light spot fusion stretching map of the previous frame to obtain a light spot weight map of the current frame.

[0180] For example, the following formula is used to perform weighted averaging of different proportions:

[0181]

[0182] in, Indicates the light spot fusion stretching image of the current processing frame, Indicates the previous frame of light spot fusion stretching image.

[0183] S210 , separating the foreground and background of the original image of the current processing frame according to the portrait segmentation model to obtain a background segmentation image and a portrait segmentation image.

[0184] S211, perform pre-processing such as enhancement, segmentation, gamma correction and stretching on the original image of the current frame, and finally obtain the background image I to be rendered for the current processing frame. image4render The specific implementation process can refer to the above-mentioned process of obtaining the background image I of the kth frame to be rendered. image4render To avoid repetition, I will not go into details here.

[0185] S212, using a spot rendering method to obtain a spot blur rendering image of the target object in the current processing frame. For example, the portrait segmentation image of the current processing frame and the background spot rendering image of the current processing frame are merged to obtain a final portrait spot blur rendering image (i.e., a specific example of the target object spot blur rendering image). That is, according to the background image I to be rendered in the current processing frame image4render A spot rendering method is adopted with the spot weight map of the current processing frame to obtain a spot blurred rendering map of the target object of the current processing frame. For the specific process, reference may be made to the exemplary description of S102.

[0186] That is, the light spot weight map of the current processing frame obtained in S209 and the pre-processed background image I to be rendered of the current processing frame are combined. image4render The light spot rendering method is input to obtain the background light spot rendering image of the current processing frame. It can be understood that performing image rendering on the light spot weight map in S101 can significantly improve the light spot flickering problem in the target video.

[0187] S213 , repeating the above process to obtain multiple frames of target object light spot blurred rendering images, and obtaining a target video according to the multiple frames of target object light spot blurred rendering images.

[0188] Please refer to Figure 4 , Figure 4 The image processing device provided by the embodiment of the present application is shown. It should be understood that the device is similar to the above-mentioned Figure 2 The method embodiment corresponds to the method embodiment and can execute each step involved in the above method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or fixed in the operating system of the device. The device for processing light spot flicker in the video blur mode includes: a light spot weight map acquisition module 101 for the kth frame and a light spot blur rendering image module 102 for the target object for the kth frame. In some embodiments of the present application, the device also includes a target video acquisition module 103.

[0189] The k-th frame light speckle weight map acquisition module 101 is configured to obtain the k-th frame light speckle weight map based on M historical frame light speckle detection maps and the k-th frame light speckle detection map, wherein the M historical frame light speckle detection maps are obtained by performing light speckle detection on M frames of original images respectively, and the k-th frame light speckle detection map is obtained by performing light speckle detection on the k-th frame original image, the M frames of original images and the k-th frame original image are located in the same video to be blurred, and the M frames of original images are one frame or multiple consecutive frames of images located before the k-th frame original image and adjacent to the k-th frame original image, and M is an integer greater than or equal to 1.

[0190] The k-th frame target object light spot blurred rendering image module 102 is configured to obtain the k-th frame target object light spot blurred rendering image according to the k-th frame light spot weight map and the k-th frame original image.

[0191] The target video acquisition module 103 is configured to acquire multiple frames of target object light spot blurred rendering images, and obtain a target video according to the multiple frames of target object light spot blurred rendering images.

[0192] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0193] Some embodiments of the present application provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the method described in any of the above embodiments.

[0194] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method described in any of the above embodiments can be implemented.

[0195] like Figure 5 As shown, some embodiments of the present application provide an electronic device 500, including a memory 510, a processor 520, and a computer program stored on the memory 510 and executable on the processor 520, wherein the processor 520 can implement the method described in any of the above embodiments when reading the program from the memory 510 through a bus 530 and executing the program.

[0196] Processor 520 can process digital signals and can include various computing architectures, such as a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0197] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the present disclosure may be used to execute the instructions in the memory 510 to implement Figure 2 The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0199] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0200] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0201] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0202] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0203] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for image processing, characterized in that: The method comprises: obtaining a k-th frame light speckle weight map according to M historical frame light speckle detection maps and a k-th frame light speckle detection map, wherein the M historical frame light speckle detection maps are obtained by performing light speckle detection on M frames of original images, the k-th frame light speckle detection map is obtained by performing light speckle detection on the k-th frame original image, the M frames of original image are one or more consecutive frames of images located before and adjacent to the k-th frame original image, and M is an integer greater than or equal to 1; Obtaining a k-th frame of target object light spot blur rendering image according to the k-th frame of light spot weight map and the k-th frame of original image, wherein the k-th frame of target object light spot blur rendering image is the acquired target image; The step of obtaining a k-th frame target object light spot blur rendering image according to the k-th frame light spot weight map and the k-th frame original image includes: Preprocessing the k-th original image to obtain the k-th background image to be rendered; Executing a light spot rendering method according to the light spot weight map of the kth frame and the background image to be rendered of the kth frame to obtain a background light spot blurred rendering image of the kth frame; The target object after saturation gain is fused with the k-th frame background light spot blurred rendering image to obtain the k-th frame target object light spot blurred rendering image.

2. The method according to claim 1, wherein Before obtaining the k-th frame light spot weight map according to the M historical frame light spot detection maps and the k-th frame light spot detection map, the method further includes: Performing image segmentation on any one of the M frames of original image and the k-th frame of original image to obtain a background segmentation map; Obtaining a single-channel image based on the maximum value of multiple channels of each pixel in the background segmentation image; A light spot detection image corresponding to any frame of the original image is obtained at least according to the value of each pixel point on the single-channel image.

3. The method according to claim 2, wherein The step of obtaining a light spot detection image corresponding to any frame of the original image at least according to the pixel value of each pixel point on the single-channel image includes: Performing fuzzification processing on the single-channel image to obtain a fuzzy single-channel image; If the value of any pixel point on the single-channel image is greater than a first threshold, and the value of any pixel point on the blurred single-channel image is less than or equal to a second threshold, it is confirmed that the any pixel point belongs to a light spot, wherein the second threshold is less than the first threshold.

4. The method according to any one of claims 1 to 3, wherein M is an integer greater than 1, where The step of obtaining the k-th frame light spot weight map according to the M historical frame light spot detection maps and the k-th frame light spot detection map includes: Selecting X historical frame light speckle detection images from the M historical frame light speckle detection images, wherein the X historical frame light speckle detection images are adjacent to the k-th frame light speckle detection image, and X is an integer greater than or equal to 1 and less than or equal to M; Fusing the X historical frame light spot detection images and the k-th frame light spot detection image to obtain the k-th frame light spot fusion image; The k-th frame light spot weight map is obtained according to the k-th frame light spot fusion map.

5. The method according to claim 4, wherein The selecting X historical frame light spot detection images from the M historical frame light spot detection images includes: The X historical frame light spot detection images are selected from the M historical frame light spot detection images according to distribution information of light spots on original images of different frames.

6. The method according to claim 4, wherein The selecting X historical frame light spot detection images from the M historical frame light spot detection images includes: If the difference value between the k-th frame light speckle detection image and the j-th historical frame light speckle detection image meets the difference threshold, the light speckle detection images from the j-th frame to the (k-1)-th frame are used as the X historical frame light speckle detection images, wherein the j-th historical frame light speckle detection image belongs to a frame image of the M historical frame light speckle detection images.

7. The method according to claim 6, wherein If the value of j is , i is an integer greater than or equal to 0, where, The confirming that the difference between the k-th light spot detection image and the j-th historical frame light spot detection image satisfies a difference threshold includes: The following process is repeated starting from 0 until the difference value obtained in the following manner meets the difference threshold: Calculate the k-th frame light spot detection image and the The absolute value of the difference between the corresponding pixel values ​​of each pixel point on the frame light spot detection image is obtained to obtain the i-th difference value; Calculate the k-th frame light spot detection map and the The absolute value of the difference between the corresponding pixel values ​​of each pixel point on the frame light spot detection image is obtained to obtain the i+1th difference value; Calculating the difference between the i-th difference value and the i+1-th difference value, and calculating the sum of the i-th difference value and the i+1-th difference value; Obtaining the difference value according to the ratio of the difference to the sum; in," "The operator symbol indicates the rounding operation.

8. The method according to claim 6, wherein The selecting X historical frame light spot detection images from the M historical frame light spot detection images further includes: If the difference values ​​between the k-th frame light speckle detection image and the j-th historical frame light speckle detection image do not meet the difference threshold, the (k-1)-th frame light speckle detection image is used as the X historical frame light speckle detection images, wherein the j-th historical frame light speckle detection image belongs to a frame image among the M historical frame light speckle detection images.

9. The method according to claim 4, wherein The step of obtaining the k-th frame light spot weight map according to the k-th frame light spot fusion map includes: Performing curve stretching on the k-th frame of the light spot fusion image to obtain the k-th frame of the light spot fusion stretched image; The k-th frame light spot weight map is obtained according to the k-th frame light spot fusion stretching map.

10. The method according to claim 4, wherein The step of obtaining the k-th frame light spot weight map according to the k-th frame light spot fusion map includes: The k-th frame light spot weight map is obtained by performing a smoothing process on the k-th frame light spot fusion map or the k-th frame light spot fusion stretched map obtained by stretching the k-th frame light spot fusion map.

11. The method according to claim 10, wherein The step of performing smoothing on the k-th frame light spot fusion image to obtain the k-th frame light spot weight image includes: Performing weighted averaging of the values ​​of corresponding pixels on the k-1th frame of the light spot fusion image and the kth frame of the light spot fusion image in different proportions to obtain a weighted average value of each pixel, and using the weighted average value of each pixel as the weight value of the corresponding pixel on the kth frame of the light spot weight map; or Performing weighted averaging of the values ​​of corresponding pixels on the k-1th frame of the light spot fusion stretching image and the kth frame of the light spot fusion image in different proportions to obtain a weighted average value of each pixel, and using the weighted average value of each pixel as the weight value of the corresponding pixel on the kth frame of the light spot weight map; or, The step of performing smoothing on the k-th frame light spot fusion stretched image obtained by stretching the k-th frame light spot fusion image to obtain the k-th frame light spot weight map includes: performing weighted averaging of the values ​​of corresponding pixels on the k-1th frame of the light spot fusion image and the kth frame of the light spot fusion stretching image in different proportions to obtain a weighted average value of each pixel, and using the weighted average value of each pixel as the weight value of the corresponding pixel on the kth frame of the light spot weight map; or The values ​​of the same pixel points on the k-1th frame light spot fusion stretching image and the kth frame light spot fusion stretching image are weighted averaged in different proportions to obtain the weighted average value of each pixel point, and the weighted average value of each pixel point is used as the weight value of the corresponding pixel point on the kth frame light spot weight map.

12. The method according to claim 1, wherein The method further includes: repeating the process of acquiring the target image to obtain a target video, wherein the target video includes multiple target images.

13. A computer program product, characterized in that The computer program product comprises a computer program, wherein when the computer program is executed by a processor, it can implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 12 can be implemented.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 12 can be implemented.

Images