Method, apparatus, computer device and storage medium for replacing sky background

Abstract

The present application relates to a method, device, computer device and storage medium for replacing a sky background. The method includes obtaining a target background and a sample to be processed selected by a user. If the type of the sample to be processed is a video type, the sample to be processed is frame-divided to obtain a set of video frames to be processed; obtaining a current frame and adjacent frames; performing segmentation processing on the sky background in the current frame to obtain a sky mask image; constructing a transformation matrix based on the sky mask image and the adjacent frames, and performing affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image; performing color harmony processing on the initial sky background image based on the target background to obtain a target sky background image of the current frame; and generating a target sky background video when all video frames are processed. The present invention replaces the sky background of the current frame through the information of adjacent frames, which is beneficial to improving the accuracy of sky background replacement.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to a method, device, computer device and storage medium for replacing the sky background. Background Art

[0002] As an important part of an image or video, the sky often makes it difficult to obtain the expected sky when shooting due to the uncontrollability of the weather. Therefore, based on algorithms, it is convenient for users to replace the unsatisfactory sky in the video or image with any desired appearance in a more convenient way. Whether it is a starry sky, a bright moon, or lightning and thunder, various weather effects can be generated with one click through algorithms.

[0003] The existing sky background replacement method is to first perform semantic segmentation on the image or video frame to obtain a mask image, which annotates the confidence of each pixel point being the sky, and then replace the original sky with a new sky background image according to the mask image to complete the sky replacement function. Since it is difficult to accurately identify the boundary between the new sky background and the sky background to be replaced, and when the scene of the front and rear frame images changes, the existing method cannot accurately capture the change of the sky background, resulting in poor accuracy of the existing method for sky background replacement. There is an urgent need for a method that can improve the accuracy of sky background replacement. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a method, device, computer device and storage medium for replacing the sky background to improve the accuracy of sky background replacement.

[0005] To solve the above technical problems, the embodiments of this application provide a method for replacing the sky background, including:

[0006] Obtain the target background and the sample to be processed selected by the user, and determine the type of the sample to be processed;

[0007] If the type of the sample to be processed is a video type, perform frame splitting on the sample to be processed to obtain a set of video frames to be processed;

[0008] Obtain the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed;

[0009] Perform segmentation processing on the sky background in the current frame to obtain a sky mask image;

[0010] Construct a transformation matrix based on the sky mask image and the adjacent frame, and perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image;

[0011] Based on the target background, perform color harmony processing on the initial sky background image to obtain the target sky background image of the current frame;

[0012] Determine whether the current frame is the last video frame in the set of video frames to be processed. If not, return to execute the step of obtaining the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed, obtaining multiple target sky background images of the current frame, and generating a target sky background video based on the multiple target sky background images of the current frame.

[0013] To solve the above technical problems, an embodiment of the present application provides a sky background replacement device, including:

[0014] A to-be-processed sample acquisition module, configured to acquire the target background and the to-be-processed sample selected by the user, and determine the type of the to-be-processed sample;

[0015] A video frame set generation module, configured to, if the type of the to-be-processed sample is a video type, perform frame splitting on the to-be-processed sample to obtain a set of video frames to be processed;

[0016] A to-be-processed image acquisition module, configured to acquire the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed;

[0017] A sky mask image generation module, configured to perform segmentation processing on the sky background in the current frame to obtain a sky mask image;

[0018] An initial sky background image generation module, configured to construct a transformation matrix based on the sky mask image and the adjacent frame, and perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image;

[0019] A color harmony processing module, configured to perform color harmony processing on the initial sky background image based on the target background to obtain the target sky background image of the current frame;

[0020] A target sky background video generation module, configured to determine whether the current frame is the last video frame in the set of video frames to be processed. If not, return to execute the step of obtaining the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed, obtaining multiple target sky background images of the current frame, and generating a target sky background video based on the multiple target sky background images of the current frame.

[0021] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors implement the method for replacing the sky background described in any one of the above.

[0022] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for replacing the sky background described in any one of the above.

[0023] The embodiments of the present invention provide a method, device, computer device and storage medium for replacing the sky background. Among them, the method includes: obtaining the target background and the sample to be processed selected by the user, and judging the type of the sample to be processed; if the type of the sample to be processed is a video type, performing frame splitting on the sample to be processed to obtain a set of video frames to be processed; obtaining the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed; performing segmentation processing on the sky background in the current frame to obtain a sky mask map; constructing a transformation matrix based on the sky mask map and the adjacent frame, and performing affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background map; performing color harmony processing on the initial sky background map based on the target background to obtain the target sky background map of the current frame; judging whether the current frame is the last video frame in the set of video frames to be processed, if not, returning to execute the step of obtaining the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed, obtaining target sky background maps of multiple current frames, and generating a target sky background video based on the target sky background maps of multiple current frames. The embodiments of the present invention perform frame splitting on the sample to be processed, obtain the current frame and the adjacent frame, and perform sky segmentation processing on the current frame, so as to realize the segmentation of the foreground and the sky background in the image, which is convenient for subsequent replacement of the sky background; then construct a transformation matrix based on the adjacent frame, and perform affine transformation processing on the target background based on the transformation matrix, so as to realize the replacement of the sky background of the current frame; when all video frames in the set of video frames are processed, generate a target sky background video, realizing the replacement of the sky background in the video, and at the same time combining the information of the adjacent frame to replace the sky background of the current frame, which is beneficial to improving the accuracy of sky background replacement. Description of the Drawings

[0024] In order to more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0025] Figure 1 It is a flowchart of an implementation of the method for replacing the sky background provided by an embodiment of the present application;

[0026] Figure 2 It is another flowchart of an implementation of a subprocess in the method for replacing the sky background provided by an embodiment of the present application;

[0027] Figure 3 It is another flowchart of an implementation of a subprocess in the method for replacing the sky background provided by an embodiment of the present application;

[0028] Figure 4 It is another flowchart of an implementation of a subprocess in the method for replacing the sky background provided by an embodiment of the present application;

[0029] Figure 5 It is another flowchart of an implementation of a subprocess in the method for replacing the sky background provided by an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram for comparing the replacement of the sky background of the first video frame provided by an embodiment of the present application;

[0031] Figure 7 It is a schematic diagram for comparing the replacement of the sky background of the middle video frame provided by an embodiment of the present application;

[0032] Figure 8 It is another flowchart of an implementation of a subprocess in the method for replacing the sky background provided by an embodiment of the present application;

[0033] Figure 9 It is a schematic diagram for comparing the replacement of the sky background of the first image provided by an embodiment of the present application;

[0034] Figure 10 It is a schematic diagram for comparing the replacement of the sky background of the middle image provided by an embodiment of the present application;

[0035] Figure 11 It is a schematic diagram of the device for replacing the sky background provided by an embodiment of the present application;

[0036] Figure 12 It is a schematic diagram of the computer device provided by an embodiment of the present application. Detailed implementation manners

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0038] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0039] In order to enable those skilled in the technical field to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0040] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0041] It should be noted that the method for replacing the sky background provided in the embodiments of this application is generally executed by a server. Correspondingly, the device for replacing the sky background is generally configured in the server.

[0042] Please refer to Figure 1 , Figure 1 which shows a specific implementation of the method for replacing the sky background.

[0043] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 the process sequence shown, and the method includes the following steps:

[0044] S1: Obtain the target background and the sample to be processed selected by the user, and determine the type of the sample to be processed.

[0045] Specifically, when the user needs to replace the sky background in a certain image or video, the user selects the final sky background to be replaced, that is, selects the target background. After the server receives the target background and the sample to be processed selected by the user, since the types of the sample to be processed include the image type and the video type, and the replacement methods adopted by the server are different for different types, it is necessary to determine the type of the sample to be processed.

[0046] S2: If the type of the sample to be processed is a video type, then perform frame splitting on the sample to be processed to obtain a set of video frames to be processed.

[0047] Specifically, if the type of the sample to be processed is a video type, then perform frame splitting on the sample to be processed to obtain each video frame, and then store each video frame in a database to obtain a set of processed video frames. Further, mark them in the order of the video frames, which is convenient for starting to replace the sky background from the first video frame, and at the same time is convenient for subsequent judgment of which frame the currently obtained current frame belongs to, facilitating the replacement of the sky background frame by frame for the video frames in turn, and is also conducive to superimposing the subsequently successfully replaced video frames into a video.

[0048] S3: Obtain the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed.

[0049] Specifically, the adjacent frame refers to the previous frame of the current frame. If the current frame is the first frame in the set of video frames to be processed, then use the current frame at this time as the adjacent frame.

[0050] S4: Perform segmentation processing on the sky background in the current frame to obtain a sky mask image.

[0051] Specifically, by identifying the sky background and foreground in the current frame, perform segmentation processing on the sky background to obtain a corresponding mask image mask, that is, obtain the sky mask image. Further, before step S4, judge whether the current frame is the last frame in the set of video frames to be processed. If it is the last frame, then after processing this video frame, all video frames will be processed.

[0052] Further, step S4 includes:

[0053] Compress the current frame to a preset size to obtain a target image of the current frame;

[0054] Input the target image of the current frame into a pre-trained segmentation model, and through the pre-trained segmentation model, identify the sky background in the target image of the current frame, and perform segmentation processing on the sky background to obtain a sky mask image.

[0055] Specifically, compress the current frame to a preset size to obtain the target image of the current frame. The preset size is set according to the actual situation and is not limited here. In a specific embodiment, the current frame is compressed to a size of 256×256×3. Then, input the target image of the current frame into a pre-trained segmentation model for encoding. That is, the pre-trained segmentation model identifies the sky background in the target image of the current frame through convolution operations and performs segmentation processing on the sky background. If the target image of the current frame is of the size 256×256×3, after encoding, a feature map of the size 16×16×112 is obtained; then, perform a decoding operation on the feature map encoded by the pre-trained segmentation model. After convolution and upsampling processing, and then perform a concatenation operation with the previous feature map, etc., to obtain a sky mask image. If the target image of the current frame is of the size 256×256×3, after decoding, a sky mask image of the size 256×256×1 is obtained. Further, the segmentation network used in the pre-trained segmentation model is Efficientnet-b0.

[0056] Further, before step S4, the segmentation network Efficientnet-b0 is trained to obtain a pre-trained segmentation model. The training process is to input the labeled sky background image and the target replacement background image. The labeled sky background image is segmented by the segmentation network Efficientnet-b0, and the obtained result is compared with the target replacement background image to obtain a loss value. Based on the loss value, adjust the parameters of the segmentation network Efficientnet-b0, and then segment the labeled sky background image until the loss value reaches a preset threshold to obtain a pre-trained segmentation model. The preset threshold is set according to the actual situation and is not limited here.

[0057] S5: Based on the sky mask image and the adjacent frames, construct a transformation matrix, and perform an affine transformation on the target background based on the transformation matrix to obtain an initial sky background image.

[0058] Specifically, according to the information comparing the sky mask image and the adjacent frames, construct the transformation matrix of the current frame, and then perform an affine transformation on the target background based on the transformation matrix to achieve the replacement of the sky background in the target background with the sky background in the sky mask image, and obtain an initial sky background image.

[0059] Please refer to Figure 2 , Figure 2 which shows a specific implementation manner of step S5, described in detail as follows:

[0060] S51: Perform corner detection on the sky mask image to obtain a corner feature map.

[0061] Please refer toFigure 3 , Figure 3 shows a specific implementation of step S51, which is described in detail as follows:

[0062] S511: Use the ShiTomasi algorithm to perform corner detection on the sky mask image to obtain initial corners.

[0063] S512: Take two adjacent initial corners as a corner combination, and calculate the distance between the corners in the corner combination to obtain the corner distance.

[0064] S513: Obtain the corner combinations with corner distances greater than the preset distance as target corner combinations, and generate a corner feature map based on the target corner combinations.

[0065] Specifically, the ShiTomasi algorithm is an improvement of the Harris algorithm. The most original definition of the Harris algorithm is to subtract the determinant value of matrix M from the trace of M, and then compare the difference with a pre-given threshold. Later, Shi and Tomasi proposed an improved method (i.e., the ShiTomasi algorithm). If the smaller of the two eigenvalues is greater than the minimum threshold, strong corners will be obtained. In the embodiments of the present application, the ShiTomasi algorithm is used to perform corner detection on the sky mask image to obtain initial corners, take two adjacent initial corners as a corner combination, calculate the distance between the corners in the corner combination to obtain the corner distance, then obtain the corner combinations with corner distances greater than the preset distance as target corner combinations, retain the target corner combinations, and delete other corner combinations, thereby generating a corner feature map. Among them, the preset distance is set according to the actual situation and is not limited here. In a specific embodiment, the preset distance is the distance between 30 pixel points.

[0066] S52: Based on the corner feature map, perform adjacent frame corner tracking processing to obtain the corners in the adjacent frame that are the same as the corner feature map as the basic corners.

[0067] Specifically, the Lucas-Kanade algorithm is used to track the corners in the corner feature map. The Lucas-Kanade algorithm inputs the adjacent frame and the current frame, as well as the corners detected in the previous step, and returns the coordinates of the tracked corners in the previous frame and the current frame, that is, returns the basic corners. The basic corners include the position information (coordinates) in the current frame and the position information (coordinates) in the adjacent frame. Among them, the Lucas–Kanade algorithm is an optical flow estimation algorithm for two-frame difference, which was proposed by Bruce D. Lucas and Takeo Kanade in 1981. Since the algorithm is easy to apply to a set of points in the input image, it has later become an important method for finding sparse optical flow.

[0068] S53: Obtain a preset number of target corner points from the basic corner points, and construct a transformation matrix based on the target corner points.

[0069] Please refer to Figure 4 , Figure 4 which shows a specific implementation manner of step S53, described in detail as follows:

[0070] S531: Obtain a preset number of target corner points from the basic corner points by using the RANSAC random sampling method.

[0071] S532: Respectively obtain the positions of the target corner points in the current frame and the adjacent frame to obtain the current position information and the adjacent position information.

[0072] S533: Obtain preset parameters, and calculate the corresponding values of the preset parameters based on the current position information and the adjacent position information through a preset formula to obtain the target parameter values.

[0073] Specifically, use the RANSAC random sampling method to obtain a preset number of target corner points from the basic corner points, and then respectively obtain the positions of the target corner points in the current frame and the adjacent frame to obtain the current position information and the adjacent position information. Then obtain the preset parameters, and calculate the corresponding values of the preset parameters based on the current position information and the adjacent position information through a preset formula to obtain the target parameter values. For example, the preset parameters are a1, a2, a3, a4, b1, b2, where a1 = s * cosθ, a2 = s * (-sinθ), a3 = s * sinθ, a4 = s * cosθ, where s represents the scaling ratio and θ represents the rotation angle, and then use the following preset formula for calculation;

[0074]

[0075] where x and y represent the adjacent position information, and x' and y' represent the current position information.

[0076] Among them, the preset number is set according to the actual situation and is not limited here. In a specific embodiment, 3 target corner points are obtained. Among them, RANSAC random sampling can estimate the parameters of a mathematical model iteratively from a set of observation data sets containing "outliers".

[0077] S534: Construct a transformation matrix based on the target parameter values.

[0078] Specifically, obtain the target parameter values through step S533, that is, obtain the corresponding values of a1, a2, a3, a4, b1, and b2, and then construct a transformation matrix through the target parameter values. Its transformation matrix is expressed as T:

[0079]

[0080] S54: Perform an affine transformation on the target background based on the transformation matrix to obtain an initial sky background image.

[0081] Specifically, perform an affine transformation on the target background through the transformation matrix so as to replace the sky background of the target background with the sky background of the current frame, thereby obtaining an initial sky background image.

[0082] S6: Based on the target background, perform color harmony processing on the initial sky background image to obtain the target sky background image of the current frame.

[0083] Please refer to Figure 5 , Figure 5 which shows a specific implementation manner of step S6 and is described in detail as follows:

[0084] S61: Calculate the color sampling means of the initial sky background image and the target background respectively to obtain the initial color sampling mean and the target color sampling mean.

[0085] S62: Calculate the difference between the initial color sampling mean and the target color sampling mean.

[0086] S63: Based on the difference, perform color harmony processing on the initial sky background image to obtain the target sky background image of the current frame.

[0087] Specifically, since the sky background of the target background is used to replace the sky background of the current frame to obtain the initial sky background image, and there may be a large color difference between the target background and the current frame, it is necessary to perform color harmony processing on the initial sky background image. The specific process is as follows: calculate the color sampling means of the initial sky background image and the target background respectively to obtain the initial color sampling mean and the target color sampling mean, and then calculate the difference between the initial color sampling mean and the target color sampling mean; based on the difference, perform color harmony processing on the initial sky background image by using a preset weight coefficient to obtain the target sky background image of the current frame. Among them, the preset weight is set according to the actual situation and is not limited here.

[0088] S7: Determine whether the current frame is the last video frame in the set of video frames to be processed. If not, return to execute the step of obtaining the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed, obtaining the target sky background images of multiple current frames, and generating a target sky background video based on the target sky background images of multiple current frames.

[0089] Specifically, it is determined whether the current frame is the last video frame in the video frame set to be processed. If so, it means that all the video frames in the video frame set to be processed have been replaced, and then the target sky background video can be generated. If not, the steps of obtaining the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed are returned, that is, returning to step S3 to replace the sky background until all the video frames are processed, so as to obtain the target sky background images of multiple current frames. Then, according to the sky mask image and the sequence of the video frames, the target sky background images of multiple current frames are superimposed to generate the target sky background video. Please refer to Figure 6 and Figure 7 , Figure 6 is a schematic diagram of sky background replacement comparison for the first video frame provided by an embodiment of the present application; Figure 7 is a schematic diagram of sky background replacement comparison for the middle video frame provided by an embodiment of the present application. In Figure 6 and Figure 7 , the left side of the image is the image before the sky background replacement of the current frame, and the right side is the image after the sky background replacement of the current frame.

[0090] In this embodiment, the target background and the sample to be processed selected by the user are obtained, and the type of the sample to be processed is determined. If the type of the sample to be processed is a video type, the sample to be processed is frame-divided to obtain a video frame set to be processed; the current frame and the adjacent frame corresponding to the current frame are obtained from the video frame set to be processed; the sky background in the current frame is segmented to obtain a sky mask image; a transformation matrix is constructed based on the sky mask image and the adjacent frame, and the target background is subjected to an affine transformation process based on the transformation matrix to obtain an initial sky background image; based on the target background, the initial sky background image is subjected to color harmony processing to obtain the target sky background image of the current frame; it is determined whether the current frame is the last video frame in the video frame set to be processed. If not, the steps of obtaining the current frame and the adjacent frame corresponding to the current frame from the video frame set to be processed are returned until all the video frames are processed, and the target sky background images of multiple current frames are obtained, and a target sky background video is generated based on the target sky background images of multiple current frames. By frame-dividing the sample to be processed, obtaining the current frame and the adjacent frame, and segmenting the sky in the current frame in the embodiment of the present invention, the foreground and the sky background in the image are segmented, which is convenient for subsequent replacement of the sky background; then, based on the adjacent frame, a transformation matrix is constructed, and the target background is subjected to an affine transformation process based on the transformation matrix, so as to realize the replacement of the sky background of the current frame; when all the video frames in the video frame set are processed, a target sky background video is generated, realizing the replacement of the sky background in the video, and at the same time combining the information of the adjacent frame to replace the sky background of the current frame, which is beneficial to improving the accuracy of sky background replacement.

[0091] Please refer to Figures 8 to 10 ,Figure 8 A specific implementation of the method for replacing the sky background is shown, Figure 9 which is a schematic diagram for comparing the replacement of the sky background of the first-frame image provided by an embodiment of the present application; Figure 10 which is a schematic diagram for comparing the replacement of the sky background of the middle-frame image provided by an embodiment of the present application.

[0092] S1: Obtain the target background and the sample to be processed selected by the user, and determine the type of the sample to be processed.

[0093] S1A: If the type of the sample to be processed is an image type, perform sky background segmentation processing on the image to be processed in the sample to be processed to obtain an image sky background mask map.

[0094] S1B: Superimpose the foreground of the sky background mask map on the target background to obtain an initial image sky background map.

[0095] S1C: Based on the target background, perform color harmony processing on the initial image sky background map to obtain a target image sky background map.

[0096] S1D: Determine whether there is still an image to be processed in the sample to be processed. If so, return to execute the step of performing sky background segmentation processing on the image to be processed in the sample to be processed to obtain an image sky background mask if the type of the sample to be processed is an image type, until all the images to be processed in the sample to be processed are processed to obtain multiple target image sky background maps.

[0097] S1E: Stitch the multiple target image sky background maps to obtain a sky background video.

[0098] Specifically, if the type of the sample to be processed is an image type, perform sky background segmentation processing on the image to be processed in the sample to be processed to obtain an image sky background mask. The segmentation processing process is the same as that in step S4. To avoid repetition, it will not be elaborated here. Then superimpose the foreground of the sky background mask map on the target background to obtain an initial image sky background map, and then based on the target background, perform color harmony processing on the initial image sky background map to obtain a target image sky background map. The color harmony processing is the same as that in step S6. To avoid repetition, it will not be elaborated here. Then determine whether there is still an image to be processed in the sample to be processed. If so, return to execute S1A until all the images to be processed in the sample to be processed are processed to obtain multiple target image sky background maps. Finally, stitch the multiple target image sky background maps in sequence to obtain a sky background video.

[0099] S1E: Stitch the multiple target image sky background maps to obtain a sky background video

[0100] Please refer to Figure 11, as an implementation of the above Figure 1 shown method, this application provides an embodiment of a sky background replacement device, and this device embodiment corresponds to Figure 1 the shown method embodiment, and this device can be specifically applied to various electronic devices.

[0101] As Figure 11 shown, the sky background replacement device of this embodiment includes: a to-be-processed sample acquisition module 81, a video frame set generation module 82, a to-be-processed image acquisition module 83, a sky mask map generation module 84, an initial sky background map generation module 85, a color harmony processing module 86, and a target sky background video generation module 87, where:

[0102] The to-be-processed sample acquisition module 81 is used to acquire the target background and the to-be-processed sample selected by the user, and judge the type of the to-be-processed sample;

[0103] The video frame set generation module 82 is used to perform frame splitting on the to-be-processed sample if the type of the to-be-processed sample is a video type, to obtain a to-be-processed video frame set;

[0104] The to-be-processed image acquisition module 83 is used to acquire the current frame and the adjacent frame corresponding to the current frame from the to-be-processed video frame set;

[0105] The sky mask map generation module 84 is used to perform segmentation processing on the sky background in the current frame to obtain a sky mask map;

[0106] The initial sky background map generation module 85 is used to construct a transformation matrix based on the sky mask map and the adjacent frame, and perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background map;

[0107] The color harmony processing module 86 is used to perform color harmony processing on the initial sky background map based on the target background to obtain the target sky background map of the current frame;

[0108] The target sky background video generation module 87 is used to judge whether the current frame is the last video frame in the to-be-processed video frame set. If not, it returns to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the to-be-processed video frame set until all video frames are processed, to obtain target sky background maps of multiple current frames, and generate a target sky background video based on the target sky background maps of multiple current frames.

[0109] Furthermore, the initial sky background map generation module 85 includes:

[0110] A corner point detection processing unit, used to perform corner point detection processing on the sky mask map to obtain a corner point feature map;

[0111] A corner point tracking processing unit, configured to perform corner point tracking processing on adjacent frames based on a corner point feature map, and obtain corner points in the adjacent frames that are the same as the corner point feature map as basic corner points;

[0112] A target corner point acquisition unit, configured to acquire a preset number of target corner points from the basic corner points, and construct a transformation matrix based on the target corner points;

[0113] An affine transformation unit, configured to perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background map.

[0114] Further, the corner point detection processing unit includes:

[0115] An initial corner point acquisition subunit, configured to perform corner point detection processing on the sky mask map by using the ShiTomasi algorithm to obtain initial corner points;

[0116] A corner point distance calculation subunit, configured to use two adjacent initial corner points as a corner point combination, and calculate the distance between the corner points in the corner point combination to obtain a corner point distance;

[0117] A corner point feature map generation subunit, configured to acquire corner point combinations with a corner point distance greater than a preset distance as target corner point combinations, and generate a corner point feature map based on the target corner point combinations.

[0118] Further, the target corner point acquisition unit includes:

[0119] A corner point sampling subunit, configured to acquire a preset number of target corner points from the basic corner points by using the RANSAC random sampling method;

[0120] A position information acquisition subunit, configured to respectively acquire the positions of the target corner points in the current frame and the adjacent frame to obtain current position information and adjacent position information;

[0121] A target parameter value calculation subunit, configured to acquire a preset parameter, and calculate the corresponding value of the preset parameter based on the current position information and the adjacent position information through a preset formula to obtain a target parameter value;

[0122] A transformation matrix construction subunit, configured to construct a transformation matrix based on the target parameter value.

[0123] Further, the color harmony processing module 86 includes:

[0124] A color sampling mean calculation unit, configured to respectively calculate the color sampling means of the initial sky background map and the target background to obtain an initial color sampling mean and a target color sampling mean;

[0125] A difference calculation unit, configured to calculate the difference between the initial color sampling mean and the target color sampling mean;

[0126] A target sky background image generation unit, configured to perform color harmony processing on an initial sky background image based on a difference to obtain a target sky background image of the current frame.

[0127] After the to-be-processed sample acquisition module 81, it further includes:

[0128] An image sky background segmentation module, configured to perform sky background segmentation processing on a to-be-processed image in the to-be-processed sample to obtain an image sky background mask image if the type of the to-be-processed sample is an image type;

[0129] An initial image sky background image generation module, configured to superimpose the foreground of the sky background mask image and a target background to obtain an initial image sky background image;

[0130] A target image sky background image generation module, configured to perform color harmony processing on the initial image sky background image based on the target background to obtain a target image sky background image;

[0131] A target image sky background image generation module, configured to determine whether there is still a to-be-processed image in the to-be-processed sample. If so, return to execute the step of performing sky background segmentation processing on the to-be-processed image in the to-be-processed sample to obtain an image sky background mask until all the to-be-processed images in the to-be-processed sample are processed to obtain multiple target image sky background images;

[0132] A sky background video generation module, configured to splice multiple target image sky background images to obtain a sky background video.

[0133] Further, the sky mask image generation module 84 includes:

[0134] A current frame target image generation unit, configured to compress the current frame to a preset size to obtain a current frame target image;

[0135] A model segmentation unit, configured to input the current frame target image into a pre-trained segmentation model, identify the sky background in the current frame target image through the pre-trained segmentation model, and perform sky background segmentation processing to obtain a sky mask image.

[0136] To solve the above technical problems, an embodiment of the present application further provides a computer device. For details, please refer to Figure 12 , Figure 12 which is the basic structural block diagram of the computer device in this embodiment.

[0137] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are communicatively connected to each other via a system bus. It should be noted that only a computer device 9 with three components, namely a memory 91, a processor 92, and a network interface 93, is shown in the figure. However, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0138] The computer device can be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touchpad, a voice control device, or other means.

[0139] The memory 91 includes at least one type of readable storage medium. The readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 9. Of course, the memory 91 can also include both the internal storage unit and the external storage device of the computer device 9. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as the program code of the method for replacing the sky background. In addition, the memory 91 can also be used to temporarily store various data that have been output or will be output.

[0140] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of the above-mentioned sky background replacement method to implement various embodiments of the sky background replacement method.

[0141] The network interface 93 may include a wireless network interface or a wired network interface, and this network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0142] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to execute the steps of a sky background replacement method as described above.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of various embodiments of this application.

[0144] Obviously, the above-described embodiments are only part of the embodiments of this application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of this application in other related technical fields is equally within the scope of the patent protection of this application.

Claims

1. A method for replacing the sky background, characterized in that, Including: Obtain the target background and the sample to be processed selected by the user, and determine the type of the sample to be processed; If the type of the sample to be processed is a video type, perform frame splitting on the sample to be processed to obtain a set of video frames to be processed; Obtain the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed; Perform segmentation processing on the sky background in the current frame to obtain a sky mask image; Based on the sky mask image and the adjacent frame, construct a transformation matrix, and perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image; Based on the target background, perform color harmony processing on the initial sky background image to obtain the target sky background image of the current frame; Determine whether the current frame is the last video frame in the set of video frames to be processed. If not, return to execute the step of obtaining the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed, obtain the target sky background images of multiple current frames, and generate a target sky background video based on the target sky background images of multiple current frames; The step of constructing a transformation matrix based on the sky mask image and the adjacent frame, and performing affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image includes: Perform corner detection processing on the sky mask image to obtain a corner feature image; Based on the corner feature image, perform corner tracking processing on the adjacent frame to obtain the corners of the adjacent frame that are the same as the corner feature image as the basic corners; Adopt the RANSAC random sampling method to obtain a preset number of target corners from the basic corners; Respectively obtain the positions of the target corners in the current frame and the adjacent frame to obtain current position information and adjacent position information; Obtain preset parameters, and calculate the corresponding values of the preset parameters based on the current position information and the adjacent position information through a preset formula to obtain target parameter values; Based on the target parameter values, construct the transformation matrix; Perform affine transformation processing on the target background based on the transformation matrix to obtain the initial sky background image.

2. The method for replacing the sky background according to claim 1, characterized in that, The step of performing corner detection processing on the sky mask image to obtain a corner feature image includes: Adopt the ShiTomasi algorithm to perform corner detection processing on the sky mask image to obtain initial corners; Take two adjacent initial corners as a corner combination, and calculate the distance between the corners in the corner combination to obtain a corner distance; Obtain the corner combinations with the corner distance greater than the preset distance as target corner combinations, and generate the corner feature image based on the target corner combinations.

3. The method for replacing the sky background according to claim 1, characterized in that, The step of performing color harmony processing on the initial sky background image based on the target background to obtain the target sky background image of the current frame includes: Respectively calculate the color sampling means of the initial sky background image and the target background to obtain an initial color sampling mean and a target color sampling mean; Calculate the difference between the initial color sampling mean and the target color sampling mean; Based on the difference value, perform color harmony processing on the initial sky background image to obtain the target sky background image of the current frame.

4. The method for replacing the sky background according to claim 1, characterized in that, After obtaining the target background and the sample to be processed selected by the user and determining the type of the sample to be processed, the method further includes: If the type of the sample to be processed is an image type, perform sky background segmentation processing on the image to be processed in the sample to be processed to obtain an image sky background mask image; Overlay the foreground of the sky background mask image with the target background to obtain an initial image sky background image; Based on the target background, perform color harmony processing on the initial image sky background image to obtain a target image sky background image; Determine whether there is still the image to be processed in the sample to be processed. If so, return to execute the step of performing sky background segmentation processing on the image to be processed in the sample to be processed if the type of the sample to be processed is an image type, until all the images to be processed in the sample to be processed are processed to obtain multiple target image sky background images; Stitch the multiple target image sky background images to obtain a sky background video.

5. The method for replacing the sky background according to any one of claims 1 to 4, characterized in that, The performing sky background segmentation processing on the sky background in the current frame to obtain a sky mask image includes: Compress the current frame to a preset size to obtain a current frame target image; Input the current frame target image into a pre-trained segmentation model, and identify the sky background in the current frame target image through the pre-trained segmentation model, and perform segmentation processing on the sky background to obtain the sky mask image.

6. A sky background replacement device, characterized in that, including: A sample to be processed acquisition module, configured to acquire the target background and the sample to be processed selected by the user, and determine the type of the sample to be processed; A video frame set generation module, configured to perform frame splitting processing on the sample to be processed to obtain a set of video frames to be processed if the type of the sample to be processed is a video type; An image to be processed acquisition module, configured to acquire the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed; A sky mask image generation module, configured to perform sky background segmentation processing on the sky background in the current frame to obtain a sky mask image; An initial sky background image generation module, configured to construct a transformation matrix based on the sky mask image and the adjacent frame, and perform affine transformation processing on the target background based on the transformation matrix to obtain an initial sky background image; A color harmony processing module, configured to perform color harmony processing on the initial sky background image based on the target background to obtain the target sky background image of the current frame; A target sky background video generation module, configured to determine whether the current frame is the last video frame in the set of video frames to be processed. If not, return to execute the step of acquiring the current frame and the adjacent frame corresponding to the current frame from the set of video frames to be processed until all video frames are processed to obtain multiple target sky background images of the current frame, and generate a target sky background video based on the multiple target sky background images of the current frame; The initial sky background image generation module includes: A corner detection processing unit for performing corner detection processing on the sky mask image to obtain a corner feature map; A corner tracking processing unit for performing adjacent frame corner tracking processing based on the corner feature map to obtain corners in the adjacent frame that are the same as those in the corner feature map as the basic corners; A corner sampling sub-unit for obtaining a preset number of target corners from the basic corners by means of RANSAC random sampling; A position information acquisition sub-unit for respectively acquiring the positions of the target corners in the current frame and the adjacent frame to obtain current position information and adjacent position information; A target parameter value calculation sub-unit for obtaining a preset parameter and calculating the corresponding value of the preset parameter based on the current position information and the adjacent position information through a preset formula to obtain a target parameter value; A transformation matrix construction sub-unit for constructing the transformation matrix based on the target parameter value; An affine transformation unit for performing affine transformation processing on the target background based on the transformation matrix to obtain the initial sky background image.

7. A computer device, characterized in that, It includes a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the method for replacing the sky background according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the method for replacing the sky background according to any one of claims 1 to 5 is implemented.

Images