A special effects video generation method, device, computer equipment and storage medium

Abstract

The present application provides a special effects video generation method, device, computer equipment and storage medium, wherein the initial video frame in the video to be processed is extracted, and the initial video frame is subject-identified with people and animals as the main objects to obtain each main object; image semantic segmentation is performed with each main object as the foreground to obtain a layer mask, and the initial background image is obtained based on the layer mask, and the target season sticker is rendered to the initial background image to obtain the target background image; expression recognition is performed on each main object, and the target foreground image is determined according to the recognition result; candidate video frames are generated based on the target foreground image and the target background image; and the target special effects video is generated after filtering each candidate video frame based on the target filter mode. The above method is adopted to reduce the workload of users in the process of obtaining videos, reduce the difficulty and complexity of obtaining special effects videos, and at the same time reduce the labor cost and time cost required, thereby improving the efficiency of video production.

Description

Technical Field

[0001] The present invention relates to the technical field of film and television special effects, and in particular to a special effects video generation method, device, computer equipment and storage medium. Background Art

[0002] With the popularity of online social networking, video has gradually become a commonly used information recording medium and online social media. In the prior art, users usually use mobile phones, cameras and other devices to record what is happening at the moment, store the recorded video for a long time, or upload the recorded video to the Internet or send it to friends. Usually, the recorded video is a documentary video, that is, a video that records environmental events and behavioral events that are happening when the video is recorded.

[0003] However, the study found that since the video obtained by the above method can only record the event information (environmental events and behavioral events) when the video is recorded, if the user wants to obtain a video containing other environmental events or other behavioral events, it is necessary to collect videos at the location where other environmental events occur, or spend a certain amount of time waiting for environmental events to be triggered before collecting videos, and actively create other behavioral events to meet the needs of video recording, and obtain special effects videos that meet user needs. The above process undoubtedly greatly increases the user's workload in the process of obtaining videos, increases the difficulty and complexity of obtaining special effects videos, increases the labor cost and time cost required, and reduces the efficiency of video production. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a special effects video generation method, device, computer equipment and storage medium to reduce the user's workload in the process of obtaining videos, reduce the difficulty and complexity of obtaining special effects videos, and at the same time reduce the required manpower and time costs, thereby improving the efficiency of video production.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a special effects video, the method comprising:

[0006] Obtaining a video to be processed and a special effect rendering target, wherein the special effect rendering target includes a target rendering season and a target rendering weather;

[0007] Extracting a plurality of initial video frames from the video to be processed, and for each initial video frame, performing subject recognition on the initial video frame with people and animals as main objects to obtain each main object in the initial video frame;

[0008] Taking each main object in the initial video frame as the foreground, performing image semantic segmentation on the initial video frame to obtain a layer mask of each main object in the initial video frame, and separating each main object from the initial video frame based on the layer mask to obtain an initial background image of the initial video frame;

[0009] Performing similarity matching on the season information of the target rendering season and the description information of each season sticker in the season sticker library, determining the target season sticker from the season sticker library according to the matching result, and then rendering the target season sticker into the initial background image of the initial video frame to obtain the target background image;

[0010] For each main object in the initial video frame, expression recognition is performed on the main object to determine whether the expression recognition result of the main object matches the conventional expression corresponding to the target rendering weather; if the expression recognition result of the main object does not match the conventional expression corresponding to the target rendering weather, expression reshaping of the main object is performed according to the conventional expression of the target rendering weather to obtain a target foreground image of the initial video frame; if the expression recognition result of the main object matches the conventional expression corresponding to the target rendering weather, the main object is directly determined as the target foreground image of the initial video frame;

[0011] Regenerate a candidate video frame corresponding to the initial video frame based on the target foreground image and the target background image of the initial video frame;

[0012] Obtaining a target filter mode corresponding to the target rendering weather, and performing filter processing on each candidate video frame based on the target filter mode to obtain a target video frame;

[0013] Generate a target special effect video based on each target video frame.

[0014] Optionally, obtaining a target filter mode corresponding to the target rendering weather includes:

[0015] Constructing a conversion function based on historical weather data and historical optical parameters, wherein the conversion function is a function representing a conversion relationship between weather data and optical parameters;

[0016] According to the weather data of the target rendering weather, the target optical parameters are determined by using the conversion function;

[0017] The target filter pattern is determined from a filter pattern database based on the target optical parameters.

[0018] Optionally, performing similarity matching on the season information of the target rendering season and the description information of each season sticker in a season sticker library, and determining the target season sticker from the season sticker library according to the matching result, comprises:

[0019] Traversing each season sticker in the season sticker library, and calculating the semantic similarity between the description information of each season sticker and the season information of the target rendering season;

[0020] Sort the season stickers in descending order according to the semantic similarity between them and the season information of the target rendering season, and determine the season sticker with a preset position in the sorting result as the target season sticker;

[0021] Alternatively, a season sticker whose semantic similarity with the season information of the target rendering season exceeds a preset threshold is determined as the target season sticker.

[0022] Optionally, the subject recognition of the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame includes:

[0023] Input the initial video frame into the trained subject region cutting model to obtain the subject region containing each subject object;

[0024] For each main body region, the region boundary of the main body region is filled according to the neighboring pixel values ​​of the region boundary of the main body region, and the filled region boundary is smoothed to obtain the target boundary of the main body region;

[0025] The regions contained in the boundaries of each target in the initial video frame are cut to obtain each main object.

[0026] In a second aspect, an embodiment of the present application provides a special effects video generation device, the device comprising:

[0027] A rendering preparation module, used to obtain a video to be processed and a special effect rendering target, wherein the special effect rendering target includes a target rendering season and a target rendering weather;

[0028] A subject object recognition module is used to extract a number of initial video frames from the video to be processed, and for each initial video frame, subject recognition is performed on the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame;

[0029] An initial background image determination module is used to use each main object in the initial video frame as a foreground, perform image semantic segmentation on the initial video frame to obtain a layer mask of each main object in the initial video frame, and separate each main object from the initial video frame based on the layer mask to obtain an initial background image of the initial video frame;

[0030] A target background image generation module is used to perform similarity matching between the season information of the target rendering season and the description information of each season sticker in the season sticker library, determine the target season sticker from the season sticker library according to the matching result, and then render the target season sticker into the initial background image of the initial video frame to obtain the target background image;

[0031] The target foreground image generation module is used to perform expression recognition on each subject object in the initial video frame, and determine whether the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather; if the expression recognition result of the subject object does not match the conventional expression corresponding to the target rendering weather, then the expression of the subject object is reshaped according to the conventional expression of the target rendering weather to obtain the target foreground image of the initial video frame; if the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather, then the subject object is directly determined as the target foreground image of the initial video frame;

[0032] A candidate video frame generation module, used to regenerate a candidate video frame corresponding to the initial video frame based on a target foreground image and a target background image of the initial video frame;

[0033] A target video frame generation module is used to obtain a target filter mode corresponding to the target rendering weather, and filter each candidate video frame based on the target filter mode to obtain a target video frame;

[0034] The target special effect video generation module is used to generate a target special effect video based on each target video frame.

[0035] Optionally, obtaining a target filter mode corresponding to the target rendering weather includes:

[0036] Constructing a conversion function based on historical weather data and historical optical parameters, wherein the conversion function is a function representing a conversion relationship between weather data and optical parameters;

[0037] According to the weather data of the target rendering weather, the target optical parameters are determined by using the conversion function;

[0038] The target filter pattern is determined from a filter pattern database based on the target optical parameters.

[0039] Optionally, performing similarity matching on the season information of the target rendering season and the description information of each season sticker in a season sticker library, and determining the target season sticker from the season sticker library according to the matching result, comprises:

[0040] Traversing each season sticker in the season sticker library, and calculating the semantic similarity between the description information of each season sticker and the season information of the target rendering season;

[0041] Sort the season stickers in descending order according to the semantic similarity between them and the season information of the target rendering season, and determine the season sticker with a preset position in the sorting result as the target season sticker;

[0042] Alternatively, a season sticker whose semantic similarity with the season information of the target rendering season exceeds a preset threshold is determined as the target season sticker.

[0043] Optionally, the subject recognition of the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame includes:

[0044] Input the initial video frame into the trained subject region cutting model to obtain the subject region containing each subject object;

[0045] For each main body region, the region boundary of the main body region is filled according to the neighboring pixel values ​​of the region boundary of the main body region, and the filled region boundary is smoothed to obtain the target boundary of the main body region;

[0046] The regions contained in the boundaries of each target in the initial video frame are cut to obtain each main object.

[0047] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the special effects video generation method described in any optional implementation manner of the first aspect are performed.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the special effects video generation method described in any optional implementation manner in the first aspect above are executed.

[0049] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0050] First, by obtaining the video to be processed and the special effects rendering targets including the target rendering season and the target rendering weather, we can clearly identify the user's requirements for the environmental time and behavioral events they expect, and generate a video that meets the user's needs with the special effects rendering targets as the production targets of the video, namely, the target special effects video.

[0051] Then, by extracting several initial video frames in the video to be processed, for each initial video frame, taking people and animals as main objects, subject recognition is performed on the initial video frame to obtain each main object in the initial video frame; then taking each main object in the initial video frame as the foreground, image semantic segmentation is performed on the initial video frame to obtain a layer mask of each main object in the initial video frame, and based on the layer mask, each main object is separated from the initial video frame to obtain an initial background image of the initial video frame; through the above steps, the foreground and background of each image frame in the video to be processed can be separated, so that special effects rendering processing of behavioral events can be performed on the foreground and special effects rendering processing of environmental events can be performed on the background.

[0052] Next, the season information of the target rendering season is matched with the description information of each season sticker in the season sticker library for similarity, and the target season sticker is determined from the season sticker library according to the matching result, and then the target season sticker is rendered into the initial background image of the initial video frame to obtain the target background image; for each main object in the initial video frame, expression recognition is performed on the main object to determine whether the expression recognition result of the main object matches the conventional expression corresponding to the target rendering weather; if the expression recognition result of the main object does not match the conventional expression corresponding to the target rendering weather, the expression of the main object is reshaped according to the conventional expression of the target rendering weather to obtain the target background image. a target foreground image of an initial video frame; if the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather, the subject object is directly determined as the target foreground image of the initial video frame; a candidate video frame corresponding to the initial video frame is regenerated based on the target foreground image and the target background image of the initial video frame; through the above steps, special effects processing can be performed on environmental events in the initial video frame according to the target rendering season expected by the user, and special effects processing can be performed on possible behavioral events according to the target rendering weather expected by the user, and the foreground image and background image after the special effects rendering processing are integrated to obtain a new special effects video frame, so as to obtain a video frame that meets the special effects rendering target.

[0053] Finally, the target filter mode corresponding to the target rendering weather is obtained, and each candidate video frame is filtered based on the target filter mode to obtain a target video frame; a target special effect video is generated based on each target video frame; through the above steps, the image parameters of each video frame can be adjusted to match the target rendering weather, so that each video frame obtained after the special effect rendering process can be more adapted to the environmental events expected by the user, thereby improving the realism and naturalness of the obtained special effect video.

[0054] By adopting the above method, after the initial video is frame-splitting, the foreground and background of each obtained video frame are separated, and then the background of the video frame is rendered according to the seasonal environmental event expected by the user, and then the expression of the foreground of the video frame is adjusted according to the expression behavior event under the weather environmental event expected by the user, and the foreground image and the background image are integrated to obtain each candidate video frame that meets the special effect rendering target, and then each candidate video frame is filtered according to the weather data in the special effect rendering target, and then the video is generated to obtain a special effect video, so that it is possible to generate a special effect video under the environmental events and behavioral events expected by the user without being restricted by the actual time and space conditions of the environmental events and behavioral events, thereby reducing the user's workload in the process of obtaining the video, reducing the difficulty and complexity of obtaining the special effect video, and at the same time reducing the required manpower cost and time cost, thereby improving the efficiency of video production.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 A flowchart of a special effects video generation method provided by Embodiment 1 of the present invention is shown;

[0058] Figure 2 A flowchart of a filter mode acquisition method provided by the first embodiment of the present invention is shown;

[0059] Figure 3 A flowchart of a method for determining target season stickers provided in the first embodiment of the present invention is shown;

[0060] Figure 4 A flow chart of a subject object determination method provided by the first embodiment of the present invention is shown;

[0061] Figure 5 A schematic diagram showing the structure of a special effects video generating device provided by the second embodiment of the present invention is shown;

[0062] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0064] Embodiment 1

[0065] To facilitate understanding of this application, Figure 1 The content described in the flowchart of a special effect video generation method provided by the first embodiment of the present invention is described in detail.

[0066] See also Figure 1 As shown, Figure 1 A flowchart of a special effect video generation method provided in Embodiment 1 of the present invention is shown, wherein the method comprises steps S101 to S105:

[0067] S101: Obtain a video to be processed and a special effect rendering target, wherein the special effect rendering target includes a target rendering season and a target rendering weather.

[0068] Specifically, the video to be processed is obtained through the uploading behavior of the user terminal, and the special effect rendering target that the user expects to achieve is obtained through the instruction issued by the user terminal, which includes at least two dimensions of season and weather.

[0069] S102: extracting a number of initial video frames from the video to be processed, and for each initial video frame, performing subject recognition on the initial video frame with people and animals as main objects to obtain each main object in the initial video frame.

[0070] Specifically, the people and animals in each initial video frame are identified and extracted as main objects, so that the people and animals are regarded as foregrounds, distinguished from the background, and special effects rendering is performed separately.

[0071] After subject recognition, if the area of ​​the recognized subject object is smaller than the preset area, it means that the area of ​​the subject object is too small, and the user cannot see its details with the naked eye. It is also difficult for the user to perceive the special effects rendering of these objects. Therefore, in order to reduce the rendering workload and reduce the computing burden, these objects can be not rendered, and these subject objects can be filtered, and the remaining subject objects can be used as the foreground in step S103.

[0072] S103: Taking each main object in the initial video frame as the foreground, performing image semantic segmentation on the initial video frame to obtain a layer mask of each main object in the initial video frame, and separating each main object from the initial video frame based on the layer mask to obtain an initial background image of the initial video frame.

[0073] Specifically, each main object in the initial video frame is used as the foreground for semantic segmentation of the image, and mask pixels of each main object in the image are generated. Then, each main object is subtracted from the background from the initial video frame according to each mask pixel. After the above process, a two-dimensional foreground image and a two-dimensional background image can be obtained. The present application can also add z-axis information to the two-dimensional coordinates of each main object in the picture and convert them into three-dimensional coordinates to obtain a three-dimensional image of each main object. If there is an overlapping relationship between the extracted three-dimensional images, it is necessary to distinguish them based on the z-axis information corresponding to each main object. If two three-dimensional images are connected by edges, it is necessary to distinguish the z-axis sizes of the two main objects and set them to different values.

[0074] S104: Perform similarity matching on the season information of the target rendering season and the description information of each season sticker in the season sticker library, determine the target season sticker from the season sticker library according to the matching result, and then render the target season sticker into the initial background image of the initial video frame to obtain the target background image.

[0075] Specifically, when performing similarity matching, the target season sticker is determined based on the semantic similarity between the season information of the target rendering season and the description information of each season sticker in the season sticker library, and the target season sticker is rendered into the initial background image of the initial video frame according to the rendering parameters (rendering position, rendering effect, etc.) pre-configured for it to obtain the target background image.

[0076] For example, it is determined that the target season sticker is a snowflake sticker, the rendering position pre-configured for the snowflake sticker is the background global, and the effective rendering effect is semi-transparent, then the snowflake sticker is rendered with a semi-transparent effect on the background global of the initial video frame to obtain the target background image.

[0077] S105: For each main object in the initial video frame, perform expression recognition on the main object to determine whether the expression recognition result of the main object matches the regular expression corresponding to the target rendering weather; if the expression recognition result of the main object does not match the regular expression corresponding to the target rendering weather, perform expression reshaping on the main object according to the regular expression of the target rendering weather to obtain the target foreground image of the initial video frame; if the expression recognition result of the main object matches the regular expression corresponding to the target rendering weather, directly determine the main object as the target foreground image of the initial video frame.

[0078] Specifically, for each subject object belonging to a person, the extracted image of the subject object is input into the face key point detection model to obtain the position coordinates of each face key point, and then the position coordinates of each face key point are matched with the standard position coordinates under each facial expression to determine the current expression of the subject object, and the current expression of the subject object is used as the expression recognition result. For each subject object belonging to a pet, the extracted image of the subject object is input into the pet recognition model to obtain the pet type of the subject object, and then the image of the subject object is input into the pet expression recognition model used to identify the expression of the pet of the pet type to obtain the expression recognition result of the subject object.

[0079] Before reshaping the expression of the main object according to the regular expression of the target rendering weather to obtain the target foreground image of the initial video frame, the method also includes: obtaining a number of human face images and pet face images under the target rendering weather, taking the expression that appears most frequently in each human face image as the regular expression of the target rendering weather, and taking the expression that appears most frequently in the pet face images of each type of pet as the regular expression of this type of pet under the target rendering weather.

[0080] The expression of the main object is reshaped according to the regular expression of the target rendered weather to obtain the target foreground image of the initial video frame, including: for the main object belonging to a person, the expression or face of the main object is replaced by the regular expression of the human face in the target rendered weather; for the main object belonging to a pet, the expression or face of the main object is replaced by the regular expression of the pet category to which the main object belongs in the target rendered weather.

[0081] For example, in the target rendering weather (such as a sunny day), the normal expression is squinting (people usually squint their eyes because of the strong light). When it is recognized that the expression of the main object in the initial video frame is eyes wide open, the expression of eyes wide open of the main object in the initial video frame is replaced with the expression of squinting eyes, and the main object with the expression of squinting eyes is obtained.

[0082] S106: Regenerate a candidate video frame corresponding to the initial video frame based on the target foreground image and the target background image of the initial video frame.

[0083] Specifically, the target foreground image and the target background image of the initial video frame are layered to obtain a new candidate video frame.

[0084] S107: Obtain a target filter mode corresponding to the target rendering weather, and perform filter processing on each candidate video frame based on the target filter mode to obtain a target video frame.

[0085] Specifically, since the image effects presented by images in different weather conditions are different, for example, the brightness and exposure of images on a sunny day are higher than those on a cloudy day, in order to achieve a more realistic special effects rendering effect, the present application also performs filter processing on each candidate video frame according to different rendering weather conditions, and performs filter processing on each candidate video frame based on the target rendering to obtain the target video frame.

[0086] S108: Generate a target special effect video based on each target video frame.

[0087] Specifically, finally, the target video frames after the special effect rendering and filter processing are arranged and integrated in the order of their timestamps from first to last, so as to obtain the target special effect video that can be played coherently.

[0088] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flow chart of a filter mode acquisition method provided by the first embodiment of the present invention is shown, wherein the acquisition of the target filter mode corresponding to the target rendering weather comprises steps S201 to S203:

[0089] S201: Constructing a conversion function based on historical weather data and historical optical parameters, wherein the conversion function is a function representing a conversion relationship between weather data and optical parameters.

[0090] S202: According to the weather data of the target rendering weather, the target optical parameters are determined using the conversion function.

[0091] S203: Determine the target filter mode from a filter mode database based on the target optical parameters.

[0092] Specifically, the conversion function is used to represent the conversion relationship between weather data and optical parameters. Optical parameters are data information that can be understood by the camera, including image brightness, exposure intensity, and exposure time. A historical image is obtained, and a conversion function is constructed based on the historical weather data when the historical image was taken and the historical optical parameters of the historical image (such as image brightness, exposure intensity, etc.). For example, it can be a conversion function between weather data such as light intensity and image grayscale information in the optical parameters. In the conversion function, the stronger the light intensity, the brighter the image, and the higher the image grayscale value in the optical parameters. After obtaining the conversion function, the target optical parameters can be solved using the conversion function according to the weather data of the target rendering weather, and then the target filter mode with the same mode identifier as the target optical parameters is determined from the pre-configured filter mode database as the filter mode for filtering the image taken under the target rendering weather.

[0093] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flowchart of a method for determining a target season sticker provided by a first embodiment of the present invention is shown, wherein the season information of the target rendering season is matched with the description information of each season sticker in the season sticker library for similarity, and the target season sticker is determined from the season sticker library according to the matching result, including steps S301~S302:

[0094] S301: traversing each season sticker in the season sticker library, and calculating the semantic similarity between the description information of each season sticker and the season information of the target rendering season.

[0095] S302: Sort the season stickers in descending order according to the semantic similarity between them and the season information of the target rendering season, and determine the season stickers with the first preset position in the sorting result as the target season stickers; or, determine the season stickers whose semantic similarity with the season information of the target rendering season exceeds a preset threshold as the target season stickers.

[0096] Specifically, the word vectors of all words in the description information of each season sticker and the season information of the target rendering season are calculated respectively, and then the average word vector is calculated. Then, the cosine similarity between the average word vector of the description information of each season sticker and the average word vector of the season information of the target rendering season is calculated respectively to obtain the semantic similarity.

[0097] Alternatively, the word vectors of the description information of each season sticker and the word vectors of the season information of the target rendering season are weighted by TF-IDF word frequency and inverse text frequency, and then the weighted word vectors are averaged. Finally, the cosine similarity between the average word vectors is calculated to obtain the semantic similarity.

[0098] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flow chart of a method for determining a subject object provided by the first embodiment of the present invention is shown, wherein the subject recognition is performed on the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame, including steps S401 to S403:

[0099] S401: Input the initial video frame into the trained subject region cutting model to obtain a subject region including each subject object.

[0100] Specifically, when the image clarity and resolution are low, or the boundary of the main object is fuzzy, the boundary of the main area obtained by cutting may be missing, so the boundary of the main area obtained by cutting needs to be filled in through subsequent steps.

[0101] S402: For each main body region, fill the region boundary of the main body region according to the neighboring pixel values ​​of the region boundary of the main body region, and smooth the filled region boundary to obtain a target boundary of the main body region.

[0102] Specifically, the position of the missing boundary pixel is first determined according to the pixel value of each pixel point in the boundary of the main area, and then the Gaussian weight of the boundary pixel value is calculated. The target boundary pixel value is calculated according to the pixel value in the main area and the Gaussian weight. Finally, the target boundary pixel value is used to fill the pixel at the position where the boundary pixel is missing, thereby filling the boundary of the main area.

[0103] After the boundary is filled, in order to reduce image noise, the present application uses filtering methods such as mean filtering and median filtering to smooth the boundary of the filled area to obtain the target boundary of the main area.

[0104] S403: Cutting the area included in each target boundary in the initial video frame to obtain each main object.

[0105] Specifically, the image is re-cut based on the target boundary to obtain a more accurate cutting result of each main object.

[0106] Embodiment 2

[0107] See also Figure 5 As shown, Figure 5 A schematic diagram of the structure of a special effect video generation device provided by Embodiment 2 of the present invention is shown, wherein the device comprises:

[0108] The rendering preparation module 501 is used to obtain a video to be processed and a special effect rendering target, wherein the special effect rendering target includes a target rendering season and a target rendering weather;

[0109] The subject object recognition module 502 is used to extract a plurality of initial video frames from the video to be processed, and for each initial video frame, perform subject recognition on the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame;

[0110] The initial background image determination module 503 is used to use each main object in the initial video frame as the foreground, perform image semantic segmentation on the initial video frame to obtain a layer mask of each main object in the initial video frame, and separate each main object from the initial video frame based on the layer mask to obtain an initial background image of the initial video frame;

[0111] A target background image generation module 504 is used to perform similarity matching between the season information of the target rendering season and the description information of each season sticker in the season sticker library, determine the target season sticker from the season sticker library according to the matching result, and then render the target season sticker into the initial background image of the initial video frame to obtain the target background image;

[0112] The target foreground image generation module 505 is used to perform expression recognition on each subject object in the initial video frame, and determine whether the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather; if the expression recognition result of the subject object does not match the conventional expression corresponding to the target rendering weather, then the expression of the subject object is reshaped according to the conventional expression of the target rendering weather to obtain the target foreground image of the initial video frame; if the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather, then the subject object is directly determined as the target foreground image of the initial video frame;

[0113] A candidate video frame generating module 506, configured to regenerate a candidate video frame corresponding to the initial video frame based on a target foreground image and a target background image of the initial video frame;

[0114] The target video frame generation module 507 is used to obtain the target filter mode corresponding to the target rendering weather, and filter each candidate video frame based on the target filter mode to obtain the target video frame;

[0115] The target special effect video generation module 508 is used to generate a target special effect video based on each target video frame.

[0116] In an optional implementation manner, obtaining the target filter mode corresponding to the target rendering weather includes:

[0117] Constructing a conversion function based on historical weather data and historical optical parameters, wherein the conversion function is a function representing a conversion relationship between weather data and optical parameters;

[0118] According to the weather data of the target rendering weather, the target optical parameters are determined by using the conversion function;

[0119] The target filter pattern is determined from a filter pattern database based on the target optical parameters.

[0120] In an optional implementation, performing similarity matching on the season information of the target rendering season and the description information of each season sticker in the season sticker library, and determining the target season sticker from the season sticker library according to the matching result, comprises:

[0121] Traversing each season sticker in the season sticker library, and calculating the semantic similarity between the description information of each season sticker and the season information of the target rendering season;

[0122] Sort the season stickers in descending order according to the semantic similarity between them and the season information of the target rendering season, and determine the season sticker with a preset position in the sorting result as the target season sticker;

[0123] Alternatively, a season sticker whose semantic similarity with the season information of the target rendering season exceeds a preset threshold is determined as the target season sticker.

[0124] In an optional implementation, the subject recognition of the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame includes:

[0125] Input the initial video frame into the trained subject region cutting model to obtain the subject region containing each subject object;

[0126] For each main body region, the region boundary of the main body region is filled according to the neighboring pixel values ​​of the region boundary of the main body region, and the filled region boundary is smoothed to obtain the target boundary of the main body region;

[0127] The regions contained in the boundaries of each target in the initial video frame are cut to obtain each main object.

[0128] Embodiment 3

[0129] Based on the same application concept, see Figure 6 As shown, Figure 6 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Figure 6 As shown, a computer device 600 provided in Embodiment 3 of the present application includes:

[0130] A processor 601, a memory 602 and a bus 603, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the computer device 600 is running, the processor 601 communicates with the memory 602 via the bus 603. When the processor 601 is running, the machine-readable instructions execute the steps of the special effects video generation method shown in the above-mentioned embodiment 1.

[0131] Embodiment 4

[0132] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the special effects video generation method described in any one of the above embodiments are executed.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0134] The computer program product for generating special effect videos provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0135] The special effects video generation device provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0136] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0139] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0140] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0141] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A special effects video generation method, characterized in that: The method comprises: Obtaining a video to be processed and a special effect rendering target, wherein the special effect rendering target includes a target rendering season and a target rendering weather; Extracting a plurality of initial video frames from the video to be processed, and for each initial video frame, performing subject recognition on the initial video frame with people and animals as main objects to obtain each main object in the initial video frame; Taking each main object in the initial video frame as the foreground, performing image semantic segmentation on the initial video frame to obtain a layer mask of each main object in the initial video frame, and separating each main object from the initial video frame based on the layer mask to obtain an initial background image of the initial video frame; Performing similarity matching on the season information of the target rendering season and the description information of each season sticker in the season sticker library, determining the target season sticker from the season sticker library according to the matching result, and then rendering the target season sticker into the initial background image of the initial video frame to obtain the target background image; For each main object in the initial video frame, expression recognition is performed on the main object to determine whether the expression recognition result of the main object matches the conventional expression corresponding to the target rendering weather; if the expression recognition result of the main object does not match the conventional expression corresponding to the target rendering weather, expression reshaping of the main object is performed according to the conventional expression of the target rendering weather to obtain a target foreground image of the initial video frame; if the expression recognition result of the main object matches the conventional expression corresponding to the target rendering weather, the main object is directly determined as the target foreground image of the initial video frame; Regenerate a candidate video frame corresponding to the initial video frame based on the target foreground image and the target background image of the initial video frame; Obtaining a target filter mode corresponding to the target rendering weather, and performing filter processing on each candidate video frame based on the target filter mode to obtain a target video frame; Generate a target special effect video based on each target video frame.

2. The method according to claim 1, characterized in that The step of obtaining a target filter mode corresponding to the target rendering weather includes: Constructing a conversion function based on historical weather data and historical optical parameters, wherein the conversion function is a function representing a conversion relationship between weather data and optical parameters; According to the weather data of the target rendering weather, the target optical parameters are determined by using the conversion function; The target filter pattern is determined from a filter pattern database based on the target optical parameters.

3. The method according to claim 1, characterized in that The similarity matching of the season information of the target rendering season with the description information of each season sticker in the season sticker library, and determining the target season sticker from the season sticker library according to the matching result, comprises: Traversing each season sticker in the season sticker library, and calculating the semantic similarity between the description information of each season sticker and the season information of the target rendering season; Sort the season stickers in descending order according to the semantic similarity between them and the season information of the target rendering season, and determine the season sticker with a preset position in the sorting result as the target season sticker; Alternatively, a season sticker whose semantic similarity with the season information of the target rendering season exceeds a preset threshold is determined as the target season sticker.

4. The method according to claim 1, characterized in that The subject recognition of the initial video frame using people and animals as the main objects to obtain the main objects in the initial video frame includes: Input the initial video frame into the trained subject region cutting model to obtain the subject region containing each subject object; For each main body region, the region boundary of the main body region is filled according to the neighboring pixel values ​​of the region boundary of the main body region, and the filled region boundary is smoothed to obtain the target boundary of the main body region; The regions contained in the boundaries of each target in the initial video frame are cut to obtain each main object.

5. A special effects video generation device, characterized in that: The device comprises: A rendering preparation module, used to obtain a video to be processed and a special effect rendering target, wherein the special effect rendering target includes a target rendering season and a target rendering weather; A subject object recognition module is used to extract a number of initial video frames from the video to be processed, and for each initial video frame, subject recognition is performed on the initial video frame using people and animals as subject objects to obtain each subject object in the initial video frame; An initial background image determination module is used to use each main object in the initial video frame as a foreground, perform image semantic segmentation on the initial video frame to obtain a layer mask of each main object in the initial video frame, and separate each main object from the initial video frame based on the layer mask to obtain an initial background image of the initial video frame; A target background image generation module is used to perform similarity matching between the season information of the target rendering season and the description information of each season sticker in the season sticker library, determine the target season sticker from the season sticker library according to the matching result, and then render the target season sticker into the initial background image of the initial video frame to obtain the target background image; The target foreground image generation module is used to perform expression recognition on each subject object in the initial video frame, and determine whether the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather; if the expression recognition result of the subject object does not match the conventional expression corresponding to the target rendering weather, then the expression of the subject object is reshaped according to the conventional expression of the target rendering weather to obtain the target foreground image of the initial video frame; if the expression recognition result of the subject object matches the conventional expression corresponding to the target rendering weather, then the subject object is directly determined as the target foreground image of the initial video frame; A candidate video frame generation module, used to regenerate a candidate video frame corresponding to the initial video frame based on a target foreground image and a target background image of the initial video frame; A target video frame generation module is used to obtain a target filter mode corresponding to the target rendering weather, and filter each candidate video frame based on the target filter mode to obtain a target video frame; The target special effect video generation module is used to generate a target special effect video based on each target video frame.

6. The device according to claim 5, characterized in that The step of obtaining a target filter mode corresponding to the target rendering weather includes: Constructing a conversion function based on historical weather data and historical optical parameters, wherein the conversion function is a function representing a conversion relationship between weather data and optical parameters; According to the weather data of the target rendering weather, the target optical parameters are determined by using the conversion function; The target filter pattern is determined from a filter pattern database based on the target optical parameters.

7. The device according to claim 5, characterized in that The similarity matching of the season information of the target rendering season with the description information of each season sticker in the season sticker library, and determining the target season sticker from the season sticker library according to the matching result, comprises: Traversing each season sticker in the season sticker library, and calculating the semantic similarity between the description information of each season sticker and the season information of the target rendering season; Sort the season stickers in descending order according to the semantic similarity between them and the season information of the target rendering season, and determine the season sticker with a preset position in the sorting result as the target season sticker; Alternatively, a season sticker whose semantic similarity with the season information of the target rendering season exceeds a preset threshold is determined as the target season sticker.

8. The device according to claim 5, characterized in that The subject recognition of the initial video frame using people and animals as the main objects to obtain the main objects in the initial video frame includes: Input the initial video frame into the trained subject region cutting model to obtain the subject region containing each subject object; For each main body region, the region boundary of the main body region is filled according to the neighboring pixel values ​​of the region boundary of the main body region, and the filled region boundary is smoothed to obtain the target boundary of the main body region; The regions contained in the boundaries of each target in the initial video frame are cut to obtain each main object.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the special effect video generation method as described in any one of claims 1 to 4 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the special effect video generation method as described in any one of claims 1 to 4.

Images