A fully automatic video fast synthesis method based on cloud template

Abstract

The present invention discloses a method for fast automatic video synthesis based on cloud templates, the process of which is: firstly, a large number of video templates with good effects containing target photo frames are produced by software such as PR and PE, then the video templates are decoded by FFmpeg, the video frames in the templates are read, a series of algorithms such as filtering, conversion, and denoising are performed on the frames containing the target photo frames in the video to achieve the result of only containing the target photo frames in the frames, and the user uploaded photos are inserted into the position specified by the video template, and the synthesized video is formed after encoding and multiplexing operations. The method of the present invention greatly improves the speed of all methods of video synthesis.

Description

Technical Field

[0001] The invention belongs to the technical field of fast video synthesis and relates to a full-automatic fast video synthesis method based on a cloud template. Background Art

[0002] With the rapid development of information technology and the increasing popularity of the Internet, short videos are gradually becoming an important way for the public to obtain social information and participate in social hot topics. In recent years, with the rapid development of major video creation platforms, more and more high-quality video works have emerged. Traditional video production methods mainly use some software such as iMovie, AE, Edius, ArcTime, PR, etc. These professional video production methods obviously have a high learning threshold, production complexity and time cost, which makes many traditional text and picture editors discouraged and difficult to participate in short video production. In addition, industries such as photo studios and publicity media and some users may prefer simple and fast methods rather than complex and complicated functions.

[0003] Video synthesis technology is applied to various fields, and deep learning knowledge is often needed for image processing and video processing. In the process of processing, high-configuration servers and GPUs are mostly required to realize image and video processing. Video synthesis technology takes a lot of time to realize video synthesis, which takes a few hours or a few days. Although some videos with good effects have been produced, the time cost is also very high. Whether it is time, financial resources, or manpower, a lot of investment is required. This is obviously a problem that needs to be improved urgently.

[0004] Traditional PR and PE software allows users to download video templates from the Internet and replace some special frames in the video templates with their own pictures to achieve the effect of making videos. For some industries such as photo studios or publicity media, the effect of this manual video production is often not good, so the fully automatic video fast synthesis technology based on cloud templates can completely solve this problem. Summary of the invention

[0005] The purpose of the present invention is to provide a fully automatic video rapid synthesis method based on cloud templates, which solves the defects of slow video production speed and high operation complexity in the existing traditional technology.

[0006] The technical solution adopted by the present invention is:

[0007] A fully automatic video fast synthesis method based on cloud templates, comprising the following steps:

[0008] Step 1: Store different sets of video templates on the OSS Alibaba Cloud server. Each set of templates requires image frames with target detection boxes.

[0009] Step 2: The user selects the video template to be created, and submits the video template ID and the image to be inserted into the video template to the server;

[0010] Step 3: The server traverses the video templates stored in the OSS object storage service according to the ID sent by the user and finds the corresponding video template;

[0011] Step 4: Use the ffmpeg library to demultiplex and decode the video template in step 3 to extract the video frames and generate the picture folder picsources;

[0012] Step 5: Traverse all pictures in picsources, convert the RGB color space to the HSV color space, and calculate the HSV histogram of all pictures;

[0013] Step 6: Obtain the HSV value of the target detection frame through the color space tracking method, and use the HSV interval value of the target detection frame as the threshold;

[0014] Step 7: Extract the target outline and target bounding rectangle of the video frame containing the target detection frame in step 6 to generate a set contours.

[0015] Step 8: Traverse contours, get the width and height of the contour, and filter out contours with width and height less than 20.

[0016] Step 9: Filter the target detection box according to the target detection box trace;

[0017] Step 10: Read the image uploaded by the user, and perform AND or operation on the image according to the target detection frame filtered in step 9 to cover the original image. Since there is a move-in and move-out effect in the video, it is necessary to make a judgment: if the target detection frame is at the boundary, it means that a move-in or move-out operation is being performed. For move-in, put the current output back into the queue. For move-out, find the nearest complete target detection frame for synthesis. If the nearest target detection frame can be found, it means it is a move-out.

[0018] Step 11: Since the current target detection box curBox may be in the boundary area, it is necessary to calculate its actual size based on the complete target detection box fullBox. According to the actual size of the target detection box realBoxSize, the shortest side fit method is used to resize the image to prevent deformation and ensure that curBox is consistent in size with the template area to prevent bit operation errors. The obtained image area and template area are subjected to bit operation.

[0019] Step 12: Save the picture folder that has been replaced in step 11, and use the ffmpeg command to re-encode the picture folder into a video in mp4 format.

[0020] The present invention is also characterized in that:

[0021] The specific steps of step 1 are: first, use PR software to classify and store the collected materials, such as videos, audios, images, etc., in different folders. You must have a clear concept of all the materials and preliminarily determine the sequence of the video; then drag the material from the panel to the timeline panel, and you can place the video material in the same track or different tracks, make a rough cut, splice the material to complete the main line, and cut and splice the material on the timeline board. Here, you need to design a 90% photo frame for each video template, that is, the photo frame size occupies 90% of the corresponding video template space, and splice the photo frame into the corresponding video template; then you need to fine-cut the video, which is mainly divided into three steps: time remapping, setting of screen special effects, and setting of transition effects;

[0022] Finally, dub the video, add subtitles, adjust the color, etc. and select the appropriate format for output.

[0023] In step 6, the tracker createTrackbar is first created, and six values ​​of Hue Min, Hue Max, SatMin, Sat Max, Val Min, and Val Max are created to determine the spatial range of the target. Then, after continuous debugging of the mask image, the spatial range values ​​of the target detection box are finally determined to be (47, 131, 48) and (76, 255, 255), where the RGB to HSV color space formula is:

[0024] The calculation formula for color space conversion is:

[0025]

[0026] Among them, r, g, and b are the three primary color space, r represents red, g represents green, and b represents blue; and h, s, and v represent the converted color space values, h represents hue, s represents saturation, and v represents brightness.

[0027] The filtering principle in step 9 is: first, detect the target detection frame and the previous frame of the previous trace. If there is an intersection, it proves that it is the movement trajectory of the same photo. Secondly, if multiple detection frames intersect with this trace, it means that they are the same photo. Finally, multiple detection frames are merged into a large target detection frame.

[0028] The specific steps of step 9 are: after traversing step 7 to obtain contours, the target detection frame needs to be filtered due to the influence of noise. What needs to be filtered here are the frames with a length and width of less than 20, the case where the target detection frame is divided into two due to the line is filtered and repaired, and whether the target detection frame intersects. The filtering method is to traverse the video track trace. If there is no target detection frame in the track, skip it and continue to traverse the next frame. For the frames with target detection frames, nested traversal is performed again. If the target detection frames intersect, the next target frame is added to the queue to determine whether the queue length is greater than 2. If it is not greater than 2, it means that there is only one target detection frame and the next step can be directly performed. If it is greater than 2, it is first necessary to merge multiple small target detection frames into a large target detection frame, delete the small target detection frame at the same time, and add the large target detection frame to the queue.

[0029] The specific steps of step 10 are as follows: first, obtain the processed target detection frame according to step 9. If the channel of the user image is 3, the RGB color space needs to be converted to the BGRA color space. Determine whether there is a target detection frame in the current frame. If not, remove all tracks and filter the target frame according to the track. Next, the synthesis state of each target detection frame needs to be processed. If it is not synthesized, add the picture and the synthesis state of each target detection frame for the first time. If it is an already synthesized frame, delete the synthesized target detection frame. Then determine whether there is a move-in operation. If not, perform a push operation, otherwise do not push (after all the moves are completed, push together). If the current frame is at the boundary (that is, the width and height are uncertain, it is necessary to calculate the actual size of the current frame based on the nearest complete target detection frame. If the detection frame is at the boundary, it means that a move-in or move-out operation is in progress. For move-in, put the current output back into the queue. For move-out, find the nearest fullBox for synthesis.

[0030] The specific steps of step 11 are:

[0031] First, use the IMRead function to read the image uploaded by the user, and classify the target detection box processed in step 10. Since curBox may be in the boundary area, it is necessary to calculate its actual size based on fullBox. According to the actual size of the box realBoxSize, the image is resized using the shortest side fit method to prevent deformation. Get the corresponding area regionRect on the image, get the area tmpRegionMat on the template, ensure that curBox is the same size as regionRect, prevent bit operation errors, and perform bit operations on tmpRegionMat and regionRect.

[0032] The present invention demultiplexes and decodes the video to extract the video frame, and re-renders the frame, encodes and multiplexes the frame to quickly produce the video, and its beneficial effects are mainly reflected in the following aspects:

[0033] 1) Compared with the existing methods, the video production method of the present invention is faster and simpler to operate. Users only need to select their favorite video templates and upload the number of pictures required for the video template, and the system can automatically complete the video production quickly in 1-6 minutes.

[0034] 2) The development and use of the present invention has low requirements on computer hardware. An ordinary desktop computer can complete the development and use of the system, which saves a lot of assets for purchasing servers for some industries.

[0035] 3) The present invention can present different video effects according to the diversity of video templates, thereby being able to adapt to the needs of various life scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of fully automatic video rapid synthesis in the method of the present invention;

[0037] Figure 2 A portion of video frames containing a target detection frame is extracted from the original video in Embodiment 1 of the present invention.

[0038] Figure 3 It is a part of the video frames corresponding to the composite video extracted in embodiment 1 of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] A fully automatic video fast synthesis method based on cloud templates, such as Figure 1 , according to the video fast synthesis flow chart, follow the steps below to implement:

[0041] Step 1: UI engineers design N sets of different video templates and store them in the OSS Alibaba Cloud server. The N sets of templates need to contain image frames with target detection frames, and include ID, video duration, video width and height, and the number of photos required for the video, ultimately achieving the purpose of recognition and replacement.

[0042] Step 2: The user selects the video template to be created, and submits the video template ID and the image to be inserted into the video template to the server.

[0043] Step 3: The server traverses the video templates stored in the OSS object storage service based on the ID sent by the user and finds the corresponding video template.

[0044] Step 4: Use the ffmpeg library to demultiplex and decode the video template in step 3 to extract video frames and generate a picture folder picsources.

[0045] Step 5: Traverse all pictures in picsources, convert the RGB color space to the HSV color space, and calculate the HSV histogram of all pictures.

[0046] Step 6: Obtain the HSV value of the target detection frame through the color space tracking method, and use the HSV interval value of the target detection frame as the threshold.

[0047] Step 7: Extract the target outline and target bounding rectangle of the video frame containing the target detection frame in step 6 to generate a set contours.

[0048] Step 8: Traverse contours, get the width and height of the contour, and filter out contours with width and height less than 20.

[0049] Step 9: Filter the target detection frame according to the target detection frame trace (excluding the case where there are occluders on the green frame and one target detection frame is divided into multiple frames);

[0050] Filtering principles: ① Detect the target detection frame and the previous frame of the previous trace. If there is an intersection, it proves that they are the movement trajectory of the same photo. ② If multiple detection frames intersect with this trace, it means they are the same photo. ③ Combine multiple detection frames into the largest one.

[0051] Step 10: Read the image uploaded by the user, and perform AND or operation on the image according to the target detection frame filtered in step 9 to cover the original image. Since there is a move-in and move-out effect in the video, it is necessary to make a judgment: if the target detection frame is at the boundary, it means that a move-in or move-out operation is being performed. For move-in, put the current output back into the queue. For move-out, find the nearest complete target detection frame for synthesis. If the nearest target detection frame can be found, it means it is a move-out.

[0052] Step 11: Since the current target detection box curBox may be in the boundary area, it is necessary to calculate its actual size based on the complete target detection box fullBox. According to the actual size of the target detection box realBoxSize, the shortest side fit method is used to resize the image to prevent deformation and ensure that curBox is consistent in size with the template area to prevent bit operation errors. The obtained image area and template area are subjected to bit operation.

[0053] Step 12: Save the picture folder that has been replaced in step 11, and use the ffmpeg command to re-encode the picture folder into a video in mp4 format.

[0054] The specific steps of step 1 are:

[0055] First, use PR software to classify and store the collected materials, such as video, audio, images, etc., in different folders. You must have a clear concept of all the materials and preliminarily determine the sequence of the video. Then drag the material from the panel to the timeline panel. You can place the video material in the same track or different tracks, make rough cuts, splice the material to complete the main line, and cut and splice the material on the timeline board. Here you need to design 90% of the photo frame for each video template, that is, the size of the photo frame occupies 90% of the corresponding video template space, and splice the photo frame to the corresponding video template. Then you need to fine-cut the video, which is mainly divided into three steps: time remapping (speeding up or slowing down the playback speed of certain clips), setting the screen special effects (through the effect panel and the effect control panel), and setting the transition effect. Finally, dub the video, add subtitles, color, etc., and select the appropriate format for output.

[0056] The specific steps of step 6 are:

[0057] First, create a tracker createTrackbar, and create six values ​​of Hue Min, Hue Max, Sat Min, Sat Max, Val Min, and Val Max to determine the spatial range of the target. Then, after continuous debugging of the mask map, the spatial range values ​​of the target detection box are finally determined to be (47, 131, 48) and (76, 255, 255). The RGB to HSV color space formula is:

[0058]

[0059] Step 9 The specific steps are:

[0060] After traversing step 7 to obtain contours, the target detection frame needs to be filtered due to the influence of noise. Here, it is necessary to filter the target detection frame whose length and width are less than 20, filter the case where the target detection frame is divided into two due to the line and repair it, and determine whether the target detection frame intersects. The filtering method is to traverse the video track trace. If there is no target detection frame in the track, skip it and continue to traverse the next frame. For the frames with target detection frames, nested traversal is performed again. If the target detection frames intersect, the next target frame is added to the queue to determine whether the queue length is greater than 2. If it is not greater than 2, it means that there is only one target detection frame and you can proceed directly to the next step. If it is greater than 2, you first need to merge multiple small target detection frames into a large target detection frame, delete the small target detection frame, and add the large target detection frame to the queue.

[0061] The specific steps of step 10 are:

[0062] First, get the processed target detection frame according to step 9. If the channel of the user image is 3, the RGB color space needs to be converted to the BGRA color space. Determine whether there is a target detection frame in the current frame. If not, remove all tracks and filter the target frame according to the track. Next, the synthesis status of each target detection frame needs to be processed. If it is not synthesized, add the picture and the synthesis status of each target detection frame for the first time. If it is an already synthesized frame, delete the synthesized target detection frame. Then determine whether there is a move-in operation. If not, push the operation, otherwise do not push (after all the moves are completed, push together). If the current frame is at the boundary (that is, the width and height are uncertain, the actual size of the current frame needs to be calculated based on the nearest complete target detection frame. If the detection frame is at the boundary, it means that a move-in or move-out operation is in progress. For move-in, put the current output back into the queue. For move-out, find the nearest fullBox for synthesis.

[0063] The specific steps of step 11 are:

[0064] First, use the IMRead function to read the image uploaded by the user, and classify the target detection box processed in step 10. Since curBox may be in the boundary area, it is necessary to calculate its actual size based on fullBox. According to the actual size of the box realBoxSize, the image is resized using the shortest side fit method to prevent deformation. Get the corresponding area regionRect on the image, get the area tmpRegionMat on the template, ensure that curBox is the same size as regionRect, prevent bit operation errors, and perform bit operations on tmpRegionMat and regionRect.

[0065] The key point of fast video synthesis is how to connect the positions of the same target photo frame between different frames. The present invention uses the concept of "motion trajectory", which is a sequence of position information of a photo frame between different frames. By filtering, merging, converting and other series of processes on the trajectory of the target photo frame, the current frame only contains the target photo frame at the final position.

[0066] Example

[0067] In this embodiment, ten sets of video templates were selected for verification, and one of them, the Tongqu video, was used to display the results. A total of 700 frames of images were extracted from the video segment captured at 10 frames per second. During the experiment, the original Tongqu video was used for decoding and any 20 video frames were selected. Figure 2 20 frames of the original Tongqu video are shown.

[0068] Step 1: UI engineers design N sets of different video templates and store them in the OSS Alibaba Cloud server. The N sets of templates require image frames with target detection frames to achieve the purpose of recognition and replacement.

[0069] Step: 2: The user selects the video template to be produced, and submits the video template ID and the image to be inserted into the video template to the server.

[0070] Step 3: The server traverses the video templates stored in the OSS object storage service based on the ID sent by the user and finds the corresponding video template.

[0071] Step 4: Use the ffmpeg library to demultiplex and decode the video template in step 3 to extract video frames and generate a picture folder picsources.

[0072] Step 5: Traverse all pictures in picsources, convert the RGB color space to the HSV color space, and calculate the HSV histogram of all pictures.

[0073] Step 6: Obtain the HSV value of the target detection frame through the color space tracking method, and use the HSV interval value of the target detection frame as the threshold.

[0074] Step 7: Extract the target outline and target bounding rectangle of the video frame containing the target detection frame in step 6 to generate a set contours.

[0075] Step 8: Traverse contours, get the width and height of the contour, and filter out contours with width and height less than 20.

[0076] Step 9: Filter the target detection frame according to the target detection frame trace (excluding the case where there are occluders on the green frame and one target detection frame is divided into multiple frames)

[0077] Filtering principles: ① Detect the target detection frame and the previous frame of the previous trace. If there is an intersection, it proves that they are the movement trajectory of the same photo. ② If multiple detection frames intersect with this trace, it means they are the same photo. ③ Combine multiple detection frames into the largest one.

[0078] Step 10: Read the image uploaded by the user, and perform AND or operation on the image according to the target detection frame filtered in step 9 to cover the original image. Since there is a move-in and move-out effect in the video, it is necessary to make a judgment: if the target detection frame is at the boundary, it means that a move-in or move-out operation is being performed. For move-in, put the current output back into the queue. For move-out, find the nearest complete target detection frame for synthesis. If the nearest target detection frame can be found, it means it is a move-out.

[0079] Step 11: Since the current target detection box curBox may be in the boundary area, it is necessary to calculate its actual size based on the complete target detection box fullBox. According to the actual size of the target detection box realBoxSize, the shortest side fit method is used to resize the image to prevent deformation and ensure that curBox is consistent in size with the template area to prevent bit operation errors. The obtained image area and template area are subjected to bit operation.

[0080] Step 12: Save the picture folder that has been replaced in step 11, and use the ffmpeg command to re-encode the picture folder into a video in mp4 format.

[0081] Figure 3 The synthesized video is decoded to obtain 20 frames of images at the corresponding position. The above experiment is repeated using the remaining 9 sets of video templates, and the experimental results are shown in Table 1. It can be seen from the experimental results that the present invention has a good effect in practical application.

[0082] Table 1

[0083]

[0084] Therefore, compared with the existing video production method, the video rapid synthesis method of the present invention has a faster video synthesis speed, simpler operation, low requirements on computer hardware for development and use, and an ordinary desktop computer can complete the development and use of the system. Various videos can be produced according to different video templates, and the applicability is wide. The present invention has a good effect on both simple slide show playback and dynamic video, and no matter what format and size of photos uploaded by users, the photos can be correctly inserted into the photo frame in the video.

Claims

1. A fully automatic video fast synthesis method based on cloud template, characterized in that: The following steps are involved: Step 1: Store different sets of video templates on the OSS Alibaba Cloud server. Each set of templates requires image frames with target detection boxes. Step 2: The user selects the video template to be created, and submits the video template ID and the image to be inserted into the video template to the server; Step 3: The server traverses the video templates stored in the OSS object storage service according to the ID sent by the user and finds the corresponding video template; Step 4: Use the ffmpeg library to demultiplex and decode the video template in step 3 to extract the video frames and generate the picture folder picsources; Step 5: Traverse all pictures in picsources, convert the RGB color space to the HSV color space, and calculate the HSV histogram of all pictures; Step 6: Obtain the HSV value of the target detection frame through the color space tracking method, and use the HSV interval value of the target detection frame as the threshold; Step 7: Extract the target outline and target bounding rectangle of the video frame containing the target detection frame in step 6 to generate a set of contours; Step 8: Traverse contours, get the width and height of the contour, and filter out the contours with width and height less than 20; Step 9: Filter the target detection box according to the target detection box trace; Step 10: Read the picture uploaded by the user, and perform AND or operation on the image according to the target detection frame filtered in step 9, move the video in or out and cover the original picture; Step 11: According to the actual size of the target detection box realBoxSize, use the shortest side fit method to resize the image to ensure that the current target detection box curBox is the same size as the template area, and perform bitwise operations on the obtained image area and template area; Step 12: Save the picture folder that has been replaced in step 11, and use the ffmpeg command to re-encode the video frames in the picture folder into a video in mp4 format.

2. The method for rapid automatic video synthesis based on cloud template according to claim 1, characterized in that: The specific steps of step 1 are: First, PR software is used to classify and store the collected data in different folders, including videos, audios, and images, and the sequence of the videos is preliminarily determined; Then drag the material from the panel to the timeline panel, place the video material in the same track or different tracks, make rough cuts, splice the material to complete the main line, cut and splice the material on the timeline board, design a frame for each video template, the size of the frame occupies 90% of the space of the corresponding video template, and splice the frame into the corresponding video template; Next, you need to fine-tune the video, which is divided into three steps: time remapping, setting of screen special effects, and setting of transition effects; Finally, dub the video, add subtitles, adjust the color and choose the appropriate format for output.

3. The method of fully automatic video rapid synthesis based on cloud template according to claim 1, characterized in that: In step 6, the tracker createTrackbar is first created, and six values ​​of Hue Min, Hue Max, Sat Min, SatMax, Val Min, and Val Max are created to determine the spatial range of the target; then, after continuous debugging of the mask image, the spatial range values ​​of the target detection box are finally determined to be (47, 131, 48) and (76, 255, 255), where the RGB to HSV color space formula is: The calculation formula for color space conversion is: Among them, r, g, and b are the three primary color space, r represents red, g represents green, and b represents blue; and h, s, and v represent the converted color space values, h represents hue, s represents saturation, and v represents brightness.

4. The method for rapid automatic video synthesis based on cloud template according to claim 1, characterized in that: The filtering principle in step 9 is: first, detect the target detection frame and the previous frame frame. If there is an intersection, it proves that it is the movement trajectory of the same photo. Secondly, if multiple detection frames intersect with this trace, it means that it is the same photo. Finally, multiple detection frames are merged into a large target detection frame.

5. The method for rapid automatic video synthesis based on cloud template according to claim 4, characterized in that: The specific steps of step 9 are: after traversing step 7 to obtain contours, the target detection frame needs to be filtered due to the influence of noise. Here, what needs to be filtered is the frame whose length and width are less than 20, the case where the target detection frame is divided into two due to the line is filtered and repaired, and whether the target detection frame intersects; the filtering method is to traverse the video track trace, if there is no target detection frame in the track, skip it, and continue to traverse the next frame; for the frame with the target detection frame, nested traversal is performed again, if the target detection frame intersects, the next target frame is added to the queue, and it is judged whether the queue length is greater than 2. If it is not greater than 2, it means that there is only one target detection frame, and the next step is directly performed; if it is greater than 2, it is first necessary to merge multiple small target detection frames into a large target detection frame, delete the small target detection frame, and add the large target detection frame to the queue.

6. The method for rapid automatic video synthesis based on cloud template according to claim 1, characterized in that: The specific steps of step 10 are: First, get the processed target detection frame according to step 9; if the channel of the user image is 3, it is necessary to convert the RGB color space to the BGRA color space; determine whether there is a target detection frame in the current frame, if not, remove all tracks and filter the target frame according to the track; next, it is necessary to process the synthesis state of each target detection frame. If it is not synthesized, add the picture and the synthesis state of each target detection frame for the first time. If it is an already synthesized frame, delete the synthesized target detection frame; then determine whether there is a move-in operation. If not, push the operation, otherwise do not push, and push together after all the moves are completed; if the current frame is at the boundary, that is, the width and height are uncertain, it is necessary to calculate the actual size of the current frame based on the nearest complete target detection frame; if the detection frame is at the boundary, it means that a move-in or move-out operation is in progress. For move-in, put the current output back into the queue, and for move-out, find the nearest fullBox for synthesis.

7. The method for rapid automatic video synthesis based on cloud template according to claim 1, characterized in that: The specific steps of step 11 are: First, use the IMRead function to read the image uploaded by the user, and classify the target detection box processed in step 10. Resize the image using the shortest side fit method according to the actual size of the box realBoxSize to prevent deformation, obtain the corresponding region regionRect on the image, and obtain the region tmpRegionMat on the template to ensure that the current target detection box curBox is the same size as regionRect to prevent bit operation errors, and perform bit operations on tmpRegionMat and regionRect.

Images