Methods to convert landscape video footage to portrait video footage

By identifying and segmenting horizontal video materials, using the motion target significance algorithm to track the screen subject and generate vertical video materials, the problem of loss of screen subject content during horizontal video material conversion is solved, and the complete retention of video material information is achieved and Improved conversion efficiency.

CN115018706BActive Publication Date: 2024-11-01新华智云科技有限公司 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210612330.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-31
Publication Date
2024-11-01
Estimated Expiration
2042-05-31

AI Technical Summary

Technical Problem

When the prior art converts horizontal video material into vertical video material, it is easy to cause the screen subject content of the video material to be lost and all the subject information of the video material cannot be retained.

Method used

By identifying the transition time nodes in the horizontal video material, segmenting them into short horizontal video material, using the subject recognition algorithm based on the significance of the motion target to identify and track the screen subject frame by frame, short vertical video material is generated, and the short vertical video material is It is spliced ​​into complete vertical video material.

Benefits of technology

It ensures that all subject information of the video material is retained, avoids the loss of the subject content of the picture, and improves the efficiency and accuracy of the video conversion process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115018706B_ABST
    Figure CN115018706B_ABST
Patent Text Reader

Abstract

The present application relates to a method for converting a horizontal video material into a vertical video material. By identifying at least one transition time node in the horizontal video material, each transition position in the horizontal video material can be automatically located. By identifying the screen subject in the short horizontal video material through a subject recognition algorithm based on the saliency of a motion target, it is possible to automatically screen out the saliency of the screen subject in combination with motion information. By tracking the screen subject frame by frame for a long time, it is possible to ensure that the screen subject can be recognized and tracked again after it moves out of the screen and when it enters the screen for the second time. Finally, a short vertical video material is generated based on the tracking result, and all the short vertical video materials are spliced ​​into a complete vertical video material, so that the final complete vertical video material can retain all the subject information in the horizontal video material without losing any subject information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method for converting horizontal screen video material into vertical screen video material. Background Art

[0002] With the popularity of mobile terminals such as mobile phones and tablets, the demand for vertical screen video materials is increasing. However, the video materials shot by camcorders and SLR cameras are all horizontal screen video materials with an aspect ratio of 16:9. How to efficiently convert horizontal screen video materials into vertical screen video materials to improve the user's viewing experience on mobile terminals has become an urgent need in the video processing and video editing fields.

[0003] The traditional solution is generally to use a video editor to convert horizontal screen video materials into vertical screen video materials.

[0004] The horizontal-to-vertical function in existing video editors is basically manually edited, which directly rotates the video material or crops it in the center of the picture. This will result in the loss of the main content of the video material, and the resulting vertical screen video material cannot retain all the main information of the video. Summary of the invention

[0005] Based on this, it is necessary to provide a method for converting horizontal screen video materials into vertical screen video materials to address the problem that the traditional method of converting horizontal screen video materials into vertical screen video materials will result in the loss of the main content of the video material, and the final vertical screen video material cannot retain all the main information of the video.

[0006] The present application provides a method for converting a horizontal screen video material into a vertical screen video material, the method comprising:

[0007] Acquire horizontal screen video material, and identify at least one transition time node in the horizontal screen video material;

[0008] Splitting the horizontal screen video material into a plurality of short horizontal screen video materials according to at least one transition time node;

[0009] In each short horizontal screen video material, a subject recognition algorithm based on the saliency of the moving target is used to identify the subject of the picture in the short horizontal screen video material;

[0010] Track the subject of each short horizontal video frame by frame.

[0011] Generate a short vertical screen video material according to the frame-by-frame tracking result of the screen subject in each short horizontal screen video material, and obtain multiple short vertical screen video materials;

[0012] Splice all short vertical video clips into a complete vertical video clip.

[0013] The present application relates to a method for converting a horizontal video material into a vertical video material. By identifying at least one transition time node in the horizontal video material, each transition position in the horizontal video material can be automatically located. By identifying the screen subject in the short horizontal video material through a subject recognition algorithm based on the saliency of a motion target, it is possible to automatically screen out the saliency of the screen subject in combination with motion information. By tracking the screen subject frame by frame for a long time, it is possible to ensure that the screen subject can be recognized and tracked again after it moves out of the screen and when it enters the screen for the second time. Finally, a short vertical video material is generated based on the tracking result, and all the short vertical video materials are spliced ​​into a complete vertical video material, so that the final complete vertical video material can retain all the subject information in the horizontal video material without losing any subject information. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of a method for converting horizontal screen video material into vertical screen video material provided in one embodiment of the present application.

[0015] Figure 2 A schematic diagram of the positional relationship between the main body of the picture, the preset search radius, and the local search area in a horizontal screen video material provided in an embodiment of the present application.

[0016] Figure 3 A schematic diagram of selecting different local sub-search areas for the main body of a horizontal screen video material provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of this application more clear, the following is a further detailed description of this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain this application and are not used to limit this application.

[0018] The present application provides a method for converting a horizontal video material into a vertical video material. It should be noted that the method for converting a horizontal video material into a vertical video material provided by the present application is applicable to horizontal video materials shot by any shooting device.

[0019] In addition, the method for converting horizontal screen video material into vertical screen video material provided in the present application does not limit its execution subject. Optionally, the execution subject of the method for converting horizontal screen video material into vertical screen video material provided in the present application can be a horizontal to vertical processing terminal.

[0020] like Figure 1 As shown, in one embodiment of the present application, the method includes the following S100 to S600:

[0021] S100, obtaining horizontal screen video material, and identifying at least one transition time node in the horizontal screen video material.

[0022] Specifically, each horizontal screen video material to be processed may be composed of short horizontal screen video materials shot by different lenses. The screen content between the short horizontal screen video materials is discontinuous, and the main body of the screen may also change accordingly. For example, when shooting a video, the shooter may use 5 different cameras set up in 5 different positions, then the final horizontal screen video material will be composed of video images shot by these 5 different cameras interlaced and spliced. Or, use the same camera to shoot, but use 5 different lenses successively.

[0023] Therefore, this embodiment first needs to identify the position of the transition in the horizontal screen video material, that is, the identification of the transition time node, so that the horizontal screen video material can be divided into multiple short horizontal screen video materials.

[0024] S200, dividing the horizontal screen video material into a plurality of short horizontal screen video materials according to at least one transition time node.

[0025] Specifically, after the horizontal screen video material is divided into a plurality of short horizontal screen video materials, it can be ensured that the picture in each short horizontal screen video material is continuous.

[0026] S300: In each short horizontal screen video material, a subject recognition algorithm based on motion target saliency is used to identify a subject in the short horizontal screen video material.

[0027] Specifically, the subject of the picture is the most important element of the picture, and the subject of the picture is not limited to people, animals, plants, and can also be any other object that brings a significant visual experience. Optionally, in this application, the form of the subject of the picture can be a rectangular frame and the video picture displayed in the rectangular frame.

[0028] S400, tracking the subject of each short horizontal screen video material frame by frame.

[0029] Specifically, the frame-by-frame tracking in this step can ensure that even if the subject of the picture moves out of the picture and then enters the picture again, the method can track the subject of the picture again.

[0030] S500, generating a short vertical screen video material according to the frame-by-frame tracking result of the screen subject in each short horizontal screen video material, and obtaining a plurality of short vertical screen video materials.

[0031] Specifically, this step is equivalent to extracting the main body of the picture in each short horizontal screen video material, and then converting the main body of the picture in the short horizontal screen video material into a short vertical screen video material.

[0032] S600, stitches all short vertical video clips into a complete vertical video clip.

[0033] Specifically, when splicing short vertical screen video materials, they should also be spliced ​​in chronological order.

[0034] In this embodiment, by identifying at least one transition time node in the horizontal video material, it is possible to automatically locate each transition position in the horizontal video material. By identifying the screen subject in the short horizontal video material based on the motion target significance, it is possible to automatically screen out the significant screen subject in combination with the motion information. By tracking the screen subject frame by frame for a long time, it is possible to ensure that the screen subject can be recognized and tracked again after moving out of the screen and when entering the screen for the second time. Finally, a short vertical video material is generated based on the tracking result, and all the short vertical video materials are spliced ​​into a complete vertical video material, so that the final complete vertical video material can retain all the subject information in the horizontal video material without losing any subject information.

[0035] In one embodiment of the present application, the S100 includes the following S110 to S145:

[0036] S110, parsing the horizontal screen video material to obtain multiple video frames.

[0037] Specifically, a horizontal screen video material is composed of multiple video frames.

[0038] S121, performing a difference calculation on every two consecutive video frames to obtain a difference image between every two consecutive video frames.

[0039] Specifically, the difference calculation is to subtract the pixel values ​​of each pixel at the same position in the two video frames to obtain the pixel difference. The pixel difference values ​​of all pixels constitute a new video image frame, namely the difference image. The difference image can show the similar parts of the two video frames and highlight the changed parts of the two video frames. The difference image can detect the outline of a moving target.

[0040] For example, video frame A and video frame B are two consecutive video frames, each with 10 pixels (for the convenience of explanation, the number of pixels is reduced, and the number of pixels in an actual video frame is much greater than 10). Subtract the pixel values ​​of pixel 1 in video frame A from pixel 1 in video frame B, subtract the pixel values ​​of pixel 2 in video frame A from pixel 2 in video frame B... until the pixel values ​​of pixel 10 in video frame A are subtracted from the pixel 10 in video frame B, and finally 10 pixel differences are obtained. These 10 pixel differences constitute an image, which is the differential image.

[0041] S122, calculating the sum of the pixel values ​​of all the pixels in each differential image to obtain the pixel sum of each differential image.

[0042] Specifically, for each differential image, the pixel value of each pixel is added to obtain the pixel sum of the differential image.

[0043] S123, performing a difference calculation on the pixel sums of every two adjacent difference images, and defining the absolute value of the difference value obtained by the difference calculation as a quadratic difference sum.

[0044] Specifically, for example, the horizontal screen video material has 16 video frames. The video frame 1 and the video frame 2 are differentially calculated to obtain the differential image A, and the video frame 2 and the video frame 3 are differentially calculated to obtain the differential image B. Then the differential image A and the differential image B are two adjacent differential images.

[0045] S131, selecting a quadratic difference sum, and determining whether the quadratic difference sum is greater than a preset quadratic difference sum threshold.

[0046] Specifically, if the quadratic difference sum is less than or equal to a preset quadratic difference sum threshold, the quadratic difference sum is discarded.

[0047] S132: If the quadratic difference sum is greater than a preset quadratic difference sum threshold, the quadratic difference sum and the time node corresponding to the quadratic difference sum are used as candidate transition time nodes.

[0048] Specifically, for example, if the quadratic difference sum of differential image A and differential image B is greater than a preset quadratic difference sum threshold, the time node corresponding to the quadratic difference sum of differential image A and differential image B is used as a candidate transition time node.

[0049] The time node corresponding to the second difference sum of the difference image A and the difference image B is defined as follows:

[0050] First, compare the pixel sum of differential image A with the pixel sum of differential image B. If the pixel sum of differential image A is larger, take video frame 2 of video frame 1 and video frame 2 as the time node corresponding to the secondary differential sum of differential image A and differential image B, that is, the time node of video frame 2 is the candidate transition time node.

[0051] If the pixel sum of differential image B is larger, video 3 in video frame 2 and video frame 3 is used as the time node corresponding to the secondary differential sum of differential image A and differential image B, that is, the time node of video frame 3 is the candidate transition time node.

[0052] In other words, on the one hand, the pixels of the differential image and the larger of the two video frames are selected, and on the other hand, the video frame which is later in time between the two video frames is selected.

[0053] S133, returning to S131, until all the secondary difference sums have been selected, and a plurality of candidate transition time nodes are obtained.

[0054] Specifically, the obtained transition time node will not be the time node of the first video frame, but may be the time node of the last video frame.

[0055] S141, sorting a plurality of candidate transition time nodes in chronological order.

[0056] S142: Taking the first candidate transition time node as a time anchor point.

[0057] S143, selecting a preset time period after the time anchor point.

[0058] S144, searching for all candidate transition time nodes within a preset time period after the time anchor point, and selecting the candidate transition time node with the largest sum of the second difference as the transition time node.

[0059] S145, taking the transition time node as a new time anchor point, and returning to S143 until all transition time nodes are obtained.

[0060] The working principle of S141 to S144 is explained below. For example, we get 5 candidate transition time nodes. After arranging them in chronological order, candidate transition time node 1 is taken as the time anchor point, and the preset time period is selected as 2 seconds. Then all candidate transition time nodes within 2 seconds after transition time node 1 are searched, and candidate transition time node 2 and candidate transition time node 3 are found. The quadratic difference sum of candidate transition time node 3 is greater than the quadratic difference sum of candidate transition time node 2, so the subsequent transition time node 3 is taken as the first transition time node.

[0061] This is because the larger the sum of the quadratic differences is, the greater the degree of change in the picture content caused by the candidate transition time node.

[0062] Subsequently, the subsequent transition time node 3 is used as a new time anchor point, and all candidate transition time nodes within 2 seconds after the transition time node 3 are continuously searched, and the above steps are repeated until all transition time nodes are obtained.

[0063] In summary, this embodiment introduces an implementation method for identifying a transition time node, which is a method of using secondary differences to identify the transition time node. The first difference is S121, and the second difference is S123.

[0064] The second difference method can effectively filter out the rapid movement of the lens. If the camera moves rapidly during shooting, it will cause blurry and trailing images. The second difference method can filter out such blurry and trailing images.

[0065] When a preset time period is selected after a time anchor point, the preset time period can be set to 1 second, 1.5 seconds, or 2 seconds.

[0066] In one embodiment of the present application, before S131, S110 further includes:

[0067] S124, calculating the average of all quadratic difference sums to obtain the average of the quadratic difference sums.

[0068] Specifically, this embodiment introduces an implementation method for setting a quadratic difference sum threshold. First, the average value of all quadratic difference sums is calculated.

[0069] S125, obtaining a preset threshold multiplier, and taking the product of the preset threshold multiplier and the quadratic difference and average value as the preset quadratic difference and threshold.

[0070] Specifically, the threshold multiplier may be set to 1.2.

[0071] In one embodiment of the present application, the S100 includes the following S150 to S180:

[0072] S150, setting a preset time length.

[0073] S160: Use a sliding window with a preset time length to traverse the horizontal screen time material and decompose the horizontal screen time material into multiple video clips.

[0074] S170, input each video segment into a video classification model, run the video classification model and determine whether each video segment is a video segment with transitions, so as to obtain a plurality of video segments with transitions.

[0075] S180, extracting a transition time node in each video segment having a transition.

[0076] Specifically, this embodiment introduces another method of identifying the transition time node, which is implemented by using a video classification model. The video classification model is a deep learning model that needs to be trained in advance.

[0077] The specific training method is to capture the familiar transition special effects clips in the industry and mark them as 1, capture the non-transition clips of the same order of magnitude and mark them as 0, and then input the marked transition special effects clips and non-transition special effects clips into the video classification model, and use them as training data to train the video classification model. The trained video classification model acquires the ability to recognize transition clips.

[0078] Optionally, the video classification model uses one or more of TSM, Slowfast, and X3D as the video classification network framework, and the video classification model uses cross entropy as the network loss function. When the network loss function drops below a preset network loss value, the training ends.

[0079] In S150, firstly, a preset time length of the sliding window is set, and the preset time length may be 16 frames.

[0080] In one embodiment of the present application, S160 includes:

[0081] S161, using a sliding window with a first preset time length of K to capture the 1st frame to the Kth frame in the horizontal screen time material, and using the 1st frame to the Kth frame as the first video segment.

[0082] S162, move the sliding window back by a second preset time length, where the second preset length is L, and use the WLth frame to the W+Lth frame as the second video segment; W is the serial number of the last frame of the previous video segment; L is less than K.

[0083] S163, repeatedly executing S162 until the horizontal screen time material is traversed to obtain multiple video clips.

[0084] It can be understood that in this embodiment, each time the sliding window is moved back by the preset time length L, although there are overlapping segments between two adjacent video segments captured, this can prevent the loss of key segments.

[0085] For example, K is 16 and L is 8, then the first video segment is frame 1 to frame 16, and the second video segment is frame 8 to frame 24. Compared with the method of directly moving the sliding window back by the first preset time length, the sliding window directly moving back by a smaller second preset time length can obviously prevent the loss of key segments to the greatest extent.

[0086] In one embodiment of the present application, the S180 includes the following S181 to S186:

[0087] S181, selecting a video clip with transition.

[0088] S182, extracting each video frame in the video segment with transition.

[0089] S183, calculating a color histogram of each video frame in the video segment with transition.

[0090] S184, calculating the inter-frame distance between the color histograms of every two adjacent video frames.

[0091] S185, selecting a time node corresponding to the video frame with the largest inter-frame distance as a transition time node in the video segment with transition.

[0092] S186, return to S181, until the transition time node in each video segment with transition is obtained.

[0093] Specifically, the previous embodiment only identifies video segments with transitions, but does not find the transition time node in the transition segment. This embodiment introduces a method for finding the transition time node in the transition segment based on the inter-frame distance of the color histogram.

[0094] The color histogram reflects the composition distribution of colors in the image, that is, which colors appear and the probability of each color appearing. Therefore, by calculating the inter-frame distance between the color histograms of two adjacent video frames, the degree of color change between the two adjacent video frames can be known. The larger the inter-frame distance, the greater the degree of color change between the two adjacent video frames. The time node corresponding to the video frame with the largest inter-frame distance can be defined as the transition time node.

[0095] In one embodiment of the present application, the S200 includes:

[0096] S210, sorting all transition time nodes in chronological order.

[0097] S220, select the first transition time node, capture all video frames between the start time node of the horizontal screen video material and the first transition time node as a short horizontal screen video material.

[0098] S230, select the last transition time node, and capture all video frames between the last transition time node and the end time node of the horizontal screen video material as a short horizontal screen video material.

[0099] S240, intercepting all video frames between every two adjacent transition time nodes as a short horizontal screen video material.

[0100] S250, outputs all short horizontal screen video materials.

[0101] Specifically, when there is only one transition time node, it is both the first transition time node and the last transition time node. After the steps are finally executed, two short horizontal screen video materials will be obtained.

[0102] In one embodiment of the present application, the S300 includes the following S310 to S390:

[0103] S310, selecting a short horizontal screen video material.

[0104] S321, selecting the first frame image in the short horizontal screen video material.

[0105] S322, performing optical flow prediction on the first frame image and the second frame image based on flownet to obtain optical flow images of the first frame image and the second frame image.

[0106] Specifically, the optical flow image requires two adjacent frames of images. The present application only obtains the optical flow images of the first frame and the second frame, finds the main body of the picture based on the optical flow images of the first frame and the second frame, and then tracks the main body of the picture through the subsequent S400. It is not necessary to perform optical flow prediction on all frame images. Therefore, the method is relatively simple, does not require complex calculations, and has high work efficiency.

[0107] Each pixel in the optical flow image has a displacement in the x and y directions.

[0108] S323, calculating the sum of the absolute value in the x direction and the absolute value in the y direction for each pixel point in the optical flow images of the first frame image and the second frame image, to obtain the optical flow absolute value sum.

[0109] Specifically, each pixel has two pieces of information, one is the absolute value in the x direction (i.e., the absolute value of the displacement in the x direction), which expresses the speed of the pixel in the x direction, and can also be understood as the movement trend of the pixel in the x direction. The other is the absolute value in the y direction (i.e., the absolute value of the displacement in the y direction), which expresses the speed of the pixel in the y direction, and can also be understood as the movement trend of the pixel in the y direction.

[0110] S324, binarizing the absolute value sum of the optical flow, and dividing a plurality of optical flow connected domains in the optical flow images of the first frame image and the second frame image according to the binarization result of the absolute value sum of the optical flow, and obtaining a set of optical flow connected domains {F1, F2, ..., Fi}, where i is the serial number of the optical flow connected domain.

[0111] Specifically, one implementation of binarizing the optical flow absolute value sum is to compare the optical flow absolute value sum of each pixel with a preset optical flow absolute value sum threshold.

[0112] If the absolute value of the optical flow of a pixel is greater than or equal to the preset absolute value and threshold of the optical flow, it is recorded as 1. If the absolute value of the optical flow of a pixel is less than the preset absolute value and threshold of the optical flow, it is recorded as 0. In this way, each pixel in the optical flow image can be expressed by binary data such as 0 and 1.

[0113] Then, it is easier to divide multiple optical flow connected domains in the optical flow image by binarizing the data.

[0114] S331, performing static saliency segmentation on the first frame image.

[0115] Specifically, the division of the optical flow connected domain is a dynamic division, while the static saliency segmentation is a static division. Both dynamic and static divisions can more comprehensively detect salient targets in short horizontal screen video materials.

[0116] S332, dividing a plurality of statically significant connected domains in the first frame image according to the statically significant segmentation result of the first frame image, and obtaining a set of statically significant connected domains {S1, S2, ..., Sm}, where m is a sequence number of the statically significant connected domain.

[0117] Specifically, m is also the total number of statically salient connected domains.

[0118] S340, calculating the intersection-and-union ratio between each optical flow connected domain in the set of optical flow connected domains {F1, F2, ..., Fi} and each statically significant connected domain in the set of statically significant connected domains {S1, S2, ..., Sm}, to obtain a plurality of intersection-and-union ratios.

[0119] Specifically, the intersection-over-union ratio, or IOU, is the ratio of the intersection and union of the optical flow connected domain and the static saliency connected domain.

[0120] In this step, when calculating the intersection-and-union ratio, it is necessary to perform a calculation once for each optical flow connected domain in the set of optical flow connected domains {F1, F2, ..., Fi} and each statically significant connected domain in the set of statically significant connected domains {S1, S2, ..., Sm}, that is, an intersection-and-union ratio is calculated for F1 and each statically significant connected domain of S1, S2, ..., Sm, and an intersection-and-union ratio is calculated for F2 and each statically significant connected domain of S1, S2, ..., Sm... and so on.

[0121] S350, selecting an IoU ratio and determining whether the IoU ratio is greater than a preset IoU ratio threshold.

[0122] Specifically, if the IoU is less than or equal to a preset IoU threshold, the IoU will be discarded, and the set of optical flow connected domains and static saliency connected domains will also be discarded. The preset IoU threshold can be set to 0.5.

[0123] S360: If the IoU is greater than a preset IoU threshold, obtain a bounding rectangle of a graphic represented by a union of an optical flow connected domain and a static saliency connected domain corresponding to the IoU, and use the bounding rectangle as a candidate screen subject.

[0124] Specifically, first, this step obtains the graph displayed by the union of the optical flow connected domain and the static saliency connected domain corresponding to the intersection-over-union ratio. However, the graph displayed by this union is irregular. In order to facilitate subsequent tracking calculations, the second step needs to take the circumscribed rectangular frame of the graph displayed by the union as the candidate screen body.

[0125] S370, returning to S350, until each intersection-over-union ratio is selected once, and at least one candidate picture subject is obtained.

[0126] S380, calculating the sum of the absolute values ​​of the optical flows of all candidate screen subjects, sorting all candidate screen subjects from large to small according to the sum of the absolute values ​​of the optical flows, and taking the candidate screen subject with the largest sum of the absolute values ​​of the optical flows as the screen subject in the short horizontal screen video material.

[0127] Specifically, the larger the absolute value of the optical flow is, the greater the motion trend is. Conversely, the smaller the absolute value of the optical flow is, the smaller the motion trend is.

[0128] S390, return to S310, until the main body of the picture in all short horizontal screen video materials is obtained.

[0129] Specifically, this embodiment adopts a long-term tracking strategy based on optical flow connected domain and static saliency segmentation, which can ensure that when the screen subject moves out of the screen and reenters the screen, the screen subject can still be identified and tracked again.

[0130] In one embodiment of the present application, the S400 includes:

[0131] S310, selecting a short horizontal screen video material.

[0132] S321, obtaining the nth frame in the short horizontal screen video material, and obtaining the position of the main body of the picture in the nth frame. n is initially set to 1.

[0133] S322, obtaining the n+1th frame in the short horizontal screen video material.

[0134] S323, taking the position of the center of the main body of the picture in the nth frame in the n+1th frame as the center of the circle, drawing a circle in the n+1th frame with a preset search radius as the radius, and taking the area covered by the circumscribed rectangle of the circle as the local search area. The preset search radius is initially defined as the local search radius.

[0135] Specifically, the center of the main body of the picture in the nth frame is the physical center of the main body of the picture in the nth frame, that is, the intersection of the two diagonals. Because the main body of the picture is a rectangular frame, the center of the main body of the picture is the intersection of the two diagonals of the rectangular frame. All "centers" in this application refer to the intersection of the two diagonals of the rectangular frame.

[0136] The preset search radius is initially defined as the local search radius for smooth execution of the initial steps.

[0137] Figure 2 A schematic diagram of the positional relationship between the main body of the horizontal screen video material, the preset search radius, and the local search area provided in an embodiment of the present application. This step can be combined with Figure 2 To understand.

[0138] S324, intercepting a plurality of local sub-search areas in the local search area, which have the same shape and the same coverage area as the main body of the picture in the nth frame.

[0139] Specifically, the coverage area of ​​the local search area is set to be much larger than the coverage area of ​​the main body of the picture in the nth frame, so that the selectivity is relatively wide. The coverage area of ​​the local search area is set to be much larger than the coverage area of ​​the main body of the picture in the nth frame, which is achieved by adjusting the size of the preset search radius. Optionally, the local search radius can be set to be larger than the longest diagonal of the rectangular frame of the main body of the picture in the first frame. The local search radius can be set to twice the length of the longest diagonal of the rectangular frame of the main body of the picture in the first frame.

[0140] like Figure 3 As shown, in Figure 3 In the local search area, four local sub-search areas with different positions are intercepted, but they have the same shape and the same coverage area.

[0141] S325, searching for the position of the subject of the picture in the nth frame in the (n+1)th frame.

[0142] S326, in the (n+1)th frame, calculating the similarity between the main body of the picture in the nth frame and each local sub-search area to obtain multiple similarities.

[0143] Specifically, the similarity between the two rectangular contours can be calculated by using a variety of image comparison methods, such as a difference method. Any image comparison method can be used to achieve the similarity comparison between the main body of the picture in the nth frame and the local sub-search area.

[0144] S327, obtaining a maximum similarity among multiple similarities, and determining whether the maximum similarity is greater than a similarity threshold.

[0145] Specifically, the similarity index can be expressed as a percentage, and the similarity threshold can be set to 92%.

[0146] S331, if the maximum similarity is greater than the similarity threshold, the position of the local sub-search area corresponding to the maximum similarity is used as the position of the picture subject in the n+1th frame, thereby completing the positioning of the picture subject in the n+1th frame.

[0147] Specifically, this step is equivalent to "updating" the position of the main body of the picture.

[0148] S332, add 1 to n, define the preset search radius as the local search radius, and return to S322 until the positioning of the picture body of all video frames in the short horizontal screen video material is completed.

[0149] S333, return to S310, until all short horizontal screen video materials have completed the positioning of the main body of all video frames.

[0150] Specifically, when the maximum similarity is greater than the similarity threshold, the present embodiment adopts a local search, which means that the main body of the picture has not moved out of the picture.

[0151] In one embodiment of the present application, after S327, S400 further includes:

[0152] S341, if the maximum similarity is less than or equal to the similarity threshold, the position of the picture subject in the nth frame is used as the position of the picture subject in the n+1th frame, thereby completing the positioning of the picture subject in the n+1th frame.

[0153] S342, adding 1 to n, defining the preset search radius as the global search radius, and returning to S322. The global search radius is larger than the local search radius.

[0154] Specifically, when the maximum similarity is less than or equal to the similarity threshold, it means that the main body of the picture has moved out of the picture, and the next frame must be transferred to the global search. When the maximum similarity is greater than the similarity threshold again, it means that the main body of the picture has returned to the picture, and the next frame will be transferred to the local search again. The global search radius can be set to twice the local search radius.

[0155] The principle of this embodiment is the same as that of the previous embodiment, and will not be described in detail. The difference is that the preset search radius is defined as a global search radius, not a local search radius. Figure 2 Understand, no additional image drawing.

[0156] In one embodiment of the present application, the S500 includes:

[0157] S510, selecting a short horizontal screen video material.

[0158] S520, obtaining the center position of the main body of each frame in the short horizontal screen video material after frame-by-frame tracking.

[0159] S530 , taking the center position of the main body of each frame as the center, cropping a rectangular video frame with a width of w and a height of h in each frame.

[0160] S540, taking each rectangular video frame as a vertical screen video frame, splicing all the obtained rectangular video frames in chronological order to obtain a short vertical screen video material.

[0161] S550, returning to S510, until all short horizontal screen video materials are converted into short vertical screen video materials.

[0162] Optionally, before S530, the center position of the main body of each frame may be filtered to obtain the center position of the filtered main body of each frame, and a rectangular video frame with a width of w and a height of h may be cropped with the center position as the center.

[0163] In S530, instead of directly taking the center position of the main body of the picture of each frame as the center, a rectangular video picture with a width of w and a height of h is cropped with the center position of the main body of the filtered picture of each frame as the center.

[0164] The function of filtering is to smooth the tracking points, thereby eliminating the jitter of the cropping frame with a width of w and a height of h in the timing, so that the final cropped video picture is smoother. Optionally, the filtering method is to calculate the weighted average of the center position of the main body of each three consecutive frames. For example, the center position of the main body of the second frame after filtering is the weighted average of the center position of the main body of the first frame, the second frame, and the third frame. The center position of the main body of the third frame after filtering is the weighted average of the center position of the main body of the second frame, the third frame, and the fourth frame.

[0165] The technical features of the above-described embodiments may be arbitrarily combined, and the execution order of the method steps is not limited. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for converting horizontal screen video material into vertical screen video material, characterized in that: The method comprises: Acquire horizontal screen video material, and identify at least one transition time node in the horizontal screen video material; Splitting the horizontal screen video material into a plurality of short horizontal screen video materials according to at least one transition time node; In each short horizontal screen video material, a subject recognition algorithm based on the saliency of the moving target is used to identify the subject of the picture in the short horizontal screen video material; Track the subject of each short horizontal video frame by frame. Generate a short vertical screen video material according to the frame-by-frame tracking result of the screen subject in each short horizontal screen video material, and obtain multiple short vertical screen video materials; Splice all short vertical video clips into a complete vertical video clip; The method of identifying the subject in each short horizontal screen video material by a subject recognition algorithm based on the saliency of the moving target comprises: Select a short horizontal screen video material; Select the first frame image and the second frame image in the short horizontal screen video material; Perform optical flow prediction on the first frame image and the second frame image based on flownet to obtain optical flow images of the first frame image and the second frame image; For each pixel in the optical flow images of the first frame image and the second frame image, the sum of the absolute value in the x direction and the absolute value in the y direction is calculated to obtain the optical flow absolute value sum; Binarize the absolute value sum of the optical flow, and divide multiple optical flow connected domains in the optical flow images of the first frame image and the second frame image according to the binarization result of the absolute value sum of the optical flow, and obtain a set of optical flow connected domains {F1, F2, ..., Fi}, where i is the sequence number of the optical flow connected domain; Perform static saliency segmentation on the first frame image; According to the static saliency segmentation result of the first frame image, a plurality of static saliency connected domains are divided in the first frame image to obtain a set of static saliency connected domains {S1, S2, ..., Sm}, where m is the serial number of the static saliency connected domain; Calculate the intersection-and-union ratio between each optical flow connected domain in the set of optical flow connected domains {F1, F2, ..., Fi} and each statically significant connected domain in the set of statically significant connected domains {S1, S2, ..., Sm} to obtain multiple intersection-and-union ratios; Select an intersection-and-union ratio and determine whether the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold; If the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, an external rectangular frame of a graphic displayed by the union of the optical flow connected domain and the static saliency connected domain corresponding to the intersection-and-union ratio is obtained, and the external rectangular frame is used as a candidate screen subject; Returning the selected IoU, determining whether the IoU is greater than a preset IoU threshold, until each IoU is selected once, and obtaining at least one candidate screen subject; Calculate the sum of the absolute values ​​of the optical flows of all candidate screen subjects, sort all candidate screen subjects from large to small according to the sum of the absolute values ​​of the optical flows, and use the candidate screen subject with the largest sum of the absolute values ​​of the optical flows as the screen subject in the short horizontal screen video material; Return to the step of selecting a short horizontal screen video material until the main body of the picture in all the short horizontal screen video materials is obtained; The frame-by-frame tracking of the subject of the picture in each short horizontal screen video material includes: Select a short horizontal screen video material; Get the nth frame in the short horizontal screen video material, and get the position of the main body in the nth frame; n is initially set to 1; Get the n+1th frame in the short horizontal screen video material; Taking the position of the center of the main body of the picture in the nth frame in the n+1th frame as the center of the circle, drawing a circle in the n+1th frame with a preset search radius as the radius, and the area covered by the circumscribed rectangle of the circle as the local search area; the preset search radius is initially defined as the local search radius; In the local search area, a plurality of local sub-search areas having the same shape and the same coverage area as the main body of the picture in the nth frame are intercepted; Search for the position of the subject in the nth frame in the n+1th frame; In the n+1th frame, the similarity between the main body of the picture in the nth frame and each local sub-search area is calculated to obtain multiple similarities; Obtaining a maximum similarity among multiple similarities, and determining whether the maximum similarity is greater than a similarity threshold; If the maximum similarity is greater than the similarity threshold, the position of the local sub-search area corresponding to the maximum similarity is used as the position of the main body of the picture in the n+1th frame, and the positioning of the main body of the picture in the n+1th frame is completed; Add 1 to n, define the preset search radius as the local search radius, and return to the step of obtaining the n+1th frame in the short horizontal screen video material until the positioning of the main body of all video frames in the short horizontal screen video material is completed; Return to the step of selecting a short horizontal screen video material until all short horizontal screen video materials have completed the positioning of the main body of all video frames.

2. The method for converting horizontal screen video material into vertical screen video material according to claim 1, characterized in that: The acquiring of the horizontal screen video material and identifying at least one transition time node in the horizontal screen video material includes: Parse the horizontal screen video material to obtain multiple video frames; Performing differential calculation on every two consecutive video frames to obtain differential images of every two consecutive video frames; Calculate the sum of the pixel values ​​of all pixels in each differential image to obtain the pixel sum of each differential image; Perform difference calculation on the pixel sum of every two adjacent difference images, and define the absolute value of the difference value obtained by the difference calculation as the quadratic difference sum; Select a quadratic difference sum, and determine whether the quadratic difference sum is greater than a preset quadratic difference sum threshold; If the quadratic difference sum is greater than a preset quadratic difference sum threshold, the time node corresponding to the quadratic difference sum and the quadratic difference sum is taken as a candidate transition time node; Returning the selection of a quadratic difference sum, determining whether the quadratic difference sum is greater than a preset quadratic difference sum threshold, until all quadratic difference sums have been selected, and obtaining a plurality of candidate transition time nodes; Sort multiple candidate transition time nodes in chronological order; The first candidate transition time node is used as the time anchor point; Select a preset time period after the time anchor point; Search all candidate transition time nodes within a preset time period after the time anchor point, and select the candidate transition time node with the largest secondary difference as the transition time node; The transition time node is used as a new time anchor point, and the method of selecting a preset time period after the time anchor point is returned until all transition time nodes are obtained.

3. The method for converting horizontal screen video material into vertical screen video material according to claim 2, characterized in that: Before selecting a quadratic difference sum and determining whether the quadratic difference sum is greater than a preset quadratic difference sum threshold, acquiring the horizontal screen video material and identifying at least one transition time node in the horizontal screen video material further comprises: Calculate the average of all quadratic difference sums to obtain the average of quadratic difference sums; A preset threshold multiplier is obtained, and the product of the preset threshold multiplier and the quadratic difference and average value is used as the preset quadratic difference and threshold.

4. The method for converting horizontal screen video material into vertical screen video material according to claim 2, characterized in that: The acquiring of the horizontal screen video material and identifying at least one transition time node in the horizontal screen video material includes: Set the preset time length; A sliding window with a preset time length is used to traverse the horizontal screen time material, and the horizontal screen time material is decomposed into multiple video clips; Input each video segment into a video classification model, run the video classification model and determine whether each video segment is a video segment with transitions, so as to obtain multiple video segments with transitions; Extract the transition time node in each video clip with transition.

5. The method for converting horizontal screen video material into vertical screen video material according to claim 4, characterized in that: The step of extracting a transition time node from each video segment having a transition includes: Select a video clip with transitions; Extract each video frame in the video clip with transitions; Calculate the color histogram of each video frame in the video clip with transitions; Calculate the inter-frame distance between the color histograms of every two adjacent video frames; Selecting a time node corresponding to a video frame with the largest inter-frame distance as a transition time node in the video segment with transition; Return to the step of selecting a video segment with transitions, until obtaining the transition time nodes in each video segment with transitions.

6. The method for converting horizontal screen video material into vertical screen video material according to claim 3 or 5, characterized in that: The step of dividing the horizontal screen video material into a plurality of short horizontal screen video materials according to at least one transition time node includes: Sort all transition time nodes in chronological order; Select the first transition time node, capture all video frames between the start time node of the horizontal screen video material and the first transition time node as a short horizontal screen video material; Select the last transition time node, and capture all video frames between the last transition time node and the end time node of the horizontal screen video material as a short horizontal screen video material; Capture all video frames between every two adjacent transition time nodes as a short horizontal screen video material; Output all short horizontal screen video materials.

7. The method for converting horizontal screen video material into vertical screen video material according to claim 1, characterized in that: The frame-by-frame tracking of the subject of the picture in each short horizontal screen video material also includes: If the maximum similarity is less than or equal to the similarity threshold, the position of the subject in the nth frame is used as the position of the subject in the n+1th frame to complete the positioning of the subject in the n+1th frame; Add 1 to n, define the preset search radius as the global search radius, and return to the n+1th frame in the acquired short horizontal screen video material; the global search radius is larger than the local search radius.

8. The method for converting horizontal screen video material into vertical screen video material according to claim 7, characterized in that: The short vertical screen video material is generated according to the frame-by-frame tracking result of the screen subject in each short horizontal screen video material, and a plurality of short vertical screen video materials are obtained, including: Select a short horizontal screen video material; Obtain the center position of the subject of each frame in the short horizontal screen video material after frame-by-frame tracking; Taking the center position of the main body of each frame as the center, cut out a rectangular video frame with a width of w and a height of h in each frame; Take each rectangular video screen as a vertical screen video frame, and splice all the obtained rectangular video screens in chronological order to obtain a short vertical screen video material; Return to the step of selecting a short horizontal screen video material until all short horizontal screen video materials are converted into short vertical screen video materials.