Video processing method and related device

Abstract

The invention discloses a video processing method and a related device, and the method comprises the steps: firstly obtaining a plurality of shot segments of a horizontal-edition video, then dividing each shot segment into a plurality of video frame sequences, then determining a target object corresponding to a cutting window corresponding to a first frame in the video frame sequences, and carrying out the corresponding cutting of the first frame in the video frame sequences; according to the accumulated offset on the pixel level of the horizontal-version video, if the accumulated offset corresponding to the ith video frame does not reach the first preset threshold value, the position of the cutting window is kept unchanged, so that compared with the related technology, when the accumulated moving distance of the target object in the horizontal-version video does not exceed a certain value, the position of the cutting window is not changed. According to the method and the device, frequent jump of the clipping window caused by small change of the target object in the same shot can be filtered, unnecessary movement of the position of the clipping window in the horizontal-version video is effectively avoided, the picture stability of the converted vertical-version video is further effectively improved, the resource burden of a system is reduced, and the user experience is improved. And the video conversion efficiency is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a video processing method and related device. BACKGROUND

[0002] With the popularity of smart phones and the rise of short video applications, users are more and more accustomed to immersive vertical screen viewing experience, which makes the efficiency and quality of converting a video adapted for horizontal screen playback into a video adapted for vertical screen playback face higher requirements.

[0003] In the related art, the most salient target object in the video frames of a horizontal video adapted for horizontal screen playback is first identified, and the tracking of the target object is realized by combining a target tracking technology. Then, the target object is taken as the focal center of the picture to determine the position of a cropping window used to generate a vertical video adapted for vertical screen playback. Finally, the horizontal video is cropped frame by frame based on the cropping window to obtain the picture in the video frame corresponding to the vertical video.

[0004] However, when the position of the identified target object in the horizontal video changes constantly, the position of the cropping window will change accordingly and move all the time. For example, taking a moving hand as the focal center of the picture makes the hand in each video frame fixed at the same position of the vertical video picture, which easily causes frequent shaking of the vertical video picture. SUMMARY

[0005] Therefore, the present application provides a video processing method and related device to crop a horizontal video by taking a group of continuously shot shots as a basic unit, and filter the influence of the slight movement of the target in the key area on the vertical cropping window in the horizontal video picture, thereby effectively avoiding the frequent jumping of the picture in the vertical video and improving the stability of the vertical screen video picture.

[0006] To solve the above problems, the technical solutions provided by the present application are as follows:

[0007] In one aspect, the present application provides a video processing method, which comprises:

[0008] obtaining a plurality of shot segments, the shot segments being used to identify a shot in a horizontal video, the horizontal video being a video adapted for horizontal screen playback;

[0009] for a target shot segment in the plurality of shot segments, respectively taking each reference frame as the first frame of a video frame sequence, and dividing the video frames contained in the target shot segment into a plurality of video frame sequences without overlap, the reference frame being a video frame in the target shot segment other than the last one;

[0010] determine a target object corresponding to a cropping window corresponding to the target reference frame, and calculate a cumulative offset of the target object in each video frame in the target video frame sequence, the target video frame sequence being one of the plurality of video frame sequences, the target reference frame being a first frame of the target video frame sequence, the cumulative offset being used to identify a distance of cumulative movement of the target object from the target reference frame on a pixel level of the horizontal video, the cropping window referring to a cropping window used to generate the vertical video in the horizontal video, the vertical video being a video adapted for vertical screen playback;

[0011] when the cumulative offset of the i-th video frame in the target video frame sequence does not reach a first preset threshold, cropping the i-th video frame based on a position of the cropping window corresponding to the target reference frame, and taking a picture obtained by cropping as a picture in a video frame corresponding to the vertical video.

[0012] In a possible implementation, the cropping window corresponding to the target reference frame is determined in the following manner:

[0013] determining a speaker in the target reference frame;

[0014] if there is a speaker in the target reference frame, the cropping window corresponding to the target reference frame includes a region where a face of the speaker is located;

[0015] if there is no speaker in the target reference frame, determining a saliency region, the saliency region being used to identify a region in the target reference frame that has a probability of attracting a viewer's line of sight greater than a preset threshold, and the cropping window corresponding to the target reference frame includes the saliency region.

[0016] In a possible implementation, the method further includes:

[0017] if the cumulative offset of the i+1-th video frame in the target video frame sequence reaches the first preset threshold, taking a first reference frame after the target video frame sequence as a first reference frame, and constructing a first order polynomial function based on a corresponding position of a center position of the cropping window corresponding to the i-th video frame in the horizontal video ( ) and a corresponding position of a center position of the cropping window corresponding to the first reference frame in the horizontal video ( ). the cumulative offset of the i-th video frame does not reach the first preset threshold;

[0018] based on the first order polynomial function interpolating between the first reference frame and the i-th video frame, and the number of video frames between the i-th video frame and the first reference frame, to determine the first cropping window corresponding to the i+1-th video frame.

[0019] In a possible implementation, the method further includes:

[0020] resetting the accumulated offset of the i+1-th video frame;

[0021] calculating, based on the first target object corresponding to the first cropping window, the accumulated offset of the first target object in each video frame after the i+1-th video frame in the target video frame sequence;

[0022] when the accumulated offset of the k-th video frame after the i+1-th video frame in the target video frame sequence does not reach a first preset threshold, cropping the k-th video frame based on the position of the first cropping window, and taking the cropped picture as the picture in the corresponding video frame of the vertical video.

[0023] In a possible implementation, the method further includes:

[0024] determining the position of the target object in each video frame in the target video frame sequence based on the target object corresponding to the center position of the cropping window in the target reference frame;

[0025] taking the distance between the position and the center position in the horizontal direction of the horizontal video as the offset of the target object;

[0026] accumulating the offset to obtain the accumulated offset of the target in each video frame in the target video frame sequence.

[0027] In a possible implementation, the method further includes:

[0028] extracting visual features corresponding to the last video frame of a previous shot segment and the first video frame of a next shot segment, the previous shot segment and the next shot segment being two adjacent shot segments in the plurality of shot segments, and the visual features being used to identify features of a region in a video frame in which the probability of attracting the viewer's line of sight is greater than a preset threshold;

[0029] If a similarity between the visual feature of the last video frame and the visual feature of the first video frame is greater than a second preset threshold, the transparency of the last video frame is increased, the first video frame is displayed at the same time, and a cropping window corresponding to the first video frame is adjusted based on a cropping window corresponding to the last video frame.

[0030] In a possible implementation, the method further includes:

[0031] uniformly extracting a plurality of video frames from the video frames included in the target shot segment, and taking the video frames as the reference frames.

[0032] In an aspect, the present application provides a video processing apparatus, the apparatus comprising an obtaining unit, a dividing unit, a determining unit, and a cropping unit:

[0033] The obtaining unit obtains a plurality of shot segments, the shot segments being used to identify a shot in a landscape video, the landscape video being a video adapted for landscape playback;

[0034] The dividing unit is configured to, for a target shot segment in the plurality of shot segments, divide video frames included in the target shot segment into a plurality of video frame sequences without overlapping, respectively taking each reference frame as a first frame of a video frame sequence, the reference frame being a video frame in the target shot segment other than a last one.

[0035] The determining unit is configured to, for a target video frame sequence, determine a target object corresponding to a cropping window corresponding to a target reference frame, and calculate a cumulative offset of the target object in each video frame in the target video frame sequence, the target video frame sequence being one of the plurality of video frame sequences, the target reference frame being a first frame of the target video frame sequence, the cumulative offset being used to identify a distance of cumulative movement of the target object on a pixel level of the landscape video starting from the target reference frame, the cropping window being a cropping window used to generate a portrait video in the landscape video, the portrait video being a video adapted for portrait playback.

[0036] The cropping unit is configured to, when a cumulative offset of an i-th video frame in the target video frame sequence does not reach a first preset threshold, crop the i-th video frame based on a position of the cropping window corresponding to the target reference frame, and take a picture cropped to be a picture in a video frame corresponding to the portrait video.

[0037] In a possible implementation, the cropping window corresponding to the target reference frame is determined in the following manner:

[0038] determining a speaker in the target reference frame;

[0039] if there is a speaker in the target reference frame, the target reference frame corresponds to a clipping window including a region where the speaker's face is located;

[0040] if there is no speaker in the target reference frame, determining a saliency region, the saliency region is used to identify a region in the target reference frame where the probability of attracting audience's visual line is greater than a preset threshold, and the target reference frame corresponds to a clipping window including the saliency region.

[0041] In a possible implementation, the apparatus further includes a constructing unit:

[0042] The constructing unit is configured to, if the accumulated offset of the i+1th video frame in the target video frame sequence reaches the first preset threshold, take the first reference frame after the target video frame sequence as a first reference frame, and construct a first-order polynomial function based on a position corresponding to a center position of a clipping window of the i th video frame in the horizontal video (x i, y i) and a position corresponding to a center position of a clipping window of the first reference frame in the horizontal video (x 1, y 1). The first-order polynomial function is f(x) = ax + b, where a = (x 1-x i) / (x i-x 1), b = y 1-a x 1. The accumulated offset of the i th video frame does not reach the first preset threshold.

[0043] The first-order polynomial function f(x) and the number of video frames between the i th video frame and the first reference frame are used for interpolation to determine a first clipping window corresponding to the i+1th video frame.

[0044] In a possible implementation, the apparatus further includes a resetting unit and a calculating unit:

[0045] The resetting unit is configured to reset the accumulated offset of the i+1th video frame.

[0046] The calculating unit is configured to take the first target object corresponding to the first clipping window as a reference to calculate an accumulated offset of the first target object in each video frame after the i+1th video frame in the target video frame sequence.

[0047] The clipping unit is further configured to, if the accumulated offset of the k th video frame after the i+1th video frame in the target video frame sequence does not reach the first preset threshold, clip the k th video frame based on the position of the first clipping window, and take a picture obtained by clipping as a picture in a video frame corresponding to the vertical video.

[0048] In a possible implementation method, the determining unit is specifically configured to:

[0049] ​​determine a position of the target object in each video frame of the target video frame sequence based on the target object corresponding to the center position of the cropping window in the target reference frame;

[0050] take the distance between the position and the center position in the horizontal direction of the horizontal version video as an offset of the target object;

[0051] accumulate the offset to obtain an accumulated offset of the target in each video frame of the target video frame sequence.

[0052] In a possible implementation, the apparatus further includes an extraction unit and a transition unit:

[0053] The extraction unit is configured to extract visual features corresponding to a last video frame of a previous shot segment and a first video frame of a next shot segment, the previous shot segment and the next shot segment being two adjacent shot segments in the plurality of shot segments, the visual features being used to identify features of a region in a video frame that attracts a viewer's line of sight with a probability greater than a preset threshold;

[0054] The transition unit is configured to, if a similarity between the visual features of the last video frame and the visual features of the first video frame is greater than a second preset threshold, increase transparency of the last video frame, simultaneously display the first video frame, and adjust a cropping window corresponding to the first video frame based on a cropping window corresponding to the last video frame.

[0055] In a possible implementation, the apparatus further includes an extraction unit:

[0056] The extraction unit is configured to uniformly extract a plurality of video frames from video frames included in the target shot segment, and take the video frames as the reference frames.

[0057] In yet another aspect, the present application provides a computer device, which includes a processor and a memory:

[0058] The memory is configured to store a computer program;

[0059] The processor is configured to execute the method according to any one of the preceding aspects based on the computer program.

[0060] In yet another aspect, the present application provides a computer readable storage medium, which is configured to store a computer program, the computer program being configured to implement the method according to any one of the preceding aspects when executed by a computer device.

[0061] In yet another aspect, the present application provides a computer program product comprising a computer program which, when run on a computer device, causes the computer device to perform any of the above-mentioned methods.

[0062] It can be seen from the above technical solution that the technical solution firstly acquires a plurality of shot segments, wherein the shot segments are used for a shot in the horizontal video, and then respectively takes each reference frame in the target shot segment as a first frame of a video frame sequence, divides the video frames contained in the target shot segment into a plurality of video frame sequences without overlapping, and then respectively determines, for each video frame in the target video frame sequence, a cumulative offset of a target object corresponding to a cropping window in the first frame on a pixel level of the horizontal video. If the cumulative offset corresponding to the i-th video frame in the target video frame sequence does not reach a first preset threshold, the i-th video frame is cropped based on the position of the cropping window in the first frame, that is, the position of the cropping window is not moved with the change of the target object, so that compared with the related art, when the cumulative moving distance of the target object in the horizontal video does not exceed a certain value, the frequent jumping of the cropping window caused by the slight change of the target object in the same shot can be filtered, so that the target object as the focus center of the picture does not have to be pulled back to the same fixed position at all times, unnecessary movement of the position of the cropping window in the horizontal video is effectively avoided, and the picture stability of the converted vertical video is effectively improved, the user's viewing experience is improved, and the position of the cropping window of a plurality of continuous video frames can be kept unchanged, the resource burden of the system is reduced, and the efficiency of video conversion is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0064] Figure 1 One of the flowcharts of the video processing method provided by the embodiments of the present application;

[0065] Figure 2 A schematic diagram of target reference frame conversion provided by the embodiments of the present application;

[0066] Figure 3 The second flowchart of the video processing method provided by the embodiments of the present application;

[0067] Figure 4 The schematic diagram of the video processing device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0068] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0069] As described in the background section, existing video processing methods primarily rely on computer vision algorithms. After determining the cropping window in each frame of a landscape video for horizontal playback, the landscape video is cropped frame by frame to obtain a portrait video for vertical playback. The cropping window can be determined as follows:

[0070] (1) Saliency detection algorithm: Identify the most eye-catching key area in the picture and make the cropping window follow the key area.

[0071] (2) Target detection algorithm: Identify key targets such as faces and bodies, and make the key targets located in the clipping window.

[0072] (3) Algorithm combining saliency detection and target detection: The score of each pixel in the image is obtained by the saliency detection algorithm, and the overall score of key areas such as face and body is combined to determine the cropping window.

[0073] The above-mentioned method of determining the cropping window frame by frame is very sensitive to minute movements of the target in the picture or slight changes in the salient area. When the position of the identified target object in the horizontal video keeps changing, the position of the cropping window will move with it. For example, if the moving hand is used as the focal center of the picture, the hand will be fixed in the same position in the vertical video picture in each video frame. This will cause the cropping window to generate a lot of unnecessary and fragmented translational movements in order to continuously track the target. This will easily cause frequent shaking of the vertical video picture, giving users a "motion sickness" feeling and destroying the immersive experience.

[0074] To solve the above problems, the application provides a video processing method and related device, which first acquires a plurality of shot segments, wherein the shot segment is used for a shot in a horizontal video, then respectively takes each reference frame in a target shot segment as a first frame of a video frame sequence, and divides the video frames contained in the target shot segment into a plurality of video frame sequences without overlapping, then respectively determines the accumulated offset of the target object corresponding to the cropping window in the first frame on the pixel level of the horizontal video for each video frame in the target video frame sequence, and if the accumulated offset corresponding to the i-th video frame in the target video frame sequence does not reach a first preset threshold, the i-th video frame is cropped based on the position of the cropping window in the first frame, that is, the position of the cropping window is not moved with the change of the target object, so that compared with related technologies, when the accumulated moving distance of the target object in the horizontal video does not exceed a certain value, the frequent jumping of the cropping window caused by the slight change of the target object in the same shot can be filtered, so that the target object as the focal center of the picture does not have to be pulled back to the same fixed position at all times, unnecessary movement of the position of the cropping window in the horizontal video is effectively avoided, and the picture stability of the converted vertical video is effectively improved, the user's viewing experience is improved, and the positions of the cropping windows of a plurality of continuous video frames can be kept unchanged, the resource burden of the system is reduced, and the efficiency of video conversion is effectively improved.

[0075] The scheme provided by the embodiments of the application relates to the field of artificial intelligence technology, and is specifically explained through the following embodiments.

[0076] Referring to Figure 1 As shown in FIG. 1, it is one of the flow diagrams of a video processing method provided by the embodiments of the application, which can be executed by a computer device in this embodiment.

[0077] S101: Acquire a plurality of shot segments.

[0078] The horizontal video is a video stored in the computer device and adapted for horizontal screen playback, which is obtained by a device for shooting.

[0079] A shot of a movie or a video is usually a continuous and complete narrative unit, and the shot segment is used to identify a shot in the horizontal video and is composed of a plurality of continuous video frames.

[0080] The horizontal video can be scanned by a shot detection algorithm to output accurate time stamps of all shot switching points, divide the horizontal video into a series of shot segments [Shot_1, Shot_2,..., Shot_N], and then process each shot segment as a target shot segment.

[0081] S102: For a target shot segment in the plurality of shot segments, divide video frames contained in the target shot segment into a plurality of video frame sequences by taking each reference frame as a first frame of a video frame sequence without overlapping.

[0082] The reference frame is a video frame in the target shot segment that is not the last one. A plurality of video frames can be determined as reference frames according to the target object in the target shot segment, for example, a video frame at which the target object first appears is taken as a reference frame, one or more reference frames are obtained, and each reference frame is taken as a first frame of a video frame sequence, so that the video frames contained in the target shot segment are divided to obtain a plurality of non-overlapping video frame sequences. The target object can be any person or object, and embodiments of the present application do not limit this.

[0083] As an example, the shot segment Shot_1 contains 5 video frames, namely frame 11, frame 12, frame 13, frame 14 and frame 15, and has 1 new appearing person in frame 12 and frame 14 respectively, so frame 12 and frame 14 are taken as reference frames, and then frame 12 and frame 14 are taken as first frames of video frame sequences respectively to obtain video frame sequences [frame 12, frame 13] and [frame 14, frame 15].

[0084] The target object in a group of shots, such as background information or a person, may not change significantly. One or more video frames can be selected as reference frames, and the position information of the target object in the reference frames in other video frames can be used to establish the association between the plurality of video frames in the same shot.

[0085] In a possible implementation, the method further includes:

[0086] Uniformly extracting a plurality of video frames from the video frames contained in the target shot segment, and taking the video frames as reference frames.

[0087] After determining the number of video frames contained in the target shot segment, a plurality of video frames can be extracted by setting a fixed interval to obtain a plurality of reference frames.

[0088] The fixed intervals corresponding to different shot segments can be the same or different, for example, a larger fixed interval can be set in a shot segment with a larger number of video frames.

[0089] In this way, a plurality of video frames are uniformly extracted in the shot segment to obtain reference frames, without the need to identify specific information of different target objects, which can effectively save system resources and reduce resource burden.

[0090] S103: For a target video frame sequence in the plurality of video frame sequences, determine a target object corresponding to a cropping window corresponding to a target reference frame, and calculate a cumulative offset of the target object in each video frame in the target video frame sequence.

[0091] The target video frame sequence is one of the plurality of video frame sequences, and the target reference frame is a first frame of the target video frame sequence.

[0092] The cropping window refers to a cropping window used to generate a vertical video in a horizontal video, where the vertical video refers to a video adapted for vertical screen playback. The cropping window is usually rectangular in shape, and the ratio between the length in the horizontal direction and the width in the vertical direction of the vertical video can be determined according to the size of the display interface of the terminal device. In the embodiments of the present application, the aspect ratio of the cropping window in different video frames does not change.

[0093] As an example, referring to Figure 2 Fig. A shows a horizontal screen playback mode, and the content displayed in the terminal device is a target reference frame in a horizontal video adapted for horizontal screen playback, which has a length of 50 in the horizontal direction and a width of 30 in the vertical direction (the picture content is not drawn). In order to adapt it to vertical screen playback in the terminal device, the video frame picture needs to be cropped. The cropping window corresponding to the target reference frame is shown in the figure, which is rectangular in shape, with a length of 20 and a width of 30. The width of the cropping window is kept as the width of the picture of the horizontal video. Fig. B shows a vertical screen playback mode, and the terminal device plays a vertical video adapted for vertical screen playback. At this time, the display of the terminal device is the content of the picture in the cropping window corresponding to the target reference frame after stretching, which has a length of 20 stretched to 30 and a width of 30 stretched to 45 in proportion.

[0094] The area in the picture that can attract the viewer's gaze the most can be determined by related technologies such as saliency detection algorithms and object detection algorithms, and the area is included in the cropping window corresponding to the reference frame, so as to determine that each reference frame corresponds to a cropping window. The size of the area that can attract the viewer's gaze the most identified by the method can be different, and the size of the cropping window in different reference frames can change.

[0095] However, in most narrative videos, dialogue is the core of driving the plot, and who is speaking is the most important focus of the picture. In a multi-person dialogue scene, the algorithm may focus on a non-speaker because of visual saliency (such as someone's clothes are brighter, or the action is larger), resulting in the key character who is speaking being cropped out of the picture.

[0096] In related technologies, when determining the cropping window in the first reference frame, only visual information is relied on to judge the area that can attract the viewer's gaze, and the key modality of audio information is ignored. In a possible implementation, the cropping window corresponding to the target reference frame is determined by the following method:

[0097] A1: determining a speaker in the target reference frame;

[0098] A2: if the speaker exists in the target reference frame, the cropping window corresponding to the target reference frame includes a region where the face of the speaker is located;

[0099] A3: if the speaker does not exist in the target reference frame, determining a saliency region, the saliency region is used to identify a region where the probability of attracting the viewer's line of sight is greater than a preset threshold in the target reference frame, and the cropping window corresponding to the target reference frame includes the saliency region.

[0100] The active speaker detection model can be used to detect the speaker in the target reference frame, wherein the active speaker detection model is a pre-trained detection model, and the audio stream of the horizontal video and the face detection result in each reference frame are input into the model to determine whether the input face is speaking.

[0101] In the embodiments of the present application, multiple people who are speaking can be identified in each reference frame, and the saliency can be determined through the position information of the people who are speaking and the size of the face detection frame, and one of the saliency is the strongest one as the speaker.

[0102] When it is determined that the speaker does not exist in the reference frame through the active speaker detection model, the saliency heat map and the human body frame boundary of each reference frame are obtained based on the saliency detection algorithm and the human body detection algorithm, the saliency region where the probability of attracting the viewer's line of sight is greater than a preset threshold is determined by combining the saliency score of each pixel point and the saliency score of the whole human body frame, and the saliency region is included in the cropping window. When it is determined that the speaker exists in the first reference frame, the region where the face of the speaker is located is included in the cropping window, so as to determine the cropping window corresponding to the reference frame according to the picture ratio of the vertical video.

[0103] Therefore, the audio and video multi-modal information are used as the key basis for determining the picture focus, which can ensure that the picture focus is preferentially aligned to the speaker and the integrity of the dialogue scene and the clarity of the narrative logic are ensured.

[0104] It should be noted that the cropping window and the center position of the cropping window can be regarded as an absolute position in the video frame, and the position on the pixel level of the horizontal video does not change with the change of the picture. The position of the target object can be regarded as a relative position, and the position on the pixel level of the horizontal video changes with the change of the picture. The movement of the target object is the movement of the target object on the pixel level of the horizontal video.

[0105] The target object corresponding to the cropping window can be a pixel point or a regular area of a picture. Since the reference frame is the first frame in the video frame sequence to which the reference frame belongs, based on the target object corresponding to the cropping window in each reference frame, the cumulative offset of the target object in each video frame in the video frame sequence to which the reference frame belongs can be determined through a target tracking technology, that is, the distance of the target object moving on the pixel level from the reference frame.

[0106] Specifically, the computer device initializes the corresponding cumulative offset of each reference frame of the target lens segment as 0, and then determines the distance of the target object corresponding to the cropping window of the reference frame moving on the pixel level from the second frame in the video frame sequence to which the reference frame belongs, and then adds the distance according to the order of each video frame in the video frame sequence to determine the cumulative offset of each video frame.

[0107] In order to retain the content contained in the picture of the vertical video as much as possible, the length of the cropping window in the vertical direction is retained as the length of the picture of the horizontal video in the vertical direction. In order to facilitate calculation, the cumulative offset can be determined based on the distance of the target object corresponding to the center position of the cropping window moving in the horizontal direction. In one possible implementation, step S103 includes:

[0108] B1: determining the position of the target object in each video frame of the target video frame sequence based on the target object corresponding to the center position of the cropping window in the target reference frame;

[0109] B2: taking the distance between the position and the center position in the horizontal direction of the horizontal video as the offset of the target object;

[0110] B3: adding the offset to obtain the cumulative offset of the target in each video frame of the target video frame sequence.

[0111] The computer device initializes the corresponding cumulative offset of each reference frame of the target lens segment as 0, and then determines the target object corresponding to the center position of the cropping window of the reference frame from the second frame in the video frame sequence to which the reference frame belongs, and determines the distance between the target object and the center position in the horizontal direction of the horizontal video in other video frames as the offset of the target object in the video frame, and then adds the offset according to the order of each video frame in the video frame sequence to determine the cumulative offset of each video frame.

[0112] As an example, the target shot shot_1 includes 5 reference frames, and frame 1 and frame 4 are taken as the reference frames, the obtained video frame sequence is [frame 1, frame 2, frame 3], [frame 4, frame 5], wherein the coordinates of the center position of the cropping window of frame 1 are [520, 500], and the coordinates of the center position of the cropping window of frame 4 are [500, 500]. Wherein the first dimension is the coordinate in the horizontal direction of the vertical video, and the second dimension is the coordinate in the vertical direction.

[0113] The computer device initializes an accumulated offset of frame 1 and frame 4 to 0 respectively, determines the target object corresponding to the center position of the cropping window of frame 1 from the second frame of the first video frame sequence, and the coordinates of the target object in frame 2 and frame 3 are [500, 520] and [530, 520] respectively, so that the distance between the center position of the cropping window and the target object in the horizontal direction in frame 2 and frame 3 is 20 and 10 respectively, and the accumulated offset corresponding to frame 2 and frame 3 is 20 and 30 respectively. From the second frame of the second video frame sequence, the target object corresponding to the center position of the cropping window of frame 4 is determined, and the coordinates of the target object in frame 5 are [520, 520], so that the distance between the center position of the cropping window and the target object in the horizontal direction in frame 5 is 20, and the accumulated offset corresponding thereto is 20.

[0114] Therefore, by calculating the offset of the target in the horizontal direction through the center position of the cropping window, the target tracking task can be completed under lower resource consumption, thereby improving the efficiency of video conversion.

[0115] S104: When the accumulated offset of the i th video frame in the target video frame sequence does not reach the first preset threshold, cropping the i th video frame based on the position of the cropping window corresponding to the target reference frame, and taking the picture obtained by cropping as the picture in the video frame corresponding to the vertical video.

[0116] In the related art, the cropping window is very sensitive to the slight change of the target, and the movement of the target in the horizontal direction will cause the corresponding change of the cropping window, for example, when the target moves to the right in the horizontal direction, the cropping window will move to the right, thereby causing frequent shaking of the picture.

[0117] The cameraman will not adjust every slight change of the target when shooting through the device. The embodiment of the application simulates this behavior through the sliding accumulation strategy, and only when the movement of the target accumulates to a certain degree, the cropping window is moved.

[0118] In this embodiment, when the cumulative movement distance of the target is small, the position of the cropping window is not adjusted. Specifically, when the cumulative offset corresponding to the i-th video frame in the video frame sequence has not reached the first preset threshold, the cropping window in the i-th video frame is determined to still be the cropping window in the reference frame of the video frame sequence to which the i-th video frame belongs, and the position of the cropping window remains unchanged.

[0119] Furthermore, the cropping window corresponding to each video frame whose cumulative offset reaches the first preset threshold can be determined using saliency detection algorithms and object detection algorithms in related technologies.

[0120] Linear interpolation can also be used to more conveniently determine the cropping window corresponding to each video frame whose cumulative offset reaches a first preset threshold. In one possible implementation, the method further includes:

[0121] C1: If the cumulative offset corresponding to the (i+1)th video frame in the target video frame sequence reaches the first preset threshold, the first reference frame after the target video frame sequence is taken as the first reference frame, and the position of the cropping window corresponding to the i-th video frame in the horizontal video is taken as the position of the cropping window in the horizontal video. ) and the position of the cropping window corresponding to the first reference frame in the landscape video ( Construct a first-order polynomial function ;

[0122] C2: Based on linear polynomial functions The first cropping window corresponding to the (i+1)th video frame is determined by interpolating the number of video frames between the i-th video frame and the first reference frame.

[0123] Based on the positions of the cropping windows corresponding to the two processed video frames before and after the (i+1)th video frame, and the number of video frames before the (i+1)th video frame, the position of the cropping window corresponding to the (i+1)th video frame is determined by linear interpolation.

[0124] Specifically, when the cumulative offset corresponding to the (i+1)th video frame in the target video frame sequence reaches the first preset threshold, the first reference frame after the target video frame sequence is taken as the first reference frame. The center position of the cropping window corresponding to the i-th video frame whose cumulative offset has not reached the first preset threshold and the first reference frame are used. ), ( Construct a first-order polynomial function y=f(x), then determine the number of video frames N between the i-th video frame and the first reference frame, in ( )and( Insert N values ​​evenly between them to satisfy the condition. the first inserted value is taken as the center position of the cropping window corresponding to the i+1th video frame , , and the calculation formula is as follows:

[0125]

[0126] ,

[0127] =f( )

[0128] As an example, the i+1th video frame is frame 13 in the target video frame sequence shot_1[frame 11, frame 12, frame 13, frame 14, frame 15], where the cumulative offset corresponding to frame 12 does not reach the first preset threshold, and the center position of the cropping window corresponding thereto is (300, 500), the first reference frame after the target video frame sequence is frame 22 in the video frame sequence shot_2[frame 21, frame 22, frame 23], and the center position of the cropping window corresponding thereto is (320, 550), and the number of video frames between frame 13 and frame 22 is 3, then taking the first dimension of the center position as x and the second dimension as y, a first order polynomial function is constructed, and the solution is obtained as:

[0129] ,

[0130] According to the above formula, the center position of the cropping window corresponding to the i+1th video frame is (305, 512.5).

[0131] According to the coordinates of the center position and the fixed size of the cropping window, the first cropping window in the i+1th video frame can be determined, and then the position adjustment of the cropping window is realized based on the first cropping window in the i+1th video frame.

[0132] Therefore, when the cumulative movement distance of the target object in the horizontal direction exceeds a certain threshold, the position of the cropping window can be adjusted so that the target object is located at the center of the cropping window.

[0133] In order to further ensure the picture stability of the vertical video, the first target object corresponding to the first cropping window corresponding to the i+1th video frame can be taken as a reference to recalculate the cumulative offset, so as to keep the cropping window corresponding to the subsequent video frames unchanged, and in a possible implementation manner, the method further includes:

[0134] D1: resetting the cumulative offset of the i+1th video frame;

[0135] D2: taking the first target object corresponding to the first cropping window as a reference, calculating a cumulative offset of the first target object in the horizontal video in each video frame after the i+1th video frame in the target video frame sequence;

[0136] D3: when the cumulative offset of the kth video frame after the i+1th video frame in the target video frame sequence does not reach the first preset threshold, cropping the kth video frame based on the position of the first cropping window, and taking the cropped picture as a picture in the vertical video corresponding video frame.

[0137] Based on the first cropping window, the position of the cropping window in the i+1th video frame is adjusted, and the cumulative offset of the first target object in each video frame after the i+1th video frame in the target video frame sequence is calculated based on the first target object corresponding to the first cropping window corresponding to the first target object. When the cumulative offset corresponding to the kth video frame after does not reach the first preset threshold, it is determined that the cropping window corresponding to the kth video frame is still the first cropping window corresponding to the i+1th video frame, and the position of the first cropping window is kept unchanged.

[0138] Therefore, the distance of the target object moving on the pixel level of the horizontal video can be recalculated based on the target object corresponding to the adjusted cropping window, the frequent jumping of the cropping window caused by the slight change of the target object in the same shot can be filtered, and the stability of the vertical video picture is further guaranteed.

[0139] In the related art, each frame is processed independently, which may cause the cropping window to jump sharply at the shot boundary, which is very harsh and unnatural. In one possible implementation, the method further includes:

[0140] E1: extracting visual features corresponding to the last video frame of the previous shot segment and the first video frame of the next shot segment;

[0141] E2: if the similarity between the visual features of the last video frame and the visual features of the first video frame is greater than a second preset threshold, increasing the transparency of the last video frame, displaying the first video frame at the same time, and adjusting the cropping window corresponding to the first video frame based on the cropping window corresponding to the last video frame.

[0142] Wherein, the previous shot segment and the next shot segment are two adjacent shot segments in the plurality of shot segments. The visual features are used to identify the region in the video frame whose probability of attracting the viewer's eyesight is greater than a preset threshold.

[0143] The computer device first selects the last video frame of the previous shot segment and the first video frame of the next shot segment, which are adjacent in time sequence, then calculates the saliency scores of regions in the adjacent video frames based on a saliency detection algorithm, and extracts features of the adjacent video frames based on a color histogram, compares the visual features of regions with saliency scores greater than a preset threshold, and determines the similarity between the visual features of the two adjacent video frames.

[0144] When the similarity is greater than a second preset threshold, it is considered that the scene change is not large, and a smooth transition between the two adjacent video frames can be realized by adding a "dissolve" transition effect, specifically, the transparency of the last video frame in the previous shot segment is increased, and the first video frame in the next shot segment is displayed at the same time, and the cropping window corresponding to the first video frame in the next shot segment is adjusted based on the cropping window corresponding to the last video frame in the previous shot segment.

[0145] When the similarity is not greater than the second preset threshold, it is considered that the scene change is large, and a hard cut of the shot may occur, and the positions of the target objects in the two shot segments before and after the cut may be very different (such as from the left side of the screen to the right side), and no transition effect is added.

[0146] Therefore, by combining the saliency detection algorithm and the color histogram comparison, the similarity of scenes between different shots can be determined, and smooth transition between adjacent shots with small scene changes can be realized through transition effects, avoiding the abrupt jump problem caused by related technologies during shot switching, making the transition effect smooth and natural, and improving the watchability of the vertical video.

[0147] Therefore, the present application takes the shot as the basic processing unit, which can fundamentally avoid the logical confusion and picture confusion caused by cross-shot analysis, and ensure the visual coherence within each shot. By using the active speaker detection model, the cropping center is determined based on the speaker as the highest priority focal center, which can preserve the integrity of the video narrative logic, and when the distance of the cumulative movement of the target object corresponding to the cropping center on the pixel level of the horizontal video is small, the position of the cropping center is kept unchanged, filtering out the picture jitter caused by the slight movement of the target object or algorithm noise, improving the stability of the picture, avoiding visual fatigue and car sickness for users, and significantly improving the immersive viewing experience. By comprehensively analyzing the similarity of scenes in adjacent video frames between adjacent shots, a smooth transition effect is realized for adjacent shots with small scene changes.

[0148] Reference Figure 3 As shown in FIG. 2, it is a flowchart of a video processing method provided by an embodiment of the present application, in order to more clearly describe the video processing method, the following will be described in combination with a specific implementation scenario.

[0149] Input: File of the horizontal video (contains video stream and audio stream).

[0150] Step 1: Shot detection and active speaker detection.

[0151] This step can be processed in parallel, providing basic data for subsequent video processing.

[0152] Shot detection: Use shot detection algorithm to scan the entire horizontal video, output the accurate timestamp of all shot cut points. The video is divided into a series of shot segments [Shot_1, Shot_2,..., Shot_N].

[0153] Active speaker detection: input the audio stream and face detection results of the horizontal video into a pre-trained active speaker detection model to determine whether the input face is speaking.

[0154] Step 2: Process by shot.

[0155] Uniformly frame extraction within each shot segment, obtain multiple reference frames, for each reference frame: query the active speaker detection result to determine whether there is a speaker at the current frame time point, if there is a speaker, determine the cropping window based on the speaker's face box. If there is no speaker, run the saliency detection model and human body detection model to obtain the saliency heat map and human body box boundary of the current frame, and determine the cropping window based on the saliency score and human body box size.

[0156] Step 3: Intra-shot transition.

[0157] Within each shot segment, for each reference frame processed by step 2, initialize a cumulative offset, the center position of the current cropping window is the focal center of the first video frame, starting from the 2nd frame to determine the position of the cropping window frame by frame:

[0158] a. Calculate the expected offset: calculate the horizontal distance between the focal center of the frame and the center position of the current cropping window.

[0159] b. Apply sliding accumulation strategy: add the expected offset to the cumulative offset, determine whether the absolute value of the cumulative offset reaches the first preset threshold, if it does, move the cropping window and reset the cumulative offset. If it does not exceed the first preset threshold, keep the cropping window position unchanged.

[0160] c. Calculate the cropping window of all frames: frames that are not selected for processing when uniformly extracting frames within the shot, use linear interpolation method to calculate the cropping window of the current frame using the results of the nearest processed reference frames before and after.

[0161] Step 4: Inter-shot transition.

[0162] For two adjacent shot segments, the last frame of the previous shot segment and the first frame of the next shot segment are obtained, the similarity of the color histogram corresponding to the region attracting the viewer's line of sight in the two frames is calculated, if the histogram similarity is high, it indicates that the scene change is small, and a "dissolve" transition transition effect is added to realize smooth transition. If the histogram similarity is low, it indicates a hard cut, and no special transition effect is added.

[0163] Step 5: Synthesize the output.

[0164] Based on the corresponding cropped window of all frames, the original horizontal video is cropped frame by frame, and the final vertical video is synthesized.

[0165] Output: a piece of vertical video with stable picture, clear focus and natural transition.

[0166] Therefore, the present application first divides the video into independent shot segments through a shot detection algorithm; within each shot segment, a sliding accumulation strategy is used to suppress unnecessary picture jitter; at the same time, active speaker detection technology is introduced to combine audio information and visual information to ensure that the picture focus is always aimed at the person who is speaking; finally, at the boundary of the shot segment, the starting position of the new shot segment is intelligently determined through saliency detection and histogram comparison to realize smooth transition. Thus, the problems of frequent picture sliding, loss of key speakers, unnatural shot switching and the like in the related art are effectively solved, and the viewing quality and user experience of the vertical video after automatic conversion are improved.

[0167] From the perspective of a company with a large number of film and television copyrights, the following beneficial effects are achieved:

[0168] (1) Make full use of massive inventory content: it can convert the horizontal video library into a vertical format suitable for distribution and promotion on short video platforms at low cost and high quality, greatly improving the utilization rate and monetization potential of content assets.

[0169] (2) Improve user experience and platform competitiveness: in the vertical screen mode of the company's short video product or long video application program, the viewing experience can be improved, thereby enhancing user stickiness and attracting more users.

[0170] (3) Reduce manual editing costs: compared with the method of manually editing horizontal videos frame by frame to produce vertical trailers or slices, a large amount of labor cost and time cost can be saved, and cost reduction and efficiency improvement can be realized.

[0171] On the basis of the above embodiments, the present application provides a video processing device, which is described with reference to Figure 4As shown, a schematic diagram of a video processing device provided by an embodiment of the present application is shown, the device 300 includes an acquisition unit 301, a division unit 302, a determination unit 303, and a clipping unit 304:

[0172] The acquisition unit acquires a plurality of shot segments, the shot segments being used to identify a shot in a landscape video, the landscape video being a video adapted for landscape playback;

[0173] The division unit is configured to, for a target shot segment in the plurality of shot segments, respectively take each reference frame as a first frame of a video frame sequence, and divide video frames contained in the target shot segment into a plurality of video frame sequences without overlap, the reference frame being a video frame in the target shot segment that is not the last one;

[0174] The determination unit is configured to, for a target video frame sequence, determine a target object corresponding to a clipping window corresponding to a target reference frame, and calculate a cumulative offset of the target object in each video frame in the target video frame sequence, the target video frame sequence being one of the plurality of video frame sequences, the target reference frame being the first frame of the target video frame sequence, the cumulative offset being used to identify a distance of cumulative movement of the target object in a pixel layer of the landscape video starting from the target reference frame, the clipping window referring to a clipping window used to generate a portrait video in the landscape video, the portrait video being a video adapted for portrait playback;

[0175] The clipping unit is configured to, when the cumulative offset of an i-th video frame in the target video frame sequence does not reach a first preset threshold, clip the i-th video frame based on a position of the clipping window corresponding to the target reference frame, and take a picture obtained by clipping as a picture in a video frame corresponding to the portrait video.

[0176] Therefore, when the cumulative movement distance of the target object in the landscape video does not exceed a certain value, the frequent jumping of the clipping window caused by the slight change of the target object in the same shot can be filtered, so that the target object as the focus center of the picture does not have to be pulled back to the same fixed position at all times, unnecessary movement of the position of the clipping window in the landscape video is effectively avoided, and the picture stability of the converted portrait video is effectively improved, the user's viewing experience is improved, and the positions of the clipping windows of a plurality of continuous video frames can be kept unchanged, the resource burden of the system is reduced, and the efficiency of video conversion is effectively improved.

[0177] In a possible implementation, the clipping window corresponding to the target reference frame is determined in the following manner:

[0178] A speaker in the target reference frame is determined;

[0179] If there is a speaker in the target reference frame, a cropping window corresponding to the target reference frame includes a region where a face of the speaker is located.

[0180] If there is no speaker in the target reference frame, a saliency region is determined, the saliency region is used to identify a region in the target reference frame where a probability of attracting a viewer's line of sight is greater than a preset threshold, and a cropping window corresponding to the target reference frame includes the saliency region.

[0181] In this way, audio and video multi-modal information are used as a key basis for determining a picture focus, which can ensure that the picture focus is preferentially aligned to a speaker, and the integrity of a dialogue scene and the clarity of narrative logic are ensured.

[0182] In a possible implementation, the apparatus further includes a constructing unit.

[0183] The constructing unit is configured to, if the accumulated offset of the i+1th video frame in the target video frame sequence reaches the first preset threshold, take a first reference frame after the target video frame sequence as the first reference frame, and construct a first-order polynomial function based on a position in the horizontal video corresponding to a center position of a cropping window of the i th video frame (x i, y i) and a position in the horizontal video corresponding to a center position of a cropping window of the first reference frame (x 0, y 0). The first-order polynomial function is f(x) = a x + b, where a = (x 0-x i) / (x i-x 0), b = y 0-a x 0. The accumulated offset of the i th video frame does not reach the first preset threshold.

[0184] The first-order polynomial function f(x) and a number of video frames between the i th video frame and the first reference frame are used for interpolation to determine the first cropping window of the i+1th video frame.

[0185] In this way, when the accumulated movement distance of a target object in the horizontal direction exceeds a certain threshold, the position of the cropping window can be adjusted so that the target object is located at the center of the cropping window.

[0186] In a possible implementation, the apparatus further includes a resetting unit and a calculating unit.

[0187] The resetting unit is configured to reset the accumulated offset of the i+1th video frame.

[0188] The calculating unit is configured to take the first target object corresponding to the first cropping window as a reference to calculate the accumulated offset of the first target object in each video frame after the i+1th video frame in the target video frame sequence.

[0189] ​​The cropping unit is further configured to, when the accumulated offset of the kth video frame after the i+1th video frame in the target video frame sequence does not reach a first preset threshold, crop the kth video frame based on the position of the first cropping window, and take the cropped picture as a picture in the corresponding video frame of the vertical version video.

[0190] Thus, the distance of the target object accumulated moving in the pixel level of the horizontal version video can be re-accumulated based on the target object corresponding to the adjusted cropping window, the frequent jumping of the cropping window caused by the slight change of the target object in the same shot can be filtered, and the stability of the vertical version video picture is further guaranteed.

[0191] In a possible implementation method, the determination unit is specifically configured to:

[0192] determine the position of the target object in each video frame of the target video frame sequence based on the target object corresponding to the center position of the cropping window in the target reference frame;

[0193] take the distance between the position and the center position in the horizontal direction of the horizontal version video as the offset of the target object;

[0194] accumulate the offset to obtain the accumulated offset of the target in each video frame of the target video frame sequence.

[0195] Thus, the offset of the target in the horizontal direction is calculated through the center position of the cropping window, the target tracking task can be completed under lower resource consumption, and thus the efficiency of video conversion is improved.

[0196] In a possible implementation, the apparatus further includes an extraction unit and a transition unit:

[0197] The extraction unit is configured to extract visual features corresponding to the last video frame of a previous shot segment and the first video frame of a next shot segment, the previous shot segment and the next shot segment being two adjacent shot segments in the plurality of shot segments, and the visual features being used to identify features of a region in a video frame in which the probability of attracting the viewer's line of sight is greater than a preset threshold;

[0198] The transition unit is configured to, if the similarity of the visual features of the last video frame and the visual features of the first video frame is greater than a second preset threshold, increase the transparency of the last video frame, simultaneously display the first video frame, and adjust the cropping window corresponding to the first video frame based on the cropping window corresponding to the last video frame.

[0199] Therefore, by combining the saliency detection algorithm and the color histogram comparison, the similarity of scenes between different shots can be determined, so that smooth transition can be realized between adjacent shots with little scene change through the transition effect, the abrupt jump problem caused by the related art when switching shots is avoided, the transition effect is smooth and natural, and the watchability of the vertical video is improved.

[0200] In a possible implementation, the apparatus further includes an extraction unit.

[0201] The extraction unit is configured to uniformly extract a plurality of video frames from the video frames included in the target shot segment, and take the video frames as the reference frames.

[0202] Therefore, the reference frames are obtained by uniformly extracting a plurality of video frames in the shot segment, without the need to identify specific information of different target objects, so that system resources can be effectively saved and resource burden can be reduced.

[0203] On the basis of the above-described embodiments, an embodiment of the present application provides a computer device, which includes a processor and a memory.

[0204] The memory is configured to store a computer program.

[0205] The processor is configured to execute the above-described video processing method according to the computer program.

[0206] On the basis of the above-described embodiments, an embodiment of the present application provides a computer readable storage medium, which is configured to store a computer program, and the computer program is configured to implement the above-described video processing method when executed by a computer device.

[0207] On the basis of the above-described embodiments, an embodiment of the present application provides a computer program product including a computer program, and the computer program product is configured to enable the computer device to execute the above-described video processing method when the computer program product is run on the computer device.

[0208] It should be noted that the embodiments in the specification adopt a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system or device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0209] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that, The method includes: Acquire multiple shot clips, each shot clip being used to identify a shot in a landscape video, which is a video adapted for landscape playback; For the target shot segment among the multiple shot segments, each reference frame is used as the first frame of the video frame sequence. The video frames contained in the target shot segment are divided into multiple video frame sequences without overlap. The reference frame is a video frame in the target shot segment that is not the last one. For a target video frame sequence, the target object corresponding to the cropping window of the target reference frame is determined, and the cumulative offset of the target object in each video frame of the target video frame sequence is calculated. The target video frame sequence is one of the plurality of video frame sequences, and the target reference frame is the first frame of the target video frame sequence. The cumulative offset is used to identify the distance that the target object has moved cumulatively at the pixel level of the horizontal video starting from the target reference frame. The cropping window refers to the cropping window in the horizontal video used to generate the vertical video. The vertical video is a video adapted for vertical screen playback. When the cumulative offset of the i-th video frame in the target video frame sequence does not reach the first preset threshold, the i-th video frame is cropped based on the position of the cropping window corresponding to the target reference frame, and the cropped image is used as the image in the corresponding video frame of the vertical video.

2. The method according to claim 1, characterized in that, The cropping window corresponding to the target reference frame is determined in the following way: Identify the speaker within the target reference frame; If a speaker is present in the target reference frame, the cropping window corresponding to the target reference frame includes the area where the speaker's face is located. If there is no speaker in the target reference frame, a salient region is determined. The salient region is used to identify the region in the target reference frame that has a greater probability of attracting the viewer's attention than a preset threshold. The cropping window corresponding to the target reference frame includes the salient region.

3. The method according to claim 1, characterized in that, The method further includes: If the cumulative offset corresponding to the (i+1)th video frame in the target video frame sequence reaches the first preset threshold, the first reference frame after the target video frame sequence is taken as the first reference frame, and the center position of the cropping window corresponding to the i-th video frame is taken as the position in the horizontal video. ) and the position of the center of the cropping window corresponding to the first reference frame in the horizontal video ( Construct a first-order polynomial function The cumulative offset of the i-th video frame has not reached the first preset threshold. Based on the aforementioned first-order polynomial function The first cropping window corresponding to the (i+1)th video frame is determined by interpolating the number of video frames between the i-th video frame and the first reference frame.

4. The method according to claim 3, characterized in that, The method further includes: Reset the cumulative offset of the (i+1)th video frame; Based on the first target object corresponding to the first cropping window, calculate the cumulative offset of the first target object in each video frame after the (i+1)th video frame in the target video frame sequence; In the target video frame sequence, when the cumulative offset of the kth video frame after the (i+1)th video frame does not reach the first preset threshold, the kth video frame is cropped based on the position of the first cropping window, and the cropped image is used as the image in the corresponding video frame of the vertical video.

5. The method according to claim 1, characterized in that, The process of determining the target object corresponding to the cropping window of the target reference frame and calculating the cumulative offset of the target object in each video frame of the target video frame sequence includes: Based on the target object corresponding to the center position of the cropping window within the target reference frame, the position of the target object within each video frame of the target video frame sequence is determined; The distance between the position and the center position in the horizontal direction of the landscape video is used as the offset of the target object; The offsets are summed to obtain the cumulative offset of the target in each video frame of the target video frame sequence.

6. The method according to claim 1, characterized in that, The method further includes: Visual features are extracted from the last video frame of the previous shot segment and the first video frame of the next shot segment, respectively. The previous shot segment and the next shot segment are two adjacent shot segments among the multiple shot segments. The visual features are used to identify areas in the video frame that have a greater probability of attracting the viewer's attention than a preset threshold. If the visual features of the last video frame are more similar to the visual features of the first video frame than a second preset threshold, the transparency of the last video frame is increased, the first video frame is displayed, and the cropping window corresponding to the first video frame is adjusted based on the cropping window corresponding to the last video frame.

7. The method according to claim 1, characterized in that, The method further includes: Multiple video frames are uniformly extracted from the video frames contained in the target shot segment, and these video frames are used as the reference frames.

8. A video processing apparatus, characterized in that, The device includes an acquisition unit, a division unit, a determination unit, and a trimming unit: The acquisition unit acquires multiple shot clips, each shot clip being used to identify a shot in a landscape video, which is a video adapted for landscape playback. The division unit is used to divide the video frames contained in the target shot segment into multiple video frame sequences without overlap, with each reference frame as the first frame of the video frame sequence. The reference frame is a video frame in the target shot segment that is not the last one. The determining unit is used to determine the target object corresponding to the cropping window of the target reference frame for the target video frame sequence, and to calculate the cumulative offset of the target object in each video frame of the target video frame sequence. The target video frame sequence is one of the plurality of video frame sequences, the target reference frame is the first frame of the target video frame sequence, and the cumulative offset is used to identify the distance that the target object has moved cumulatively at the pixel level of the horizontal video starting from the target reference frame. The cropping window refers to the cropping window in the horizontal video used to generate the vertical video, and the vertical video is a video adapted for vertical screen playback. The cropping unit is used to crop the i-th video frame based on the position of the cropping window corresponding to the target reference frame when the cumulative offset of the i-th video frame in the target video frame sequence does not reach the first preset threshold, and to use the cropped image as the image in the corresponding video frame of the vertical video.

9. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs; The processor is configured to perform the method according to any one of claims 1-7 according to the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a computer device, performs the method described in any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-7.

Images