Video cropping method, apparatus and electronic device

Abstract

This application provides a video cropping method, apparatus, and electronic device, relating to the field of video processing technology. The method includes: acquiring a video frame sequence of an original video; performing subject-aware analysis on the video frame sequence to determine at least one target to be tracked, and acquiring spatiotemporal continuity information of the at least one target to be tracked in the video frame sequence; determining corresponding cropping regions for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information; and cropping the video frame sequence according to the cropping regions to generate a cropped video conforming to the target cropping aspect ratio. Thus, this method can automatically generate high-quality cropped videos without continuous human intervention.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video cropping method, apparatus, and electronic device. Background Technology

[0002] With the widespread adoption of mobile internet and multiple terminal devices, video content often needs to be adapted to different platforms and display specifications (such as 9:16 portrait screen, 1:1 square aspect ratio, etc.) to meet the needs of mobile viewing and social sharing. Therefore, intelligent video cropping technology has emerged.

[0003] However, the relevant video cropping solutions all have significant limitations in practical applications. Specifically, cropping methods based on fixed regions and sizes, by completely ignoring video content, easily result in the main subject (such as key figures) being cropped out of the frame, causing content loss. Cropping schemes based on image features (such as saliency detection), while effective for single frames, lack continuous tracking of the target's cross-frame motion, resulting in unstable cropping boxes in the temporal dimension, leading to jitter and jumps in the output video. While manual cropping can ensure quality, it relies on professional operators, is inefficient, and difficult to scale up.

[0004] Therefore, how to automatically generate high-quality cropped videos is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a video cropping method, apparatus, and electronic device that can automatically generate high-quality cropped videos without continuous human intervention.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a video cropping method, the method comprising: Obtain the video frame sequence of the original video; Subject-aware analysis is performed on the video frame sequence to identify at least one target to be tracked, and the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence is obtained. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames. Based on the target cropping aspect ratio and the spatiotemporal continuity information, a corresponding cropping region is determined for each video frame in the video frame sequence; Based on the cropping region, the video frame sequence is cropped to generate a cropped video that conforms to the target cropping aspect ratio.

[0007] Optionally, subject-aware analysis is performed on the video frame sequence to determine at least one target to be tracked, and spatiotemporal continuity information of at least one target to be tracked in the video frame sequence is obtained, including: At least one target object is detected from the video frame sequence using a target detection model; The detected target object is tracked across frames to obtain the position sequence of the target object in consecutive video frames, forming a motion trajectory. The target object with a continuous motion trajectory is identified as the target to be tracked, and the motion trajectory of the target to be tracked is used as the spatiotemporal continuity information.

[0008] Optionally, the spatiotemporal continuity information includes the motion trajectory of at least one of the targets to be tracked; Based on the target cropping aspect ratio and the spatiotemporal continuity information, the corresponding cropping region is determined for each video frame in the video frame sequence, including: Based on the motion trajectory, determine the first position of the target to be tracked in the current video frame; Based on the historical position information of the motion trajectory, the first position is smoothed to obtain the smoothed second position; Using the second position as a reference and in conjunction with the target cropping aspect ratio, the cropping area of ​​the current video frame is determined.

[0009] Optionally, taking the second position as a reference and combining it with the target cropping aspect ratio, the cropping region of the current video frame is determined, including: The size of the cropping region is dynamically determined based on the size information of the target to be tracked in the video frame sequence; The boundary coordinates of the cutting area are determined based on the second position and the size of the cutting area; If the boundary coordinates indicate that the cropped area exceeds the boundary of the current video frame, the boundary coordinates are adjusted so that the cropped area is within the boundary of the current video frame.

[0010] Optionally, the method further includes: Image segmentation and / or optical character recognition methods are used to detect and analyze interfering visual elements in the video frame sequence to obtain content restoration information. The content restoration information includes at least location information for indicating the area to be restored in the video frame sequence. The interfering visual elements include at least one of watermarks, graphic stickers, and native subtitles. The video frame sequence is cropped according to the cropping region, including: The video frame sequence is cropped based on the cropped area and the content restoration information.

[0011] Optionally, interfering visual elements in the video frame sequence are detected and analyzed to obtain content restoration information, including: Identify the regions where interfering visual elements are located in the video frame sequence, and determine the location information of the regions where the interfering visual elements are located as the location information of the region to be repaired in the content repair information.

[0012] Optionally, if the interfering visual element is a watermark or graphic sticker, the video frame sequence is cropped based on the cropping area and the content restoration information, including: Based on the location information in the content restoration information, an image restoration model is used to generate and fill pixel-level content in the area where the watermark or graphic sticker is located, replacing the watermark or graphic sticker, to obtain the restored video frame. Extract the image content defined by the cropped area from the repaired video frames.

[0013] Optionally, when the interfering visual element is a native subtitle, performing additional content analysis on the video frame sequence to obtain content repair information further includes: performing text recognition on the region of the native subtitle to obtain the subtitle text content; Based on the cropping region and the content restoration information, the video frame sequence is cropped, including: Extract the image content defined by the cropped region from the video frame sequence; Based on the subtitle text content and the target cropping aspect ratio, determine the layout parameters of the new subtitle; The new subtitles are re-rendered on the screen content according to the layout parameters.

[0014] Optionally, the target cropping aspect ratio includes multiple preset aspect ratios; Generating a cropped video that conforms to the target aspect ratio includes: For each of the multiple preset aspect ratios, a corresponding cropping area is determined and processed to simultaneously generate multiple cropped videos with different aspect ratios.

[0015] Optionally, performing subject-aware analysis on the video frame sequence to determine at least one target to be tracked further includes: From at least one of the targets to be tracked, a primary target is selected according to a preset rule; The spatiotemporal continuity information refers to the motion trajectory information of the main tracking target.

[0016] A second aspect of this application discloses a video cropping apparatus, the apparatus comprising: The video decoding module is used to obtain the video frame sequence of the original video. The subject perception module is used to perform subject perception analysis on the video frame sequence, determine at least one target to be tracked, and obtain the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames. The cropping decision module is used to determine the corresponding cropping region for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information. The video compositing module is used to perform cropping processing on the video frame sequence according to the cropping area to generate a cropped video that conforms to the target cropping aspect ratio.

[0017] A third aspect of this application discloses an electronic device, including a memory, a processor, and a program or instructions stored in the memory and executable on the processor, wherein the processor executes the program or instructions to implement the steps of the video cropping method described in the first aspect of this application.

[0018] A fourth aspect of this application discloses a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implements the steps of the video cropping method described in the first aspect of this application.

[0019] The embodiments of this application have the following advantages: In this embodiment of the application, by performing subject-aware analysis on the video frame sequence and obtaining the spatiotemporal continuity information of the target to be tracked, the core moving subject in the video can be automatically and continuously tracked. This makes the determination of the cropping area based on a complete spatiotemporal understanding of the subject's motion trajectory, rather than the analysis of isolated frames or fixed rules, fundamentally avoiding the problem of subject loss caused by cropping and ensuring the complete preservation of key content.

[0020] Furthermore, based on the spatiotemporal continuity information representing positional changes, the cropping region for each frame is dynamically determined, allowing the cropping region to smoothly follow the subject's movement. This overcomes the inter-frame jumps and visual jitters caused by fixed cropping or single-frame feature-based cropping methods, thus generating cropped videos with natural transitions in the subject's perspective and a stable and smooth viewing experience.

[0021] In this way, the method integrates subject tracking perception and intelligent dynamic cropping into a coherent automated process. Users only need to specify the target cropping aspect ratio to automatically generate cropped videos that meet the requirements, reducing reliance on professional editing skills and manual operation. At the same time, the method can flexibly adapt to various preset aspect ratio requirements, meet the specifications of different publishing platforms and terminal devices, and achieve batch and intelligent video content adaptation processing. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the steps of a video cropping method provided in an embodiment of this application; Figure 2 This is a flowchart of another video cropping method provided in an embodiment of this application; Figure 3 This is an overall flowchart of a video cropping method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a video cropping device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The video cropping method provided in this application can be applied to the field of multi-platform intelligent distribution and adaptation of video content. With the rapid development of the mobile internet, video content needs to be presented on different platforms (such as short video platforms, social media, and video websites) and different terminal devices (such as mobile phones, tablets, and desktop monitors). These platforms and devices often have different recommended aspect ratios. For example, traditional film and television content is mostly 16:9 landscape, while short video platforms prefer 9:16 portrait, and social media commonly uses a 1:1 square aspect ratio. In this application scenario, if it is desired to efficiently adapt the same high-quality video material (such as film and television clips or promotional videos) to multiple channels, the relevant technologies rely on professional editors to manually crop, scale, and adjust it. This process is time-consuming and labor-intensive, and it is difficult to guarantee the continuity of subject tracking and the consistency of image quality during batch processing. This application aims to address this practical production need by integrating subject tracking perception, intelligent cropping, and content restoration into a coherent automated process, which can automatically generate high-quality cropped videos without continuous human intervention.

[0026] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a video cropping method provided in an embodiment of this application. Figure 1 As shown, a video cropping method provided in this application embodiment may include steps S110 to S140: Step S110: Obtain the video frame sequence of the original video.

[0027] In this embodiment, the received original video (e.g., a segment) After extracting video clips (at a specific resolution), video decoding techniques (such as calling multimedia processing libraries like FFmpeg or OpenCV) are used to extract each frame of the original video and convert it into static images. This process parses the video into a series of static images arranged in chronological order, i.e., a video frame sequence. For example, the original video can be decoded into RGB images at 30 frames per second.

[0028] In some embodiments, metadata information of the original video is also extracted, such as frame rate (fps), resolution (width and height), audio, total number of frames and encoding format, etc.; and the metadata information is associated with the video frame sequence to provide basic parameters for subsequent processing.

[0029] Step S120: Perform subject perception analysis on the video frame sequence to determine at least one target to be tracked, and obtain the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames.

[0030] In this embodiment, a deep learning-based object detection model (such as YOLO, SSD, etc.) can be used to perform subject perception analysis on the video frame sequence to identify one or more important visual targets (such as people, vehicles, animals, etc.) in the original video. These targets are defined as "targets to be tracked". For example, in a dialogue scene, two people may be identified; in a racing scene, multiple race cars may be identified.

[0031] Target tracking algorithms (such as DeepSORT and ByteTrack) can be used to perform cross-frame correlation tracking of the target, continuously tracking its position (usually represented by bounding box coordinates) in consecutive video frames. By integrating the position data within a time window, the position sequence of each target in consecutive frames is obtained, forming a motion trajectory. This motion trajectory constitutes the spatiotemporal continuity information, which includes the instantaneous position of the target in each frame and describes its trajectory, velocity, and direction as its position changes over time, achieving a coherent understanding of the target's motion state.

[0032] Step S130: Determine the corresponding cropping region for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information.

[0033] In this embodiment, the target cropping aspect ratio can be user-defined or predefined. For example, a target cropping aspect ratio of 9:16 is used for vertical short videos, and 1:1 is used for social media. Combining the spatiotemporal continuity information (i.e., the motion trajectory of the target to be tracked) obtained in step S120, an optimal cropping window (i.e., cropping area) is calculated for each video frame to provide accurate positional information for intelligent cropping, ensuring that the target to be tracked is always located in a suitable position in the frame (such as centered or at the golden ratio point) in the cropped video.

[0034] Specifically, for each frame in the video frame sequence, a cropping center point is predicted or calculated based on the position of the target to be tracked in the current frame and neighboring historical frames (derived from spatiotemporal continuity information). Subsequently, using this center point as a reference, a cropping region is calculated according to the target's aspect ratio. The size of the cropping region can be adaptively adjusted according to the size of the target to be tracked.

[0035] Step S140: Based on the cropping area, crop the video frame sequence to generate a cropped video that conforms to the target cropping aspect ratio.

[0036] In this embodiment, for each video frame in the video frame sequence, the image content defined by the cropping region corresponding to that frame, determined in step S130, is extracted from the original video frame. Then, the extracted content is scaled or placed onto a canvas that conforms to the target cropping aspect ratio (if the size of the extracted region is inconsistent with the target canvas size, scaling is performed). Finally, all processed video frames are re-encoded according to the original frame rate and mixed with the audio stream of the original video to ultimately generate a complete cropped video that conforms to the target cropping aspect ratio.

[0037] The technical solution adopted in this application realizes full-process automation from video input to cropped video output, eliminating the need for manual frame-by-frame selection or adjustment and improving processing efficiency. The dynamic cropping strategy based on spatiotemporal continuity information ensures that the tracked subject remains at the visual focus throughout the cropped video, and the transitions in cropping perspective are smooth and natural, avoiding subject loss and image jumps. Furthermore, this method can generate cropped videos based on any set target aspect ratio, possessing multi-platform and multi-terminal adaptability, meeting diverse content distribution needs. By integrating subject perception analysis and dynamic intelligent cropping into a coherent automated process, it solves multiple challenges faced in high-quality video cropping in one go, providing an end-to-end solution.

[0038] In an optional embodiment, step S120 above, "performing subject-aware analysis on the video frame sequence, determining at least one target to be tracked, and obtaining spatiotemporal continuity information of at least one target to be tracked in the video frame sequence," may include steps S120-1 to S120-3: Step S120-1: Detect at least one target object from the video frame sequence using the target detection model.

[0039] This step is the initial stage of subject-aware analysis, used to automatically identify potential target objects from each video frame. Specifically, pre-trained or scene-optimized object detection models (such as YOLO, SSD, etc.) can be used to analyze the video frame sequence frame by frame. This model is trained on a dataset containing common video subject categories (such as "people," "vehicles," and "animals") and can identify objects belonging to preset categories in video frames. It outputs a category label and a precise bounding box coordinate for each identified object, defining the object's position and extent within the frame. These detected objects are the target objects.

[0040] There can be multiple target objects. For example, when processing a street interview video, the object detection model may detect multiple target objects in a single frame, such as "reporter," "interviewee," and "pedestrian" in the distance. This step achieves preliminary semantic understanding of the video content, transforming the image from a collection of pixels into a structured, labeled collection of objects, providing the basic input for subsequent continuity analysis.

[0041] Step S120-2: Perform cross-frame correlation tracking on the detected target object to obtain the position sequence of the target object in consecutive video frames and form a motion trajectory.

[0042] This step is used to correlate discrete, frame-independent detection results into a continuous, stable motion trajectory. Multi-target tracking algorithms (such as DeepSORT, ByteTrack, etc., which combine appearance features and motion models) can be used to perform cross-frame correlation tracking on the target objects identified frame by frame in step S120-1.

[0043] Specifically, the process first determines which target object in the current video frame belongs to the same physical entity as any target object in the previous video frame (e.g., by calculating the overlap of bounding boxes, similarity of appearance features, etc.). If they belong to the same target, the association is successful. Then, a motion trajectory is created and updated for each successfully associated target object, recording its historical positions (i.e., position sequences) in consecutive video frames, forming the motion trajectory of that target object. Through this cross-frame association, different target objects can be distinguished, and their motion can be continuously tracked.

[0044] Step S120-3: The target object with a continuous motion trajectory is identified as the target to be tracked, and the motion trajectory of the target to be tracked is used as the spatiotemporal continuity information.

[0045] In this step, from all the target objects tracked in step S120-2, those target objects with continuous motion trajectories are selected, that is, target objects that have been successfully tracked and whose trajectory length exceeds a preset threshold (e.g., appearing continuously for more than 5 frames), and these are identified as targets to be tracked. Target objects that have not been successfully tracked or whose trajectories are discontinuous (such as interfering objects that appear briefly) are filtered out.

[0046] Subsequently, the motion trajectory of the target to be tracked is used as spatiotemporal continuity information to guide subsequent video cropping. This spatiotemporal continuity information includes the instantaneous position of the target in each frame, depicting its trajectory, velocity, and direction as its position changes over time, thus achieving a coherent understanding of the target's motion state. If multiple target objects with continuous motion trajectories exist, they can all be used as targets to be tracked, or one can be selected as the primary tracking target based on preset rules (such as largest area, closest to the center of the frame, etc.).

[0047] The technical solution adopted in this application decomposes subject perception analysis into three hierarchical and collaborative sub-steps: target detection, cross-frame correlation tracking, and target determination. Target detection provides accurate semantic segmentation and localization for each frame, cross-frame tracking strings these instantaneous snapshots into a coherent motion narrative, and the target determination step filters based on trajectory continuity to ensure that only truly stable objects are included in subsequent processing. This makes the spatiotemporal continuity information a continuous trajectory with a clear semantic identity based on accurate target recognition, providing a reliable basis for subsequent intelligent cropping centered on a specific subject, and avoiding cropping logic confusion caused by misjudging transient interference objects as subjects. Furthermore, the detection-then-tracking strategy can effectively handle complex situations such as temporary target occlusion, entering and exiting the frame, and appearance changes. Even when facing complex scenes common in real-world videos, it can still maintain stable tracking performance, improving the practicality and reliability of the method.

[0048] In an optional embodiment, step S120 above, "performing subject-aware analysis on the video frame sequence to determine at least one target to be tracked," further includes: Step S120-4: Select a primary tracking target from at least one of the targets to be tracked according to a preset rule; wherein the spatiotemporal continuity information is the motion trajectory information of the primary tracking target.

[0049] In this embodiment, if there are multiple targets to be tracked, the target most likely to be the main character of the video or the target of greatest interest to the user is selected as the primary tracking target, so that the cropping is performed around the primary tracking target. Specifically, a set of configurable preset rules is used to evaluate and score all the targets to be tracked determined in step S120-3, and finally one of them is selected as the primary tracking target.

[0050] The preset rules can be viewed as a multi-factor comprehensive evaluation model. These rules can be determined by comprehensively considering one or more factors, including spatial saliency, positional centrality, and temporal persistence. Spatial saliency refers to calculating the area occupied by the target in the frame; a larger area generally indicates greater prominence and a higher likelihood of it being a subject. Positional centrality refers to calculating the distance between the center point of the target and the center point of the video frame; a closer distance suggests the target may be more important in the composition. Temporal persistence refers to counting the number of consecutive frames or duration for which the target has been successfully tracked; a longer duration indicates it is likely a stable subject in the scene rather than a transient disturbance.

[0051] Finally, by weighting and combining the scores of one or more of the above factors, a subject importance score is calculated for each target to be tracked, and the target with the highest score is selected as the primary tracking target. After selection, the motion trajectory information corresponding to the primary tracking target is used as spatiotemporal continuity information representing the core narrative clues of the entire video, and is used in subsequent steps to calculate the cropping region of each video frame.

[0052] The technical solution adopted in this application solves the problem of focus selection in cropping decisions in multi-target scenarios, ensuring that the output cropped video has a consistent and clear narrative focus, and avoiding the visual confusion caused by the cropping frame hesitating or frequently switching between different targets. Furthermore, after selecting the primary target for tracking, the entire cropping process will unfold around its stable motion trajectory, ensuring that the final video has a highly consistent perspective and coherent visual flow in the time dimension, improving the professional quality and viewing experience of the cropped video. In addition, this method selects the primary tracking target based on preset rules, making the end-to-end automation of the entire intelligent cropping process more advanced and universally applicable, especially suitable for batch processing scenarios of massive amounts of video.

[0053] In an optional embodiment, the spatiotemporal continuity information includes the motion trajectory of at least one of the targets to be tracked; the step S130 above, "determining the corresponding cropping region for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information," may include steps S130-1 to S130-3: Step S130-1: Determine the first position of the target to be tracked in the current video frame based on the motion trajectory.

[0054] In this embodiment, the spatiotemporal continuity information is specifically manifested as the motion trajectory of the target to be tracked (or the main target to be tracked). This motion trajectory is the historical location history of the target in a series of consecutive video frames, which is usually a sequence ordered by timestamps. Each element contains the target's location information in the corresponding video frame (e.g., represented by the coordinates of the center point of the bounding box).

[0055] When determining the corresponding cropping region for each video frame in the video frame sequence, when processing the current video frame, the instantaneous position of the target to be tracked in that frame is queried from the motion trajectory or directly received, i.e., the first position, which is the original position obtained by single-frame detection.

[0056] Step S130-2: Based on the historical position information of the motion trajectory, smooth the first position to obtain the smoothed second position.

[0057] In this embodiment, considering that directly using the first position as the cropping center may cause the cropping area to fluctuate slightly due to detection noise, resulting in image jitter, a smoothed second position is output based on the historical position information of the motion trajectory to filter out high-frequency noise and unnecessary slight jitter in the position data. This second position eliminates the jitter caused by detection noise and better reflects the true motion intention of the target.

[0058] The historical position information of the motion trajectory can be a sequence of the positions of the target to be tracked in several frames (e.g., 15 frames) before the current video frame. Specifically, based on the historical position information of the motion trajectory, a smoothing algorithm is applied to the first position to obtain the smoothed second position.

[0059] In some embodiments, a Savitzky-Golay filter can be used to smooth the data by performing local multinomial regression within a moving window, effectively filtering out high-frequency noise while preserving motion trends. In other embodiments, a Kalman filter can be used, which can combine the motion model to provide an optimal estimate of observation noise.

[0060] Step S130-3: Using the second position as a reference and in conjunction with the target cropping aspect ratio, determine the cropping area of ​​the current video frame.

[0061] The second position is used as a reference, with the smoothed second position serving as the center point of the ideal cropping area. Based on this center point and the user-specified or preset target cropping aspect ratio, the size of the cropping area is calculated. Then, based on the center point and the size of the cropping area, the boundary coordinates of the cropping area (usually the coordinates of the top left and bottom right corners) can be obtained through geometric calculations.

[0062] Understandably, the size can be determined using a fixed value strategy, such as setting a fixed height or width; or it can be dynamically adjusted according to the size of the target to be tracked, such as making the clipping box height a fixed multiple of the target bounding box height.

[0063] The technical solution of this application, by introducing smoothing processing based on historical trajectories, eliminates the instability caused by directly using the detection position, ensures the continuity of the cropping area movement, and thus solves the problems of screen jumping and shaking, so that the final generated video has a professional and stable camera movement effect.

[0064] In an optional embodiment, step S130-3 above, "using the second position as a reference and combining the target cropping aspect ratio, to determine the cropping area of ​​the current video frame," may include steps S130-3-1 to S130-3-3: Step S130-3-1: Dynamically determine the size of the cropping region based on the size information of the target to be tracked in the video frame sequence.

[0065] The size information can be historical data of the bounding box of the target to be tracked (or the main target to be tracked), such as the width and height of the bounding box.

[0066] Specifically, the height of the cropping region can be determined by a preset scaling factor based on the average size (such as height) of the target in the current video frame or multiple adjacent consecutive frames; then, given the aspect ratio of the target cropping region, the corresponding cropping region width can be calculated.

[0067] In this way, the size of the cropping area can adaptively match the visual proportion of the target being tracked, ensuring that the main body is of a relatively appropriate size and the composition is reasonable in the output image. This avoids the problem of excessive white space or over-cropping of the main body that may be caused by fixed-size cropping, and improves the visual quality of the cropping result.

[0068] Step S130-3-2: Determine the boundary coordinates of the cutting area based on the second position and the size of the cutting area.

[0069] The boundary coordinates of the cropping area can include the coordinates of the top-left corner and the bottom-right corner of the cropping area. The second position is used as the center point of the cropping area. Based on the dimensions of the cropping area (height H and width W), the coordinates of the top-left and bottom-right corners are derived through geometric calculations. The top-left corner coordinates can be expressed as: The coordinates of the lower right corner can be represented as: .

[0070] Thus, by defining the boundary coordinates of the cropping region, an ideal cropping window is defined that can be centered at a smooth position and meet the target aspect ratio and adaptive size requirements.

[0071] Step S130-3-3: If the boundary coordinates indicate that the cropping area exceeds the boundary of the current video frame, adjust the boundary coordinates so that the cropping area is within the boundary of the current video frame.

[0072] In this embodiment, considering that the target to be tracked may move to the edge of the screen, the calculated boundary coordinates may cause part of the cropped area to exceed the visible range of the original video frame (i.e., go beyond the boundary), resulting in invalid black or filled areas and ruining the viewing experience. Therefore, boundary checks and boundary coordinate adjustments are required.

[0073] Specifically, the boundary coordinates are compared with the boundary range of the current video frame. If the left side of the cropped area exceeds the boundary, the entire cropped area is shifted to the right; if the right side exceeds the boundary, the entire cropped area is shifted to the left; if the top side exceeds the boundary, the entire cropped area is shifted downwards; and if the bottom side exceeds the boundary, the entire cropped area is shifted upwards. The principle of these directional adjustments is to minimize the deviation from the ideal center point while ensuring that the cropped area remains completely within the frame boundary. For example, priority is given to ensuring that the cropped area includes the target. After this adjustment step, it is ensured that the final cropped area is completely within the pixel matrix of the original video frame.

[0074] The technical solution adopted in this application ensures the proportional coordination between the cropped image and the subject through adaptive size determination; it transforms smooth tracking into specific spatial regions through coordinate calculation, and boundary constraint processing guarantees the integrity of the output video content. Thus, it solves the problems of inflexible fixed size and out-of-bounds cropping of edge targets encountered in dynamic cropping. This allows the entire intelligent cropping method to not only work stably in the center of the image but also to perform reasonable processing when the target moves to the edge of the image, enhancing the robustness, practicality, and stability of the output quality.

[0075] In an optional embodiment, the method further includes: Step A1: Using image segmentation and / or optical character recognition methods, detect and analyze interfering visual elements in the video frame sequence to obtain content repair information. The content repair information includes at least the location information for indicating the area to be repaired in the video frame sequence. The interfering visual elements include at least one of watermarks, graphic stickers, and native subtitles.

[0076] In this embodiment, visually interfering elements refer to non-main content in the image that may affect the viewing experience or requires adaptation; among them, watermarks are usually located in the corner of the image and refer to semi-transparent or low-transparency logos, platform identifiers, etc. Graphic stickers refer to static or dynamic image elements added to the image to enhance the effect, such as emoticons, brand icons, etc.

[0077] Specifically, image segmentation methods (such as deep learning-based segmentation models) can be used to perform pixel-level detection of watermarks and graphic stickers, or optical character recognition (OCR) methods can be used to detect text regions in native subtitles. These techniques are used to detect the presence of visually interfering elements in the video frame sequence. If such elements are detected, the pixel coordinates (i.e., location information) of the element's location are recorded, and this information is integrated into content restoration information. In some implementations, the location information of the area to be restored can be represented in the form of a mask. This information provides a clear operational target for subsequent image restoration or content reconstruction.

[0078] Furthermore, the step S140 above, "to perform cropping processing on the video frame sequence according to the cropping area", specifically includes: cropping the video frame sequence according to the cropping area and the content restoration information.

[0079] In this embodiment, the following operations can be performed on each video frame in the video frame sequence: First, based on the content restoration information obtained in step A1, image restoration processing is performed on the identified area to be restored (such as a watermark) (e.g., using an image restoration algorithm based on a generative adversarial network or a diffusion model) to generate an intermediate frame. Then, from this intermediate frame (or the original video frame, if no restoration is required), the target screen content is extracted according to the cropping area corresponding to the frame determined in step S130. Next, the extracted content is scaled or placed on a canvas that conforms to the target cropping aspect ratio. If the content restoration information includes subtitle text, the subtitles may also be re-rendered (e.g., adjusting the font, size, and position) according to the size and layout of the new screen in this step. Finally, all processed video frames are re-encoded according to the original frame rate and mixed with the audio stream of the original video to finally generate a complete cropped video that conforms to the target cropping aspect ratio.

[0080] By incorporating the detection and analysis of interfering visual elements and processing based on content restoration information into the cropping process, the technical solution of this application achieves intelligent identification and automated processing of various interfering elements in videos. This solves the problems of traditional cropping methods being unable to handle overlapping elements such as watermarks and stickers, and the truncation or loss of subtitles during aspect ratio conversion. It further improves the professional quality and applicability of the output video and expands the practical scenarios of the method.

[0081] In an optional embodiment, step A1 above, "detecting and analyzing interfering visual elements in the video frame sequence to obtain content repair information", may include: identifying the region where the interfering visual elements are located in the video frame sequence, and determining the location information of the region where the interfering visual elements are located as the location information of the region to be repaired in the content repair information.

[0082] Specifically, for watermarks and graphic stickers, since they typically have specific visual patterns (such as fixed position, color, shape, or transparency), pixel-level semantic segmentation can be performed using deep learning-based image segmentation models (such as the U-Net network or its variants). After being trained on a large number of images containing / not containing such elements, this model can predict and output binary masks identifying these interfering elements frame by frame. For native subtitles, optical character recognition engines are typically used for processing. These engines can not only detect areas in the image where text may exist (especially the bottom area), but also preliminarily identify the text content, thereby separating the subtitle area from the background.

[0083] For each identified visually disruptive element, its precise location information is further output through its corresponding detection algorithm. For example, for watermarks and stickers, the image segmentation model outputs a segmentation mask or polygonal contour that precisely defines the pixel area occupied by the watermark or sticker in the image; for captions, the optical character recognition engine outputs the coordinates of one or more text boxes (e.g., the coordinates of the top left and bottom right corners). This location information precisely indicates the areas in the image that require special attention and need to be repaired.

[0084] Subsequently, this location information is stored and organized as part of the content repair information. Specifically, the location information of the area where the interfering visual element is located is determined as the location information of the area to be repaired in the content repair information. The content repair information may exist in the form of a coordinate list, a mask image sequence, or metadata, explicitly indicating which areas of the image need to be repaired (e.g., removed, repaired, or rearranged).

[0085] The technical solution adopted in this application replaces manual identification and annotation, and can automatically locate various types of interfering visual elements such as watermarks, stickers, and subtitles. It integrates the precise location information into content repair information, providing accurate operational basis for subsequent repair processing, improving the automation level and processing speed of the processing flow, improving the pertinence, efficiency and processing quality of subsequent repair steps, and reducing unnecessary computational overhead and the risk of accidental damage to normal screen content.

[0086] In an optional embodiment, when the interfering visual element is a watermark or graphic sticker, cropping the video frame sequence based on the cropping area and the content restoration information may include steps B1 to B2: Step B1: Based on the location information in the content restoration information, use an image restoration model to generate and fill pixel-level content in the area where the watermark or the graphic sticker is located, so as to replace the watermark or the graphic sticker and obtain the restored video frame.

[0087] For two types of visually disruptive elements, watermarks and graphic stickers, the location information in the content restoration information can be the precise coordinates or mask of the watermark or graphic sticker area.

[0088] Specifically, using location information as a guide for the restoration operation, an image restoration model (e.g., the DeepFillv2 model based on generative adversarial networks, or the LaMa model based on diffusion) is invoked to restore the image of the area containing the watermark or graphic sticker. This image restoration model can learn and infer the original reasonable texture, color, and structure of the occluded or interfered content based on the intact pixel information (context) surrounding the area to be restored, thereby performing pixel-level content generation and filling, replacing the original watermark or graphic sticker with newly generated pixels that are visually natural and coherent with the surrounding background. After processing, a restored video frame is obtained, in which the original watermark or sticker has been replaced by newly generated content that blends naturally with the surrounding background.

[0089] Step B2: Extract the image content defined by the cropped area from the repaired video frame.

[0090] After obtaining the repaired video frame (i.e., a clean image with distracting visual elements removed), the cropping region (i.e., the coordinates of the smart cropping box) determined in step S130 is applied to the repaired video frame to extract the corresponding image content. This image content is the core image content after the specified visual distracting elements (watermarks / stickers) have been removed and the image has been intelligently composed around the subject. Subsequently, this content is scaled or adapted to the target canvas size for the final output, resulting in the cropped video.

[0091] The technical solution adopted in the application embodiment uses an image restoration model to generate and fill pixel-level content in the area where the watermark or graphic sticker is located. This achieves automated and intelligent removal of visual interference and generates fill content that is highly coordinated with the original background, improving the professionalism and visual purity of the cropped video. This method seamlessly integrates the restoration and cropping operations, completing the restoration on a complete frame before performing the cropping. This avoids the problem of insufficient contextual information that may occur when cropping first and then restoring, thus ensuring the quality of the restoration. Furthermore, by locating and replacing interfering elements such as watermarks and stickers, the focus of the cropped video is more concentrated on the main content itself, improving the viewing experience.

[0092] In an optional embodiment, when the interfering visual element is a native subtitle, step A1 above, "performing additional content analysis on the video frame sequence to obtain content repair information," may further include steps A1-3: Step A1-3: Perform text recognition on the region of the original subtitle to obtain the subtitle text content.

[0093] For visually disruptive elements such as native subtitles, the region of the native subtitle is identified in step S140-1. In this step, text recognition is performed on the region of the native subtitle to understand the specific information it carries (i.e., the subtitle text content).

[0094] Specifically, optical character recognition (OCR) technology can be used to perform text recognition on the area of ​​the original subtitles to detect the position of each character within that area, and then convert the segmented character images into text symbols to obtain the subtitle text content. In some embodiments, in addition to the subtitle text content, metadata such as timestamps and font styles are also output. This achieves the transformation from a visual area to editable text information, laying the data foundation for subsequent intelligent reflow.

[0095] Further, the video frame sequence is cropped based on the cropped area and the content restoration information, which may include steps B3 to B5: Step B3: Extract the image content defined by the cropping region from the video frame sequence.

[0096] Specifically, based on the cropping area determined for the current video frame in step S130, the image content centered on the target subject is extracted from the original video frame (or after other repair processing in steps B1 to B2). The image content obtained in this step is a clean main scene without subtitles, and its size and shape are determined by the aspect ratio of the target cropping.

[0097] Step B4: Determine the layout parameters of the new subtitle based on the subtitle text content and the target cropping aspect ratio.

[0098] To present the subtitle text in a clearer and more suitable manner for the new screen size, the layout parameters for the new subtitles are determined based on the target crop aspect ratio and the length of the subtitle text. These layout parameters include font and style (font type, size, color, weight), position (the coordinates of the subtitle within the image, typically located at the bottom and potentially adjusted based on the image content), layout (maximum number of characters per line, alignment, line spacing), and background (whether to add a semi-transparent background to enhance readability).

[0099] In this way, this step can adapt to different canvas sizes, ensuring that the subtitles are clear, readable, and aesthetically pleasing in various aspect ratios without obscuring key visual content.

[0100] Step B5: Re-render the new subtitles on the screen content according to the layout parameters.

[0101] Specifically, the layout parameters determine the rendering position of the new subtitle, and the graphics rendering library is used to draw the new subtitle text at that rendering position with the specified font, size, color and other styles.

[0102] Understandably, if a background is set, a semi-transparent rectangle will be drawn first, and the new subtitle text will be drawn within this rectangle. Furthermore, anti-aliasing and other techniques can be considered during the rendering process to improve visual quality. Ultimately, a video frame integrating the intelligently cropped main image and optimized subtitle layout is generated. On this video frame, the new subtitles are clearly readable, and their position is not lost or incomplete due to cropping of the original image.

[0103] The technical solution adopted in this application, by recognizing and then rearranging the text, ensures that key text information is completely and accurately preserved after format conversion, thereby improving the information integrity of the video content and solving the problem of subtitles being lost or incomplete due to cropping. Furthermore, the new subtitles can automatically adapt to any target aspect ratio, ensuring they are in the optimal readable position on any size screen, improving the professionalism and visual appeal of video cropping, and avoiding problems such as messy fonts, improper placement, or insufficient contrast with the background that may exist in the original subtitles.

[0104] In an optional embodiment, the target cropping aspect ratio includes multiple preset aspect ratios; the step S140 above, "generating a cropped video that conforms to the target cropping aspect ratio", may include: for each of the multiple preset aspect ratios, determining the corresponding cropping area and processing it to simultaneously generate multiple cropped videos with different aspect ratios.

[0105] In this embodiment, the target cropping aspect ratio is a list (set) containing multiple preset aspect ratios, such as ["9:16", "1:1", "16:9"]. These preset aspect ratios represent different video specifications that need to be output simultaneously (such as vertical short videos, square social media videos, traditional landscape videos, etc.).

[0106] For the same subject location and the same frame, different target aspect ratios will lead to completely different strategies for centering, size, and boundary processing of the optimal cropping frame. For example, in a 9:16 portrait screen, vertical cropping may be necessary to keep the subject centered, while in a 1:1 square screen, horizontal cropping may be necessary. Therefore, for each preset aspect ratio, the corresponding cropping region is determined for each video frame in the video frame sequence through the above step S130; then, in step S150, based on the content restoration information generated in step S140 and the cropping regions calculated for each preset aspect ratio, the video frame sequence is processed in parallel to generate cropped videos that conform to multiple different aspect ratios.

[0107] The technical solution adopted in this application embodiment can simultaneously generate multiple cropped videos with different aspect ratios, avoiding the computational overhead and time cost of processing each aspect ratio individually, making the batch production of multi-specification videos efficient and feasible. Furthermore, cropped videos with different aspect ratios all originate from the same set of subject tracking and content analysis results, ensuring consistency across multiple versions in terms of core subject focus, motion coherence, and handling of interfering elements. For example, the main character is stably tracked in all versions, and the same watermark is corrected in all versions, avoiding deviations that may occur during separate processing.

[0108] like Figure 2 As shown, Figure 2 This is a flowchart of another video cropping method provided in this application embodiment. Specifically, the method includes the following steps S210 to S250: Step S210: Obtain the video frame sequence of the original video.

[0109] Step S220: Perform subject perception analysis on the video frame sequence to determine at least one target to be tracked, and obtain the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames.

[0110] Step S230: Determine the corresponding cropping region for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information.

[0111] Step S240: Using image segmentation and / or optical character recognition methods, detect and analyze interfering visual elements in the video frame sequence to obtain content repair information. The content repair information includes at least the location information for indicating the area to be repaired in the video frame sequence. The interfering visual elements include at least one of watermarks, graphic stickers, and native subtitles.

[0112] Step S250: Perform cropping processing on the video frame sequence based on the cropping area and the content restoration information.

[0113] Through the above implementation process, the entire process from video input to cropped video output is automated, eliminating the need for manual frame-by-frame selection or adjustment and improving processing efficiency. Furthermore, the dynamic cropping strategy based on spatiotemporal continuity information ensures that the tracked subject remains in the visual focus throughout the cropped video, and the transitions in cropping perspective are smooth and natural, effectively avoiding subject loss and image jumps. By detecting and analyzing interfering visual elements in the video frame sequence, it can intelligently identify and process interfering elements such as watermarks and subtitles, improving the visual purity and professionalism of the output video. In addition, this method can generate cropped videos based on any set target aspect ratio, possessing strong multi-platform and multi-terminal adaptability, meeting diverse content distribution needs.

[0114] To better understand the video cropping method of this application, a specific example will be used to illustrate the implementation process. The original video is a 1920... A 1080p (i.e., 16:9 aspect ratio) landscape video clip needs to be cropped to a 1080p output suitable for short video platforms. A 1920 (i.e., 9:16 aspect ratio) video version. Specifically, the processing steps are as follows: Step 1: Decode the original video into a sequence of video frames, for example, decode the original video into an RGB image at 30 frames per second.

[0115] Step 2: Perform subject perception analysis on the video frame sequence. The detection module identifies the target to be tracked, "Person A", and starts tracking. It continuously outputs the bounding box of Person A in each frame to obtain the corresponding motion trajectory.

[0116] Step 4: Use a smoothing algorithm to smooth the path points (first position) in the motion trajectory of character A. Using the smoothed path point (second position) as the center, calculate a cropping region with an aspect ratio of 9:16. For example, in a certain frame, the first position is (950, 500), and the smoothed second position is (945, 505). Then, a cropping region of approximately 607 pixels is generated centered at (945, 505). 1080 cropping area.

[0117] Step 5: Detection of visual interference elements: Using a watermark detection model, a semi-transparent logo of a video platform was found in the upper right corner. A mask was generated for this logo, which is the area to be repaired. Using an image inpainting model, a similar sky texture was generated based on the sky and clouds surrounding the logo to cover this area. OCR was used to identify the Chinese subtitle "I believe you can do it" at the bottom, located at (100, 1000).

[0118] Step 6: Compositing and Output: Crop the 607 calculated in step 4 from the current video frame. In the 1080p area, overlay this area onto the same frame background where the top-right logo has been fixed; on the black border at the bottom of the canvas (coordinates (0, 1800)), re-render the text "I believe you can do it" in white font. Scale this processed image to 1080p. The canvas of 1920 is used to repeat this process for each frame, encoding it into a target format (such as MP4).

[0119] In this way, a vertical video is obtained in which the main character moves naturally in the center of the screen, the watermark in the upper right corner disappears, and the subtitles are clearly located at the bottom black border, resulting in a professional and natural overall viewing experience.

[0120] The following description uses a video cropping system (hereinafter referred to as the system) as the execution entity to illustrate the video cropping method provided in this application. Figure 3 As shown, Figure 3 This is an overall flowchart of a video cropping method provided in an embodiment of this application. Specifically, the system includes: a video decoding module, a subject perception analysis module, an intelligent cropping region calculation module, an additional content processing module, and an image compositing and encoding output module. The method includes the following steps: First, the video frame sequence of the original video is processed by the video decoding module.

[0121] Secondly, the video frame sequence is subjected to subject perception analysis through a subject perception analysis module. Specifically, keyframes are detected, and at least one target object is detected from the video frame sequence using a target detection model; cross-frame correlation tracking is performed on the detected target object to obtain the position sequence of the target object in consecutive video frames, forming a motion trajectory; the target object with a continuous motion trajectory is identified as the target to be tracked, and the motion trajectory of the target to be tracked is used as the spatiotemporal continuity information.

[0122] Then, the intelligent cropping region calculation module determines the corresponding cropping region for each video frame in the video frame sequence. Specifically, the first position of the target to be tracked in the current video frame is determined based on the motion trajectory, and the first position is smoothed based on the historical position information of the motion trajectory to obtain a smoothed second position; the size of the cropping region is dynamically determined based on the size information of the target to be tracked in the video frame sequence; the boundary coordinates of the cropping region are determined based on the second position and the size of the cropping region; if the cropping region exceeds the boundary of the current video frame, the boundary coordinates are adjusted so that the cropping region is within the boundary of the current video frame, thus obtaining the final cropping region.

[0123] Next, the additional content processing module detects and analyzes interfering visual elements in the video frame sequence and performs video repair. Specifically, interfering visual elements in the video frame sequence, such as watermarks, graphic stickers, and native subtitles, are identified. The location information of the areas where the interfering visual elements are located is determined as the location information of the area to be repaired in the content repair information. Based on the location information in the content repair information, an image repair model is used to generate and fill pixel-level content in the areas where the watermarks or graphic stickers are located to replace the watermarks or graphic stickers, resulting in the repaired video frames. For native subtitles, text recognition is performed on the areas of the native subtitles to obtain the subtitle text content, and the layout parameters of the new subtitles are determined based on the subtitle text content and the target cropping aspect ratio.

[0124] Finally, through the image compositing and encoding output module, the image content defined by the cropping area is extracted from the repaired video frame according to the cropping area corresponding to each frame. The extracted content is scaled or placed on a canvas that conforms to the target cropping aspect ratio. All processed video frames are re-encoded according to the original frame rate and mixed with the audio stream of the original video to finally generate a complete cropped video that conforms to the target cropping aspect ratio.

[0125] Through the above implementation process, subject tracking and perception, intelligent cropping, and content restoration are integrated into a coherent automated workflow. Users only need to specify the target aspect ratio for cropping to automatically generate cropped videos that meet the requirements, reducing reliance on professional editing skills and manual operation. At the same time, this method can flexibly adapt to various preset aspect ratio requirements, efficiently meeting the specifications of different publishing platforms and terminal devices, and achieving batch and intelligent video content adaptation processing.

[0126] This application also provides a video cropping device, referring to... Figure 4 As shown, Figure 4 This is a schematic diagram of a video cropping device provided in an embodiment of this application. The device includes: Video decoding module 410 is used to acquire the video frame sequence of the original video; The subject perception module 420 is used to perform subject perception analysis on the video frame sequence, determine at least one target to be tracked, and obtain the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames. The cropping decision module 430 is used to determine the corresponding cropping region for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information. The video compositing module 440 is used to perform cropping processing on the video frame sequence according to the cropping area to generate a cropped video that conforms to the target cropping aspect ratio.

[0127] In an optional embodiment, the subject sensing module 420 is further configured to: At least one target object is detected from the video frame sequence using a target detection model; The detected target object is tracked across frames to obtain the position sequence of the target object in consecutive video frames, forming a motion trajectory. The target object with a continuous motion trajectory is identified as the target to be tracked, and the motion trajectory of the target to be tracked is used as the spatiotemporal continuity information. In an optional embodiment, the spatiotemporal continuity information includes the motion trajectory of at least one of the tracked targets; the clipping decision module 430 is further configured to: Based on the motion trajectory, determine the first position of the target to be tracked in the current video frame; Based on the historical position information of the motion trajectory, the first position is smoothed to obtain the smoothed second position; Using the second position as a reference and in conjunction with the target cropping aspect ratio, the cropping area of ​​the current video frame is determined.

[0128] In an optional embodiment, the cropping decision module 430 is further configured to: The size of the cropping region is dynamically determined based on the size information of the target to be tracked in the video frame sequence; The boundary coordinates of the cutting area are determined based on the second position and the size of the cutting area; If the boundary coordinates indicate that the cropped area exceeds the boundary of the current video frame, the boundary coordinates are adjusted so that the cropped area is within the boundary of the current video frame.

[0129] In an optional embodiment, the device further includes: The content analysis module 450 is used to detect and analyze interfering visual elements in the video frame sequence using image segmentation methods and / or optical character recognition methods, and to obtain content repair information. The content repair information includes at least location information for indicating the area to be repaired in the video frame sequence. The interfering visual elements include at least one of watermarks, graphic stickers, and native subtitles. In an optional embodiment, the content analysis module 450 is further configured to: Identify the regions where interfering visual elements are located in the video frame sequence, and determine the location information of the regions where the interfering visual elements are located as the location information of the region to be repaired in the content repair information.

[0130] In an optional embodiment, when the interfering visual element is a watermark or a graphic sticker, the video synthesis module 440 is further configured to: Based on the location information in the content restoration information, an image restoration model is used to generate and fill pixel-level content in the area where the watermark or the graphic sticker is located, in order to replace the watermark or the graphic sticker, and obtain the restored video frame. Extract the image content defined by the cropped area from the repaired video frames.

[0131] In an optional embodiment, when the interfering visual element is a native subtitle, the content analysis module 450 is further configured to: perform text recognition on the region of the native subtitle to obtain the subtitle text content; The video synthesis module 440 is also used for: Extract the image content defined by the cropped region from the video frame sequence; Based on the subtitle text content and the target cropping aspect ratio, determine the layout parameters of the new subtitle; The new subtitles are re-rendered on the screen content according to the layout parameters.

[0132] In one optional embodiment, the target cropping aspect ratio includes multiple preset aspect ratios; the video compositing module 440 is further configured to: determine and process the corresponding cropping area for each of the multiple preset aspect ratios to simultaneously generate multiple cropped videos with different aspect ratios.

[0133] In an optional embodiment, the subject perception module 420 is further configured to: select a primary tracking target from at least one of the targets to be tracked according to a preset rule; wherein the spatiotemporal continuity information is the motion trajectory information of the primary tracking target.

[0134] This application also provides an electronic device, see embodiments thereof. Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 500 includes a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus for communication. The memory 510 stores a computer program that can run on the processor 520 to implement the steps of the video cropping method described in the embodiments of this application.

[0135] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the video cropping method described in this application.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0137] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0141] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0142] The above provides a detailed description of a video cropping method, apparatus, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of video cropping, characterized by, include: Obtain the video frame sequence of the original video; Subject-aware analysis is performed on the video frame sequence to identify at least one target to be tracked, and the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence is obtained. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames. Based on the target cropping aspect ratio and the spatiotemporal continuity information, a corresponding cropping region is determined for each video frame in the video frame sequence; Based on the cropping region, the video frame sequence is cropped to generate a cropped video that conforms to the target cropping aspect ratio.

2. The method of claim 1, wherein, Performing subject-aware analysis on the video frame sequence to identify at least one target to be tracked, and obtaining spatiotemporal continuity information of at least one target to be tracked in the video frame sequence, including: At least one target object is detected from the video frame sequence using a target detection model; The detected target object is tracked across frames to obtain the position sequence of the target object in consecutive video frames, forming a motion trajectory. The target object with a continuous motion trajectory is identified as the target to be tracked, and the motion trajectory of the target to be tracked is used as the spatiotemporal continuity information.

3. The method of claim 1, wherein, The spatiotemporal continuity information includes the motion trajectory of at least one of the targets to be tracked; Based on the target cropping aspect ratio and the spatiotemporal continuity information, the corresponding cropping region is determined for each video frame in the video frame sequence, including: Based on the motion trajectory, determine the first position of the target to be tracked in the current video frame; Based on the historical position information of the motion trajectory, the first position is smoothed to obtain the smoothed second position; Using the second position as a reference and in conjunction with the target cropping aspect ratio, the cropping area of ​​the current video frame is determined.

4. The method of claim 3, wherein, Using the second position as a reference and in conjunction with the target cropping aspect ratio, the cropping region of the current video frame is determined, including: The size of the cropping region is dynamically determined based on the size information of the target to be tracked in the video frame sequence; The boundary coordinates of the cutting area are determined based on the second position and the size of the cutting area; If the boundary coordinates indicate that the cropped area exceeds the boundary of the current video frame, the boundary coordinates are adjusted so that the cropped area is within the boundary of the current video frame.

5. The method of claim 1, wherein, The method further includes: Image segmentation and / or optical character recognition methods are used to detect and analyze interfering visual elements in the video frame sequence to obtain content restoration information. The content restoration information includes at least location information for indicating the area to be restored in the video frame sequence. The interfering visual elements include at least one of watermarks, graphic stickers, and native subtitles. The video frame sequence is cropped according to the cropping region, including: The video frame sequence is cropped based on the cropped area and the content restoration information.

6. The method of claim 5, wherein, The video frame sequence is subjected to interference visual elements for detection and analysis to obtain content restoration information, including: Identify the regions where interfering visual elements are located in the video frame sequence, and determine the location information of the regions where the interfering visual elements are located as the location information of the region to be repaired in the content repair information.

7. The method according to claim 6, characterized in that, When the interfering visual element is a watermark or graphic sticker, the video frame sequence is cropped according to the cropping area and the content restoration information, including: Based on the location information in the content restoration information, an image restoration model is used to generate and fill pixel-level content in the area where the watermark or the graphic sticker is located, in order to replace the watermark or the graphic sticker, and obtain the restored video frame. Extract the image content defined by the cropped area from the repaired video frames.

8. The method according to claim 6, characterized in that, In the case where the interfering visual element is a native subtitle, additional content analysis is performed on the video frame sequence to obtain content repair information. This also includes: performing text recognition on the region of the native subtitle to obtain the subtitle text content. Based on the cropping region and the content restoration information, the video frame sequence is cropped, including: Extract the image content defined by the cropped region from the video frame sequence; Based on the subtitle text content and the target cropping aspect ratio, determine the layout parameters of the new subtitle; The new subtitles are re-rendered on the screen content according to the layout parameters.

9. The method according to any one of claims 1-8, characterized in that, The target cropping aspect ratio includes multiple preset aspect ratios; Generating a cropped video that conforms to the target aspect ratio includes: For each of the multiple preset aspect ratios, a corresponding cropping area is determined and processed to simultaneously generate multiple cropped videos with different aspect ratios.

10. The method according to any one of claims 1-8, characterized in that, Performing subject-aware analysis on the video frame sequence to determine at least one target to be tracked further includes: From at least one of the targets to be tracked, a primary target is selected according to a preset rule; The spatiotemporal continuity information refers to the motion trajectory information of the main tracking target.

11. A video cropping device, characterized in that, include: The video decoding module is used to obtain the video frame sequence of the original video. The subject perception module is used to perform subject perception analysis on the video frame sequence, determine at least one target to be tracked, and obtain the spatiotemporal continuity information of at least one target to be tracked in the video frame sequence. The spatiotemporal continuity information is used to characterize the positional change relationship of the target to be tracked in consecutive video frames. The cropping decision module is used to determine the corresponding cropping region for each video frame in the video frame sequence based on the target cropping aspect ratio and the spatiotemporal continuity information. The video compositing module is used to perform cropping processing on the video frame sequence according to the cropping area to generate a cropped video that conforms to the target cropping aspect ratio.

12. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the video cropping method as described in any one of claims 1-10.

Images