Video key point migration method and system

Abstract

The application discloses a video key point migration method and system, relates to the technical field of multimedia data processing, and takes the clip trace file of the source video file clipped into a target video file as the basis for key point migration, breaks through the limitation of video key point migration through video matching, does not need to perform full decoding and feature extraction on the source video file and the target video file, only needs to realize the mapping of the video key point in the target video file according to the clip trace file, greatly reduces the computing power requirement, and even if the visual features of the video change greatly, the accurate migration of the video key point can be realized according to the clip trace file.

Description

Technical Field

[0001] This application relates to the field of multimedia data processing technology, and in particular to a method and system for migrating key points in video. Background Technology

[0002] In long-form video operations, OTT (Over The Top) (Internet TV platforms), and broadcast media asset systems, operators mark numerous key points on the video timeline, such as the start / end of highlights, intro / outro times, tag points, and high-energy moments. However, after a video is uploaded, it is often "replaced" due to content review, image quality optimization, watermark removal, or the addition of pre-roll ads, causing the key points in the original video file to not correspond to those in the replaced video file.

[0003] Currently, matching is generally done using the image or audio fingerprints of two video segments (such as SIFT feature point matching). Since each replacement requires full decoding and feature extraction of both the original and the replacement video, it consumes a lot of computing power. Furthermore, if operations such as image cropping or adding borders are performed during replacement, the visual features of the video will change drastically, making it impossible to accurately identify key points in the original video in the replacement file. Summary of the Invention

[0004] In view of the above problems, this application provides a method and system for video key point migration, the specific solution of which is as follows:

[0005] The first aspect of this application provides a method for video keypoint migration, including:

[0006] In response to a replacement video request, obtain the source video file and the target video file corresponding to the replacement video request;

[0007] Obtain the key points in the source video file;

[0008] Obtain the editing trace file of the source video file edited into the target video file;

[0009] The target location of the key points in the target video file is determined based on the editing trace file;

[0010] Mark the target key points corresponding to the key points at the target location.

[0011] In one possible implementation, determining the target location of the key points in the target video file based on the clipping trace file includes:

[0012] For each of the key points, determine whether the timestamp of the key point is within the time interval of the source video of a clip in the editing trace file;

[0013] If the key point is within the time interval of the source video of the clip, then the target timestamp of the key point in the target video file is determined based on the timestamp of the key point, the time interval of the source video of the clip, and the time interval of the target video.

[0014] If the key point does not include coordinate information, then the target timestamp is determined as the target location.

[0015] In one possible implementation, determining the target location of the key points in the target video file based on the clipping trace file further includes:

[0016] If the key point includes coordinate information, then the target coordinate information of the target key point in the target frame is determined according to the coordinate information of the key point and the spatial transformation matrix parameters of the clip. The target frame is the video frame corresponding to the target timestamp in the target video file. The spatial transformation matrix parameters include at least one of the following: scaling ratio, displacement coordinates, and cropping edge value.

[0017] If the target coordinate information of the key point is within the visible area of ​​the target frame, then the target timestamp and the target coordinate information are determined as the target location;

[0018] If the target coordinates of the key point are not within the visible area of ​​the target frame, then the key point is marked as an invisible key point.

[0019] In one possible implementation, the video keypoint migration method further includes:

[0020] If the key point is not within the time range of the source video of any clip in the editing trace file, the key point will be marked as a discarded key point or a key point pending review.

[0021] In one possible implementation, obtaining the editing trace file of editing the source video file into the target video file includes:

[0022] Based on the association between different versions of video files in the version genealogy database, the migration path between the source video file and the target video file is determined;

[0023] Obtain the editing trace file corresponding to each pair of adjacent video files in the migration path.

[0024] In one possible implementation, determining the target location of the key points in the target video file based on the clipping trace file includes:

[0025] If the migration path includes two or more video files, the key point migration between each pair of adjacent video files shall be performed sequentially according to the migration order of the video files in the migration path.

[0026] For two adjacent video files, during the keypoint migration process, the target position of the keypoint in the previous video file is determined in the next video file based on the corresponding editing trace file.

[0027] In one possible implementation, the video keypoint migration method further includes:

[0028] Acquire the target key point and the N frames corresponding to the key point, where N≥1;

[0029] A lightweight hash comparison is performed on the N frames corresponding to the target key point and the N frames corresponding to the key point to obtain the similarity between the N frames corresponding to the target key point and the N frames corresponding to the key point.

[0030] If the similarity is greater than or equal to the threshold, then the target video file is published;

[0031] If the similarity is less than the threshold, the target key point is marked as a key point to be manually reviewed.

[0032] In one possible implementation, the method for generating the clipping trace file includes:

[0033] Obtain the source video file;

[0034] After editing the source video file into the target video file, each clip on the editing operation timeline is traversed to obtain the editing attributes of each clip.

[0035] The clip trace file is generated in a standard format based on the clip attributes of each clip block.

[0036] In one possible implementation, traversing each clip block on the clip operation timeline includes:

[0037] In the process of editing the source video file into the target video file, if there are multiple camera angles or multiple layers of compositing, the bottom track or the user-specified track is determined as the editing operation timeline;

[0038] Iterate through each of the clip blocks on the clip operation timeline.

[0039] In one possible implementation, generating a standard-format clip trace file based on the clip attributes of each clip block includes:

[0040] For each of the clip blocks, the source video time interval of the clip segment is generated based on the clip in point and clip out point in the clip attributes of the clip block;

[0041] The target video time interval of the clip is generated based on the new segment start point and new segment end point in the clip attributes of the clip block;

[0042] The rate parameter of the clip segment is generated based on the playback rate in the clip attributes of the clip block;

[0043] The spatial transformation matrix parameters of the clip segment are generated based on the motion attributes and cropping effect parameters in the clip properties of the clip block.

[0044] Generate the clip trace file, which includes at least one of the clip segments, according to a standard format data structure.

[0045] A second aspect of this application provides a video keypoint migration system, comprising: a client device and a server device;

[0046] After the client device edits the source video file into the target video file, it generates an editing trace file and uploads the editing trace file and the target video file to the server device.

[0047] In response to a replacement request, the server device obtains the source video file and the target video file corresponding to the replacement request, obtains the key points and editing traces file in the source video file, determines the target position of the key points in the target video file based on the editing traces file, and marks the target key points corresponding to the key points at the target position.

[0048] By employing the above technical solution, this application provides a video keypoint migration method that uses the editing trace file of the source video file to the target video file as the basis for keypoint migration. This method overcomes the limitations of video keypoint migration through video matching, eliminating the need for full decoding and feature extraction of both the source and target video files. It only requires mapping the video keypoints to the target video file based on the editing trace file, significantly reducing computational requirements. Even if the visual features of the video undergo significant changes, accurate migration of video keypoints can still be achieved based on the editing trace file. Attached Figure Description

[0049] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A schematic diagram of a video key point migration system architecture provided in this application embodiment;

[0051] Figure 2 A flowchart illustrating a video key point migration method provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart illustrating a method for generating editing trace files, provided in an embodiment of this application. Detailed Implementation

[0053] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] See Figure 1 , Figure 1 A schematic diagram of a video keypoint migration system architecture is shown. The system may include a client device 100 and a server device 200. The server device 200 can provide the video keypoint migration method provided in this embodiment to one or more terminals.

[0057] The client device 100 can have video editing software installed. After the video editing software edits the source video file into the target video file, the plugin deployed in the video editing software generates an editing trace file and uploads it to the server device 200. This plugin is compatible with various video editing software. Taking Adobe Premiere Pro as an example, the plugin can be developed using the CEP (Common Extensibility Platform) architecture and ExtendScript scripts.

[0058] In response to the replacement request, the server device 200 obtains the source video file and target video file corresponding to the replacement request, obtains the key points and editing trace files in the source video file, determines the target position of the key points in the target video file based on the editing trace files, and marks the target key points corresponding to the key points at the target position.

[0059] It should be understood that in some optional implementations, the client device 100 can also generate an editing trace file by itself after the video editing software edits the source video file into the target video file, and realize the migration of video key points based on the editing trace file, without the need for the cooperation of the server device. This application embodiment is not limited to this.

[0060] The following description Figure 1 The product form of the client device is 100.

[0061] The client device 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0062] Client device 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on client device 100; it may include more or fewer components, or combine certain components, or use different components.

[0063] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the client device 100. Specifically, the input unit may include a touch screen (optional) and / or other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0064] Among them, the input device can receive input data, etc.

[0065] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In this embodiment, the display unit can be used to display the interface of video editing software, editing trace files, etc.

[0066] The memory can be used to store software code related to the video key point migration method, the processor can execute the relevant steps of the video key point migration method, and can also schedule other units (such as the above-mentioned input unit and display unit) to achieve the corresponding functions.

[0067] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0068] In this embodiment of the application, the radio frequency unit can send data to the server device 200.

[0069] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0070] The client device 100 also includes a power source (such as a battery) to power the various components.

[0071] The client device 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the client device 100 to other devices for communication, or to connect a charger to charge the client device 100.

[0072] The server device 200 includes a bus, a processor, a communication interface, and a memory. The processor, memory, and communication interface communicate with each other via the bus.

[0073] The memory can be used to store software code related to the video keypoint migration method, the processor can execute the steps of the video keypoint migration method, and can also schedule other units to achieve the corresponding functions.

[0074] This application provides a method for migrating video key points. The method is described in detail below with reference to the accompanying drawings.

[0075] Reference Figure 2 , Figure 2 This is a flowchart illustrating a video keypoint migration method provided in an embodiment of this application, as shown below. Figure 2 As shown in the figure, the video key point migration method provided in this application embodiment may include steps 201 to 205, which are described in detail below.

[0076] 201: In response to a replacement video request, retrieve the source and target video files corresponding to the replacement video request.

[0077] The replacement request includes a source video file identifier and a target video file identifier. The replacement request indicates that the source video file should be replaced with the target video file.

[0078] The target video file can be directly obtained by editing the source video file. For example, if the source video file is version A and the target video file is version B, editing version A will directly yield version B.

[0079] The target video file can also be indirectly obtained by editing the source video file. For example, if the source video file is version A and the target video file is version C, version A can be edited to obtain version B, and version B can be edited to obtain version C.

[0080] 202: Obtain key points from the source video file.

[0081] By reading the key point list P from the source video file old = {p1, p2, ..., p n} retrieves key points from the source video file. n represents the number of key points, n≥1, meaning the list of key points must include at least one key point.

[0082] 203: Obtain the editing trace file of the source video file edited into the target video file.

[0083] After editing the source video file into the target video file, an editing trace file is obtained. This editing trace file is stored in association with both the source and target video files; specifically, the version association information of the editing trace file includes the source video file identifier and the target video file identifier. Based on this version association information, the corresponding editing trace files for the source and target video files can be retrieved.

[0084] Editing trace files are structured data files that record editing operations and parameters during the video editing process. They can be in JSON format. Their core function is to capture the process logic of video editing, rather than storing the original audio and video data. They are the core data basis for realizing automated processing such as video key point migration and editing trace comparison.

[0085] When the target video file is indirectly obtained by editing the source video file, the number of editing trace files corresponding to the source video file and the target video file is more than one.

[0086] 204: Determine the target location of key points in the target video file based on the editing trace file.

[0087] For each key point, the mapping relationship between the key point's position in the source video file and its target position in the target video file is determined by traversing the editing trace file, thereby determining the target position of the key point in the target video file.

[0088] The target location of key points in the target video file corresponds to the location of key points in the source video file. It can include only time dimension information, or it can include both time dimension information and spatial dimension information.

[0089] It should be noted that if a video frame containing a key point is deleted during the editing of the source video file, then it is not necessary to relocate that key point.

[0090] 205: Mark the target key points corresponding to the key points at the target location.

[0091] In one possible implementation, key points corresponding to key points are marked at the target location. Specifically, the structured data of the key points is written into the metadata of the target video file. The structured data of the key points includes: type, timestamp, and coordinate information. The type includes highlights, advertisements, tags, opening and closing credits, etc., and the coordinate information can be empty.

[0092] This embodiment provides a video keypoint migration method that uses the editing trace file of the source video file to the target video file as the basis for keypoint migration. This method overcomes the limitations of video keypoint migration through video matching. It does not require full decoding and feature extraction of the source and target video files. It only needs to map the video keypoints to the target video file based on the editing trace file. It does not require audio-video comparison and requires very little text processing computing power, which greatly reduces the computing power requirement. Even if the visual features of the video have changed significantly, the video keypoints can still be accurately migrated based on the editing trace file.

[0093] The editing trace files in the above embodiments are generated by a plugin deployed in the video editing software on the client device. The plugin is a self-developed plugin that can be adapted to various different video editing software.

[0094] Please see Figure 3 This application provides a method for generating editing trace files, specifically including the following steps 301-303:

[0095] 301: Get the source video file.

[0096] The plugin pulls source video files from cloud or local storage into the video editing software track. The plugin records the source video file identifier (Source_Video_ID) and initializes the capture task.

[0097] 302: After editing the source video file into the target video file, iterate through each clip on the editing timeline and obtain the editing attributes of each clip.

[0098] Users can use video editing software to edit source video files. Editing operations can include cutting, splicing, speed adjustment, adding black borders, etc.

[0099] When a user clicks the render or submit button in the video editing software, the plugin intercepts the corresponding render or export event, calls the video editing software API to traverse each clip item on the editing operation timeline, and obtains the clip attributes of each clip item.

[0100] A clip is a native concept in video editing software project files. It refers to a single block of footage on the editing timeline, corresponding to a raw, unsplit clip (such as a 10-second video clip). It is the smallest object to be edited. Clip attributes include: operation type, in-point, out-point, start point, end point, speed, motion properties, and cropping parameters.

[0101] The editing operation timeline is the main timeline for editing operations, and all time parameters for editing operations are extracted based on this timeline.

[0102] If the project file of the video editing software contains only a single video track (Video Track 1), or although there are multiple tracks, there is no multi-camera switching or multiple layers of core content overlay (such as only subtitles or simple effects on the top layer), then Video Track 1 will be used directly as the editing operation timeline.

[0103] If there is multi-camera footage (such as footage switching between Camera 1, Camera 2, and Camera 3 in the track), the core of multi-camera editing is "scene switching". However, the overall length of the timeline and the segment division are still based on the footage logic of the underlying track. Other camera angles are only used for scene replacement and do not change the core structure of the timeline (such as the start and end time of the clip and speed parameters). Therefore, the underlying track can be determined as the editing operation timeline.

[0104] In multi-layered compositing scenarios, there are multiple video tracks, each carrying core content (e.g., the bottom layer is the main feature, the middle layer is a picture-in-picture interview, and the top layer is key scene effects). Since the content of the upper-layer track is superimposed on the timeline of the bottom-layer track (e.g., the appearance or disappearance time of the picture-in-picture depends on the time node of the bottom-layer main feature), the bottom-layer track is usually the "time reference" of the entire video. Therefore, the bottom-layer track can be determined as the timeline for editing operations.

[0105] If the user explicitly specifies a track (such as manually selecting "Video Track 3" as the main track in the plugin), the user's instruction will be followed first, and the timeline of the specified track will be used as the editing timeline.

[0106] In the aforementioned complex editing scenarios, the underlying track is prioritized as the editing operation timeline by default, while also allowing for user customization. This ensures the uniqueness of the main timeline, resolves the issue of unclear time references in complex editing scenarios, and provides a stable and unified foundation for subsequent editing trace collection, editing trace file generation, and key point migration, ultimately guaranteeing the accuracy of key point migration.

[0107] 303: Generates a standard format clip trace file based on the clip attributes of each clip block.

[0108] To mask the differences in project files between different video editing software, this embodiment defines a JSON-based Common Editing Description Language (CEDL) format. This format does not store specific audio or video data, but only the operational behavior.

[0109] For example, the CEDL data protocol specification is as follows:

[0110] Root (root node): contains source_ver_id (source video file identifier), target_ver_id (target video file identifier), and canvas_resolution (canvas resolution). canvas_resolution can be empty.

[0111] Timeline: An ordered list of clip segments, each segment containing the following fields:

[0112] index: serial number.

[0113] type: Fragment type, with enumeration values ​​of mapping, insert, and black.

[0114] source_time_range: The time range of the source video [in_point, out_point] (unit: milliseconds).

[0115] target_time_range: The target video time range [start_time, end_time] (unit: milliseconds).

[0116] speed_rate: Rate parameter (1.0 for normal speed playback, 2.0 for multiplied speed playback).

[0117] spatial_transform: Spatial transformation matrix parameters, including scale (scaling ratio), position (displacement coordinates x, y), and crop (crop edge values ​​top, bottom, left, right).

[0118] Specifically, a standard-format clip file is generated based on the clip attributes of each clip block, including the following steps A1-A5:

[0119] A1: For each clip, generate the source video time range of the clip segment based on the clip in point and clip out point in the clip's clip attributes;

[0120] Use the clip in-point (Clip.InPoint) in the clip properties as the in_point in the source video time range (source_time_range) of the clip;

[0121] Use the clip's outpoint (Clip.OutPoint) in the clip properties as the outpoint in the source video time range (source_time_range) of the clip.

[0122] A2: Generate the target video time range of the clip based on the new segment start point and new segment end point in the clip's clip attributes;

[0123] Use the new clip start point (Clip.Start) in the clip properties as the start_time in the target video time range (target_time_range) of the clip;

[0124] Use the start and end points (Clip.End) of the new clip in the clip properties as the end_time in the target video time range (target_time_range) of the clip.

[0125] A3: Generate the rate parameter of the clip segment based on the playback rate in the clip properties of the clip block;

[0126] Use the playback speed (Clip.Speed) in the clip properties as the speed parameter (speed_rate) of the clip.

[0127] A4: Generate the spatial transformation matrix parameters of the clip segment based on the motion attributes and cropping effect parameters in the clip's clip properties;

[0128] Read the scaling and position parameters and the cropping effect parameters from the Motion property in the clip properties, and generate the scale (scaling ratio), position (displacement coordinates x, y), and crop (crop edge values ​​top, bottom, left, right) parameters from the spatial transformation matrix of the clip.

[0129] A5: Generate a clip file containing at least one clip segment, following a standard data structure format.

[0130] The standard format data structure is the same as the data structure specified in the CEDL data protocol. An example of data in a clipping trace file is as follows:

[0131] {

[0132] "source_video_id": "vid_version_A",

[0133] "target_video_id": "vid_version_B", / / Identifier of the pre-generated target video file

[0134] "total_duration":1200.5,

[0135] "edit_segments": [

[0136] {

[0137] "segment_index": 1,

[0138] " source_time_range ": [0, 100], / / Source video file 0-100 seconds

[0139] "target_time_range": [0, 100], / / Maps to 0-100 seconds of the target video file

[0140] "speed_ratio": 1.0,

[0141] "operation": "copy"

[0142] },

[0143] {

[0144] "segment_index": 2,

[0145] "source_time_range": [null, null], / / Corresponding to sourceless video files

[0146] "target_time_range": [100, 115], / / Target video file 100-115 seconds

[0147] "operation": "insert", / / Operation: insert (e.g., an advertisement)

[0148] "metadata": "ad_clip_01.mp4

[0149] },

[0150] {

[0151] "segment_index": 3,

[0152] "source_time_range": [120, 200], / / Source video file 120-200 seconds (100-120 seconds were cut off)

[0153] "target_time_range": [115, 195], / / Maps to 115-195 seconds of the target video file.

[0154] "speed_ratio": 1.0, / / Double speed: 1.0

[0155] "operation": "move"

[0156] } ]

[0158] }

[0159] After the client device generates the editing trace file, it uploads the editing trace file and the target video file to the server device, so that the server device associates and stores the editing trace file with the source video file and the target video file.

[0160] In one possible implementation, the target video file is directly edited from the source video file. One implementation of step 204 in the above embodiments includes the following steps 2041-2043:

[0161] 2041: For each key point, determine whether the timestamp of the key point is within the time interval of the source video of a clip in the editing trace file.

[0162] List of key points P old = {p1, p2, ..., p n A key point p in} i For example, iterate through each clip in the clip file and determine the key point p. i Whether the timestamp falls within the source video time range of a certain clip.

[0163] 2042: If the key point is within the time interval of the source video of the clip, then determine the target timestamp of the key point in the target video file based on the timestamp of the key point, the time interval of the source video of the clip, and the time interval of the target video.

[0164] If in_point≤p i If ≤out_point, then the key point p i The target key point p' in the target video file i The formula for calculating the target timestamp is as follows:

[0165] p' i = start_time + (p i -in_point) / speed_rate;

[0166] Where start_time is the start time within the target video time interval of the clip, (p i -in_point) indicates p i The relative offset within the original segment, where speed_rate is the rate parameter.

[0167] It should be noted that if the key point p i The key point p is not within the time frame of any clip in the clip file. i If it falls within the cut-off interval, then the key point p will be... i Mark as obsolete critical points or critical points pending review. In one possible implementation, p can also be... i Automatically snaps to the nearest clipping point, about to be edited. i Automatically adjust to the nearest valid clipping boundary to the cut interval, ensuring p i It can still be associated with the valid content of the target video file, avoiding p i Completely ineffective.

[0168] Assuming the source video file (version A) is 10 minutes long (0-600 seconds), the editor performed the following operations:

[0169] 100-150 seconds (the clipped section) of the original video file were deleted.

[0170] The source_time_range corresponding to the retained segments is [0-100 seconds] or [150-600 seconds];

[0171] In the target video file (version B), the target_time_range of these two preserved segments is [0-100 seconds] and [100-550 seconds].

[0172] At this point, there are two key editing points at the source video file level: 100 seconds (the end point of the first retained segment) and 150 seconds (the beginning point of the second retained segment).

[0173] If the key point p i = 120 seconds (falling within the 100-150 second interval of the clipped interval), then calculate p. i The distance to the two clipping points, p i The distance to 100 seconds is 20 seconds, p i The distance to 150 seconds is 30 seconds. Therefore, the nearest edit point is 100 seconds (at the source video file level), corresponding to the target_time_range edit point of 100 seconds in the target video file. Finally, p i Automatically snapped to 100 seconds of the target video file and marked as p. i The corresponding target key points.

[0174] 2043: If the key point does not include coordinate information, then the target timestamp is determined as the target location.

[0175] If the key points include coordinate information, such as key points of ROI region, face frame, bullet screen blocking area, etc., the implementation of step 204 in the above embodiment also includes the following steps 2044-2046:

[0176] 2044: Determine the target coordinates of the key points in the target frame based on the coordinates of the key points and the spatial transformation matrix parameters of the clip.

[0177] The target frame is the video frame whose key points are located at the target timestamp in the target video file. The spatial transformation matrix parameters include at least one of the following: scaling ratio, displacement coordinates, and cropping edge values.

[0178] Assume the coordinates of the key points are represented by a rectangle R. old = (x, y, w, h), where the spatial transformation parameters of the corresponding clip are the scaling factor k and the displacement coordinates (Δx, Δy).

[0179] Then the coordinate information R of the key point in the target video file. new The calculation methods for (x', y', w', h') include:

[0180] 1. Scaling calculation: w' = w × k, h' = h × k;

[0181] 2. Displacement calculation: x' = (x × k) + Δx, y' = (y × k) + Δy.

[0182] 2045: If the target coordinates of the key point are within the visible area of ​​the target frame, then the target timestamp and target coordinates are determined as the target location.

[0183] 2046: If the target coordinates of a key point are not within the visible area of ​​the target frame, then the key point will be marked as an invisible key point.

[0184] If the spatial transformation parameters of the clip corresponding to the key points also include crop edge values, it is also necessary to calculate the visible area of ​​the target frame based on the crop edge values, and then calculate the coordinate information R of the target key points. new The intersection of (x', y', w', h') and the visible region determines R. new Whether (x', y', w', h') is within the visible area of ​​the target frame.

[0185] The following is a specific example illustrating the video key point migration method provided in the embodiments of this application.

[0186] For example, a 10-minute source video file has a key point called "High Energy Highlight" at 5:00 (300 seconds), marked with p i The following editing operations were performed by the editor on the client device:

[0187] 1. The first 10 seconds of the opening sequence have been removed.

[0188] 2. A 15-second advertisement was inserted at 2:00 in the original video.

[0189] 3. The segments containing "high-energy highlights" were sped up by 2x.

[0190] Based on two clips in the editing trace file, the temporal mapping relationship between the source video file and the target video file can be determined as follows:

[0191] Clip 1: The source video file [10s-120s] is mapped to the target video file [0s-110s].

[0192] Clip 2: The newly inserted ad is mapped to the target video file [110s-125s].

[0193] Clip 3: The source video file [120s-400s] is mapped to the target video file [125s-265s] (because the source video file [120s-400s] is fast-forwarded at 2x speed, the duration = (400-120) / 2 = 140s, therefore, the target video file is 125+140=265s).

[0194] Key point p i If the source video time interval [120s, 400s] falls within clip 3, then the following parameters are determined based on the above mapping relationship:

[0195] start_time = 125s;

[0196] in_point = 120s;

[0197] speed_rate = 2.0;

[0198] Then p i The corresponding target key point p' i The formula for calculating the target timestamp is as follows:

[0199] p' i = start_time + (p i -in_point) / speed_rate =125 + (300 - 120) / 2.0 = 125 + 90 = 215s.

[0200] In summary, a "high-energy highlight" marker is generated at 3 minutes and 35 seconds (215 seconds) of the target video file without manual intervention.

[0201] In scenarios where the target video file is indirectly obtained by editing the source video file, it is necessary to determine the migration path between the source and target video files based on the relationships between different versions of video files in the version hierarchy database. This version hierarchy database stores the relationships between various versions of the data. This allows us to obtain the editing trace files corresponding to each pair of adjacent video files along the migration path.

[0202] For example: when online videos need to be downloaded from version V A Replace with the latest version V C In the scenario, version A (V) A This is the original online version, containing key locations marked by operations personnel (P). old (timestamp T) A Coordinates R A Version B (V) B This is an intermediate version (may not yet be released), by V. A Edited (e.g., V was cut) A A certain fragment); Version C (V) C This is the final version released, by V. B Edited (e.g., from V) B (If speed adjustment or screen compression has been performed), then the following operations need to be performed:

[0203] 1. Check V C The direct parent node is V. B ;

[0204] 2. Check V B The direct parent node is V. A ;

[0205] 3. Construct V A With V C Migration path between: Path = V A -Trace1-> V B -Trace2-> V C .

[0206] In scenarios where the target video file is indirectly obtained by editing the source video file, the migration of video key points needs to be performed sequentially according to the migration order of the video files in the migration path, and the key point migration between each two adjacent video files needs to be performed in turn. Specifically, for two adjacent video files, during the key point migration process, the target position of the key point of the previous video file in the next video file is determined based on the corresponding editing trace file.

[0207] Furthermore, the aforementioned online videos require a version V. A Replace with the latest version V C Taking this scenario as an example, two key point migrations need to be performed:

[0208] First critical point migration (V) A to V B ), read the clip trace file Trace1 (CEDL) A -> B}), for key point P old The key point P is obtained by performing intermediate state calculation. old Corresponding intermediate key point T B .

[0209] Second keypoint migration (VB to VC), reading the clip trace file Trace2 (CEDL) B ->C}), with the central key point T B As input, the target key point T is calculated. C .

[0210] In one possible implementation, after marking the target key points at the target location according to any of the above embodiments, a closed-loop verification can be performed on the target key points to obtain N frames of images corresponding to the target key points and N frames of images corresponding to the key points, where N≥1. In one example, N=5, which includes the target frame where the target key point is located, the two frames before the target frame, and the two frames after the target frame. Then, a lightweight hash comparison is performed on the N frames of images corresponding to the target key points and the N frames of images corresponding to the key points to obtain the similarity between the N frames of images corresponding to the target key points and the N frames of images corresponding to the key points. For example, a perceptual hash algorithm is used to calculate the similarity between the first hash value and the second hash value of the N frames of images corresponding to the target key points. If the similarity is greater than or equal to a threshold (which can be set to 0.8), it is marked as "high confidence" and the target video file is released; if the similarity is less than the threshold, the similarity is too low, and the target key point is marked as a key point to be manually reviewed, and the operations personnel are notified.

[0211] The closed-loop verification mechanism provided in this embodiment ensures data security in extremely complex special effects compositing scenarios.

[0212] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0213] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0214] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0215] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for migrating key points in a video, characterized in that, include: In response to a replacement video request, obtain the source video file and the target video file corresponding to the replacement video request; Obtain the key points in the source video file; Obtain the editing trace file of the source video file edited into the target video file; The target location of the key points in the target video file is determined based on the editing trace file; Mark the target key points corresponding to the key points at the target location.

2. The video key point migration method according to claim 1, characterized in that, Determining the target location of the key points in the target video file based on the editing trace file includes: For each of the key points, determine whether the timestamp of the key point is within the time interval of the source video of a clip in the editing trace file; If the key point is within the time interval of the source video of the clip, then the target timestamp of the key point in the target video file is determined based on the timestamp of the key point, the time interval of the source video of the clip, and the time interval of the target video. If the key point does not include coordinate information, then the target timestamp is determined as the target location.

3. The video key point migration method according to claim 2, characterized in that, The step of determining the target location of the key points in the target video file based on the editing trace file further includes: If the key point includes coordinate information, then the target coordinate information of the target key point in the target frame is determined according to the coordinate information of the key point and the spatial transformation matrix parameters of the clip. The target frame is the video frame corresponding to the target timestamp in the target video file. The spatial transformation matrix parameters include at least one of the following: scaling ratio, displacement coordinates, and cropping edge value. If the target coordinate information of the key point is within the visible area of ​​the target frame, then the target timestamp and the target coordinate information are determined as the target location; If the target coordinates of the key point are not within the visible area of ​​the target frame, then the key point is marked as an invisible key point.

4. The video key point migration method according to claim 2, characterized in that, The video key point migration method also includes: If the key point is not within the time range of the source video of any clip in the editing trace file, the key point will be marked as a discarded key point or a key point pending review.

5. The video keypoint migration method according to claim 1, characterized in that, The step of obtaining the editing trace file of editing the source video file into the target video file includes: Based on the association between different versions of video files in the version genealogy database, the migration path between the source video file and the target video file is determined; Obtain the editing trace file corresponding to each pair of adjacent video files in the migration path.

6. The video key point migration method according to claim 5, characterized in that, Determining the target location of the key points in the target video file based on the editing trace file includes: If the migration path includes two or more video files, the key point migration between each pair of adjacent video files shall be performed sequentially according to the migration order of the video files in the migration path. For two adjacent video files, during the keypoint migration process, the target position of the keypoint in the previous video file is determined in the next video file based on the corresponding editing trace file.

7. The video keypoint migration method according to any one of claims 1-6, characterized in that, The video key point migration method also includes: Acquire the target key point and the N frames corresponding to the key point, where N≥1; A lightweight hash comparison is performed on the N frames corresponding to the target key point and the N frames corresponding to the key point to obtain the similarity between the N frames corresponding to the target key point and the N frames corresponding to the key point. If the similarity is greater than or equal to the threshold, then the target video file is published; If the similarity is less than the threshold, the target key point is marked as a key point to be manually reviewed.

8. The video keypoint migration method according to claim 1, characterized in that, The method for generating the clipping trace file includes: Obtain the source video file; After editing the source video file into the target video file, each clip on the editing operation timeline is traversed to obtain the editing attributes of each clip. The clip trace file is generated in a standard format based on the clip attributes of each clip block.

9. The video keypoint migration method according to claim 8, characterized in that, The traversal of each clip block on the timeline of the clipping operation includes: In the process of editing the source video file into the target video file, if there are multiple camera angles or multiple layers of compositing, the bottom track or the user-specified track is determined as the editing operation timeline; Iterate through each of the clip blocks on the clip operation timeline.

10. The video keypoint migration method according to claim 8, characterized in that, The step of generating the clip trace file in a standard format based on the clip attributes of each clip block includes: For each of the clip blocks, the source video time interval of the clip segment is generated based on the clip in point and clip out point in the clip attributes of the clip block; The target video time interval of the clip is generated based on the new segment start point and new segment end point in the clip attributes of the clip block; The rate parameter of the clip segment is generated based on the playback rate in the clip attributes of the clip block; The spatial transformation matrix parameters of the clip segment are generated based on the motion attributes and cropping effect parameters in the clip properties of the clip block. Generate the clip trace file, which includes at least one of the clip segments, according to a standard format data structure.

11. A video keypoint migration system, characterized in that, include: Client equipment and server equipment; After the client device edits the source video file into the target video file, it generates an editing trace file and uploads the editing trace file and the target video file to the server device. In response to a replacement request, the server device obtains the source video file and the target video file corresponding to the replacement request, obtains the key points and editing traces file in the source video file, determines the target position of the key points in the target video file based on the editing traces file, and marks the target key points corresponding to the key points at the target position.

Images