Multi-dimensional video content intelligent editing method and device, equipment and medium

Abstract

The invention relates to a multi-dimensional video content intelligent editing method and device, equipment and a medium. The method comprises the following steps: extracting multi-dimensional video features including visual features, audio features, semantic features and motion trail features according to a multi-camera video stream; fusing the multi-dimensional video features to obtain a fused feature vector; classifying the video frames based on the fusion feature vector to generate a scene label sequence; reasoning and generating an editing decision sequence according to the scene label sequence through a decision tree reasoning engine; positioning a lens switching point according to the editing decision sequence to obtain an editing time point set; and splicing the multi-camera video stream based on the set to generate a target edited video. By adopting the method, the problems of one-sided editing decision and poor scene adaptability in the prior art can be solved, the content integrity and the scene adaptability of multi-camera video editing are improved, and the high-quality presentation requirement is met.

Description

Technical Field

[0001] This invention belongs to the field of video editing technology, and in particular relates to a method, apparatus, device and medium for intelligent editing of multi-dimensional video content. Background Technology

[0002] With the rapid development of technology in video production and distribution, multi-camera shooting technology has been widely used in scenarios such as live conference broadcasting, sports event broadcasting, and wedding recording. This technology, by deploying multiple cameras to simultaneously capture video streams, can capture scene content from different perspectives. Compared to single-camera shooting, it provides richer visual material, meeting users' needs for "multi-angle, full-scene" video content presentation. At the same time, video editing technology is gradually evolving towards automation and intelligence. Traditional manual editing, due to its low efficiency and reliance on experience, is no longer suitable for handling the "large amount of material and diverse perspectives" of multi-camera video streams. Therefore, the industry's demand for intelligent multi-camera video editing methods is increasingly urgent, and various intelligent editing technologies based on single features or simple rules are emerging.

[0003] In existing multi-camera video intelligent editing technologies, the mainstream processing methods mainly revolve around single-dimensional features: First, some technologies make editing decisions based on visual features, that is, by extracting visual information such as image clarity, color saturation, and edge sharpness of video frames, they select shots with better visual effects for splicing. For example, in sports event editing, they prioritize camera shots with clear images and prominent subjects. Second, a few technologies introduce audio features to assist in decision-making, by detecting parameters such as the volume, tone, and noise intensity of audio signals, to retain segments with higher audio quality. For example, in conference editing, they select shots with clear speaking voices and no background noise. Third, some technologies use fixed rules for editing, such as mechanically splicing multi-camera video streams according to preset rules such as "switching camera positions every 5 seconds" and "prioritizing main camera shots," without the need for complex feature analysis and decision-making reasoning processes.

[0004] However, existing multi-camera video intelligent editing technologies still have significant shortcomings, failing to meet the demands for high-quality and highly adaptable editing. On the one hand, existing technologies generally rely on single-dimensional features, such as visual or audio features alone, ignoring the semantic information of the video content (such as the meaning expressed by the image and the theme of the scene) and motion trajectory features (such as the camera's zoom and pan trajectories and the path of object movement). This results in a lack of comprehensiveness in editing decisions. For example, in conference editing, relying solely on visual clarity may lead to the accidental deletion of "keyframes of the speaker's gestures" (which, although the image clarity is slightly lower, contain important semantic information), or the splicing of disjointed segments with "violent camera shake" may occur due to the failure to consider camera motion trajectories. On the other hand, editing technologies based on fixed rules have extremely poor flexibility and cannot adapt to the editing needs of different scenarios. For example, the rule of "switching cameras every 5 seconds" may result in "frequent interruptions to close-up shots of the newlyweds" when used for wedding editing, while in sports editing, it may lead to missing "key goal moments" due to untimely switching. These problems collectively result in videos edited by existing technologies having "insufficient content integrity and poor scene adaptability," making it difficult to meet users' demands for high-quality multi-camera video presentation. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, equipment, and medium for intelligent editing of coherent and accurate multi-dimensional video content that can automatically generate the above-mentioned technical problems.

[0006] Firstly, this application provides a multi-dimensional intelligent video content editing method, including:

[0007] Multi-dimensional video features are extracted from multi-camera video streams to obtain multi-dimensional video features, including visual features, audio features, semantic features, and motion trajectory features.

[0008] Feature fusion processing is performed based on multi-dimensional video features to obtain a fused feature vector;

[0009] Based on the fused feature vector, each frame of video is classified into content and scene, generating a scene label sequence;

[0010] Based on the scene label sequence, the editing strategy is inferred through the decision tree inference engine to generate the editing decision sequence;

[0011] Based on the editing decision sequence, the camera switching points of the multi-camera video stream are located, and a set of editing time points is generated.

[0012] Based on the set of editing time points, the multi-camera video streams are spliced ​​together to generate the target edited video.

[0013] In one embodiment, based on the fused feature vector, content scene classification is performed on each frame of video to generate a scene label sequence, including:

[0014] Key feature points are extracted from the fused feature vector of each frame of video to obtain a set of key feature points;

[0015] Based on the set of key feature points, the feature point matching relationship between adjacent frames is calculated to obtain an inter-frame feature association map;

[0016] Based on the inter-frame feature association map and combined with the preset scene classification model, the content scene is classified for each video frame, and a scene tag sequence is generated.

[0017] In one embodiment, based on the scene label sequence, a decision tree inference engine is used to perform editing strategy inference to generate an editing decision sequence, including:

[0018] Key editing events are identified from the scene tag sequence to obtain a key event index set;

[0019] Based on the key event index set, the scene label sequence is sliced ​​using a context window to obtain a set of label fragments with temporal context;

[0020] The decision tree reasoning engine performs rule matching on a set of labeled fragments with temporal context to obtain a preliminary editing decision sequence.

[0021] The initial editing decision sequence is checked for temporal continuity to obtain the time-corrected editing decision sequence;

[0022] Perform a logical consistency check on the time-corrected editing decision sequence to generate the editing decision sequence.

[0023] In one embodiment, a decision tree inference engine is used to perform rule matching on a set of labeled fragments with temporal context to obtain a preliminary editing decision sequence, including:

[0024] Based on a predefined editing rule library, the node judgment logic of the decision tree inference engine is configured to generate a rule matching decision tree;

[0025] By traversing the set of label fragments with time context through a rule-matching decision tree, the original decision result for each label fragment is obtained;

[0026] The original decision results are verified for temporal continuity to obtain the verified decision results;

[0027] The conflict resolution process is applied to the verified decision results to generate a preliminary editing decision sequence.

[0028] In one embodiment, based on the editing decision sequence, the camera switching points of the multi-camera video stream are located to generate a set of editing time points, including:

[0029] Extract timestamps from the editing decision sequence to obtain a set of decision time points;

[0030] Based on the decision time point set, frame-level matching processing is performed on the multi-camera video stream to generate a candidate switching frame set.

[0031] Boundary point localization is performed based on the candidate switching frame set to obtain the editing time point set.

[0032] In one embodiment, frame-level matching processing is performed on the multi-camera video stream based on the decision time point set to generate a candidate switching frame set, including:

[0033] The set of decision time points is subjected to time window truncation processing to obtain a set of frame segments to be matched;

[0034] Based on the set of frame segments to be matched, the similarity of the inter-frame visual features of the multi-camera video stream is calculated to generate a frame similarity matrix;

[0035]

[0036] in, Let be the multimodal fusion similarity between the i-th frame of host video A and the j-th frame of slave video B. Let H(·) be the frame-level semantic feature vector, H(·) be the three-dimensional histogram feature extraction function in the HSV color space, and M(·) be the motion feature extraction function based on optical flow field. , , For adaptive weighting coefficients, The scaling parameter for color features;

[0037] A set of candidate switching frames is generated by filtering local extrema using the frame similarity matrix.

[0038] In one embodiment, based on a set of editing time points, the multi-camera video stream is spliced ​​to generate a target edited video, including:

[0039] The set of edited time points is sorted along the timeline to obtain an ordered sequence of edited time points;

[0040] Based on the ordered editing timeline, the video stream from multiple cameras is processed to extract shot segments and generate a set of shot segments to be spliced.

[0041] Based on the set of shot segments to be spliced, a shot transition effect is added to generate a set of transitioned shot segments.

[0042] The target edited video is generated by sequentially splicing together a collection of transitional shot clips.

[0043] Secondly, this application also provides a multi-dimensional video content intelligent editing device, including:

[0044] The multi-dimensional feature extraction module is used to extract multi-dimensional features from multi-camera video streams to obtain multi-dimensional video features, including visual features, audio features, semantic features, and motion trajectory features.

[0045] The feature fusion processing module is used to perform feature fusion processing based on multi-dimensional video features to obtain a fused feature vector.

[0046] The content scene classification module is used to classify the content scene of each video frame based on the fused feature vector and generate a scene label sequence.

[0047] The editing strategy reasoning module is used to perform editing strategy reasoning based on scene label sequences and generate editing decision sequences through a decision tree reasoning engine.

[0048] The shot switching point positioning module is used to locate shot switching points in multi-camera video streams based on the editing decision sequence and generate a set of editing time points.

[0049] The shot stitching module is used to stitch together multiple camera video streams based on a set of editing time points to generate the target edited video.

[0050] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the multi-dimensional video content intelligent editing method as described in the first aspect.

[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-dimensional video content intelligent editing method as described in the first aspect.

[0052] The aforementioned multi-dimensional intelligent video content editing methods, devices, equipment, and media extract visual, audio, semantic, and motion trajectory features from multi-camera video streams, covering video images, sound, content meaning, and dynamic information, avoiding the limitation of single features reflecting only a part of the content. The fusion of multi-dimensional features yields a fused feature vector, integrating scattered information. Based on the fused vector, scene label sequences are generated for classification, enabling accurate understanding of video content and clarifying the scene type of each frame. An editing decision sequence is generated through a decision tree inference engine, transforming scene understanding into specific editing strategies, replacing fixed rules. The editing time points are located according to the decision sequence, accurately locking the shot switching positions and avoiding arbitrary switching. The target video is generated by splicing shots based on time points, ensuring the coherent integration of multi-camera materials to form a complete finished product. Existing technologies suffer from problems such as reliance on single features for editing decisions and poor scene adaptability. In this method, multi-dimensional feature extraction and fusion solve the problem of one-sided decision-making based on single features, ensuring that no key information is missed; the decision tree inference engine replaces fixed rules, solving the problem of poor scene adaptability, and can generate adaptation strategies according to different scenes; precise switching point positioning and coherent shot splicing further guarantee the editing quality, improve the integrity of multi-camera video editing content and scene adaptability, and meet the needs of high-quality video presentation. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a multi-dimensional intelligent video content editing method provided by the present invention;

[0055] Figure 2 A flowchart illustrating a method for generating a clipping decision sequence in an optional embodiment of the present invention;

[0056] Figure 3 This is a structural schematic diagram of a multi-dimensional video content intelligent editing device provided by the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In one embodiment, such as Figure 1As shown, a multi-dimensional intelligent video content editing method is provided. This embodiment illustrates the application of this method to a video editing terminal. It is understood that this method can also be applied to a server, or to a system including both a video editing terminal and a server, and is implemented through the interaction between the video editing terminal and the server. In this embodiment, the method includes the following steps:

[0059] S111. Multi-dimensional feature extraction is performed on the multi-camera video stream to obtain multi-dimensional video features; the multi-dimensional video features include visual features, audio features, semantic features and motion trajectory features.

[0060] Specifically, the multi-camera video stream is decoded frame by frame. Visual features are obtained by jointly calculating the directional gradient histogram and color moments on the image channel. Simultaneously, the audio frames are extracted by short-time Fourier transform to form Mel frequency cepstral coefficients to form audio features. A pre-trained bag-of-words model is run on the decoded frame sequence to merge the object labels and speech-to-text of the scene into semantic features. The camera motion vector is calculated by accumulating the optical flow fields of the previous and next frames and the trajectory of the main subject in the scene is recorded to form motion trajectory features. The extracted features are aligned by frame number to obtain multi-dimensional video features.

[0061] S112. Perform feature fusion processing based on multi-dimensional video features to obtain a fused feature vector.

[0062] Optionally, the features of each dimension can be standardized to ensure that the numerical range of different features is consistent. Then, the vectors of visual features, audio features, semantic features, and motion trajectory features can be combined according to their dimensions to form a fused feature vector. Alternatively, an attention mechanism can be used to assign different weights to each feature according to the importance of different features in subsequent scene classification, and then perform weighted summation to obtain the fused feature vector. This process allows the fused feature vector to comprehensively reflect multi-dimensional information.

[0063] S113. Based on the fused feature vector, classify the content and scene of each video frame and generate a scene label sequence.

[0064] Specifically, when classifying the content scene for each video frame, the fused feature vector is input into a pre-trained classification model, such as a support vector machine or a classification neural network. This classification model has been trained on a large number of video samples with scene labels in the early stage, and has the ability to determine the scene category based on the fused features. For example, it can identify "classroom scene" and "sports competition scene". The classification results corresponding to each video frame are arranged in chronological order to generate a scene label sequence. This sequence can reflect the scene category of the video frame as time changes.

[0065] S114. Based on the scene label sequence, the editing strategy is inferred through the decision tree inference engine to generate the editing decision sequence.

[0066] The decision tree inference engine is a tree structure built based on preset editing rules. Each node of the tree represents a judgment condition related to a scene label, and the branches represent the corresponding editing strategies when the conditions are met. For example, when the scene label for multiple consecutive frames is "speaker close-up", the editing strategy of "keeping the current camera position" is inferred. When the scene label changes from "audience panorama" to "speaker close-up", the editing strategy of "switching to the speaker's camera position" is inferred. The scene label sequence is input into the decision tree inference engine in sequence, and the editing strategy for each step is obtained according to the judgment logic of the tree structure, thereby generating an editing decision sequence.

[0067] S115. Based on the editing decision sequence, locate the shot switching points of the multi-camera video stream and generate a set of editing time points.

[0068] Specifically, when locating the camera switching point in a multi-camera video stream, the "camera switching" decision in the editing decision sequence is used to mark the corresponding video frame time point as a potential switching point. Furthermore, the potential switching points are verified and adjusted by combining the visual abrupt change characteristics of the video frames, such as the pixel difference between adjacent frames exceeding a preset threshold, or the amplitude change of the audio signal, to ensure accurate switching point location. All determined switching points are then arranged in chronological order to generate an editing time point set.

[0069] S116. Based on the set of editing time points, perform shot splicing processing on the multi-camera video stream to generate the target edited video.

[0070] For example, based on the set of editing time points, a shot segment is extracted from the video stream of the corresponding camera position at each editing time point. For example, video frame alignment technology is used to ensure that the content of the shot segments from different camera positions is continuous at the splicing point and to avoid the appearance of jumps in the picture. At the same time, the audio signal at the splicing point is smoothed, such as by using fade-in and fade-out, to reduce the discomfort caused by abrupt changes in audio. All the extracted and processed shot segments are spliced ​​together in chronological order to generate the target edited video.

[0071] The aforementioned multi-dimensional intelligent video content editing method comprehensively acquires video information through multi-dimensional feature extraction, integrates key data through feature fusion, achieves accurate scene classification based on fused features, obtains reasonable editing strategies by combining decision tree reasoning, and then accurately locates switching points and completes shot splicing. This method realizes the automation and precision of multi-camera video editing, effectively reduces manual intervention, improves editing efficiency, and ensures high quality in terms of content coherence and visual and auditory experience of the generated target edited video.

[0072] In one embodiment, based on the fused feature vector, content scene classification is performed on each frame of video to generate a scene label sequence, including:

[0073] S211. Extract key feature points from the fused feature vector of each video frame to obtain a set of key feature points.

[0074] Key feature point extraction can employ either Scale-Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF) algorithms. Taking SIFT as an example, firstly, extreme value detection is performed on the video frames corresponding to the fused feature vectors at different scales to locate potential key feature points. Then, by calculating the direction and magnitude of the gradient in the neighborhood of each feature point, feature descriptors with scale and rotation invariance are generated. Points that can represent the core information within the frame are selected as key feature points. For example, from video frames containing buildings, the outline corner points of buildings are extracted as key feature points. Finally, the key feature points of all frames are integrated to form a set of key feature points.

[0075] S212. Based on the set of key feature points, calculate the feature point matching relationship between adjacent frames to obtain an inter-frame feature association map.

[0076] Optionally, the similarity of key feature point descriptors between two adjacent frames is first calculated using Euclidean distance, which is the square root of the sum of the squares of the differences between corresponding elements of the two descriptor vectors. A smaller distance indicates higher similarity. Then, a nearest neighbor matching strategy is employed: for each key feature point in the previous frame, the feature point with the highest similarity in subsequent frames is found as the matching point. Simultaneously, a similarity threshold is set to remove incorrect matches. The corresponding matching relationships are then presented graphically, where nodes represent key feature points and edges represent matching relationships, resulting in an inter-frame feature association graph. This graph visually reflects the association between features in adjacent frames.

[0077] S213. Based on the inter-frame feature association map and combined with the preset scene classification model, classify the content scene of each video frame and generate a scene label sequence.

[0078] Specifically, the inter-frame feature association graph is converted into vector form. A graph convolutional network is then used to process the graph structure data, extracting global features and converting them into feature vectors. These vectors are then combined with the fused feature vectors of the corresponding frames to form a comprehensive feature. The pre-defined scene classification model needs to be trained in advance using video samples labeled with scene categories. For example, a random forest or deep neural network can be used. The comprehensive features are input into the scene classification model, which, based on the scene classification rules learned during training, outputs the scene category for each frame. The corresponding category results are arranged in chronological order of the frames to generate a scene label sequence.

[0079] In the above embodiments, key feature points are extracted from the fused feature vector to focus on core information. Then, an association graph is constructed by calculating the matching relationship between feature points of adjacent frames. The association graph is combined with a preset scene classification model to complete scene classification. This method can make full use of inter-frame association information to optimize scene classification, effectively improve the accuracy and stability of scene classification, and ensure the coherence and rationality of the edited video content.

[0080] In an optional embodiment, such as Figure 2 As shown, based on the scene label sequence, a decision tree inference engine is used to infer the editing strategy and generate an editing decision sequence, including:

[0081] S311. Perform key editing event identification on the scene label sequence to obtain a key event index set.

[0082] Optionally, key clip event identification requires pre-setting scene label patterns corresponding to the key events. For example, "switching from the 'meeting speech' label to the 'audience interaction' label" can be defined as a key clip event, or maintaining the "wonderful goal" label for multiple consecutive frames is also considered a key event. By traversing the scene label sequence and using a pattern matching algorithm, the consistency between the label combinations in the sequence and the preset key event label patterns is compared. When a match is successful, the time index of the corresponding frame is recorded. All successfully matched time indices are integrated to form a key event index set, which can accurately locate the clip nodes in the video that require key processing.

[0083] S312. Based on the key event index set, perform context window slicing on the scene label sequence to obtain a set of label fragments with time context.

[0084] Specifically, a context window is formed by extending a fixed number of frames forward and backward from the key event index. The window length can be preset according to the characteristics of the video content. For example, the context window can be formed by extending 10 frames forward and 10 frames backward from the frame corresponding to the key event index. For each context window corresponding to a key event index, scene labels of all frames within the window are extracted to form label fragments with a chronological order. All such label fragments are integrated to obtain a set of label fragments with temporal context, which can retain the scene association information before and after the key event.

[0085] S313. Using the decision tree reasoning engine, perform rule matching on the set of labeled fragments with time context to obtain a preliminary editing decision sequence.

[0086] Specifically, during the rule matching process, the decision tree inference engine needs to be pre-built based on editing rules. The internal nodes of the tree represent scene label conditions in the tagged segments, such as "whether the tagged segment contains the 'intense movement' label," and the leaf nodes represent the corresponding editing decisions, such as "switch to close-up camera" or "maintain panoramic camera." The tagged segments with time context are sequentially input into the decision tree inference engine, which traverses layer by layer according to the label conditions starting from the root node until it reaches the leaf node to obtain the corresponding editing decision. All decisions are arranged in chronological order to form a preliminary editing decision sequence.

[0087] S314. Perform time continuity verification on the preliminary editing decision sequence to obtain the time-corrected editing decision sequence.

[0088] The time continuity check requires traversing the initial editing decision sequence, calculating the time index difference between two adjacent editing decisions, and determining whether this difference meets the preset minimum time interval requirement. This minimum time interval needs to be set according to the requirements of video playback smoothness, such as avoiding continuous camera switching within 1 second. When it is detected that the time interval between adjacent decisions is less than the preset value, the earlier decision is retained, the later redundant decision is deleted, or the time index of the later decision is adjusted to meet the interval requirement. After adjustment, the time-corrected editing decision sequence is obtained.

[0089] S315. Perform a logical consistency check on the time-corrected editing decision sequence and generate an editing decision sequence.

[0090] For example, logical consistency verification requires pre-setting editing decision logic rules, such as "if the previous decision is 'switch to camera A,' then the decision 'switch to camera B, which has the same view as camera A,' should not appear in the short term." The time-corrected editing decision sequence is iterated, and the compliance of adjacent decisions with the pre-set logical rules is compared. When a logical conflict is found, the reasonable decision is retained according to the rules, and the conflicting decision is replaced. For example, the conflicting "switch to camera B" decision is adjusted to "keep camera A," ultimately generating an editing decision sequence that conforms to logical consistency.

[0091] In the above embodiments, by identifying key editing events from the scene tag sequence and constructing tag segments with temporal context, and then obtaining preliminary editing decisions through decision tree reasoning, and optimizing the decision sequence after checking for temporal continuity and logical consistency, this method can accurately locate key editing nodes, ensure the temporal fluency and logical rationality of editing decisions, and effectively improve the quality and efficiency of multi-camera video editing.

[0092] In one embodiment, a decision tree inference engine is used to perform rule matching on a set of labeled fragments with temporal context to obtain a preliminary editing decision sequence, including:

[0093] S411. Based on the predefined editing rule library, configure the node judgment logic of the decision tree inference engine to generate a rule matching decision tree.

[0094] For example, a predefined editing rule base needs to be built in advance, containing corresponding rules for scene tags and editing decisions. For instance, rules such as "if the tag clip contains 'intense movement,' then it corresponds to 'switch to close-up camera position,'" and "if the tag clip is 'meeting speech,' then it corresponds to 'maintain medium shot position for the person,'" etc., must be followed. The rules must also clearly define the priority of the judgment logic, such as "in case of conflict, the rule with higher scene importance takes precedence." When configuring the node judgment logic, each rule in the rule base is broken down into nodes of a decision tree. Internal nodes correspond to scene tag judgment conditions in the rule, such as "whether it contains the 'intense movement' tag," while leaf nodes correspond to editing decisions in the rule. Simultaneously, the node hierarchy is determined according to rule priority to complete the generation of the rule matching decision tree, ensuring that the decision tree accurately maps to the rule base logic.

[0095] S412. By traversing the set of label fragments with time context through the rule matching decision tree, the original decision result of each label fragment is obtained.

[0096] Specifically, when traversing the set of tag segments with temporal context, each tag segment is sequentially input into the rule matching decision tree. Starting from the root node, based on whether the tag segment meets the scene tag judgment condition of the current node, the corresponding branch is selected to continue traversing until a leaf node is reached. The editing decision corresponding to the leaf node is the original decision result of that tag segment. For example, if a tag segment contains the tag "intense movement", and the node condition of "containing 'intense movement' tag" is met when traversing the decision tree, the branch leads to the leaf node "switch to close-up camera position". This result is the original decision result of that segment. The original decision results of all segments are then organized in chronological order.

[0097] S413. Perform a time-series continuity check on the original decision results to obtain the checked decision results.

[0098] Optionally, the temporal continuity verification needs to preset a temporal interval standard. This standard is determined based on the requirements for smooth video playback. For example, the time index interval between two adjacent original decision results must be no less than a certain fixed duration to avoid frequent camera switching within a short period of time. All original decision results are iterated, and the time index difference between adjacent results is calculated. If the difference is less than the preset standard, it is determined whether the subsequent decision is redundant. For example, if the previous decision is "switch to close-up camera position," and the subsequent decision is also "switch to close-up camera position" with a very short interval, it is determined to be redundant, and the subsequent decision is deleted. If the subsequent decision is of a different type and the interval is too short, the time index of the subsequent decision is adjusted to meet the interval standard, resulting in the verified decision result.

[0099] S415. Perform conflict resolution processing on the verified decision results to generate a preliminary editing decision sequence.

[0100] Specifically, conflict resolution requires pre-setting conflict judgment and resolution rules. Conflict judgment rules include "multiple different editing decisions corresponding to the same time index" and "conflicting logic between adjacent decisions (e.g., the previous decision is 'keep camera A', and the next decision is immediately 'switch to camera B, which repeats the viewpoint of A')". Resolution rules include "the decision corresponding to the tag segment with richer scene information shall prevail" and "the conflicting decision that meets the needs of the mainstream scene shall be retained". After traversing and verifying the decision results, conflicts are identified according to the judgment rules and then processed according to the resolution rules. For example, if there are two decisions, "switch to close-up" and "keep medium shot", at the same time index, if the corresponding tag segment "intense movement" has richer scene information, the "switch to close-up" decision shall be retained. After the corresponding video stream is processed, a preliminary editing decision sequence is generated.

[0101] In the above embodiments, a rule matching decision tree is generated based on a predefined editing rule base configuration, and then the original decision results are obtained by traversing the tag segments. After temporal continuity verification and conflict resolution processing, a preliminary editing decision sequence is generated. This method ensures that the preliminary editing decision sequence conforms to the preset rules and has temporal fluency and logical rationality through accurate mapping between the rule base and the decision tree and multiple rounds of verification and optimization, effectively improving the accuracy and efficiency of multi-camera video editing.

[0102] In one embodiment, based on the editing decision sequence, the camera switching points of the multi-camera video stream are located to generate a set of editing time points, including:

[0103] S511. Extract timestamps from the editing decision sequence to obtain a set of decision time points.

[0104] Optionally, the timing information corresponding to each decision in the editing decision sequence is first defined. Each decision in the editing decision sequence is associated with a specific video frame, and the timestamp is the absolute time position of that frame in the video stream, usually in seconds or frame counts. For example, a "switch close-up camera position" decision corresponds to the 120th second frame in the video stream, and its timestamp is 120 seconds. During extraction, the editing decision sequence is traversed, the timing information of the frames associated with each decision is read, and the corresponding timing information is organized according to the decision order to form a set of decision time points.

[0105] S512. Based on the decision time point set, perform frame-level matching processing on the multi-camera video stream to generate a candidate switching frame set.

[0106] Specifically, the timelines of each camera in the multi-camera video stream are first obtained to ensure synchronization, meaning that the same timestamp corresponds to the same moment in the video footage from each camera. Based on each time point in the decision time point set, the corresponding video frame with that timestamp is found in each camera's video stream. This corresponding video frame is the candidate frame for different cameras at that time point. All candidate frames corresponding to each decision time point are categorized. For example, decision time point 120 seconds corresponds to the 120th second frame of camera 1, the 120th second frame of camera 2, etc. All categorized candidate frames are combined to form a candidate switching frame set, which contains the selectable switching frame options for each decision time point.

[0107] S513. Based on the candidate switching frame set, perform boundary point localization to obtain the editing time point set.

[0108] For example, boundary point localization requires analyzing the continuity and content relevance of each candidate frame in the candidate switching frame set. This is achieved by calculating the pixel difference value (e.g., Mean Squared Error, MSE) between adjacent frames; the smaller the pixel difference value, the more continuous the two frames are. For each candidate switching frame corresponding to a decision time point, the pixel difference between it and the last frame in the previous decision time period is compared, and the frame with the smallest difference value that meets the editing requirements is selected as the boundary point. The time corresponding to this boundary point is the editing time point. For example, among the candidate frames at the decision time point of 120 seconds, the pixel differences between each camera frame and the last frame at 119 seconds are compared, and the 120-second frame corresponding to the camera frame with the smallest difference is selected as the editing time point. All determined editing time points are arranged in order to form an editing time point set.

[0109] In the above embodiments, decision time points are extracted by editing decision sequences, multi-camera candidate switching frames are matched based on time points, and finally the set of editing time points is determined by boundary point positioning. This method ensures accurate positioning of editing time points and smooth picture through precise time correlation and frame-level picture analysis.

[0110] In one embodiment, frame-level matching processing is performed on the multi-camera video stream based on the decision time point set to generate a candidate switching frame set, including:

[0111] S611. Perform time window truncation processing on the set of decision time points to obtain a set of frame segments to be matched.

[0112] The time window extraction process requires pre-setting the length of the time window, which is determined based on the frame rate of the video frames and the editing precision requirements. For example, the window length can be set to include a range of several frames before and after the decision time point. Centered on each time point in the decision time point set, consecutive video frames of the pre-set length before and after that time point in the corresponding multi-camera video stream are extracted to form a frame segment. For example, if a decision time point is 120 seconds, and the window length is set to 5 frames before and after, then consecutive frames from 119.8 seconds to 120.2 seconds (assuming a frame rate of 25 frames / second) are extracted as a frame segment. All such frame segments generated around the decision time point are integrated to obtain the set of frame segments to be matched.

[0113] S612. Based on the set of frame segments to be matched, perform similarity calculation on the inter-frame visual features of the multi-camera video stream to generate a frame similarity matrix.

[0114]

[0115] in, Let be the multimodal fusion similarity between the i-th frame of host video A and the j-th frame of slave video B. Let H(·) be the frame-level semantic feature vector, H(·) be the three-dimensional histogram feature extraction function in the HSV color space, and M(·) be the motion feature extraction function based on optical flow field. , , For adaptive weighting coefficients, These are the scale adjustment parameters for color features.

[0116] Specifically, features are first extracted from each frame of the host-side video A and slave-side video B in the set of frame segments to be matched. Among these features are frame-level semantic feature vectors. The image semantic model, pre-trained, captures semantic information about objects and scenes within a frame. The HSV color space 3D histogram feature extraction function H(·) first converts the frame to the HSV color space, then statistically analyzes the pixel distribution across hue, saturation, and brightness to generate a 3D histogram as color features. The motion feature extraction function M(·) based on optical flow calculates the motion vectors of pixels in adjacent frames to obtain the frame's motion features. Then, the process is calculated according to a given formula. Adaptive weight coefficients , , The scaling parameters are set according to the degree of influence of different features on similarity. The calculated differences in color features are used to balance all Arrange the frames according to the master frame number i and the slave frame number j to generate a frame similarity matrix.

[0117] S613. Filter local extreme points using the frame similarity matrix to generate a set of candidate switching frames.

[0118] For example, first, the extreme value judgment condition is determined: if the value of an element in the matrix is ​​greater than the values ​​of its adjacent elements (up, down, left, and right), it is determined to be a local maximum point. A local maximum point represents the highest similarity between the corresponding host frame and slave frame. The frame similarity matrix is ​​traversed, and all local maximum points are identified according to the extreme value judgment condition. The host frame index i and slave frame index j corresponding to these points are recorded, and the corresponding video frames are found. These video frames are the candidate switching frames. All candidate switching frames are arranged in order of decision time points to generate a candidate switching frame set. This set ensures that at each decision time point, there is a frame with the highest similarity as a switching candidate.

[0119] In the above embodiments, by extracting frame segments to be matched around the decision time point, calculating the frame similarity matrix through multi-feature fusion, and filtering local extreme points to obtain a set of candidate switching frames, this method ensures that the candidate switching frames have high similarity and visual coherence through precise time window limitation and multi-dimensional visual feature similarity calculation.

[0120] In one embodiment, based on a set of editing time points, the multi-camera video stream is spliced ​​to generate a target edited video, including:

[0121] S711. Perform timeline sorting on the set of edited time points to obtain an ordered sequence of edited time points.

[0122] Optionally, the timeline sorting process requires first determining the sorting benchmark. Based on the absolute timeline of the video stream, all time points in the clip time point set are converted into a uniform time unit (e.g., seconds). A sorting algorithm, such as bubble sort / quick sort, is then used to arrange these time points in ascending order, ensuring that the video frame corresponding to a later time point is later than the video frame corresponding to a previous time point. For example, if the clip time point set contains 120 seconds, 90 seconds, and 150 seconds, sorting yields an ordered sequence of 90 seconds, 120 seconds, and 150 seconds, which clearly defines the order of shot transitions.

[0123] S712. Based on the ordered editing timeline, extract shot segments from the multi-camera video stream to generate a set of shot segments to be spliced.

[0124] Specifically, the time intervals of the multi-camera video stream are divided based on the ordered editing timeline. Two adjacent ordered time points constitute a time interval, and each time interval corresponds to a shot segment. The target camera position for each time interval is determined based on the initial editing decisions. From the video stream of that target camera position, consecutive video frames within the corresponding time interval are extracted to form the shot segment. For example, ordered time points 90 seconds and 120 seconds correspond to the time interval 90 seconds to 120 seconds. If the decision for this interval is "main camera position A," then the frame sequence from 90 seconds to 120 seconds is extracted from the video stream of main camera position A as the shot segment. All such shot segments extracted according to time intervals and target camera positions are integrated to generate a set of shot segments to be spliced.

[0125] S713. Based on the set of shot segments to be spliced, add shot transition effects to generate a set of transitioned shot segments.

[0126] For example, adding transition effects requires a pre-set transition effect library, including common transition effects such as fade-in / fade-out, dissolve, and wipe, each corresponding to a specific scene transition algorithm. Based on the content and scene of adjacent segments in the set of shots to be stitched together, a matching transition effect is selected. For example, the dissolve effect is selected when adjacent segments are both "meeting scenes," and the wipe effect is selected when the scene switches from "outdoor" to "indoor." Using video editing algorithms, the selected transition effect is applied at the junction of adjacent shot segments. For example, the dissolve effect is achieved by gradually reducing the transparency of the previous segment while increasing the transparency of the subsequent segment. After processing, a set of transitioned shot segments is generated.

[0127] S714. The target edited video is generated by sequentially splicing the set of transitional shot clips.

[0128] Specifically, the segments in the transitioned shot clip set are sequentially connected according to the chronological order corresponding to the ordered editing timeline. During the connection process, it is ensured that the end frame of the previous segment and the beginning frame of the next segment are seamlessly connected through the added transition effect. At the same time, the audio signals between the segments are synchronized to ensure that the time correspondence between the audio and the video is consistent. For example, first splice the shot clips from 90 seconds to 120 seconds, then splice the shot clips from 120 seconds to 150 seconds, and use a dissolve effect to transition between the two segments, ultimately forming the complete target edited video.

[0129] In the above embodiments, by sorting the editing time points to clarify the switching order, extracting the corresponding shot segments, adding transition effects, and finally splicing them together in time sequence to generate the target video, this method ensures that the target edited video meets high-quality standards in terms of time logic, screen connection, and visual experience through standardized time sorting, accurate shot extraction, and natural transition processing, effectively improving the integrity and viewing experience of multi-camera video editing.

[0130] The aforementioned multi-dimensional intelligent video content editing method, device, equipment, and medium extract multi-dimensional features (visual, audio, semantic, and motion trajectory) from multi-camera video streams and fuse them into a fused feature vector. Next, scene classification is optimized by combining key feature point extraction and inter-frame feature association graphs to generate a scene label sequence. Then, through key editing event recognition and context window slicing, a preliminary editing decision is generated based on a decision tree inference engine configured with a predefined rule base, and optimized into a final editing decision sequence after time continuity and logical consistency checks. Finally, based on the decision extraction time point, frame-level matching, similarity calculation, and extreme value filtering are used to locate and switch frames. The target edited video is then generated through timeline sorting, shot extraction, transition addition, and time-series splicing, forming a complete process from feature processing to editing output. This method improves information integrity through multi-dimensional feature fusion, optimizes decision rationality through multiple rounds of verification, and ensures visual continuity through precise frame matching and transition processing. It solves the technical problems of low efficiency, poor accuracy, and unsatisfactory visual effects in multi-camera video editing in the background technology, achieving efficient, accurate, and visually smooth intelligent multi-camera video editing.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0132] Based on the same inventive concept, this application also provides an apparatus for implementing the aforementioned intelligent editing method for multi-dimensional video content. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the intelligent editing apparatus for multi-dimensional video content provided below can be found in the limitations of the intelligent editing method for multi-dimensional video content described above, and will not be repeated here.

[0133] In one exemplary embodiment, such as Figure 3 As shown, a multi-dimensional video content intelligent editing device 10 is provided, including:

[0134] The multi-dimensional feature extraction module 11 is used to extract multi-dimensional features from the multi-camera video stream to obtain multi-dimensional video features; the multi-dimensional video features include visual features, audio features, semantic features and motion trajectory features;

[0135] Feature fusion processing module 12 is used to perform feature fusion processing based on multi-dimensional video features to obtain a fused feature vector;

[0136] The content scene classification module 13 is used to classify the content scene of each video frame according to the fused feature vector and generate a scene label sequence.

[0137] The editing strategy reasoning module 14 is used to perform editing strategy reasoning based on scene label sequences and through a decision tree reasoning engine to generate editing decision sequences.

[0138] The shot switching point positioning module 15 is used to locate the shot switching points of the multi-camera video stream according to the editing decision sequence and generate a set of editing time points.

[0139] The shot stitching processing module 16 is used to perform shot stitching processing on multi-camera video streams based on the set of editing time points to generate the target edited video.

[0140] In one embodiment, the content scene classification module includes:

[0141] The key feature point extraction unit is used to extract key feature points from the fused feature vector of each frame of video to obtain a set of key feature points.

[0142] The inter-frame feature association map calculation unit is used to calculate the feature point matching relationship between adjacent frames based on the key feature point set to obtain the inter-frame feature association map.

[0143] The scene classification unit is used to classify the content and scene of each video frame based on the inter-frame feature association graph and a preset scene classification model, and generate a scene tag sequence.

[0144] In one embodiment, the clipping strategy reasoning module includes:

[0145] The key clip event identification unit is used to identify key clip events in the scene label sequence to obtain a key event index set.

[0146] The context window slicing unit is used to perform context window slicing on the scene label sequence based on the key event index set, so as to obtain a set of label fragments with temporal context;

[0147] The preliminary editing decision generation unit is used to perform rule matching on a set of labeled fragments with time context through a decision tree inference engine to obtain a preliminary editing decision sequence;

[0148] The time continuity verification unit is used to verify the time continuity of the preliminary editing decision sequence to obtain the time-corrected editing decision sequence.

[0149] The logical consistency verification unit is used to perform logical consistency verification on the time-corrected editing decision sequence and generate the editing decision sequence.

[0150] In one embodiment, the preliminary editing decision generation unit includes:

[0151] The decision tree configuration subunit is used to configure the node judgment logic of the decision tree inference engine according to the predefined clipping rule library, and generate a rule matching decision tree;

[0152] The original decision result acquisition sub-unit is used to traverse the set of label fragments with time context through rule matching decision tree to obtain the original decision result of each label fragment;

[0153] The temporal continuity verification subunit is used to verify the temporal continuity of the original decision results and obtain the verified decision results.

[0154] The conflict resolution subunit is used to resolve conflicts in the verified decision results and generate a preliminary editing decision sequence.

[0155] In one embodiment, the lens switching point positioning module includes:

[0156] The timestamp extraction unit is used to extract timestamps from the editing decision sequence to obtain a set of decision time points;

[0157] The frame-level matching unit is used to perform frame-level matching processing on the multi-camera video stream based on the decision time point set, and generate a candidate switching frame set.

[0158] The boundary point localization unit is used to locate boundary points based on the candidate switching frame set to obtain the editing time point set.

[0159] In one embodiment, the frame-level matching unit includes:

[0160] The time window truncation subunit is used to perform time window truncation processing on the set of decision time points to obtain a set of frame segments to be matched;

[0161] The inter-frame visual feature similarity calculation subunit is used to calculate the similarity of inter-frame visual features of multi-camera video streams based on the set of frame segments to be matched, and generate a frame similarity matrix.

[0162]

[0163] in, Let be the multimodal fusion similarity between the i-th frame of host video A and the j-th frame of slave video B. Let H(·) be the frame-level semantic feature vector, H(·) be the three-dimensional histogram feature extraction function in the HSV color space, and M(·) be the motion feature extraction function based on optical flow field. , , For adaptive weighting coefficients, The scaling parameter for color features;

[0164] The local extremum point filtering subunit is used to filter local extremum points through the frame similarity matrix and generate a set of candidate switching frames.

[0165] In one embodiment, the lens stitching processing module includes:

[0166] The timeline sorting unit is used to sort the set of edit time points along the timeline to obtain an ordered sequence of edit time points.

[0167] The shot segment extraction unit is used to extract shot segments from multi-camera video streams based on an ordered editing time sequence, and generate a set of shot segments to be spliced.

[0168] The transition effect addition unit is used to add transition effects to the set of shot segments to be spliced, and generate a set of transitioned shot segments.

[0169] The temporal splicing unit is used to splice the transitional shot clips in a temporal sequence to generate the target edited video.

[0170] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the multi-dimensional video content intelligent editing method as described above.

[0171] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-dimensional video content intelligent editing method as described above.

[0172] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0173] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A multi-dimensional intelligent video content editing method, characterized in that, The method includes: Multi-dimensional video features are extracted from multi-camera video streams to obtain multi-dimensional video features, which include visual features, audio features, semantic features, and motion trajectory features. Based on the multi-dimensional video features, feature fusion processing is performed to obtain a fused feature vector; Based on the fused feature vector, each frame of video is classified into content and scene, and a scene tag sequence is generated. Based on the scene label sequence, the editing strategy is inferred through the decision tree inference engine to generate an editing decision sequence; Based on the editing decision sequence, the camera switching points of the multi-camera video stream are located to generate a set of editing time points; Based on the set of editing time points, the multi-camera video stream is spliced ​​to generate the target edited video.

2. The method according to claim 1, characterized in that, The step of classifying the content and scene of each video frame based on the fused feature vector and generating a scene label sequence includes: For each frame of video, key feature points are extracted from the fused feature vector to obtain a set of key feature points; Based on the set of key feature points, the feature point matching relationship between adjacent frames is calculated to obtain an inter-frame feature association map; Based on the inter-frame feature association graph and combined with the preset scene classification model, the content scene is classified for each video frame, and a scene tag sequence is generated.

3. The method according to claim 1, characterized in that, The step of generating an editing decision sequence by reasoning the editing strategy through a decision tree inference engine based on the scene label sequence includes: Key editing events are identified from the scene tag sequence to obtain a key event index set; Based on the key event index set, the scene label sequence is processed by context window slicing to obtain a set of label fragments with time context; The decision tree reasoning engine is used to perform rule matching on the set of labeled fragments with time context to obtain a preliminary editing decision sequence; The initial editing decision sequence is subjected to time continuity verification to obtain the time-corrected editing decision sequence; The time-corrected editing decision sequence is subjected to a logical consistency check to generate an editing decision sequence.

4. The method according to claim 3, characterized in that, The step of using a decision tree inference engine to perform rule matching on the set of labeled fragments with temporal context to obtain a preliminary editing decision sequence includes: Based on a predefined editing rule library, the node judgment logic of the decision tree inference engine is configured to generate a rule matching decision tree; The set of label fragments with time context is traversed by the rule-matching decision tree to obtain the original decision result for each label fragment; The original decision results are subjected to a time-series continuity check to obtain the checked decision results; The verified decision results are subjected to conflict resolution processing to generate the preliminary editing decision sequence.

5. The method according to claim 1, characterized in that, The step of locating shot switching points in the multi-camera video stream based on the editing decision sequence and generating an editing time point set includes: The timestamps of the editing decision sequence are extracted to obtain a set of decision time points; Based on the set of decision time points, the multi-camera video stream is subjected to frame-level matching processing to generate a set of candidate switching frames; Boundary point localization is performed based on the candidate switching frame set to obtain the editing time point set.

6. The method according to claim 5, characterized in that, The step of performing frame-level matching processing on the multi-camera video stream based on the decision time point set to generate a candidate switching frame set includes: The set of decision time points is subjected to time window truncation processing to obtain a set of frame segments to be matched; Based on the set of frame segments to be matched, the similarity of the inter-frame visual features of the multi-camera video stream is calculated to generate a frame similarity matrix. in, Let be the multimodal fusion similarity between the i-th frame of host video A and the j-th frame of slave video B. Let H(·) be the frame-level semantic feature vector, H(·) be the three-dimensional histogram feature extraction function in the HSV color space, and M(·) be the motion feature extraction function based on optical flow field. , , For adaptive weighting coefficients, The scaling parameter for color features; The frame similarity matrix is ​​used to filter local extreme points and generate a set of candidate switching frames.

7. The method according to claim 1, characterized in that, The step of performing shot splicing processing on the multi-camera video stream based on the set of editing time points to generate the target edited video includes: The set of edit time points is sorted along the timeline to obtain an ordered sequence of edit time points; Based on the ordered editing timeline, the multi-camera video stream is used to extract shot segments and generate a set of shot segments to be spliced. Based on the set of shot segments to be spliced, a shot transition effect is added to generate a set of transitioned shot segments. The target edited video is generated by sequentially splicing the set of transitional shot segments.

8. A multi-dimensional video content intelligent editing device, characterized in that, The device includes: The multi-dimensional feature extraction module is used to extract multi-dimensional features from multi-camera video streams to obtain multi-dimensional video features; the multi-dimensional video features include visual features, audio features, semantic features, and motion trajectory features; The feature fusion processing module is used to perform feature fusion processing based on the multi-dimensional video features to obtain a fused feature vector. The content scene classification module is used to classify the content scene of each video frame according to the fused feature vector and generate a scene tag sequence. The editing strategy reasoning module is used to perform editing strategy reasoning based on the scene label sequence and generate an editing decision sequence through a decision tree reasoning engine. The shot switching point positioning module is used to locate the shot switching points of the multi-camera video stream according to the editing decision sequence and generate a set of editing time points; The shot stitching processing module is used to perform shot stitching processing on the multi-camera video stream based on the set of editing time points to generate the target edited video.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images