A Deep Learning-Based Automatic Video Editing Method and Medium

Through deep learning technology, the target feature points and multimodal timing features of video frames are extracted and editing strategies are generated, which solves the problem of insufficient flexibility in the existing video automatic editing method and achieves a more accurate and natural video editing effect.

CN118612508BActive Publication Date: 2024-11-12SHENZHEN JIUFEN CULTURE MEDIA CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410655930.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-24
Publication Date
2024-11-12
Estimated Expiration
2044-05-24

AI Technical Summary

Technical Problem

The existing automatic video editing methods are based on rules, with limited flexibility and adaptability, which leads to inaccurate and natural enough to meet the needs of high-quality video editing.

Method used

Using a deep learning-based method, we extract the target feature points of the video frame, perform image enhancement and scene division, extract multimodal timing features, generate editing strategies, and realize automatic video editing. Specific steps include: image clarity extraction, image enhancement, feature point filtering, scene division, multimodal feature extraction and editing strategy generation.

Benefits of technology

It improves the accuracy and nature of video editing, can describe the video scene more accurately, and generate higher-quality edited videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118612508B_ABST
    Figure CN118612508B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a method and medium for automatic video editing based on deep learning, the method comprising: extracting a target video frame sequence from a video frame to be edited, and performing image enhancement on the target video frame sequence to obtain an enhanced image frame sequence; extracting an initial feature point set of each enhanced image frame, and performing feature point screening on the initial feature point set to obtain a target feature point; performing scene division on the enhanced image frame sequence according to the target feature point to obtain a scene image frame set, and extracting scene audio corresponding to the scene image frame set; extracting multimodal temporal features of the scene image frame set and the scene audio, and calculating a scene description of the scene image frame set according to the multimodal temporal features; generating an editing strategy according to the scene description, and editing the video to be edited according to the editing strategy to obtain an edited video. The present invention can improve the effect of automatic video editing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a deep learning-based automatic video editing method and medium. Background Art

[0002] With the rapid development of the Internet and social media, the creation and sharing of video content has become increasingly popular. However, high-quality video editing usually requires professional skills and a lot of time. The traditional video editing process includes multiple steps such as shooting, importing, cutting, transitions, adding special effects, etc. These steps often require video editors to have certain professional skills and need to be done manually, which is inefficient.

[0003] In recent years, deep learning technology has made significant progress in the field of image and video analysis, especially in automatic content recognition, scene detection and object tracking. The development of these technologies has provided new possibilities for automatic video editing. Existing automatic video editing methods are mainly rule-based methods. For example, the scene corresponding to each video frame is calculated, and the appropriate key frames are selected to form a segment according to the scene using predefined rules and editing goals. However, rule-based methods rely heavily on manually defined editing rules, and their flexibility and adaptability are limited, resulting in inaccurate and unnatural video editing results. Therefore, how to improve the effect of automatic video editing has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a deep learning-based automatic video editing method and medium, the main purpose of which is to solve the problem of poor effect of automatic video editing.

[0005] To achieve the above object, the present invention provides a deep learning-based automatic video editing method, comprising:

[0006] Obtain a video to be edited, extract a target video frame sequence from the video to be edited according to the image clarity of each video frame in the video to be edited, and perform image enhancement on the target video frame sequence to obtain an enhanced image frame sequence; wherein, performing image enhancement on the target video frame sequence to obtain an enhanced image frame sequence includes: converting each target video frame in the target video frame sequence to a luminance color space to obtain a hue component, a saturation component, and a luminance component; performing image decomposition on the luminance component to obtain an illumination image and a reflection image, and dividing the illumination image into blocks to obtain illumination image blocks; performing histogram equalization on the illumination image blocks respectively to obtain an enhanced illumination image; calculating directional gradients of the reflection image in multiple directions, and enhancing the reflection image according to the directional gradients to obtain an enhanced reflection image;

[0007] The reflected image is enhanced using the following formula to obtain an enhanced reflected image:

[0008]

[0009] Among them, F e represents the enhanced reflection image, F represents the reflection image, M represents the total number of directional gradients, represents convolution calculation, O m represents the mth directional gradient;

[0010] Performing inverse image decomposition on the enhanced illumination image and the enhanced reflection image to obtain an enhanced brightness component; converting the hue component, the saturation component and the enhanced brightness component into an RGB color space to obtain an enhanced image frame sequence;

[0011] Extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence, calculating a grayscale change value of each initial feature point in the initial feature point set, and screening feature points of the initial feature point set according to the grayscale change value to obtain target feature points;

[0012] Dividing the enhanced image frame sequence into scenes according to the target feature points to obtain a scene image frame set, and extracting scene audio corresponding to the scene image frame set;

[0013] Extracting multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio, and calculating a scene description of the scene image frame set according to the multimodal temporal features;

[0014] An editing strategy for the video to be edited is generated according to the scene description, and the video to be edited is edited according to the editing strategy to obtain an edited video.

[0015] Optionally, extracting a target video frame sequence from the video to be edited according to the image definition of each video frame in the video to be edited includes:

[0016] Calculate the directional gradient of each pixel point in the video frame in the coordinate direction;

[0017] Calculate the grayscale gradient of each pixel in the video frame according to the directional gradient;

[0018] The grayscale gradient of each pixel is calculated using the following formula:

[0019]

[0020] Among them, G(x,y) represents the grayscale gradient of the pixel point (x,y), G x(x,y) represents the directional gradient of the pixel point (x,y) in the horizontal coordinate direction, G y (x,y) represents the directional gradient of the pixel point (x,y) in the ordinate direction;

[0021] The definition value of the video frame is calculated according to the grayscale gradient, and a target video frame sequence is determined from the video frame according to the definition value.

[0022] Optionally, calculating the directional gradient of each pixel point in the video frame in a coordinate direction includes:

[0023] The directional gradient of each pixel in the video frame in the coordinate direction is calculated using the following formula:

[0024] G x (x,y)=g(x-1,y+1)+g(x+1,y+1)+2g(x,y+1)-g(x-1,y-1)

[0025] +g(x+1,y-1)-2g(x,y-1)

[0026] G y (x,y)=g(x+1,y-1)+g(x+1,y+1)+2g(x+1,y)-g(x-1,y-1)

[0027] -g(x-1, y+1)-2g(x-1, y)

[0028] Among them, G x (x, y) represents the directional gradient of the pixel point (x, y) in the horizontal coordinate direction, g(x-1, y+1) represents the grayscale value of the pixel point (x-1, y+1), g(x+1, y+1) represents the grayscale value of the pixel point g(x+1, y+1), g(x-1, y-1) represents the grayscale value of the pixel point g(x-1, y-1), (x+1, y-1) represents the grayscale value of the pixel point (x+1, y-1), g(x, y-1) represents the grayscale value of the pixel point g(x, y-1), G y (x, y) represents the directional gradient of the pixel point (x, y) in the ordinate direction, (x+1, y) represents the grayscale value of the pixel point (x+1, y), and (x-1, y) represents the grayscale value of the pixel point (x-1, y).

[0029] Optionally, extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence includes:

[0030] Counting the grayscale value of each pixel in the enhanced image frame, and calculating the grayscale value extreme value according to the grayscale value; and calculating the characteristic grayscale value threshold according to the grayscale value extreme value;

[0031] The following formula is used to calculate the feature gray value threshold:

[0032]

[0033] Wherein, T represents the characteristic gray value threshold, I1 represents the high brightness extreme value in the gray value extreme value, I2 represents the low brightness extreme value in the gray value extreme value, and N represents the total number of pixels in the calculated gray value extreme value;

[0034] A neighborhood point of each pixel point is selected using a preset neighborhood boundary, and an initial feature point set is determined according to the grayscale value of the neighborhood point and the grayscale value threshold.

[0035] Optionally, calculating the grayscale change value of each initial feature point in the initial feature point set includes:

[0036] Selecting a neighborhood feature point set of each initial feature point in the initial feature point set using a preset window;

[0037] Calculating the pixel gradient of the pixel value of each neighborhood feature point in the neighborhood feature point set, and constructing a feature matrix according to the pixel gradient;

[0038] The feature matrix is ​​constructed using the following formula:

[0039]

[0040] Among them, M represents the feature matrix, D x Represents the pixel gradient of the neighborhood feature point in the horizontal direction, D y Pixel gradient of neighborhood feature points in the vertical direction;

[0041] Calculating the matrix determinant and the matrix rank of the feature matrix, and calculating the grayscale change value of each of the initial feature points according to the matrix determinant and the matrix rank;

[0042] The grayscale change value of each initial feature point is calculated using the following formula:

[0043] R = det(M) - k[t(M)] 2

[0044] Among them, R represents the grayscale change value, det(M) represents the matrix determinant of the feature matrix M, k represents the preset adjustment parameter, and t(M) represents the matrix rank of the feature matrix M.

[0045] Optionally, dividing the enhanced image frame sequence into scenes according to the target feature points to obtain a scene image frame set includes:

[0046] Dividing the image region of the corresponding enhanced image frame in the enhanced image frame sequence according to the target feature point to obtain a feature point region;

[0047] Extracting regional features of the feature point area, and performing feature splicing on the regional features to obtain video frame features;

[0048] Calculating the scene similarity between the video frame feature and the preset scene feature, and determining the enhanced image frame scene corresponding to each enhanced image frame in the enhanced image frame sequence according to the scene similarity;

[0049] The enhanced image frames with the same enhanced image frame scene and adjacent to each other in the enhanced image frame sequence are divided to obtain a scene image frame set.

[0050] Optionally, extracting multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio includes:

[0051] Calculating an image frame scene action set corresponding to the scene image frame set;

[0052] Performing position encoding and feature activation on the image frame scene actions in the image frame scene action set to obtain image frame temporal features;

[0053] Calculating an audio feature sequence of the scene audio, and performing average pooling on the audio feature sequence to obtain a target audio feature sequence;

[0054] Calculating the semantic similarity between each target audio feature in the target audio feature sequence and the preset speech scene action feature to obtain a semantic temporal feature;

[0055] The image frame temporal features and the semantic temporal features are fused to obtain multimodal temporal features.

[0056] Optionally, the fusing the image frame temporal features with the semantic temporal features to obtain multimodal temporal features includes:

[0057] Fully connecting the image frame temporal features and the semantic temporal features respectively to obtain fully connected features;

[0058] Adding the image frame temporal features and the semantic temporal features element by element to obtain an added feature sequence;

[0059] The added feature sequence and the fully connected feature are concatenated column by column to obtain a multimodal time series feature.

[0060] Optionally, generating an editing strategy for the video to be edited according to the scene description includes:

[0061] Calculating a video theme corresponding to the video to be edited according to the scene description;

[0062] Calculating the semantic similarity between the video theme and the scene description, and determining the editing order and editing time of the scene description according to the semantic similarity;

[0063] An editing strategy is generated according to the editing sequence and the editing time.

[0064] In order to solve the above problems, the present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the automatic video editing method based on deep learning as described in any one of the above is implemented.

[0065] The embodiment of the present invention extracts the target video frame sequence through the image clarity of each video frame in the video to be edited, and performs image enhancement on the target video frame sequence, which can remove the image noise in the target video frame and enhance the detail information of the target video frame to obtain a more accurate enhanced image frame sequence; extracts the target feature points of each enhanced image frame in the enhanced image frame sequence, and can select the target feature points with more significant pixel value changes to improve the accuracy of subsequent scene division; performs scene division according to the target feature points to obtain a scene image frame set, and extracts the corresponding scene audio, which can obtain the multimodal temporal features of the multimodal feature information, more accurately and intuitively obtain the information contained in the scene image frame set, obtain the scene description, generate the editing strategy through the scene description, and use the editing strategy to edit the video to be edited, which can more accurately describe the information in the scene image frame set, improve the accuracy of video editing, and obtain a more accurate edited video. Therefore, the automatic video editing method and medium based on deep learning proposed by the present invention can solve the problem of poor effect of automatic video editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a process flow of a deep learning-based automatic video editing method provided by an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a process for calculating the grayscale change value of each initial feature point in an initial feature point set provided by an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of a process for dividing an enhanced image frame sequence into scenes according to an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of a process for extracting multimodal temporal features of a scene image frame set provided by an embodiment of the present invention;

[0070] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0071] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0072] The embodiment of the present application provides a method for automatic video editing based on deep learning. The execution subject of the automatic video editing method based on deep learning includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the automatic video editing method based on deep learning can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0073] Reference Figure 1 FIG. 1 is a flow chart of a method for automatic video editing based on deep learning provided by an embodiment of the present invention. In this embodiment, the method for automatic video editing based on deep learning includes:

[0074] S1. Obtain a video to be edited, extract a target video frame sequence from the video to be edited according to the image clarity of each video frame in the video to be edited, and perform image enhancement on the target video frame sequence to obtain an enhanced image frame sequence.

[0075] In an embodiment of the present invention, the video to be edited is a video that needs to be edited, which may be obtained by splicing multiple videos or a long video. Video frames are extracted from the video frames to be edited according to the video frame rate of the video to be edited, but the image clarity of each video frame may be different, and the clarity reflects the clarity and recognizability of the details in the video frame. Therefore, the video frames of the video to be edited need to be screened according to the clarity of each video frame to obtain a target video frame sequence with higher quality.

[0076] In the embodiment of the present invention, extracting a target video frame sequence from the video to be edited according to the image definition of each video frame in the video to be edited includes:

[0077] Calculate the directional gradient of each pixel point in the video frame in the coordinate direction;

[0078] Calculate the grayscale gradient of each pixel in the video frame according to the directional gradient;

[0079] The definition value of the video frame is calculated according to the grayscale gradient, and a target video frame sequence is determined from the video frame according to the definition value.

[0080] In the embodiment of the present invention, the directional gradient is to calculate the gradient value of the pixel point, and the clarity of each video frame is reflected by the directional gradient, wherein the coordinate direction is the directional gradient in the horizontal and vertical coordinate directions.

[0081] Specifically, the directional gradient of each pixel in the video frame in the coordinate direction is calculated using the following formula:

[0082] G x (x,y)=g(x-1,y+1)+g(x+1,y+1)+2g(x,y+1)-g(x-1,y-1)

[0083] +g(x+1,y-1)-2g(x,y-1)

[0084] G y (x,y)=g(x+1,y-1)+g(x+1,y+1)+2g(x+1,y)-g(x-1,y-1)

[0085] -g(x-1, y+1)-2g(x-1, y)

[0086] Among them, G x (x, y) represents the directional gradient of the pixel point (x, y) in the horizontal coordinate direction, g(x-1, y+1) represents the grayscale value of the pixel point (x-1, y+1), g(x+1, y+1) represents the grayscale value of the pixel point g(x+1, y+1), g(x-1, y-1) represents the grayscale value of the pixel point g(x-1, y-1), (x+1, y-1) represents the grayscale value of the pixel point (x+1, y-1), g(x, y-1) represents the grayscale value of the pixel point g(x, y-1), G y (x, y) represents the directional gradient of the pixel point (x, y) in the ordinate direction, (x+1, y) represents the grayscale value of the pixel point (x+1, y), and (x-1, y) represents the grayscale value of the pixel point (x-1, y).

[0087] Furthermore, the rate of change of the brightness of each pixel, ie, the grayscale gradient, can be calculated by using the gradient operator in the horizontal and vertical directions, thereby reflecting the image clarity of the entire video frame.

[0088] Specifically, the grayscale gradient of each pixel can be calculated using the following formula:

[0089]

[0090] Among them, G(x, y) represents the grayscale gradient of the pixel point (x, y), G x (x, y) represents the directional gradient of the pixel point (x, y) in the horizontal direction, G y (x,y) represents the directional gradient of the pixel point (x,y) in the ordinate direction.

[0091] In an embodiment of the present invention, by summing the absolute values ​​of the grayscale gradients of each pixel point of the video frame in the horizontal and vertical directions, the clarity value of the entire video frame can be obtained, and by selecting video frames whose clarity values ​​are greater than a preset threshold as target video frames, the target video frames are sorted according to their appearance time in the video to be edited to obtain a target video frame sequence.

[0092] In the embodiment of the present invention, the step of performing image enhancement on the target video frame sequence to obtain an enhanced image frame sequence includes:

[0093] Convert each target video frame in the target video frame sequence into a brightness color space to obtain a hue component, a saturation component, and a brightness component;

[0094] Decomposing the brightness component to obtain an illumination image and a reflection image, and dividing the illumination image into blocks to obtain illumination image blocks;

[0095] Performing histogram equalization on the illumination image blocks respectively to obtain enhanced illumination images;

[0096] Calculating directional gradients of the reflected image in multiple directions, and enhancing the reflected image according to the directional gradients to obtain an enhanced reflected image;

[0097] Performing inverse image decomposition on the enhanced illumination image and the enhanced reflection image to obtain an enhanced brightness component;

[0098] The hue component, the saturation component and the enhanced brightness component are converted into an RGB color space to obtain an enhanced image frame sequence.

[0099] In the embodiment of the present invention, the brightness color space is to convert the target video frame into the HSI color space, wherein the HSI color space is based on three parameters of hue (Hue), saturation (Saturation) and brightness (Intensity), which is more in line with human perception of color. Therefore, the hue component, saturation component and brightness component under the hue, saturation and brightness parameters can be obtained. The brightness component and color information (hue component and saturation component) in the HSI color space are independent of each other, and the image color is not affected by the lighting conditions, but depends on the reflection characteristics of the object to the lighting. Therefore, the image can be regarded as the result of the interaction between the lighting and the reflection of the object to the lighting, and the brightness component can be decomposed into the lighting image and the reflection image through Retinex.

[0100] Furthermore, the illuminated image is divided into equal parts to obtain multiple image blocks, and histogram equalization is performed on each image block. Image enhancement can be performed locally on the illuminated image to improve the image enhancement effect. However, histogram balancing can remove noise, but the detailed information of some dark areas cannot be obtained. Therefore, the directional gradient can be used to enhance the contrast of the emitted image and enrich the texture details of the image.

[0101] The directional gradients in multiple directions are extracted by using gradient operators in multiple different directions. For example, the coordinate axis is set with the center of the reflected image as the zero point, and the directional gradients are calculated at multiple angles in the positive direction of the horizontal coordinate. For example, the directional gradients are calculated in the directions of 0 degrees, 30 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees in the positive direction of the horizontal coordinate, and then the gradient mean is calculated and fused with the reflected image to obtain an enhanced reflected image with enhanced details. The gradient operator can be a preset 3*3 matrix.

[0102] Specifically, the reflected image is enhanced using the following formula to obtain an enhanced reflected image:

[0103]

[0104] Among them, F e represents the enhanced reflection image, F represents the reflection image, M represents the total number of directional gradients, represents convolution calculation, O m represents the mth directional gradient.

[0105] In the embodiment of the present invention, image noise in the target video frame can be removed by image enhancement, and detail information of the target video frame can be enhanced to obtain a more accurate enhanced image frame sequence.

[0106] S2. Extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence, calculating a grayscale change value of each initial feature point in the initial feature point set, and screening feature points of the initial feature point set according to the grayscale change value to obtain target feature points.

[0107] In the embodiment of the present invention, the feature point refers to a point with unique attributes in the enhanced image frame, for example, a pixel point with obvious mutation in the image pixel point, thereby representing the feature information in the enhanced image frame.

[0108] In the embodiment of the present invention, the step of extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence includes:

[0109] Counting the grayscale value of each pixel in the enhanced image frame, and calculating the grayscale value extreme value according to the grayscale value;

[0110] Calculate a characteristic gray value threshold according to the gray value extreme value;

[0111] A neighborhood point of each pixel point is selected using a preset neighborhood boundary, and an initial feature point set is determined according to the grayscale value of the neighborhood point and the grayscale value threshold.

[0112] In an embodiment of the present invention, the grayscale value extreme values ​​include high-brightness extreme values ​​calculated from multiple pixels with the largest grayscale values ​​and low-brightness extreme values ​​calculated from multiple pixels with the smallest grayscale values. For example, the grayscale value average of the 20 pixels with the largest grayscale values ​​is selected as the high-brightness extreme value, and the grayscale value average of the 20 pixels with the smallest grayscale values ​​is selected as the low-brightness extreme value.

[0113] Furthermore, the characteristic gray value threshold is calculated using the following formula:

[0114]

[0115] Wherein, T represents the characteristic gray value threshold, I1 represents the high brightness extreme value in the gray value extreme value, I2 represents the low brightness extreme value in the gray value extreme value, and N represents the total number of pixels in the calculated gray value extreme value.

[0116] Furthermore, neighborhood points can be selected using a circle with a radius of 3 as the neighborhood boundary. For example, 16 pixels are selected as neighborhood points by a circle with a radius of 3. When the grayscale value of a number of neighborhood pixels greater than a preset threshold in the neighborhood points is greater than the sum of the grayscale value of the pixel point and the feature grayscale value threshold, the pixel point corresponding to the neighborhood point is the initial feature point. However, since the initial feature point is a sparsely extracted feature point, the accuracy of the initial feature point selection cannot be guaranteed. Therefore, the initial feature points need to be further screened to obtain the target feature points.

[0117] In the embodiment of the present invention, refer to Figure 2As shown, the step of calculating the grayscale change value of each initial feature point in the initial feature point set includes:

[0118] S21, using a preset window to select a neighborhood feature point set of each initial feature point in the initial feature point set;

[0119] S22, calculating the pixel gradient of the pixel value of each neighborhood feature point in the neighborhood feature point set, and constructing a feature matrix according to the pixel gradient;

[0120] S23, calculating the matrix determinant and the matrix rank of the feature matrix, and calculating the grayscale change value of each of the initial feature points according to the matrix determinant and the matrix rank.

[0121] In an embodiment of the present invention, a preset sliding window can be translated up, down, left, and right in the horizontal and vertical directions to obtain a set of neighborhood feature points, and the Sobel operator or other differential operators are used to calculate the pixel gradient of each neighborhood feature point in the horizontal and vertical directions, thereby constructing a feature matrix based on the pixel gradients.

[0122] Specifically, the feature matrix is ​​constructed using the following formula:

[0123]

[0124] Among them, M represents the feature matrix, D x Represents the pixel gradient of the neighborhood feature point in the horizontal direction, D y The pixel gradient of the neighborhood feature point in the vertical direction.

[0125] Furthermore, the matrix determinant is a specific value. For a 2x2 matrix, the matrix determinant is the difference of the products of the diagonal elements. The matrix rank represents the maximum number of linearly independent rows or columns in the characteristic matrix.

[0126] Specifically, the grayscale change value of each initial feature point is calculated using the following formula:

[0127] R = det(M) - k[t(M)] 2

[0128] Among them, R represents the grayscale change value, det(M) represents the matrix determinant of the feature matrix M, k represents the preset adjustment parameter, and t(M) represents the matrix rank of the feature matrix M.

[0129] Furthermore, if the grayscale change value is greater than the preset change value threshold, it means that the initial feature point is the target feature point. The grayscale change value can be used to effectively identify the change in pixel values ​​in the local area, so that the target feature point with more significant pixel value changes can be selected, thereby improving the accuracy of subsequent scene segmentation.

[0130] S3. Divide the enhanced image frame sequence into scenes according to the target feature points to obtain a scene image frame set, and extract scene audio corresponding to the scene image frame set.

[0131] In the embodiment of the present invention, scene division is to identify the scene in each enhanced image in the enhanced image sequence according to the target feature points. For example, the enhanced image may be a scene of a person giving a speech or talking or a specific location object.

[0132] In the embodiment of the present invention, refer to Figure 3 As shown, the enhanced image frame sequence is divided into scenes according to the target feature points to obtain a scene image frame set, including:

[0133] S31, dividing the image region of the corresponding enhanced image frame in the enhanced image frame sequence according to the target feature point to obtain a feature point region;

[0134] S32, extracting regional features of the feature point area, and performing feature splicing on the regional features to obtain video frame features;

[0135] S33, calculating the scene similarity between the video frame feature and the preset scene feature, and determining the enhanced image frame scene corresponding to each enhanced image frame in the enhanced image frame sequence according to the scene similarity;

[0136] S34, dividing the enhanced image frames in the enhanced image frame sequence that have the same scene and are adjacent to each other to obtain a scene image frame set.

[0137] In an embodiment of the present invention, an image area of ​​a preset size can be extracted in the enhanced image frame with the target feature point as the center to obtain a feature point area, and then a pre-built neural network is used to extract image features of each feature point area to obtain regional features, and video frame features are obtained through the regional features.

[0138] Furthermore, the scene feature is a feature corresponding to multiple pre-constructed image scenes. By calculating the scene similarity through the video frame features, the overlap between the enhanced image frame and the image scene can be calculated, and the image scene corresponding to the scene feature with the largest scene similarity is selected as the enhanced image frame scene.

[0139] Among them, by extracting enhanced image frames with the same and adjacent enhanced image frames as a scene image frame set, the scene switching in the video to be edited can be determined, for example, switching from a classroom to a playground, which can improve the accuracy of subsequent editing strategies.

[0140] In an embodiment of the present invention, a preset speech extraction tool can be used to extract video speech in the video to be edited, and the video speech can be cropped through the timeline according to the timeline of the scene image frame set in the video to be edited to obtain the scene speech corresponding to each scene image frame set.

[0141] S4. Extract multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio, and calculate a scene description of the scene image frame set according to the multimodal temporal features.

[0142] In the embodiment of the present invention, the multimodal temporal features are video temporal features in a set of scene image frames and semantic temporal features corresponding to scene speech, and a multimodal temporal feature containing multimodal feature information can be obtained.

[0143] In the embodiment of the present invention, refer to Figure 4 As shown, the step of extracting the multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio comprises:

[0144] S41, calculating an image frame scene action set corresponding to the scene image frame set;

[0145] S42, performing position encoding and feature activation on the image frame scene actions in the image frame scene action set to obtain image frame temporal features;

[0146] S43, calculating an audio feature sequence of the scene audio, and performing average pooling on the audio feature sequence to obtain a target audio feature sequence;

[0147] S44, calculating the semantic similarity between each target audio feature in the target audio feature sequence and the preset speech scene action feature to obtain a semantic temporal feature;

[0148] S45, performing feature fusion on the image frame temporal features and the semantic temporal features to obtain multimodal temporal features.

[0149] In an embodiment of the present invention, the image frame scene action set is a set of specific action proposals appearing in each scene image frame. For example, in the scene image frame set, a user is waving his arms, running, or looking at static objects. Specifically, the video features in the image frame scene set can be extracted through a pre-built two-stream network, and then the pre-trained BMN (Bottom-up Multi-person Action Detection) is used to detect the action categories appearing in the scene image frames to obtain the image frame scene action set.

[0150] Furthermore, position encoding is to encode the position of the scene action, and then use the tanh nonlinear activation function and the fully connected layer to embed the scene action of the action-encoded image frame into the high-dimensional feature space to obtain the temporal features of the image frame.

[0151] In an embodiment of the present invention, an audio feature sequence corresponding to the scene audio speech signal is extracted, and then average pooling is performed to obtain a target audio feature sequence expressed by video-level features. The semantic similarity between each target audio feature in the target audio feature sequence and the preset speech scene action feature is calculated through the scaled point product attention mechanism to obtain the semantic information corresponding to the scene audio, that is, the semantic temporal feature.

[0152] Specifically, the fusing the image frame temporal features with the semantic temporal features to obtain multimodal temporal features includes:

[0153] Fully connecting the image frame temporal features and the semantic temporal features respectively to obtain fully connected features;

[0154] Adding the image frame temporal features and the semantic temporal features element by element to obtain an added feature sequence;

[0155] The added feature sequence and the fully connected feature are concatenated column by column to obtain a multimodal time series feature.

[0156] In an embodiment of the present invention, the image frame timing features and the semantic timing features can be converted into one-dimensional fully connected features through full connection. By splicing the additive feature sequence with the fully connected features by column, the fusion of the additive feature sequence and the fully connected features can be achieved to obtain multimodal timing features.

[0157] In the embodiment of the present invention, the scene description is a text description of the scene, action and emotion in the scene image frame set, so as to more intuitively obtain the information contained in the scene image frame set and improve the accuracy of video editing.

[0158] In an embodiment of the present invention, the multimodal temporal features can be encoded through a fully connected layer and then passed through an activation function to obtain a word prediction probability vector, thereby generating a text of the scene description.

[0159] In the embodiment of the present invention, by calculating the scene description of the scene image frame set through multimodal temporal features, more accurate and intuitive information contained in the scene image frame set can be obtained through multimodal feature information, thereby improving the accuracy of video editing.

[0160] S5. Generate an editing strategy for the video to be edited according to the scene description, and edit the video to be edited according to the editing strategy to obtain an edited video.

[0161] In the embodiment of the present invention, the editing strategy is a method for editing the video to be edited, including strategies such as the segments to be retained in the video frames to be edited, the order and duration of different segments, and the emotional tone of the video.

[0162] In the embodiment of the present invention, the step of generating the editing strategy of the video to be edited according to the scene description includes:

[0163] Calculating a video theme corresponding to the video to be edited according to the scene description;

[0164] Calculating the semantic similarity between the video theme and the scene description, and determining the editing order and editing time of the scene description according to the semantic similarity;

[0165] An editing strategy is generated according to the editing sequence and the editing time.

[0166] In an embodiment of the present invention, a video summary can be generated based on the keywords in each scene description, and the video theme can be determined through the video summary. The keywords can be determined based on the frequency of occurrence of the words in the scene description, and the context information of the keywords can be obtained through the self-attention layer through the keywords, so that the video theme can be determined.

[0167] Furthermore, the fit between the scene description and the video theme is calculated by semantic similarity, and then the time sequence and time when each scene description should appear are determined, wherein the semantic similarity includes the semantic similarity between the scene description and the video theme, and the similarity of the occurrence order of the scene description in the video theme. For example, the greater the semantic similarity, the longer the editing time; the greatest similarity between the semantic features of the scene description and the early features of the video theme indicates that the scene description is in the early stage of the video theme, and the editing order corresponding to the scene description should be in the front, and the editing order of each scene description is obtained. Specifically, the semantic similarity can be calculated based on the distance between the feature vectors of the video theme and the scene description.

[0168] Furthermore, video clips may be taken from the video to be edited according to the scene image frame set corresponding to the scene description, and then the video clips may be spliced ​​according to the editing strategy to obtain a final edited video.

[0169] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0170] The present invention also provides an electronic device, which may include a processor, a memory, a communication bus and a communication interface, and may also include a computer program stored in the memory and executable on the processor, such as a method program for improving welding stability of heterogeneous titanium alloy laser welding technology.

[0171] In some embodiments, the processor may be composed of an integrated circuit, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and a combination of various control chips.

[0172] The memory includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device.

[0173] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory and at least one processor, etc.

[0174] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0175] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor through a power management system, so that the power management system can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0176] Specifically, the specific implementation method of the processor for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0177] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0178] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0179] Obtain a video to be edited, extract a target video frame sequence from the video to be edited according to the image clarity of each video frame in the video to be edited, and perform image enhancement on the target video frame sequence to obtain an enhanced image frame sequence; wherein, performing image enhancement on the target video frame sequence to obtain an enhanced image frame sequence includes: converting each target video frame in the target video frame sequence to a luminance color space to obtain a hue component, a saturation component, and a luminance component; performing image decomposition on the luminance component to obtain an illumination image and a reflection image, and dividing the illumination image into blocks to obtain illumination image blocks; performing histogram equalization on the illumination image blocks respectively to obtain an enhanced illumination image; calculating directional gradients of the reflection image in multiple directions, and enhancing the reflection image according to the directional gradients to obtain an enhanced reflection image;

[0180] The reflected image is enhanced using the following formula to obtain an enhanced reflected image:

[0181]

[0182] Among them, F e represents the enhanced reflection image, F represents the reflection image, M represents the total number of directional gradients, represents convolution calculation, O m represents the mth directional gradient;

[0183] Performing inverse image decomposition on the enhanced illumination image and the enhanced reflection image to obtain an enhanced brightness component; converting the hue component, the saturation component and the enhanced brightness component into an RGB color space to obtain an enhanced image frame sequence;

[0184] Extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence, calculating a grayscale change value of each initial feature point in the initial feature point set, and screening feature points of the initial feature point set according to the grayscale change value to obtain target feature points;

[0185] Dividing the enhanced image frame sequence into scenes according to the target feature points to obtain a scene image frame set, and extracting scene audio corresponding to the scene image frame set;

[0186] Extracting multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio, and calculating a scene description of the scene image frame set according to the multimodal temporal features;

[0187] An editing strategy for the video to be edited is generated according to the scene description, and the video to be edited is edited according to the editing strategy to obtain an edited video.

[0188] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0189] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0191] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0192] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0193] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0194] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for automatic video editing based on deep learning, characterized in that: The method comprises: Obtain a video to be edited, extract a target video frame sequence from the video to be edited according to the image clarity of each video frame in the video to be edited, and perform image enhancement on the target video frame sequence to obtain an enhanced image frame sequence; wherein, performing image enhancement on the target video frame sequence to obtain an enhanced image frame sequence includes: converting each target video frame in the target video frame sequence to a luminance color space to obtain a hue component, a saturation component, and a luminance component; performing image decomposition on the luminance component to obtain an illumination image and a reflection image, and dividing the illumination image into blocks to obtain illumination image blocks; performing histogram equalization on the illumination image blocks respectively to obtain an enhanced illumination image; calculating directional gradients of the reflection image in multiple directions, and enhancing the reflection image according to the directional gradients to obtain an enhanced reflection image; The reflected image is enhanced using the following formula to obtain an enhanced reflected image: Among them, F e represents the enhanced reflection image, F represents the reflection image, M represents the total number of directional gradients, represents the convolution calculation, O m represents the mth directional gradient; Performing inverse image decomposition on the enhanced illumination image and the enhanced reflection image to obtain an enhanced brightness component; converting the hue component, the saturation component and the enhanced brightness component into an RGB color space to obtain an enhanced image frame sequence; Extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence, calculating a grayscale change value of each initial feature point in the initial feature point set, and screening feature points of the initial feature point set according to the grayscale change value to obtain target feature points; Dividing the enhanced image frame sequence into scenes according to the target feature points to obtain a scene image frame set, and extracting scene audio corresponding to the scene image frame set; Extracting multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio, and calculating a scene description of the scene image frame set according to the multimodal temporal features; An editing strategy for the video to be edited is generated according to the scene description, and the video to be edited is edited according to the editing strategy to obtain an edited video.

2. The deep learning-based automatic video editing method according to claim 1, characterized in that: The step of extracting a target video frame sequence from the video to be edited according to the image definition of each video frame in the video to be edited comprises: Calculate the directional gradient of each pixel point in the video frame in the coordinate direction; Calculate the grayscale gradient of each pixel in the video frame according to the directional gradient; The grayscale gradient of each pixel is calculated using the following formula: Among them, G(x,y) represents the grayscale gradient of the pixel point (x,y), G x (x, y) represents the directional gradient of the pixel point (x, y) in the horizontal coordinate direction, G y (x,y) represents the directional gradient of the pixel point (x,y) in the ordinate direction; The definition value of the video frame is calculated according to the grayscale gradient, and a target video frame sequence is determined from the video frame according to the definition value.

3. The deep learning-based automatic video editing method according to claim 2, characterized in that: The calculating the directional gradient of each pixel point in the video frame in the coordinate direction includes: The directional gradient of each pixel in the video frame in the coordinate direction is calculated using the following formula: G x (x,y)=g(x-1,y+1)+g(x+1,y+1)+2g(x,y+1)-g(x-1,y-1)+g(x+1,y-1)-2g(x,y-1) G y (x,y)=g(x+1,y-1)+g(x+1,y+1)+2g(x+1,y)-g(x-1,y-1)-g(x-1,y+1)-2g(x-1,y) Among them, G x (x,y) represents the directional gradient of the pixel point (x,y) in the horizontal coordinate direction, g(x-1,y+1) represents the grayscale value of the pixel point (x-1,y+1), g(x+1,y+1) represents the grayscale value of the pixel point g(x+1,y+1), g(x-1,y-1) represents the grayscale value of the pixel point g(x-1,y-1), (x+1,y-1) represents the grayscale value of the pixel point (x+1,y-1), g(x,y-1) represents the grayscale value of the pixel point g(x,y-1), G y (x,y) represents the directional gradient of the pixel point (x,y) in the ordinate direction, (x+1,y) represents the grayscale value of the pixel point (x+1,y), and (x-1,y) represents the grayscale value of the pixel point (x-1,y).

4. The deep learning-based automatic video editing method according to claim 1, characterized in that: The extracting an initial feature point set of each enhanced image frame in the enhanced image frame sequence comprises: Counting the grayscale value of each pixel in the enhanced image frame, and calculating the grayscale value extreme value according to the grayscale value; and calculating the characteristic grayscale value threshold according to the grayscale value extreme value; The following formula is used to calculate the feature gray value threshold: Wherein, T represents the characteristic gray value threshold, I1 represents the high brightness extreme value in the gray value extreme value, I2 represents the low brightness extreme value in the gray value extreme value, and N represents the total number of pixels in the calculated gray value extreme value; A neighborhood point of each pixel point is selected using a preset neighborhood boundary, and an initial feature point set is determined according to the grayscale value of the neighborhood point and the grayscale value threshold.

5. The deep learning-based automatic video editing method according to claim 1, characterized in that: The calculating the grayscale change value of each initial feature point in the initial feature point set includes: Selecting a neighborhood feature point set of each initial feature point in the initial feature point set using a preset window; Calculating the pixel gradient of the pixel value of each neighborhood feature point in the neighborhood feature point set, and constructing a feature matrix according to the pixel gradient; The feature matrix is ​​constructed using the following formula: Among them, M represents the feature matrix, D x Represents the pixel gradient of the neighborhood feature point in the horizontal direction, D y Pixel gradient of neighborhood feature points in the vertical direction; Calculating the matrix determinant and the matrix rank of the feature matrix, and calculating the grayscale change value of each of the initial feature points according to the matrix determinant and the matrix rank; The grayscale change value of each initial feature point is calculated using the following formula: R=det(M)-k[t(M)] 2 Among them, R represents the grayscale change value, det(M) represents the matrix determinant of the feature matrix M, k represents the preset adjustment parameter, and t(M) represents the matrix rank of the feature matrix M.

6. The method for automatic video editing based on deep learning as claimed in claim 1, characterized in that: The step of dividing the enhanced image frame sequence into scenes according to the target feature points to obtain a scene image frame set includes: Dividing the image region of the corresponding enhanced image frame in the enhanced image frame sequence according to the target feature point to obtain a feature point region; Extracting regional features of the feature point area, and performing feature splicing on the regional features to obtain video frame features; Calculating the scene similarity between the video frame feature and the preset scene feature, and determining the enhanced image frame scene corresponding to each enhanced image frame in the enhanced image frame sequence according to the scene similarity; The enhanced image frames with the same enhanced image frame scene and adjacent to each other in the enhanced image frame sequence are divided to obtain a scene image frame set.

7. The method for automatic video editing based on deep learning as claimed in claim 1, characterized in that: The extracting multimodal temporal features of the scene image frame set according to the scene image frame set and the scene audio comprises: Calculating an image frame scene action set corresponding to the scene image frame set; Performing position encoding and feature activation on the image frame scene actions in the image frame scene action set to obtain image frame temporal features; Calculating an audio feature sequence of the scene audio, and performing average pooling on the audio feature sequence to obtain a target audio feature sequence; Calculating the semantic similarity between each target audio feature in the target audio feature sequence and the preset speech scene action feature to obtain a semantic temporal feature; The image frame temporal features and the semantic temporal features are fused to obtain multimodal temporal features.

8. The deep learning-based automatic video editing method according to claim 7, characterized in that: The step of fusing the image frame temporal features with the semantic temporal features to obtain multimodal temporal features includes: Fully connecting the image frame temporal features and the semantic temporal features respectively to obtain fully connected features; Adding the image frame temporal features and the semantic temporal features element by element to obtain an added feature sequence; The added feature sequence and the fully connected feature are concatenated column by column to obtain a multimodal time series feature.

9. The method for automatic video editing based on deep learning as claimed in claim 1, characterized in that: The step of generating the editing strategy of the video to be edited according to the scene description includes: Calculating a video theme corresponding to the video to be edited according to the scene description; Calculating the semantic similarity between the video theme and the scene description, and determining the editing order and editing time of the scene description according to the semantic similarity; An editing strategy is generated according to the editing sequence and the editing time.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the deep learning-based automatic video editing method as described in any one of claims 1 to 9 is implemented.