Video intelligent editing method and device based on AI technology, and terminal

Abstract

The invention discloses an intelligent video editing method and device based on an AI technology and a terminal. The method comprises the following steps: extracting a target face of a target picture from the target picture; extracting each frame of face of the video source; performing similarity comparison on each frame of face of the video source and a target face of the target picture, finding out key frames of the video source of which the similarity reaches a preset threshold value, and determining a time point when a target person appears in the key frames; determining the starting time and the ending time of the video source clip needing to be clipped, and clipping the video clip; and splicing the edited video clips in sequence according to a time sequence, compensating a transition effect at frame splicing positions of discontinuous time by using an artificial intelligence technology, and synthesizing a final target video file. According to the method and the device, the corresponding target person can be searched from massive video resources through person recognition, and finally, the personal album of the exclusive person is formed through automatic editing, so that the editing efficiency is greatly improved, and convenience is provided for the use of a user.

Description

Technical Field

[0001] The present invention relates to the field of video editing technology, and in particular to an AI-based video intelligent editing method, device, intelligent terminal and storage medium. Background Art

[0002] With the development of technology, the rise of social media and the popularity of mobile devices, massive amounts of video data are generated every day.

[0003] Sometimes people want to keep some videos they are interested in, but they often don't need every clip in the video. Maybe they just want to see the footage of their children, or only pay attention to the programs starring their favorite stars, or they may need to use the footage of some specific people to make a new video.

[0004] However, in the prior art, if a user wants to create a new video from massive video resources using shots of certain people, such as only watching programs starring their favorite stars or saving shots of their children, they can only do so by manually editing them one by one, which is time-consuming and laborious.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the problems and defects of the above-mentioned prior art, a video intelligent editing method, device, intelligent terminal and storage medium based on AI technology are provided. The present invention can realize the search for the corresponding target person from massive video resources through character recognition, and finally automatically edit to form a personal album of the exclusive person, which greatly improves the editing efficiency and provides convenience for users.

[0007] The technical solution adopted by the present invention to solve the problem is as follows:

[0008] A video intelligent editing method based on AI technology, including:

[0009] Obtain and determine the target picture with the target person that needs to be edited, and obtain the video source that needs to be edited;

[0010] Performing face recognition detection on the target person in the target image to extract the target face of the target image;

[0011] Recognize the face of each frame of the video source, and extract the face of each frame of the video source;

[0012] Compare the similarity of each face frame in the video source with the target face in the target image, find the key frame of the video source whose similarity reaches a predetermined threshold, and determine the time point when the target person appears in the key frame;

[0013] Based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited;

[0014] The edited video clips are spliced ​​in chronological order, and artificial intelligence technology is used to compensate for the transition effect at the splicing of frames in discontinuous time to synthesize the final target video file.

[0015] The AI-based video intelligent editing method, wherein the steps of obtaining and determining a target picture with a target person to be edited, and obtaining a video source to be edited include:

[0016] Receive the user's operation instruction to select the target picture with the target person to be edited;

[0017] Receive the user's operation instruction to select the video source that needs to be edited;

[0018] Load the target image and video source described by source.

[0019] The AI-based video intelligent editing method, wherein the step of performing face recognition detection on the target person in the target image and extracting the target face of the target image comprises:

[0020] For the target person in the target image, a face detection algorithm is used to locate the position of the target person's face, and a feature vector of the target person's face is extracted.

[0021] The AI-based video intelligent editing method, wherein the step of recognizing the face of each frame of the video source and extracting the face of each frame of the video source comprises:

[0022] The face of each frame image in the video source is recognized, the position of the face in each frame image is located using a face detection algorithm, and the feature vector of the face in each frame image is extracted.

[0023] The AI-based video intelligent editing method, wherein the similarity comparison of each face frame of the video source with the target face of the target picture is performed to find the key frame of the video source whose similarity reaches a predetermined threshold, comprises:

[0024] Compare the feature vector of the face of each frame extracted from the video source with the feature vector of the target face extracted;

[0025] Calculate the similarity between the feature vector of the face in each frame image and the feature vector of the target face;

[0026] When the similarity between the feature vector of the face in the frame image extracted from the video source and the feature vector of the target face is greater than a predetermined threshold, it is determined that the face features match;

[0027] The frames with matching facial features are marked as key frames containing the target person in the video source.

[0028] In the AI-based video intelligent editing method, the step of determining the time point at which the target person appears in the key frame comprises:

[0029] The key frames marked as containing the target person in the video source are analyzed to determine the time point when the target person appears.

[0030] The method for intelligent video editing based on AI technology, wherein the step of determining the start and end time of the video source segment to be edited based on the key frame of the video source found and the time point when the target person appears in the key frame, and editing the video segment includes:

[0031] Based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited using the video editing library.

[0032] A video intelligent editing device based on AI technology, wherein the device comprises:

[0033] The acquisition module is used to acquire and determine the target picture with the target person that needs to be edited, and to acquire the video source that needs to be edited;

[0034] An image recognition module is used to perform face recognition detection on a target person in the target image and extract the target face of the target image;

[0035] A video frame recognition module, used to recognize the face of each frame image in the video source and extract the face of each frame of the video source;

[0036] A similarity comparison module is used to compare the similarity of each face frame of the video source with the target face of the target image, find the key frame of the video source whose similarity reaches a predetermined threshold, and determine the time point when the target person appears in the key frame;

[0037] The video clip editing module is used to determine the start and end time of the video source clip to be edited based on the key frames of the video source found and the time points when the target person appears in the key frames, and perform video clip editing;

[0038] The video synthesis module is used to splice the edited video clips in chronological order, and use artificial intelligence technology to compensate for the transition effect at the splicing points of frames in discontinuous time to synthesize the final target video file.

[0039] An intelligent terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.

[0040] A computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described above.

[0041] Beneficial effects of the invention: The invention provides a method, device, intelligent terminal and storage medium for intelligent video editing based on AI technology. The invention can find a person in multiple complex video scenes, and automatically identify and extract all clips of the person to form a new video album. This process combines video segmentation, object detection, face recognition, object tracking, video editing and video synthesis technology; it searches for the corresponding target person through character recognition from massive video resources, and finally automatically edits to form a personal album for the exclusive person, which greatly improves the editing efficiency and provides convenience for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flowchart of the video intelligent editing method based on AI technology provided in Example 1 of the present invention.

[0044] Figure 2 It is a flowchart of the video intelligent editing method based on AI technology provided in Example 2 of the present invention.

[0045] Figure 3 A functional block diagram of an embodiment of an intelligent video editing device based on AI technology provided by the present invention.

[0046] Figure 4 It is a block diagram of the internal structure principle of the intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0049] With the development of technology, a huge amount of video data is generated every day, but people often do not need every clip in the video. Maybe you just want to see the shots of your children, or only pay attention to the programs starring your favorite stars. You may also need to use the shots of some specific people to make new videos, find the corresponding target person through character recognition from the massive video resources, and finally form a personal album for the exclusive person. To this end, an embodiment of the present invention provides a method for completing intelligent video editing based on AI technology.

[0050] like Figure 1 As shown, a video intelligent editing method based on AI technology in Embodiment 1 of the present invention includes the following steps:

[0051] Step S100, obtaining and determining a target picture with a target person that needs to be edited, and obtaining a video source that needs to be edited;

[0052] In the video editing of the embodiment of the present invention, it is first necessary to obtain and determine the image material and video source with the target face.

[0053] In the embodiment of the present invention, first, it is necessary to clarify who the target person to be edited is, which can be a shot of one's own child, or a celebrity, other family members, or a specific character. Then, target pictures with the target person are collected; clear and appropriate pictures of the target person can be obtained through pictures stored by oneself, or from social media, online galleries, or personal storage devices. These pictures can be static portraits, activity photos, or pictures related to specific situations.

[0054] Then, you can filter out the collected pictures that are high-quality, have clean backgrounds, and are visually attractive and meet your editing needs.

[0055] As for obtaining the video source, it is to select the video source part that needs to be cut, such as the highlights of an event, a character interview, or a daily life scene. The video source in this embodiment can come from multiple places, including newly shot videos, existing movie clips, online video platforms (such as YouTube, Vimeo), and private video archives.

[0056] Wherein, the step S100 specifically includes:

[0057] S101, receiving an operation instruction from a user to select a target picture with a target person to be edited;

[0058] That is, in the embodiment of the present invention, the user can first select the target picture with the target person to be edited. The system of the embodiment of the present invention receives the user's instruction to select the picture of the target person, and the user can select a specific file or enter a search condition through the interface to ensure that the selected picture contains the specific person.

[0059] S102, receiving a user's operation instruction to select a video source that needs to be edited;

[0060] In this step embodiment, the system also receives the video source that the user selects to be edited, such as local storage, cloud service or third-party platform, and then proceeds to step S103.

[0061] S103: Load the target image and video source.

[0062] Step S200, performing face recognition detection on the target person in the target image, and extracting the target face of the target image;

[0063] In this step, face recognition technology can be used to analyze the target image to identify the face area in the image. The face can be accurately located in the image and framed by algorithms (such as convolutional neural networks, CNN).

[0064] In the embodiment of the present invention, once the target face is detected, it will be cut out from the background containing other elements to form a separate facial image. The extracted target face can be saved as a separate file for subsequent video editing and processing.

[0065] In the embodiment of the present invention, the face detection algorithm can be used to locate the face of the target person in the target image, and the feature vector of the target face can be extracted. The feature vector is a fixed-length numerical array, which reflects the characteristics of the target face, such as facial contour, eye shape, nose size, lip thickness, etc., so as to facilitate the comparison of the similarity of the target face in the subsequent editing step.

[0066] Step S300, identifying the face of each frame of the video source, and extracting the face of each frame of the video source;

[0067] In the embodiment of this step, the face of each frame image in the video source is recognized, the position of the face in each frame image is located using a face detection algorithm, and the feature vector of the face in each frame image is extracted.

[0068] In the embodiment of the present invention, specifically, first, the video source to be edited is split into separate frame images. The video is composed of fast and continuous static images, so each frame is an independent image. And the face recognition algorithm is applied to each frame image for cyclic processing.

[0069] Regarding face detection, a convolutional neural network (CNN) or other detection algorithms (such as the Haar cascade classifier) ​​is used to analyze each frame and identify the face area in the image. The algorithm finds the location of each face in the image and marks it with a rectangular box.

[0070] If there are multiple faces in a certain frame of image, the face detection algorithm of the embodiment of the present invention can detect all faces at the same time.

[0071] Then extract the feature vector of the face in each frame of the image; the specific process is: first standardize the detected face area (such as scaling and normalization) to ensure the consistency of the input data. Then use the deep learning model to extract the feature vector of the face. The feature vector is a fixed-length numerical sequence, such as 128 dimensions.

[0072] In the embodiment of the present invention, the extracted feature vectors may be stored in a database or used for subsequent processing to facilitate matching, comparison or analysis.

[0073] The benefit of this step is that the facial feature vectors provide data that can be used for subsequent operations. These vectors allow for a variety of analyses, such as face similarity comparisons. And because feature vectors are numerical data, they can be easily stored, managed, and queried, allowing the system to quickly expand in new application scenarios and integrate with other data sources.

[0074] For example, if you want to edit a video clip containing a child in a surveillance video stream, then in this step, you will analyze the surveillance video stream frame by frame, possibly generating 30 frames per second, and perform face detection.

[0075] If three faces are detected in a certain frame, the face detection algorithm used in the embodiment of the present invention will accurately frame the facial area of ​​each person (for example, it can be marked with a red rectangle). Then, feature extraction is performed on the face area, and three feature vectors are extracted for each face area, for example:

[0076] Character A’s feature vector: [0.25, -0.14, 0.80, ..., 0.03];

[0077] Character B feature vector: [0.30, -0.30, 0.60, ..., 0.05];

[0078] Character C feature vector: [0.28, -0.15, 0.75, ..., 0.04];

[0079] In this way, the feature vector automatically extracted from the video source to be edited can be matched with the target face feature vector in the target image for similarity in the subsequent steps.

[0080] It can be seen that in the embodiment of the present invention, face recognition, detection and feature vector extraction for each frame of the video source are important steps to achieve efficient video analysis. This process not only improves the real-time and accuracy of face recognition, but also provides support for target face similarity comparison in the subsequent video editing process.

[0081] Step S400, comparing the similarity of each face frame of the video source with the target face of the target image, finding the key frame of the video source whose similarity reaches a predetermined threshold, and determining the time point when the target person appears in the key frame;

[0082] In this step, each face frame in the video source is compared with the target image for similarity, and the key frame and time point of the video clip are found. By comparing the similarity of each face frame in the video stream with the face in the predefined target image, the key frame that needs to be video clipped is identified, and the time point when the target person appears is determined.

[0083] Specifically, the present invention is about the similarity comparison process of each frame of face:

[0084] Feature extraction: First, face detection is performed on each frame in the video source, and the feature vector of each face detected in each frame is extracted.

[0085] Target face feature vector: Similarly, extract the feature vector of the target face from the target image as a comparison benchmark.

[0086] Then, similarity calculation is performed, specifically, using a similarity measurement method (such as Euclidean distance, cosine similarity, etc.) to calculate the similarity between each frame face feature vector and the target face feature vector.

[0087] The process of finding the key frames whose similarity reaches a predetermined threshold in the embodiment of the present invention is as follows:

[0088] A similarity threshold is set, that is, in the embodiment of the present invention, a similarity threshold (eg, 0.8) is defined before specific implementation, and only when the calculated similarity is higher than the threshold is the face in the frame considered to be similar to the target face.

[0089] Regarding key frame recognition, in the embodiment of the present invention, all frames with faces are traversed, and frames with similarity higher than a threshold are marked as key frames to facilitate subsequent video editing, and the time points of these frames in the video are recorded.

[0090] In the embodiment of the present invention, the process of determining the time point at which the target person appears in the key frame is as follows:

[0091] By recording by timestamp, that is, in the embodiment of the present invention, for each image marked as a key frame, its timestamp in the video is recorded, which is usually expressed in the form of seconds or frame numbers.

[0092] Output results: The present invention will sort out all key frames and their corresponding time points to facilitate subsequent video editing and merging.

[0093] For example, taking a security surveillance video as an example, the goal is to identify the growth movements of a child in the surveillance video. The specific process is as follows:

[0094] Taking the acquisition of security surveillance video as an example, the target face to be recognized is set. Specifically, a picture with a child as the target face can be used as the target picture.

[0095] Frame-by-frame analysis: The present invention performs face detection on each frame in the video and extracts the feature vector of each detected face.

[0096] Similarity comparison: For each face detected in a frame, the present invention calculates the similarity between its feature vector and the target face feature vector.

[0097] Key frame identification: When the similarity between the detected face and the target face in the frames of the 5th, 12th and 30th seconds is 0.85, 0.75 and 0.82 respectively, only the similarity of the 5th and 30th seconds exceeds the predetermined threshold of 0.8, so these two frames are marked as key frames.

[0098] Time point recording: The present invention will record the 5th second and the 30th second as the time points when a child appears, and output the results to facilitate video editing and saving in subsequent steps.

[0099] In the present invention, the similarity comparison between each face frame in the video source and the target image is performed to find the key frame and its time point, which is an important recognition and analysis process. This method not only improves the recognition efficiency of the target person, but also provides a basis for subsequent data processing and analysis.

[0100] In the embodiment of the present invention, by setting a similarity threshold, frames similar to the target face can be quickly screened out, reducing the complexity of subsequent processing. In addition, the time point when the target person appears in the video is determined, so that the user can quickly find the relevant scene and improve the retrieval efficiency. Only key frames are processed, which reduces the computational burden, which is particularly important when processing long videos.

[0101] In a further embodiment, the present application compares the similarity of each face frame of the video source with the target face of the target image, and finds the key frame of the video source whose similarity reaches a predetermined threshold, including:

[0102] S401, comparing the feature vector of the face of each frame image extracted from the video source with the feature vector of the target face extracted;

[0103] In this step, each frame of the video is first extracted and the facial features are identified. These feature vectors are then compared with the pre-defined target face feature vectors to detect similarities.

[0104] S402, calculating the similarity between the feature vector of the face in each frame image and the feature vector of the target face;

[0105] In this step, the similarity between the extracted facial feature vector and the target facial feature vector is calculated by using an algorithm (such as cosine similarity or Euclidean distance). This process is usually based on a trained model in machine learning.

[0106] S403, when the similarity between the feature vector of the human face in the frame image extracted from the video source and the feature vector of the target face is greater than a predetermined threshold, it is determined that there is a facial feature match;

[0107] In this step, if the similarity between the facial feature vector in a frame and the target face exceeds the set threshold, the frame is judged as a "match", which means that the system believes that there is a high probability that this face and the target face are the same person.

[0108] S404: Mark the frames with matched facial features as key frames containing the target person in the video source.

[0109] In this step, all frames with detected facial feature matches are marked for subsequent operations. These marked frames can be used for further analysis, extraction, or visualization, such as creating video clips to facilitate the observation of the appearance of the target person.

[0110] In the embodiment of the present invention, the automated face matching process can quickly process videos, reduce the time for manual review, and improve overall recognition efficiency.

[0111] In the present invention, key frames marked as containing the target person in the video source are analyzed to determine the time point when the target person appears.

[0112] In this step, the key frames marked as containing the target person in the video source are analyzed: In the previous step, the system has identified and marked the key frames containing the target person's facial feature matches. At this point, these key frames need to be analyzed to extract more specific information, such as the target person's appearance duration, frequency, and position in the video.

[0113] To determine the time points when the target person to be edited appears, we can analyze the key frames to identify the time points when the target person first appears and the time points when the target person last appears. These time points will be used as the start and end marks of the edit to ensure that the edited clip only contains the relevant moments of the target person.

[0114] For example, a video surveillance company is processing a video recording of a public event to identify and extract a speaker's speech. Here is the specific operation process:

[0115] Marking key frames: In the surveillance video, the present invention marks all key frames containing the speaker through face recognition technology and records the timestamps of these frames.

[0116] Analyzing key frames: Next, the present invention analyzes these key frames to identify the first appearance time (eg, 00:05:30 of the video) and the last appearance time (eg, 00:10:15 of the video) of the speaker.

[0117] Determine the editing time point: Based on these analyses, the present invention finally determines the editing time period to be 00:05:30 to 00:10:15. In this way, the edited clip will only contain the speaker's speech content, which is convenient for subsequent sharing or publishing.

[0118] Through such a process, the present invention can efficiently extract relevant segments of the target person, improve work efficiency, and also ensure the accuracy and relevance of the content.

[0119] By automatically analyzing key frames, the present invention allows users to quickly obtain the appearance time of the target person in the video without having to manually check frame by frame, saving time and human resources. After determining the specific time point, the editing can be more accurate, ensuring that the generated video clips only contain content related to the target person, improving the relevance and usability of the video. In addition, this step can not only be used for editing, but also provide basic data for subsequent behavior analysis, event detection, etc., enhancing the intelligence level of the video analysis system.

[0120] Step S500: based on the found key frames of the video source and the time points when the target person appears in the key frames, determine the start and end times of the video source segments to be edited, and perform video segment editing;

[0121] In the embodiment of the present invention, based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited using a video editing library.

[0122] Specifically, in the previous steps, the embodiment of the present invention has identified and marked key frames in the video, which contain the appearance of the target person. Each key frame corresponds to a specific timestamp, indicating the appearance time of the target person in the video.

[0123] In addition, in the embodiment of the present invention, these key frames are analyzed to extract the first appearance time and the last appearance time of the target person. These time points will serve as the basis for editing to ensure the integrity and relevance of the editing content.

[0124] Then, based on the appearance time of the target character in the keyframe, the start and end time of the clip will be determined. For example, if the target character first appears at 00:05:30 and leaves at 00:10:15, these two time points will be set as the start and end time of the clip.

[0125] In the embodiment of the present application, once the start and end times are determined, a video editing library (such as OpenCV, FFmpeg, etc.) can be called to perform the actual editing operation. These libraries provide powerful video processing functions, which can efficiently edit video clips of a specified time period and save them as new files.

[0126] For example, taking a news organization producing a report on an important press conference as an example, the following is a specific operation flow of the method of the present invention:

[0127] In the step of identifying key frames, the key frames of the spokesperson containing the target person can be marked in the video by using face recognition technology, and the timestamps of these frames can be recorded. For example, the target spokesperson first appears at 00:02:15 and ends his speech at 00:05:45.

[0128] In the stage of determining the editing time, this embodiment analyzes the key frames and determines that the start time of the editing is 00:02:15 and the end time is 00:05:45.

[0129] Use video editing library to edit: Next, the present invention can call the video editing library, such as FFmpeg, and execute the following command:

[0130] Css

[0131] ffmpeg-iinput_video.mp4-ss 00:02:15-to 00:05:45-c copy output_clip.mp4;

[0132] This command will extract the clip from 00:02:15 to 00:05:45 from the input video and save it to a new file output_clip.mp4.

[0133] In this way, it is possible to quickly edit and generate accurate clips containing the target person's speech, which is convenient for subsequent reporting and dissemination. This not only improves work efficiency, but also ensures the accuracy and timeliness of news content.

[0134] It can be seen that in the embodiment of the present invention, the editing time point is determined in an automated manner, which saves the time of manual intervention and improves work efficiency, especially when processing long videos. And the precise time point ensures that the edited clips only contain content related to the target person, avoiding unnecessary interference information, making the final video more concise and clear.

[0135] Step S600: splice the edited video clips in chronological order, and use artificial intelligence technology to compensate for transition effects at the splicing points of discontinuous time frames to synthesize the final target video file.

[0136] In the embodiment of this step, the edited video clips are spliced ​​in chronological order, and artificial intelligence technology is used to compensate for the transition effect at the splicing of frames in discontinuous time, and finally a target video file is synthesized.

[0137] Specifically, in the previous steps, multiple clips have been generated, and these clips may be extracted according to different time points when the target person appears in the video. At this time, the present invention will arrange these clips according to their chronological order to ensure the coherence and logic of the video content.

[0138] Regarding the use of artificial intelligence technology to compensate for transition effects in the present invention, specifically, during the splicing process, a time interval may appear between two clips, causing the screen to appear abrupt when switching. In order to solve this problem, the embodiment of the present invention will apply artificial intelligence technology (such as a deep learning model) to generate transition effects. This may include:

[0139] Image Interpolation: Generates new in-between frames to smoothly connect two clips.

[0140] Audio Smoothing: Adjust the volume and effects of the audio to ensure that the sound does not change abruptly when switching.

[0141] Dynamic Adjustment: Automatically adjusts color and brightness based on changes in scene and action to enhance visual consistency.

[0142] In the embodiment of the present invention, the use of image interpolation, audio smoothing and dynamic adjustment is an important technology to ensure the final video quality.

[0143] Specifically, the image interpolation technology used in the embodiment of the present invention specifically uses artificial intelligence to generate new intermediate frames to smoothly connect two video clips. This process is implemented through computer vision algorithms, aiming to reduce visual jumps caused by cutting and make the picture transition more natural.

[0144] For example, in a short video, the camera first shows a skier rushing down a mountain, and then quickly switches to another skier jumping at a different location. In this scene, the switch between the two skiers may make the audience feel abrupt. The present invention can use image interpolation in the video editing process. The editor can generate new frames between the two clips, for example, by inserting the intermediate state of the skier turning in the air to achieve a smooth transition, as if the audience witnessed the smooth process of the entire action.

[0145] The audio smoothing adopted by the present invention is to adjust the volume and effect of the audio to ensure that the transition of the sound is natural when the visual is switched, and no abrupt changes will occur. This process includes audio fading in, fading out, and mixing technology.

[0146] For example, in a video documentary, the narrator's commentary discusses different city scenes. When the camera switches from one city to another, some new background sound effects (such as street noise) may appear. In order to make a smooth transition, the present invention can use audio smoothing technology in the video editing process to gradually reduce the background sound effects of the first city, while gradually increasing the sound effects of the second city. Through this gradual change, the sound heard by the audience will not suddenly change, but it will feel like experiencing a natural journey.

[0147] The dynamic adjustment adopted by the present invention is to automatically adjust the color and brightness of the video according to the changes in the scene and action to enhance the visual consistency. This technology can ensure that the color tone and brightness between different shots maintain a certain degree of coordination.

[0148] For example, take a promotional film about environmental protection, showing a series of shots, from barren land to vibrant forests. When switching from one shot to another, the color difference may be obvious due to different lighting conditions. The present invention will automatically make dynamic adjustments during video editing, and in later video editing, it can automatically identify the hue of each shot, and adjust the brightness, contrast and saturation, so that the transition between shots is smoother. For example, during video editing, the green hue and brightness of the forest shot can be enhanced to make it visually consistent with the previous shot, creating a coherent and harmonious viewing experience.

[0149] It can be seen that in the video editing embodiment of the present invention, image interpolation is used to make the switching of video actions smoother, audio smoothing ensures the natural transition of sound, and dynamic adjustment improves the visual effect of the entire video.

[0150] Finally, the present invention will synthesize the edited videos into a final target video file. Specifically, once all the clips are spliced ​​and the transition effect is compensated, the present invention will generate a final video file. This file will contain all the edited clips, and the transition is natural and smooth, which is suitable for display or release.

[0151] For example, in a documentary video about a sports event, you need to edit together the wonderful moments of multiple athletes in the video. The following is the specific operation process:

[0152] Clip generation: Automatically extract multiple clips containing multiple target faces from the game video, such as the moments when athletes score goals and celebrate at different times. The time points are 00:01:15 to 00:01:30, 00:02:45 to 00:03:00, and 00:04:10 to 00:04:25.

[0153] Splice in chronological order: Then arrange these clips in chronological order to form a preliminary video sequence. When splicing the video, it is detected that there is a blank time between 00:01:30 and 00:02:45. Then the present invention uses artificial intelligence technology to generate a transition effect, insert a new intermediate frame, and smoothly connect the two clips. At the same time, adjust the audio so that the cheers of the athletes' goals transition naturally when switching, and there will be no abrupt volume changes.

[0154] After the above processing, a complete sports event highlights video is generated. The file name is highlights.mp4, which contains all the wonderful moments and the transition effect is natural and smooth.

[0155] In this way, the present invention can efficiently edit high-quality videos.

[0156] It can be seen that the present invention uses artificial intelligence technology to compensate for transition effects, and the final generated video will be smoother and have better visual effects, avoiding the stiffness caused by editing. In addition, the coherent picture and natural sound transition can enhance the audience's viewing experience, making the video content easier to understand and attractive. In addition, the automated transition effect generation reduces the need for manual intervention, saves time and cost of post-production, and improves work efficiency.

[0157] The present invention is further described in detail below through a specific application example:

[0158] like Figure 2 As shown, a video intelligent editing method based on AI technology in a specific application embodiment of the present invention includes the following steps:

[0159] S31, start, and enter S32.

[0160] S32, prepare pictures and videos, and proceed to steps S33 and S35 respectively.

[0161] In this step, the target picture containing the target person to be edited and the video to be edited are prepared.

[0162] S33, detect the face in the picture and go to step S34.

[0163] That is, the face of the target person in the target image is detected and the position of the image where the face is located is located is located using a face detection algorithm (MTCNN / dlbi).

[0164] Among them, MTCNN is a deep learning model for face detection and key point location. It first generates candidate regions and refines features in a cascade manner, and then processes them through multiple tasks (such as bounding box regression and key point location of the face). Dlbi refers to a specific project, library or application related to deep learning.

[0165] S34, extract the face of the image and proceed to step S37;

[0166] That is, the target face detected from the picture is converted into a set of feature vectors that can represent the unique attributes of the face.

[0167] S35, detecting faces in the video; and proceeding to step S36;

[0168] That is, detecting the face of the target person included in the video to be edited.

[0169] S36, extracting the target face contained in the video frame image, and proceeding to step S37;

[0170] Each target face detected in the video frame is converted into a set of feature vectors that can represent the unique attributes of the face.

[0171] S37, face feature matching, and entering S38;

[0172] In this step, the similarity between the facial features of each frame in the video and the target facial features in the image is calculated. A threshold value (for example, the similarity reaches 0.8) can be set in advance, and the features are considered matched if the threshold is exceeded.

[0173] S38, detect the time when the target person appears in the video, and go to step S39.

[0174] In the embodiment of this step, video analysis is performed to find the target person included in the video; each frame in the video is analyzed to determine the time point when the target person appears.

[0175] S39, video clip editing, and enter step S40.

[0176] In the embodiment of this step, video editing is mainly performed. Specifically, the start and end time of the video segment to be edited are determined according to the analysis result; and the video segment is edited using a video editing library (such as OpenCV).

[0177] S40, perform video synthesis, and then proceed to step S41.

[0178] In this step, video synthesis is performed, specifically, the edited video clips are spliced ​​in order; OpenCV or FFmeng can be used to synthesize the final video file. In the embodiment of the present invention, AI artificial intelligence technology is used to add transition effects in video synthesis.

[0179] Specifically, the present invention uses AI technology to synthesize videos as follows:

[0180] 1) Input image: Given two input images I0 and I1 (the images come from the last frame screenshot of synthetic video 1 and the first frame screenshot of video 2), the goal of DiffMorpher is to generate a smooth transition sequence between the two images.

[0181] 2) Understanding the content of the image: The LoRA model is used to enable it to fit (understand) the semantic content of the image.

[0182] 3) Obtaining image noise: Through the DDIM inversion process, the potential noise of the two images is obtained from the model.

[0183] 4) Interpolation operation: interpolate between LoRA parameters and potential noise to ensure smooth semantic transition.

[0184] 5) Generate intermediate images: Use the interpolation ratio α to interpolate between the LoRA parameters and the underlying noise to generate a series of high-fidelity images depicting the smooth transition between I_0 and I_1.

[0185] Through these steps, the transition effect of the two video compositions is optimized to make the transition smoother.

[0186] S41, end.

[0187] It can be seen from the above that the embodiment of the present invention provides a video intelligent editing method based on AI technology, which can automatically detect abnormal data in a data set and issue an early warning when an abnormality is detected, thereby providing protection for data security.

[0188] Exemplary Devices

[0189] like Figure 3 As shown, an embodiment of the present invention provides a video intelligent editing device based on AI technology, the device comprising:

[0190] The acquisition module 310 is used to acquire and determine the target picture with the target person that needs to be edited, and acquire the video source that needs to be edited;

[0191] The image recognition module 320 is used to perform face recognition detection on the target person in the target image and extract the target face of the target image;

[0192] The video frame recognition module 330 is used to recognize the face of each frame image in the video source and extract the face of each frame of the video source;

[0193] A similarity comparison module 340 is used to compare the similarity of each face frame of the video source with the target face of the target image, find the key frame of the video source whose similarity reaches a predetermined threshold, and determine the time point when the target person appears in the key frame;

[0194] The video clip editing module 350 is used to determine the start and end time of the video source clip to be edited based on the key frames of the video source found and the time points when the target person appears in the key frames, and perform video clip editing;

[0195] The video synthesis module 360 ​​is used to splice the edited video clips in chronological order, and use artificial intelligence technology to compensate for the transition effect at the splicing of frames at discontinuous times to synthesize the final target video file, as described above.

[0196] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 4As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a database connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video intelligent editing method based on AI technology is implemented. The database of the intelligent terminal is used to store a video intelligent editing program based on AI technology.

[0197] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0198] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:

[0199] Obtain and determine the target picture with the target person that needs to be edited, and obtain the video source that needs to be edited;

[0200] Performing face recognition detection on the target person in the target image to extract the target face of the target image;

[0201] Recognize the face of each frame of the video source, and extract the face of each frame of the video source;

[0202] Compare the similarity of each face frame in the video source with the target face in the target image, find the key frame of the video source whose similarity reaches a predetermined threshold, and determine the time point when the target person appears in the key frame;

[0203] Based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited;

[0204] The edited video clips are spliced ​​in chronological order, and artificial intelligence technology is used to compensate for transition effects at the splicing of frames at discontinuous times to synthesize the final target video file, as described above.

[0205] The steps of obtaining and determining a target picture with a target person to be edited, and obtaining a video source to be edited include:

[0206] Receive the user's operation instruction to select the target picture with the target person to be edited;

[0207] Receive the user's operation instruction to select the video source that needs to be edited;

[0208] Load the target image and video source described by source.

[0209] The step of performing face recognition detection on the target person in the target image to extract the target face of the target image includes:

[0210] For the target person in the target image, a face detection algorithm is used to locate the position of the target person's face, and a feature vector of the target person's face is extracted.

[0211] The step of identifying the face of each frame of the video source and extracting the face of each frame of the video source comprises:

[0212] The face of each frame image in the video source is recognized, the position of the face in each frame image is located using a face detection algorithm, and the feature vector of the face in each frame image is extracted.

[0213] The step of comparing the similarity of each face frame of the video source with the target face of the target image to find the key frame of the video source whose similarity reaches a predetermined threshold comprises:

[0214] Compare the feature vector of the face of each frame extracted from the video source with the feature vector of the target face extracted;

[0215] Calculate the similarity between the feature vector of the face in each frame image and the feature vector of the target face;

[0216] When the similarity between the feature vector of the human face in the frame image extracted from the video source and the feature vector of the target face is greater than a predetermined threshold, it is determined that the face features match;

[0217] The frames with matching facial features are marked as key frames containing the target person in the video source.

[0218] The step of determining the time point at which the target person appears in the key frame includes:

[0219] The key frames marked as containing the target person in the video source are analyzed to determine the time point when the target person appears.

[0220] The step of determining the start and end time of the video source segment to be edited based on the key frame of the found video source and the time point when the target person appears in the key frame, and editing the video segment includes:

[0221] Based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited using the video editing library, as described above.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0223] In summary, the present invention provides a video intelligent editing method, device, intelligent terminal and storage medium based on AI technology. The present invention can find a person in multiple complex video scenes, and automatically identify and extract all clips of the person to form a new video album. This process combines video segmentation, object detection, face recognition, object tracking, video editing and video synthesis technology; it searches for the corresponding target person through character recognition from massive video resources, and finally automatically edits to form a personal album for the exclusive person, which greatly improves the editing efficiency and provides convenience for users.

Claims

1. A video intelligent editing method based on AI technology, characterized in that: include: Obtain and determine the target picture with the target person that needs to be edited, and obtain the video source that needs to be edited; Performing face recognition detection on the target person in the target image to extract the target face of the target image; Recognize the face of each frame of the video source, and extract the face of each frame of the video source; Compare the similarity of each face frame in the video source with the target face in the target image, find the key frame of the video source whose similarity reaches a predetermined threshold, and determine the time point when the target person appears in the key frame; Based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited; The edited video clips are spliced ​​in chronological order, and artificial intelligence technology is used to compensate for the transition effect at the splicing of frames in discontinuous time to synthesize the final target video file.

2. The video intelligent editing method based on AI technology according to claim 1 is characterized in that: The steps of obtaining and determining the target picture with the target person to be edited, and obtaining the video source to be edited include: Receive the user's operation instruction to select the target picture with the target person to be edited; Receive the user's operation instruction to select the video source that needs to be edited; Load the target image and video source described by source.

3. The video intelligent editing method based on AI technology according to claim 1 is characterized in that: The step of performing face recognition detection on the target person in the target image to extract the target face of the target image comprises: For the target person in the target image, a face detection algorithm is used to locate the position of the target person's face, and a feature vector of the target person's face is extracted.

4. The video intelligent editing method based on AI technology according to claim 3 is characterized in that: The step of identifying the face of each frame of the video source and extracting the face of each frame of the video source comprises: The face of each frame image in the video source is recognized, the position of the face in each frame image is located using a face detection algorithm, and the feature vector of the face in each frame image is extracted.

5. The AI-based video intelligent editing method according to claim 4 is characterized in that: The process of comparing the similarity of each face frame of the video source with the target face of the target image to find the key frame of the video source whose similarity reaches a predetermined threshold comprises: Compare the feature vector of the face of each frame extracted from the video source with the feature vector of the target face extracted; Calculate the similarity between the feature vector of the face in each frame image and the feature vector of the target face; When the similarity between the feature vector of the face in the frame image extracted from the video source and the feature vector of the target face is greater than a predetermined threshold, it is determined that the face features match; The frames with matching facial features are marked as key frames containing the target person in the video source.

6. The AI-based video intelligent editing method according to claim 5, characterized in that: The step of determining the time point at which the target person appears in the key frame comprises: The key frames marked as containing the target person in the video source are analyzed to determine the time point when the target person appears.

7. The video intelligent editing method based on AI technology according to claim 1 is characterized in that: The step of determining the start and end time of the video source segment to be edited based on the found key frames of the video source and the time point when the target person appears in the key frames, and editing the video segment includes: Based on the key frames of the video source found and the time points when the target person appears in the key frames, the start and end times of the video source segments to be edited are determined, and the video segments are edited using the video editing library.

8. A video intelligent editing device based on AI technology, characterized in that: The device comprises: The acquisition module is used to acquire and determine the target picture with the target person that needs to be edited, and to acquire the video source that needs to be edited; An image recognition module is used to perform face recognition detection on a target person in the target image and extract the target face of the target image; A video frame recognition module, used to recognize the face of each frame image in the video source and extract the face of each frame of the video source; A similarity comparison module is used to compare the similarity of each face frame of the video source with the target face of the target image, find the key frame of the video source whose similarity reaches a predetermined threshold, and determine the time point when the target person appears in the key frame; The video clip editing module is used to determine the start and end time of the video source clip to be edited based on the key frames of the video source found and the time points when the target person appears in the key frames, and perform video clip editing; The video synthesis module is used to splice the edited video clips in chronological order, and use artificial intelligence technology to compensate for the transition effect at the splicing points of frames in discontinuous time to synthesize the final target video file.

9. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in any one of claims 1 to 7.

Images