Automatic video editing method, system, terminal and storage medium

Abstract

The present invention discloses an automatic video editing method, system, terminal and storage medium. The method comprises: obtaining key frames of a video to be edited, and self-labeling the key frames using an image comparison algorithm to generate an unsupervised vector representation of the key frames; obtaining corpus information of the video to be edited, and obtaining an unsupervised vector representation of the corpus information using a text comparison algorithm; segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of key frames; calculating the similarity of adjacent video segments according to the unsupervised vector representation of the key frames and the unsupervised vector representation of the corpus information, merging adjacent video segments whose similarity is greater than a set similarity threshold, and generating a video editing result of the video to be edited. The embodiment of the present invention utilizes image and text information, avoids manual data labeling, realizes automatic video editing, and greatly improves the efficiency of video editing.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to an automatic video editing method, system, terminal and storage medium. Background Art

[0002] With the development of 4G networks, short video technology has flourished. With the emergence of a large number of video apps such as Douyin, Kuaishou, and Bilibili, the number of videos has increased exponentially. Although videos are more intuitive than text and pictures, watching videos takes a lot of time. For a long video, the valuable or user-interesting clips often only account for a part of the total length of the video, so the demand for video editing is also increasing day by day.

[0003] Video editing in existing technologies usually requires reliance on human resources, which is both costly and inefficient, and to some extent hinders the development of short video technology. Summary of the invention

[0004] The present invention provides an automatic video editing method, system, terminal and storage medium, aiming to solve the technical problems of the existing video editing, such as the need to rely on human resources, the waste of money, the low efficiency of video editing, etc.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] An automatic video editing method, the method comprising:

[0007] Obtain key frames of the video to be edited, and use an image comparison algorithm to self-label the key frames to generate an unsupervised vector representation of the key frames;

[0008] Obtaining corpus information of the video to be edited, and using a text comparison algorithm to obtain an unsupervised vector representation of the corpus information;

[0009] Segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of the key frames;

[0010] The similarity between adjacent video clips is calculated based on the unsupervised vector representation of the key frames and the unsupervised vector representation of the corpus information, and the adjacent video clips whose similarity is greater than a preset similarity threshold are merged to generate a video editing result of the video to be edited.

[0011] The technical solution adopted by the embodiment of the present invention also includes: the key frame is a frame where a key action in the motion change of a character or object in the video to be edited is located, and obtaining the key frame of the video to be edited includes:

[0012] Using ffmpeg to perform frame extraction processing on the video to be edited;

[0013] For all extracted images, the similarity between adjacent images is calculated, and the images with similarity lower than the set threshold are taken as key frames.

[0014] The technical solution adopted by the embodiment of the present invention also includes: the self-marking of the key frame by using an image contrast algorithm includes:

[0015] Based on the acquired key frames, an unsupervised algorithm is used to train the Selflabel model. The Selflabel model uses an image contrast algorithm to learn the unsupervised vector representation of the key frame image, self-labels the key frame through clustering and representation learning, and outputs the self_label (frame k ), where frame k Represents the kth key frame image.

[0016] The technical solution adopted by the embodiment of the present invention also includes: the step of obtaining the corpus information of the video to be edited includes:

[0017] ASR technology is used to collect ASR voice information of the video to be edited, and the collected ASR voice information is cut into ASR text information of a set length;

[0018] Use OCR technology to obtain OCR text information from the extracted frame image;

[0019] The cut ASR text information and OCR text information are used as the corpus information of the video to be edited.

[0020] The technical solution adopted by the embodiment of the present invention also includes: the unsupervised vector representation of the corpus information obtained by using a text comparison algorithm includes:

[0021] The SimCSE model is trained based on the corpus information. The SimCSE model uses a text comparison algorithm to learn the unsupervised vector representation of ASR text information and OCR text information, and outputs the text vector simcse (asr k ) and simcse(ocr k ), where asr k Indicates the kth ASR text information of the video to be edited, ocr k Represents the OCR text information of the kth key frame image.

[0022] The technical solution adopted by the embodiment of the present invention also includes: segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of key frames includes:

[0023] Each key frame is used as a cutting point, and the video to be edited is divided into video segments corresponding to the number of key frames, and each video segment includes a key frame image and ASR text information and OCR text information corresponding to the video segment.

[0024] The technical solution adopted by the embodiment of the present invention also includes: the calculating the similarity of adjacent video clips according to the unsupervised vector representation of the key frame and the unsupervised vector representation of the corpus information includes:

[0025] The similarities of the key frames, ASR text information and OCR text information of the adjacent video clips are calculated respectively:

[0026] simi1=cos(self_label(frame k ),self_label(frame k+1 ))

[0027] simi2=cos(simcse(asr k ),simcse(asr k+1 ))

[0028] simi3=cos(simcse(ocr k ),simcse(ocr k+1 ))

[0029] Among them, simi1, simi2 and simi3 represent the similarities of key frames, ASR text information and OCR text information in adjacent video clips respectively;

[0030] The similarity of adjacent video clips is calculated based on the similarity of the key frames, ASR text information, and OCR text information:

[0031] simi=α*simi1+β*simi2+(1―α―β)*simi3

[0032] simi represents the similarity between the adjacent video clips, and α and β are adjustable parameters.

[0033] Another technical solution adopted by the embodiment of the present invention is: an automatic video editing system, comprising:

[0034] The first acquisition module is used to acquire key frames of the video to be edited, and self-label the key frames using an image comparison algorithm to generate an unsupervised vector representation of the key frames;

[0035] The second acquisition module is used to acquire the corpus information of the video to be edited, and to acquire the unsupervised vector representation of the corpus information by using a text comparison algorithm;

[0036] Video segmentation module: used for segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of key frames;

[0037] Video merging module: used to calculate the similarity of adjacent video clips based on the unsupervised vector representation of the key frames and the unsupervised vector representation of the corpus information, merge the adjacent video clips whose similarity is greater than a set similarity threshold, and generate the video editing result of the video to be edited.

[0038] Another technical solution adopted by an embodiment of the present invention is: a terminal, the terminal includes a processor and a memory coupled to the processor, wherein:

[0039] The memory stores program instructions for implementing the above-mentioned automatic video editing method;

[0040] The processor is used to execute the program instructions stored in the memory to perform the automatic video editing operation.

[0041] Another technical solution adopted by the embodiment of the present invention is: a storage medium stores program instructions executable by a processor, and the program instructions are used to execute the above-mentioned automatic video editing method.

[0042] The automatic video editing method, system, terminal and storage medium of the embodiment of the present invention collect key frames and corpus information of the video to be edited, divide the video to be edited into multiple video segments by key frames, calculate the similarity of adjacent video segments based on the vector representation of key frames and corpus information, merge video segments with high similarity, and obtain the final video editing result. The embodiment of the present invention utilizes image and text information at the same time, avoids manual data labeling, realizes automatic video editing, and greatly improves the efficiency of video editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of an automatic video editing method according to a first embodiment of the present invention;

[0044] Figure 2 is a flow chart of an automatic video editing method according to a second embodiment of the present invention;

[0045] Figure 3 is a schematic structural diagram of an automatic video editing system according to an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of the terminal structure of an embodiment of the present invention;

[0047] Figure 5 It is a schematic diagram of the storage medium structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms "first", "second" and "third" in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] See also Figure 1 , is a flowchart of the automatic video editing method of the first embodiment of the present invention. The automatic video editing method of the first embodiment of the present invention comprises the following steps S101-S104:

[0052] S101: obtaining key frames of the video to be edited, and self-labeling the key frames using an image comparison algorithm to generate an unsupervised vector representation of the key frames;

[0053] Among them, the key frame is the frame where the key action of the character or object in the motion change of the video to be edited is located. The key frame is obtained by using ffmpeg to extract frames from the video to be edited; FFmpeg is an open source computer program that can be used to record and convert digital audio and video and convert them into streams. FFmpeg has functions such as video acquisition, video format conversion, video capture, and video watermarking. For all the images after frame extraction, the similarity between adjacent images is calculated, and the images with similarity lower than the set threshold are used as key frames.

[0054] The image contrast algorithm is used to self-label the key frames. Specifically, based on the acquired key frames, the unsupervised algorithm is used to train the Self label model. The Self label model uses the image contrast algorithm to learn the unsupervised vector representation of the key frame image, and the key frames are self-labeled through clustering and representation learning, and the self_label (frame k ), frame k Represents the kth key frame image.

[0055] S102: Obtaining corpus information of the video to be edited, and using a text comparison algorithm to obtain an unsupervised vector representation of the corpus information;

[0056] Among them, the specific method of obtaining corpus information is: using ASR technology to collect ASR voice information of the video to be edited, and cutting the collected ASR voice information into ASR text information of a set length; using OCR technology to obtain OCR text information from the image after frame extraction; and using the cut ASR text information and OCR text information as the corpus information of the video to be edited.

[0057] The text comparison algorithm is used to obtain the unsupervised vector representation of the corpus information. Specifically, the SimCSE model is trained based on the corpus information. The SimCSE model uses the text comparison algorithm to learn the unsupervised vector representation of ASR text information and OCR text information, and outputs the text vector simcse (asr k ) and simcse(ocr k ), where asr k Indicates the kth ASR text information of the video to be edited, ocr k Represents the OCR text information of the kth key frame image.

[0058] S103: Segment the video to be edited according to the key frames to generate video segments corresponding to the number of key frames;

[0059] The specific video segmentation method is as follows: each key frame is used as a cutting point, and the video to be edited is divided into video segments corresponding to the number of key frames, and each video segment includes a key frame image and ASR text information and OCR text information corresponding to the video segment.

[0060] S104: calculating the similarity between adjacent video segments according to the unsupervised vector representation of the key frames and the unsupervised vector representation of the corpus information, merging adjacent video segments whose similarity is greater than a preset similarity threshold, and generating a video editing result of the video to be edited;

[0061] The similarity calculation method of adjacent video clips is specifically as follows:

[0062] First, calculate the similarity of key frames, ASR text information, and OCR text information of adjacent video clips:

[0063] simi1=cos(self_label(frame k ),self_label(frame k+1 )) (1)

[0064] simi2=cos(simcse(asr k ),simcse(asr k+1 )) (2)

[0065] simi3=cos(simcse(ocr k ),simcse(ocr k+1 )) (3)

[0066] Among them, simi1, simi2 and simi3 represent the similarities of key frames, ASR text information and OCR text information in adjacent video clips respectively;

[0067] Then, the similarity of adjacent video clips is calculated based on the similarity of key frames, ASR text information, and OCR text information:

[0068] simi=α*simi1+β*simi2+(1―α―β)*simi3 (4)

[0069] simi represents the similarity between adjacent video clips, and α and β are adjustable parameters.

[0070] The automatic video editing method of the first embodiment of the present invention obtains the key frames and corpus information of the video to be edited, uses an image comparison algorithm to learn the unsupervised vector representation of the key frame image, uses a text comparison algorithm to learn the unsupervised vector representation of the corpus information, divides the video to be edited into multiple video clips through the key frames, and calculates the similarity of adjacent video clips based on the vector representation of the key frames and corpus information, merges video clips with high similarity, and obtains the final video editing result. The embodiment of the present invention utilizes both image and text information, avoids manual data labeling, realizes automatic video editing, and greatly improves the efficiency of video editing.

[0071] See also Figure 2 , is a flow chart of the automatic video editing method of the second embodiment of the present invention. The automatic video editing method of the second embodiment of the present invention comprises the following steps S201-S209:

[0072] S201: Collect at least one video to be edited;

[0073] S202: extracting frames from the video to be edited to obtain key frames of the video to be edited;

[0074] In this step, the ffmpeg program is used to extract frames from the collected videos to be edited, wherein FFmpeg is an open source computer program that can be used to record and convert digital audio and video and convert them into streams. FFmpeg has functions such as video acquisition, video format conversion, video capture, and video watermarking. Key frames refer to frames where key actions are located in the motion changes of characters or objects in the video to be edited. In an embodiment of the present invention, the key frames are obtained by calculating the similarity between adjacent images for all images after frame extraction, and taking image frames whose similarity is lower than a set threshold as key frames. While obtaining the key frames, a certain number of remaining frames are retained according to a set ratio, and the remaining frames are non-key frames. The number k of remaining frames can be set randomly.

[0075] S203: Based on the acquired key frames, an unsupervised algorithm is used to train a Self-label model, and the Self-label model self-labels the key frames through clustering and representation learning;

[0076] In this step, the Self label model is a self-supervised algorithm that calibrates labels by maximizing the mutual information between data and labels. The Self label model uses the image contrast algorithm to learn the unsupervised vector representation of key frame images, self-labels the key frame images through clustering and representation learning, and outputs the self_label (frame k ), frame k Represents the kth key frame image.

[0077] S204: using ASR (Automatic Speech Recognition) technology and OCR (Optical Character Recognition) recognition technology to collect corpus information of the video to be edited;

[0078] In this step, the video is a typical multimodal data, including images and rich text information. The corpus information of the video to be edited includes the ASR voice information in the video to be edited and the OCR text information in the frame-extracted image. In an embodiment of the present invention, the method for obtaining the corpus information of the video to be edited is as follows: ASR technology is used to collect the ASR voice information of the video to be edited, and the collected ASR voice information is cut into ASR text information of a set length; at the same time, OCR technology is used to obtain OCR text information from the frame-extracted image, and the cut ASR text information and OCR text information are used as the corpus information of the video to be edited. Among them, the cutting length of the ASR voice information is 100, which can be set according to the actual application.

[0079] S205: training a SimCSE model based on the collected corpus information, and outputting a SimCSE text vector of the video to be edited through the SimCSE model;

[0080] In the embodiment of the present invention, the SimCSE model can be trained unsupervised, based on the self-supervised training of the BERT model, and the natural language data enhancement with semantic equivalence is maintained through dropout. The unsupervised vector representation of the text is learned by the text comparison algorithm, and the text vector simcse (asr k ) and simcse(ocr k ), where asr k Indicates the kth ASR text information of the video to be edited, ocr k Represents the OCR text information of the kth key frame image.

[0081] S206: Using each key frame as a cutting point, the video to be edited is divided into a plurality of video segments, and each video segment includes a key frame and ASR text information and OCR text information corresponding to the video segment;

[0082] In this step, the long video is divided into multiple shorter video clips based on the key frames of the video to be edited. In each video clip, there is a key frame image and the ASR text and OCR text corresponding to the video clip, that is, each video clip is represented as (frame, asr, ocr).

[0083] S207: Calculate the similarity of two adjacent video clips based on the key frames, ASR text information and OCR text of the video to be edited;

[0084] In this step, the similarity calculation method of adjacent video segments is as follows: first, the similarity of the key frames, ASR text and OCR text of the adjacent video segments is calculated respectively, and then the similarity of the adjacent video segments is calculated based on the similarity of the key frames, ASR text and OCR text. The specific calculation formula is as follows:

[0085] simi1=cos(self_label(frame k ),self_label(frame k+1 )) (1)

[0086] simi2=cos(simcse(asr k ),simcse(asr k+1 )) (2)

[0087] simi3=cos(simcse(ocr k ),simcse(ocr k+1 )) (3)

[0088] simi=α*simi1+β*simi2+(1―α―β)*simi3 (4)

[0089] Wherein, simi1, simi2 and simi3 respectively represent the similarity of key frames, ASR text and OCR text in adjacent video segments, and simi represents the similarity of adjacent video segments. α and β are adjustable parameters. Preferably, the embodiment of the present invention sets the values ​​of α and β to 0.45.

[0090] S208: Determine whether the similarity between two adjacent video segments is greater than a preset similarity threshold. If the similarity between the two adjacent video segments is greater than the preset similarity threshold, execute S209;

[0091] In this step, the similarity threshold is set to 0.5, that is, if the similarity between two adjacent video clips is greater than 0.5, the two video clips are considered to be similar enough to be merged. Otherwise, the two video clips are discarded.

[0092] S209: merging adjacent video clips whose similarity is greater than a preset similarity threshold to obtain a final video editing result;

[0093] In this step, by merging video clips with high similarity, an edited short video is obtained, so that the edited short video is smoother and the viewing experience of the viewer is improved.

[0094] Based on the above, the automatic video editing method of the second embodiment of the present invention collects key frames and corpus information of the video to be edited, uses an image comparison algorithm to learn the unsupervised vector representation of the key frame image, uses a text comparison algorithm to learn the unsupervised vector representation of the corpus information, divides the video to be edited into multiple video segments through key frames, and calculates the similarity of adjacent video segments based on the vector representation of the key frames and corpus information, merges video segments with high similarity, and obtains the final video editing result. The embodiment of the present invention utilizes both image and text information, avoids manual data labeling, realizes automatic video editing, and greatly improves the efficiency of video editing.

[0095] In an optional implementation, it is also possible to upload the result of the automatic video editing method to the blockchain.

[0096] Specifically, the corresponding summary information is obtained based on the result of the automatic video editing method. Specifically, the summary information is obtained by hashing the result of the automatic video editing method, such as by using the sha256s algorithm. Uploading the summary information to the blockchain can ensure its security and fairness and transparency to users. Users can download the summary information from the blockchain to verify whether the result of the automatic video editing method has been tampered with. The blockchain referred to in this example is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0097] See also Figure 3 , is a schematic diagram of the structure of an automatic video editing system according to an embodiment of the present invention. The automatic video editing system 40 according to an embodiment of the present invention comprises:

[0098] The first acquisition module 41 is used to acquire key frames of the video to be edited, and self-label the key frames using an image comparison algorithm to generate an unsupervised vector representation of the key frames; wherein the key frames are frames where key actions are located in the motion changes of the characters or objects in the video to be edited. The key frame acquisition method is: using ffmpeg to extract frames from the video to be edited; for all extracted images, the similarity between adjacent images is calculated, and images with similarity lower than a set threshold are used as key frames.

[0099] The first acquisition module 41 uses an image contrast algorithm to self-label the key frame. Specifically, based on the acquired key frame, an unsupervised algorithm is used to train a Self label model. The Self label model uses an image contrast algorithm to learn an unsupervised vector representation of the key frame image, and self-labels the key frame through clustering and representation learning, and outputs the self_label (frame k ), frame k Represents the kth key frame image.

[0100] The second acquisition module 42 is used to obtain the corpus information of the video to be edited, and use a text comparison algorithm to obtain an unsupervised vector representation of the corpus information; wherein, the corpus information acquisition method is specifically as follows: using ASR technology to collect ASR voice information of the video to be edited, and cutting the collected ASR voice information into ASR text information of a set length; using OCR technology to obtain OCR text information from the image after frame extraction; and using the cut ASR text information and OCR text information as the corpus information of the video to be edited.

[0101] The second acquisition module 42 uses a text comparison algorithm to obtain an unsupervised vector representation of the corpus information. Specifically, the SimCSE model is trained based on the corpus information. The SimCSE model uses a text comparison algorithm to learn the unsupervised vector representation of the ASR text information and the OCR text information, and outputs the text vector simcse (asr k ) and simcse(ocr k ), where asr k Indicates the kth ASR text information of the video to be edited, ocr k Represents the OCR text information of the kth key frame image.

[0102] Video segmentation module 43: used to segment the video to be edited according to key frames, and generate video segments corresponding to the number of key frames; wherein the video segmentation method of the video segmentation module is specifically: each key frame is used as a cutting point, and the video to be edited is segmented into video segments corresponding to the number of key frames, and each video segment includes a key frame image and ASR text information and OCR text information corresponding to the video segment.

[0103] The video merging module 44 is used to calculate the similarity of adjacent video clips according to the unsupervised vector representation of the key frame and the unsupervised vector representation of the corpus information, merge the adjacent video clips whose similarity is greater than the set similarity threshold, and generate the video editing result of the video to be edited; wherein the similarity of adjacent video clips is calculated in the following manner:

[0104] First, calculate the similarity of key frames, ASR text information, and OCR text information of adjacent video clips:

[0105] simi1=cos(self_label(frame k ),self_label(frame k+1 )) (1)

[0106] simi2=cos(simcse(asr k ),simcse(asr k+1 )) (2)

[0107] simi3=cos(simcse(ocr k ),simcse(ocr k+1 )) (3)

[0108] Among them, simi1, simi2 and simi3 represent the similarities of key frames, ASR text information and OCR text information in adjacent video clips respectively;

[0109] Then, the similarity of adjacent video clips is calculated based on the similarity of key frames, ASR text information, and OCR text information:

[0110] simi=α*simi1+β*simi2+(1―α―β)*simi3 (4)

[0111] simi represents the similarity between adjacent video clips, and α and β are adjustable parameters.

[0112] The automatic video editing system of the embodiment of the present invention obtains the key frames and corpus information of the video to be edited, uses an image comparison algorithm to learn the unsupervised vector representation of the key frame image, uses a text comparison algorithm to learn the unsupervised vector representation of the corpus information, divides the video to be edited into multiple video clips through the key frames, and calculates the similarity of adjacent video clips based on the vector representation of the key frames and corpus information, merges video clips with high similarity, and obtains the final video editing result. The embodiment of the present invention utilizes both image and text information, avoids manual data labeling, realizes automatic video editing, and greatly improves the efficiency of video editing.

[0113] See also Figure 4 , is a schematic diagram of a terminal structure of an embodiment of the present invention. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51 .

[0114] The memory 52 stores program instructions for implementing the above-mentioned automatic video editing method.

[0115] The processor 51 is used to execute program instructions stored in the memory 52 to perform automatic video editing operations.

[0116] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0117] See also Figure 5 , Figure 5 Schematic diagram of the structure of the storage medium of the embodiment of the present invention. The storage medium of the embodiment of the present invention stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0118] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0119] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present invention.

Claims

1. An automatic video editing method, characterized in that: The method comprises: Obtain key frames of the video to be edited, and use an image comparison algorithm to self-label the key frames to generate an unsupervised vector representation of the key frames; the key frames are frames where key actions in the motion changes of characters or objects in the video to be edited are located, and obtaining the key frames of the video to be edited includes using ffmpeg to extract frames from the video to be edited; for all extracted images, calculate the similarity between adjacent images, and use images with a similarity lower than a set threshold as key frames; Acquire the corpus information of the video to be edited, and use a text comparison algorithm to obtain an unsupervised vector representation of the corpus information; the acquisition of the corpus information of the video to be edited includes: using ASR technology to collect ASR voice information of the video to be edited, and cutting the collected ASR voice information into ASR text information of a set length; using OCR technology to obtain OCR text information from the image after frame extraction; and using the cut ASR text information and OCR text information as the corpus information of the video to be edited; Segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of the key frames; The similarity between adjacent video clips is calculated based on the unsupervised vector representation of the key frames and the unsupervised vector representation of the corpus information, and the adjacent video clips whose similarity is greater than a preset similarity threshold are merged to generate a video editing result of the video to be edited.

2. The automatic video editing method according to claim 1, characterized in that: The self-marking of the key frame by using an image contrast algorithm includes: Based on the acquired key frames, an unsupervised algorithm is used to train the Selflabel model. The Selflabel model uses an image contrast algorithm to learn the unsupervised vector representation of the key frame image, self-labels the key frames through clustering and representation learning, and outputs the key frames. ,in, Indicates k Key frame images.

3. The automatic video editing method according to claim 1, characterized in that: The step of using a text comparison algorithm to obtain an unsupervised vector representation of the corpus information includes: The SimCSE model is trained based on the corpus information. The SimCSE model uses a text comparison algorithm to learn the unsupervised vector representation of the ASR text information and the OCR text information, and outputs the text vector of the video to be edited. and ;in, Indicates the first k ASR text information, Indicates k OCR text information of a key frame image.

4. The automatic video editing method according to claim 3, characterized in that: The step of segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of the key frames includes: Each key frame is used as a cutting point, and the video to be edited is divided into video segments corresponding to the number of key frames, and each video segment includes a key frame image and ASR text information and OCR text information corresponding to the video segment.

5. The automatic video editing method according to claim 2 or 3, characterized in that: The calculating the similarity of adjacent video clips according to the unsupervised vector representation of the key frame and the unsupervised vector representation of the corpus information includes: The similarities of the key frames, ASR text information and OCR text information of the adjacent video clips are calculated respectively: in, Respectively represent the similarity of key frames, ASR text information, and OCR text information in adjacent video clips; The similarity of adjacent video clips is calculated based on the similarity of the key frames, ASR text information, and OCR text information: ， represents the similarity of the adjacent video clips, , They are adjustable parameters respectively.

6. An automatic video editing system, characterized in that: The system comprises: The first acquisition module is used to acquire key frames of the video to be edited, and self-label the key frames using an image comparison algorithm to generate an unsupervised vector representation of the key frames; the key frames are frames where key actions in the motion changes of characters or objects in the video to be edited are located, and the acquisition of the key frames of the video to be edited includes using ffmpeg to extract frames from the video to be edited; for all extracted images, the similarity between adjacent images is calculated, and images with a similarity lower than a set threshold are used as key frames; The second acquisition module is used to acquire the corpus information of the video to be edited, and to acquire the unsupervised vector representation of the corpus information by using a text comparison algorithm; the acquisition of the corpus information of the video to be edited includes: using ASR technology to collect ASR voice information of the video to be edited, and cutting the collected ASR voice information into ASR text information of a set length; using OCR technology to acquire OCR text information from the image after frame extraction; and using the cut ASR text information and OCR text information as the corpus information of the video to be edited; Video segmentation module: used for segmenting the video to be edited according to the key frames to generate video segments corresponding to the number of key frames; Video merging module: used to calculate the similarity of adjacent video clips based on the unsupervised vector representation of the key frames and the unsupervised vector representation of the corpus information, merge the adjacent video clips whose similarity is greater than a set similarity threshold, and generate the video editing result of the video to be edited.

7. A terminal, characterized in that: The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the automatic video editing method according to any one of claims 1 to 5; The processor is used to execute the program instructions stored in the memory to perform the automatic video editing method.

8. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the automatic video editing method according to any one of claims 1 to 5.

Images