A video automatic editing method

Abstract

The present application discloses a method for automatic video editing, which belongs to the field of electronic digital data processing, including publishing a first video based on a first account; when the publishing time of the first video reaches a first preset time, segmenting the first video based on a first model to obtain multiple video clips; for each of the video clips, extracting all bullet comments corresponding to the video clip to obtain a bullet comment set corresponding to the video clip; determining the segment description text corresponding to the video clip; using a second model, extracting candidate images from the video clip based on the segment description text and the bullet comment set; fusing the candidate images corresponding to each of the video clips to obtain a second video, which is the automatic editing result of the first video. The scheme automatically obtains high-quality video editing results.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular to a method for automatic video editing. Background Art

[0002] In video applications, more and more self-media users can exchange information with other users by publishing videos. By supporting self-media users to edit videos, we can help them produce high-quality and more attractive videos, thereby promoting information interaction using videos as the medium. However, the existing technology lacks a sufficiently intelligent solution that can provide high-quality video editing capabilities. Summary of the invention

[0003] The embodiment of the present application provides a method for automatic video editing, which can automatically achieve high-quality video editing and promote information interaction using video as the medium.

[0004] According to one aspect of an embodiment of the present application, a method for automatic video editing is provided, the method comprising:

[0005] Release the first video based on the first account;

[0006] When the release duration of the first video reaches a first preset duration, segmenting the first video based on a first model to obtain a plurality of video segments;

[0007] For each of the video clips, extract all bullet comments corresponding to the video clip to obtain a bullet comment set corresponding to the video clip; determine a clip description text corresponding to the video clip; and use a second model to extract a candidate image in the video clip based on the clip description text and the bullet comment set;

[0008] The candidate images corresponding to the respective video clips are fused to obtain a second video, where the second video is an automatic editing result of the first video.

[0009] In one embodiment, the method further includes publishing the second video based on a second account, where the second account is an associated account of the first account; before the second video is published, the method further includes: deleting the first video published by the first account.

[0010] In one embodiment, the second model is used to perform the following operations:

[0011] Intelligently classify the barrage set to obtain at least one first barrage set and at least one second barrage set, wherein the first barrage set is composed of barrages with similar general semantics, and the second barrage set is composed of barrages with similar semantics determined based on key information, wherein the key information is information related to the vertical category corresponding to the first video;

[0012] Extracting a plurality of sub-segments from the video segment based on each of the first bullet comment sets and each of the second bullet comment sets;

[0013] For each of the sub-segments, a third model is used to obtain candidate images in the sub-segment based on the segment description text.

[0014] In one embodiment, the general semantic distance between any two bullet comments in the first bullet comment set is less than a first distance threshold, the semantic distance between any two bullet comments in the second bullet comment set determined based on key information is less than a second distance threshold, and the first distance threshold is less than the second distance threshold;

[0015] The number of barrages in the first barrage set is greater than a first number threshold, the number of barrages in the second barrage set is greater than a second number threshold, and the first number threshold is greater than the second number threshold.

[0016] In one embodiment, extracting a plurality of sub-segments from the video segment based on each of the first bullet comment sets and each of the second bullet comment sets includes:

[0017] For each of the first bullet chat sets, according to the first target moments respectively corresponding to the bullet chats in the first bullet chat set, at least one first time period is obtained, and the number of first target moments included in each of the first time periods is greater than a third quantity threshold; in the video segment, a candidate sub-segment corresponding to each of the first time periods is extracted;

[0018] For each of the second bullet chat sets, according to the second target moments respectively corresponding to the bullet chats in the second bullet chat set, at least one second time period is obtained, and the number of second target moments included in each of the second time periods is greater than the third number threshold; in the video segment, a candidate sub-segment corresponding to each of the second time periods is extracted;

[0019] The candidate sub-segments that generate the intersection are merged to obtain a fused sub-segment;

[0020] Based on the candidate sub-segments that do not generate an intersection and the fused sub-segment, a plurality of sub-segments in the video segment are obtained.

[0021] In one embodiment, the segment description text includes a plurality of text attribute values, each of which indicates a target value of a corresponding category, and the third model performs the following operations:

[0022] Obtain all images in the sub-segment, and perform the following operations on each of the images:

[0023] Perform a cross-category and cross-modality comparison between the image and all the text attribute values ​​to obtain a horizontal fit degree of the image relative to each of the text attribute values, and the accumulation of all horizontal fit degrees output by the cross-category and cross-modality comparison is 1;

[0024] For each of the text attribute values, the following operations are performed: determining all values ​​of the category corresponding to the text attribute value; performing a cross-modal comparison of the same category between all the values ​​and the image to obtain a first degree of fit of the image relative to each of the values; normalizing each of the first degrees of fit to obtain a second degree of fit corresponding to each of the first degrees of fit; taking the second degree of fit of the text attribute value as the vertical degree of fit of the image relative to the text attribute value;

[0025] For each of the text attribute values ​​of the image, averaging the corresponding horizontal fit and the corresponding vertical fit to obtain a corresponding target fit;

[0026] Averaging the target matching degrees of the image with respect to each of the text attribute values ​​to obtain the target matching degree of the image, wherein the target matching degree indicates the degree of matching between the image and the segment description text;

[0027] Determining a video segment to be extracted within the sub-segment, wherein the video segment to be extracted includes an image with the largest target fitness;

[0028] A candidate image is determined in the video segment to be extracted.

[0029] In one embodiment, the fusing of the candidate images corresponding to the respective video clips to obtain a second video, where the second video is an automatic editing result of the first video, includes:

[0030] According to the playback order of each candidate image in the first video, the candidate images are spliced ​​to obtain the second video; and manual editing of the second video is supported.

[0031] In one embodiment, the first model is used to perform the following operations:

[0032] For each preset image in the first video, identifying whether the preset image and its previous image belong to the same scene, and setting a mark for the preset image, the mark being used to indicate whether the corresponding preset image and its previous image belong to the same scene, and the preset image is not the first image of the first video;

[0033] The multiple video clips are obtained according to the marks corresponding to the preset images.

[0034] In one embodiment, the categories covered by the segment description text include: clarity category, contrast category, expression category and atmosphere category.

[0035] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned automatic video editing method.

[0036] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0037] The embodiment of the present application proposes an automatic video editing method, which can pre-publish the initial video (first video) to collect user evaluation information (bullet comments). Candidate images in each video segment are screened based on the collected bullet comments and the segment description text corresponding to the video segment in the first video. These automatically screened candidate images reflect the content that the viewer focuses on based on the bullet comments, and also reflect the key content of the first video itself determined based on the segment description text. Only images that cover these two types of key content can be used as candidate images. The second video obtained based on the candidate images has the following significant advantages:

[0038] (1) The second video is composed of images that are the focus of the user and are the focus of the first video itself. Each image in the second video is important enough, representative enough, attractive enough to the viewers, and can arouse discussion. Therefore, the second video has sufficient appeal, dissemination power, and interactivity, and is a high-quality video.

[0039] (2) The second video is generated based on some images in the first video, reflecting the editing result of the first video. The process of generating the second video based on the first video has a high degree of automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 is a flow chart of a method for automatic video editing provided by an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of video segment segmentation provided by an embodiment of the present application;

[0043] Figure 3 is a flowchart of the second model execution steps provided by an embodiment of the present application;

[0044] Figure 4 It is a flowchart of a method for extracting sub-segments based on a bullet screen set provided by an embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of a process of processing each image of each sub-segment provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] Please refer to Figure 1 , which shows a flow chart of a video automatic editing method provided by an embodiment of the present application. The method can be applied to a video application, and the video application can include a client and a server. The method can include the following steps:

[0047] S101. Publish a first video based on a first account.

[0048] The stage in which the first account releases the first video belongs to the pre-release stage, which is only for some users and the duration is strictly controlled. The video application can determine a user subset from the user group that uses the video application, and only users belonging to the user subset can receive the first video released by the first account. The method for determining the user subset is not limited and does not constitute an implementation obstacle. The number of users in the user subset is very small compared to the user group, so the pre-release of the first video basically does not affect the subsequent release effect of the second video for the entire user group. In addition, the duration of the pre-release is strictly controlled to further ensure that the pre-release of the first video basically does not affect the subsequent release effect of the second video for the entire user group.

[0049] The first account and the subsequent second account are different accounts belonging to the same publisher. The first account is used for pre-release and is targeted at a small subset of users. The second account is used for formal release and is targeted at the entire user group. The first video is a video uploaded by the publisher using the first account, and this application does not limit the content of the first video.

[0050] S102. When the release duration of the first video reaches a first preset duration, the first video is segmented based on a first model to obtain a plurality of video segments.

[0051] The first preset duration is the duration corresponding to the pre-release stage. The embodiment of the present application does not limit its value, and the value can be set by the publisher. In one embodiment, before the second video edited based on the first video is released, the method also includes: deleting the first video released by the first account. Because the first video is still saved in the background of the video application, the publisher can still access the first video. The step of "deleting the first video released by the first account" only makes the aforementioned small subset of users unable to see the first video, but does not affect the implementation of any steps in this application except step S101.

[0052] The first model in this application can use existing technology, so there is no need to elaborate on it. The technical essence of the first model is to segment the first video based on the scene to obtain multiple video clips. Specifically, the first model is used to perform the following operations: for each preset image in the first video, identify whether the preset image and its previous image belong to the same scene, and set a mark for the preset image, the mark is used to indicate whether the corresponding preset image belongs to the same scene as the previous image, and the preset image is not the first image of the first video; according to the marks corresponding to each of the preset images, the multiple video clips are obtained. The first model can directly use existing technology, and its acquisition method does not constitute an implementation obstacle, so there is no need to elaborate on it.

[0053] Please refer to Figure 2 , which shows a schematic diagram of video segment segmentation. Assume that the first video includes 20 images. Except for image 1, the other images are preset images. Each preset image corresponds to a tag. If the corresponding tag is Y, it means that it belongs to the same scene as the previous image, otherwise it belongs to a different scene from the previous image. The default tag of image 1 is N. For the first video, we can get the tag sequence "N" "Y" "Y" "Y" "Y" "Y" "Y" "Y" "Y" "Y" "N" "Y" "Y" "Y" "Y" "Y" "Y" "Y" "Y", then the first video can be divided into two video segments, the first 10 images are video segment 1, and the last 10 images are video segment 2. Obviously, each video segment starts with "N" and ends with "Y".

[0054] S103. For each of the video clips, extract all the bullet comments corresponding to the video clip to obtain the bullet comment set corresponding to the video clip; determine the clip description text corresponding to the video clip; and use a second model to extract candidate images in the video clip based on the clip description text and the bullet comment set.

[0055] In the present application, the bullet comments corresponding to each video clip can be collected. These bullet comments are published by users in the user subset after being pre-published in step S101. Continuing with the previous example, we can get the bullet comment set 1 corresponding to video clip 1 and the bullet comment set 2 corresponding to video clip 2. The bullet comments in bullet comment set 1 are published by users in the user subset when playing the image in video clip 1. The bullet comments in bullet comment set 2 are published by users in the user subset when playing the image in video clip 2.

[0056] For each video clip, the corresponding clip description text can be preset by the publisher, and the categories covered by the clip description text include: clarity category, contrast category, expression category and atmosphere category. The clip description text includes multiple text attribute values, each of which indicates the target value of the corresponding category.

[0057] For example, the values ​​of the clarity category can be "high definition", "normal definition", and "ultra-high definition"; the values ​​of the contrast category can be "low contrast", "medium contrast", and "high contrast"; the values ​​of the expression category can be "smile", "laugh", "cry", "frightened", "serious", "frown", "concentrated", and "staring". The values ​​of the atmosphere category can be "warm", "sweet", "scary", "suspenseful", "sad", and "funny".

[0058] Taking video clip 1 as an example, the publisher can determine its corresponding clip description text 1. The clip description text 1 includes 4 text attribute values, which correspond to the clarity category, contrast category, expression category and atmosphere category respectively. The 4 text attribute values ​​are "ultra-clear", "high contrast", "laugh" and "funny", respectively. Then "ultra-clear" is the target value of the clarity category corresponding to the clip description text 1, "high contrast" is the target value of the contrast category corresponding to the clip description text 1, "laugh" is the target value of the expression category corresponding to the clip description text 1, and "funny" is the target value of the atmosphere category corresponding to the clip description text 1. Based on the same inventive concept, the clip description text 2 corresponding to the video clip 2 can be determined.

[0059] For each video segment, the second model can be used to extract candidate images in the video segment based on its corresponding segment description text and corresponding bullet comment set. Figure 3 , which shows a flowchart of the execution steps of the second model in this application. The second model is used to perform the following operations:

[0060] S301. Intelligently classify the barrage set to obtain at least one first barrage set and at least one second barrage set, wherein the first barrage set is composed of barrages with similar general semantics, and the second barrage set is composed of barrages with similar semantics determined based on key information, and the key information is information related to the vertical category corresponding to the first video.

[0061] In one embodiment, the first feature extraction model can be used to extract common features from each barrage to obtain a common feature pool. Taking barrage set 1 corresponding to video clip 1 as an example, common features are extracted from each barrage in barrage set 1 to obtain a common feature pool corresponding to barrage set 1. The first feature extraction model is a model for extracting features from text that is trained using common corpus. The first feature extraction model is a prior art and does not constitute an implementation obstacle. The common features extracted by the model reflect common semantics.

[0062] Next, the universal features in the universal feature pool can be clustered to obtain at least one first barrage set, in which the universal semantic distance between any two barrages in the first barrage set is less than the first distance threshold, and the number of barrages in the first barrage set is greater than the first number threshold. The embodiments of the present application do not limit the specific clustering algorithm and universal semantic distance quantization method used, and do not constitute an implementation obstacle. Taking video clip 1 as an example, based on the universal feature pool corresponding to barrage set 1, the first barrage set 10 and the first barrage set 20 can be obtained.

[0063] The key information of the embodiment of the present application is information indicating the vertical category of the first video. In layman's terms, it is information about the professional field corresponding to the first video, which is known information. Using the previous example, if the first video is a popular law video, the key information indicates the professional field of law.

[0064] In one embodiment, a second feature extraction model can be used to extract special features from each barrage to obtain a special feature pool. The second feature extraction model is a feature extraction model determined based on key information. Continuing with the previous example, the second feature extraction model is a feature extraction model corresponding to the legal professional field. Different key information can indicate different professional fields, and then a second feature extraction model adapted to the corresponding professional field is selected. Taking barrage set 1 corresponding to video clip 1 as an example, the second feature extraction model is used to extract special features from barrage set 1 to obtain a special feature pool corresponding to barrage set 1. The training method of the second feature extraction model can use existing technology, or refer to the following text. The special features extracted by the second feature extraction model reflect the semantics determined based on the key information. The second barrage set obtained based on the special features is a barrage set composed of barrages with similar semantics determined based on the key information.

[0065] Next, each of the special features in the special feature pool can be clustered to obtain at least one second barrage set, wherein the semantic distance between any two barrages in the second barrage set determined based on the key information is less than the second distance threshold, and the number of barrages in the second barrage set is greater than the second number threshold. The embodiment of the present application does not limit the specific clustering algorithm used and the semantic distance quantification method of the barrage of the second barrage set, and does not constitute an implementation obstacle. Taking video clip 1 as an example, based on the special feature pool corresponding to barrage set 1, a second barrage set 30 and a second barrage set 40 are obtained.

[0066] Of course, the aforementioned video clip 2 can also be based on the same inventive concept to obtain corresponding at least one first barrage set and at least one second barrage set.

[0067] In the present application, the first distance threshold is smaller than the second distance threshold, and the first quantity threshold is larger than the second quantity threshold. It can be seen that the requirements for clustering the first barrage set are higher than those for clustering the second barrage set. The first barrage set has a higher degree of aggregation and contains more barrages, which reflects that the requirements for the first barrage set obtained based on general semantics are higher, because general semantics is relatively less correlated with the professional knowledge of the video clip itself than the semantics determined based on key information. In other words, from the perspective of the professional knowledge of the video clip itself, the general semantics contains a higher content of invalid information, so higher aggregation requirements are proposed to avoid invalid general semantics from being aggregated. This setting makes the aggregated first barrage set and the first barrage set more reasonable, and performs barrage aggregation from the perspective of general semantics and professional semantics determined based on key information, so that the aggregation results include both the general semantics that users are very concerned about and the professional semantics that the publisher hopes to popularize, so that the second video obtained subsequently based on this can not only attract user attention, but also achieve the publisher's popularization purpose.

[0068] S302. Based on each of the first barrage sets and each of the second barrage sets, extract multiple sub-segments from the video segment; for each of the sub-segments, use a third model to obtain candidate images in the sub-segment based on the segment description text.

[0069] Please refer to Figure 4 , which shows a schematic flow chart of a method for extracting sub-segments based on a bullet-screen set. The method of extracting a plurality of sub-segments from the video segment based on each of the first bullet-screen sets and each of the second bullet-screen sets includes:

[0070] S401. For each of the first barrage sets, according to the first target moments corresponding to each barrage in the first barrage set, at least one first time period is obtained, and the number of first target moments included in each first time period is greater than the third quantity threshold; in the video segment, the candidate sub-segment corresponding to each first time period is extracted; for each of the second barrage sets, according to the second target moments corresponding to each barrage in the second barrage set, at least one second time period is obtained, and the number of second target moments included in each second time period is greater than the third quantity threshold; in the video segment, the candidate sub-segment corresponding to each second time period is extracted.

[0071] The target time in the embodiment of the present application is the playback time of the first video when the corresponding bullet screen is released. In order to distinguish, the target time of the bullet screen in the first bullet screen set is called the first target time, and the target time of the bullet screen in the second bullet screen set is called the second target time.

[0072] For each first barrage set, at least one first time period can be obtained according to the first target moments corresponding to each barrage in the first barrage set. The number of first target moments included in each first time period is greater than the third quantity threshold. The present application does not limit the specific method of dividing the first time period, does not constitute an implementation obstacle, and does not limit the third quantity threshold, which can be set by the publisher.

[0073] For example, assuming that video clip 1 is 10 seconds in total, it corresponds to the 0th to the 10th second of the first video, which corresponds to the first barrage set 10, the first barrage set 20, the second barrage set 30, and the second barrage set 40. The first target moments of the barrages in the first barrage set 10 are concentrated between the 3rd and 4th seconds, and between the 5th and 6th seconds, thereby determining two first time periods. Correspondingly, the candidate sub-segments corresponding to these two first time periods are [3,4] and [5,6], that is, the segments formed by the content played between the 3rd and 4th seconds (including the endpoints), and between the 5th and 6th seconds (including the endpoints). Based on the same inventive concept, the barrages in the first barrage set 20 can be used to obtain candidate sub-segments [8,9].

[0074] For each second bullet chat set, a corresponding candidate sub-segment can also be obtained. This process is based on the same inventive concept as the method for obtaining a candidate sub-segment for each first bullet chat set, and no further explanation is given. For example, the second bullet chat set 30 is used to obtain the candidate sub-segment [3, 5], but the distribution of the second target time of the bullet chat of the second bullet chat set 40 is not concentrated in a certain time period, so no corresponding candidate sub-segment is obtained.

[0075] In video segment 2, also based on the same inventive concept, several candidate sub-segments can be obtained.

[0076] S402. Fuse the candidate sub-segments that generate the intersection to obtain a fused sub-segment.

[0077] The so-called fusion is to perform a union operation on the candidate sub-segments that have an intersection. The candidate sub-segment [3, 5] mentioned above has an intersection with the candidate sub-segments [3, 4] and [5, 6], and the fused sub-segment [3, 6] is obtained.

[0078] S403. Based on the candidate sub-segments that do not generate intersections and the fused sub-segment, a plurality of sub-segments in the video segment are obtained.

[0079] The candidate sub-segments and fused sub-segments that do not generate intersections can both be used as sub-segments. Using the previous example, video segment 1 has two sub-segments, namely, the candidate sub-segments [8,9] that do not generate intersections and the fused sub-segment [3,6].

[0080] For each sub-segment in each video segment, a candidate image can be determined based on the third model. Specifically, all images in the sub-segment are obtained, and steps S501-S504 are performed for each of the images. Taking video segment 1 as an example, it includes two sub-segments, namely [8,9] and [3,6]. Taking the sub-segment [8,9] as an example, all images therein can be extracted, that is, images played between the 8th second and the 9th second. Steps S501-S504 are performed for each image. Other sub-segments are processed based on the same inventive concept, and other video segments, such as the sub-segments of video segment 2, are also processed based on the same inventive concept. Please refer to Figure 5 , which shows a schematic diagram of the process of processing each image of each sub-segment.

[0081] S501. Perform a cross-category and cross-modality comparison between the image and all the text attribute values ​​to obtain a horizontal fit of the image relative to each of the text attribute values. The total of all horizontal fits output from the cross-category and cross-modality comparison is 1.

[0082] Taking the sub-segment [8,9] as an example, any image in it is taken as an example, and we call it image X to describe steps S501-S504 in detail. The sub-segment [8,9] corresponds to video segment 1. Continuing with the previous example, the segment description text 1 corresponding to video segment 1 includes 4 text attribute values, which are "ultra-clear", "high contrast", "laughing", and "funny". Then, the third model is used to take image X and the text sequence "ultra-clear", "high contrast", "laughing", and "funny" as input to complete a picture-text matching, and output the horizontal matching degrees corresponding to image X and "ultra-clear", "high contrast", "laughing", and "funny" at one time. These horizontal matching degrees are the output results after the picture-text matching of image X with "ultra-clear", "high contrast", "laughing", and "funny" and normalization processing.

[0083] Because "ultra-clear", "high contrast", "laughing", and "funny" correspond to the clarity category, contrast category, expression category, and atmosphere category respectively. So this kind of matching is cross-category matching. Image X is image information, and "ultra-clear", "high contrast", "laughing", and "funny" are text information, so this kind of matching is also cross-modal matching. This task is a cross-category multimodal comparison task, which can also be called a cross-category multimodal matching task.

[0084] The significant advantage of cross-category and cross-modal comparison based on image-text matching over existing technologies is that the comparison results can be obtained based on image-text matching. For example, if the horizontal matching degree between image X and "ultra-clear" is high, it means that image X is ultra-clear, and there is no need to use an additional definition recognition model. If the horizontal matching degree between image X and "laughing" is high, it means that the person in image X is laughing, and there is no need to use an additional expression recognition model.

[0085] The embodiments of the present application do not elaborate on the specific image-text matching process, because the execution details of image-text matching do not constitute an implementation obstacle. There are many existing technologies that can already implement image-text matching based on image-text matching models. The third model is a type of image-text matching model. The most typical image-text matching model is the clip model and various models derived from the clip model.

[0086] Continuing with the previous example, compared to professional clarity recognition models or expression recognition models, although the image-text matching task can directly obtain matching results, its accuracy is not high. The use of the image-text matching model in this application can reduce the model cost, and there is no need to use a corresponding recognition model for each category, but it is necessary to overcome the defect of low precision of the image-text matching model. Therefore, the innovation of this application lies in determining the target fitness of image X by setting cross-category cross-modal comparison and the same-category cross-modal comparison below, so that the degree of coordination between image X and the fragment description text can be obtained based only on the image-text matching model. Even if the accuracy of the image-text matching itself is not high, this application specifically sets up cross-category cross-modal comparison and the same-category cross-modal comparison below to determine the target fitness of image X, so that the target fitness is very accurate. In the case of low accuracy of image-text matching in the prior art, the accuracy of the target fitness is still ensured, which is the invention point of this application.

[0087] S502. For each of the text attribute values, perform the following operations: determine all values ​​of the category corresponding to the text attribute value; perform a cross-modal comparison of the same category between all the values ​​and the image to obtain a first degree of fit of the image relative to each of the values; normalize each of the first degrees of fit to obtain a second degree of fit corresponding to each of the first degrees of fit; and use the second degree of fit of the text attribute value as the vertical degree of fit of the image relative to the text attribute value.

[0088] Taking image X and the text attribute value "ultra-high definition" as an example, the text attribute value "ultra-high definition" corresponds to the definition category, which also includes "high definition" and "normal definition". Therefore, the third model is used to take image X and the text sequence "ultra-high definition", "high definition", and "normal definition" as input to complete a picture-text matching, and output the first matching degrees corresponding to image X and "ultra-high definition", "high definition", and "normal definition" at one time. These first matching degrees are normalized to obtain the corresponding second matching degrees. Since the target value is "ultra-high definition", the second matching degree corresponding to image X and "ultra-high definition" is used as the vertical matching degree of image X relative to "ultra-high definition".

[0089] The third model uses image X and text sequences "UHD", "HD", and "Normal HD" as input to complete the image-text matching and output the first matching degree of each at one time. In this process, "UHD", "HD", and "Normal HD" are in the same category, so this matching is the same category matching. Image X is image information, and "UHD", "HD", and "Normal HD" are text information, so this matching is also cross-modal matching. This task is a multi-modal comparison task of the same category, which can also be called a multi-modal matching task of the same category.

[0090] Based on the same inventive concept, the vertical matching degrees of the image X corresponding to “ultra-clear”, “high contrast”, “laughing” and “funny” are obtained.

[0091] S503. For each of the text attribute values ​​of the image, average the corresponding horizontal fit and the corresponding vertical fit to obtain the corresponding target fit.

[0092] In this step, the target matching degrees of the image X corresponding to “ultra-clear”, “high contrast”, “laughing” and “funny” are obtained.

[0093] S504. Averaging the target matching degrees of the image with respect to each of the text attribute values ​​to obtain the target matching degree of the image, wherein the target matching degree indicates the degree of conformity between the image and the segment description text.

[0094] In this step, the target fitness of image X is obtained. Similarly, the target fitness of each image in each sub-segment in each video segment can be obtained.

[0095] Furthermore, a video segment to be extracted is determined in the sub-segment, wherein the video segment to be extracted includes an image with the largest target fitness in the sub-segment; and a candidate image is determined in the video segment to be extracted.

[0096] Taking the sub-segment [8,9] of video segment 1 as an example, the image with the largest target fitness can be selected and recorded as image Y, and then a video segment to be extracted is determined based on the image Y. The embodiment of the present application does not limit the method for determining a video segment to be extracted based on the image Y, as long as it is located in the sub-segment [8,9] and includes a segment of image Y. For example, it can be a segment consisting of three consecutive images including image Y. All images in the video segment to be extracted can be determined as candidate images. Based on the same reason, corresponding candidate images can be determined for each sub-segment in each video segment.

[0097] S104. The candidate images corresponding to the respective video clips are merged to obtain a second video, where the second video is an automatic editing result of the first video.

[0098] The fusion of the candidate images corresponding to each of the video clips to obtain a second video, wherein the second video is the automatic editing result of the first video, includes: splicing each of the candidate images in the order in which each of the candidate images is played in the first video to obtain the second video; and supporting manual editing of the second video. Specifically, the candidate images of each sub-segment in each video clip are spliced ​​in sequence to obtain the second video, wherein the second video is the automatic editing result of the first video. The embodiment of the present application also supports manual editing of the second video, so that the edited second video can better meet the needs of the user. Of course, the manual editing part can use existing technologies, such as adding, deleting, and modifying certain image frames, and this part will not be elaborated on.

[0099] Furthermore, the second video may be published based on a second account, where the second account is an associated account of the first account, and the second video is an automatic editing result of the first video.

[0100] In an exemplary embodiment, a new training method for the second feature extraction model is proposed:

[0101] (1) Obtain multiple first training samples and multiple second training samples that carry professional field knowledge. In the embodiment of the present application, multiple first training samples and multiple second training samples all have professional knowledge corresponding to a certain professional field. Continuing with the previous example, professional knowledge corresponding to the legal professional field can be obtained. Specifically, a large amount of document data in the legal professional field can be obtained, and core words can be extracted from each of the document data to form a core vocabulary. Extract multiple question-answer pairs from the large amount of document data, and for each question-answer pair, extract the target core words in the answer that are hit by the core vocabulary to form a question-core word pair. Use a part of the question-core word pairs as the first training samples, and another part of the question-core word pairs as the second training samples.

[0102] (2) Each of the first training samples is input into the initial feature extraction model to obtain a first question feature and a corresponding first core word feature, and the parameters of the initial feature extraction model are adjusted to obtain a first intermediate model with the training goal of narrowing the distance between the two features. The present embodiment of the application does not elaborate on the parameter adjustment process based on the training goal, and does not constitute an implementation obstacle. The initial feature extraction model can use a known model of the prior art.

[0103] (3) Input each question in the second training sample into the first intermediate model to obtain a corresponding second question feature. Input each core word in the core word library into the first intermediate model to obtain a corresponding core word reference feature. Select the 10 target core word reference features that are closest to the second question feature from each of the core word reference features. If the core word set formed by the core words corresponding to each of the 10 target core word reference features includes the core word in the second training sample, add a corresponding preset tag to the corresponding second training sample.

[0104] (4) Calculate the total number N1 of all second training samples and the number N2 of second training samples with preset labels, use N1 / N2 as the weight corresponding to the second training samples with preset labels, and use 1 as the weight corresponding to the second training samples without preset labels.

[0105] (5) Input all the second training samples into the first intermediate model, and for each second training sample, obtain the corresponding third question feature and the corresponding second core word feature, and use the product of the feature distance between the third question feature and the corresponding second core word feature and the weight corresponding to the second training sample as the loss component of the second training sample; accumulate the loss components of each second training sample to obtain the total loss, and adjust the first intermediate model parameters based on the total loss to obtain the second intermediate model. The embodiment of the present application does not elaborate on the parameter adjustment process, and does not constitute an implementation obstacle. The embodiment of the present application does not limit the method for calculating the feature distance, and existing technologies can be used.

[0106] (6) For each core word in the core word library, the corresponding explanation text is searched in the document data to form a core word-explanation text pair. Each core word-explanation text pair is input into the second intermediate model to obtain the features corresponding to the core word and the features corresponding to the explanation text. With the training goal of narrowing the distance between the two features, the parameters of the second intermediate model are adjusted to obtain a second feature extraction model. The embodiment of the present application does not elaborate on the parameter adjustment process based on the training goal, and it does not constitute an implementation obstacle.

[0107] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and when the at least one instruction, at least one program, code set or instruction set is executed by a processor, the automatic video editing method is implemented. The method comprises:

[0108] Release the first video based on the first account;

[0109] When the release duration of the first video reaches a first preset duration, segmenting the first video based on a first model to obtain a plurality of video segments;

[0110] For each of the video clips, extract all bullet comments corresponding to the video clip to obtain a bullet comment set corresponding to the video clip; determine a clip description text corresponding to the video clip; and use a second model to extract a candidate image in the video clip based on the clip description text and the bullet comment set;

[0111] The candidate images corresponding to the respective video clips are fused to obtain a second video, where the second video is an automatic editing result of the first video.

[0112] In one embodiment, the method further includes publishing the second video based on a second account, where the second account is an associated account of the first account; before the second video is published, the method further includes: deleting the first video published by the first account.

[0113] In one embodiment, the second model is used to perform the following operations:

[0114] Intelligently classify the barrage set to obtain at least one first barrage set and at least one second barrage set, wherein the first barrage set is composed of barrages with similar general semantics, and the second barrage set is composed of barrages with similar semantics determined based on key information, wherein the key information is information related to the vertical category corresponding to the first video;

[0115] Extracting a plurality of sub-segments from the video segment based on each of the first bullet comment sets and each of the second bullet comment sets;

[0116] For each of the sub-segments, a third model is used to obtain candidate images in the sub-segment based on the segment description text.

[0117] In one embodiment, the general semantic distance between any two bullet comments in the first bullet comment set is less than a first distance threshold, the semantic distance between any two bullet comments in the second bullet comment set determined based on key information is less than a second distance threshold, and the first distance threshold is less than the second distance threshold;

[0118] The number of barrages in the first barrage set is greater than a first number threshold, the number of barrages in the second barrage set is greater than a second number threshold, and the first number threshold is greater than the second number threshold.

[0119] In one embodiment, extracting a plurality of sub-segments from the video segment based on each of the first bullet comment sets and each of the second bullet comment sets includes:

[0120] For each of the first bullet chat sets, according to the first target moments respectively corresponding to the bullet chats in the first bullet chat set, at least one first time period is obtained, and the number of first target moments included in each of the first time periods is greater than a third quantity threshold; in the video segment, a candidate sub-segment corresponding to each of the first time periods is extracted;

[0121] For each of the second bullet chat sets, according to the second target moments respectively corresponding to the bullet chats in the second bullet chat set, at least one second time period is obtained, and the number of second target moments included in each of the second time periods is greater than the third number threshold; in the video segment, a candidate sub-segment corresponding to each of the second time periods is extracted;

[0122] The candidate sub-segments that generate the intersection are merged to obtain a fused sub-segment;

[0123] Based on the candidate sub-segments that do not generate an intersection and the fused sub-segment, a plurality of sub-segments in the video segment are obtained.

[0124] In one embodiment, the segment description text includes a plurality of text attribute values, each of which indicates a target value of a corresponding category, and the third model performs the following operations:

[0125] Obtain all images in the sub-segment, and perform the following operations on each of the images:

[0126] Perform a cross-category and cross-modality comparison between the image and all the text attribute values ​​to obtain a horizontal fit degree of the image relative to each of the text attribute values, and the accumulation of all horizontal fit degrees output by the cross-category and cross-modality comparison is 1;

[0127] For each of the text attribute values, the following operations are performed: determining all values ​​of the category corresponding to the text attribute value; performing a cross-modal comparison of the same category between all the values ​​and the image to obtain a first degree of fit of the image relative to each of the values; normalizing each of the first degrees of fit to obtain a second degree of fit corresponding to each of the first degrees of fit; taking the second degree of fit of the text attribute value as the vertical degree of fit of the image relative to the text attribute value;

[0128] For each of the text attribute values ​​of the image, averaging the corresponding horizontal fit and the corresponding vertical fit to obtain a corresponding target fit;

[0129] Averaging the target matching degrees of the image with respect to each of the text attribute values ​​to obtain the target matching degree of the image, wherein the target matching degree indicates the degree of matching between the image and the segment description text;

[0130] Determining a video segment to be extracted within the sub-segment, wherein the video segment to be extracted includes an image with the largest target fitness;

[0131] A candidate image is determined in the video segment to be extracted.

[0132] In one embodiment, the fusing of the candidate images corresponding to the respective video clips to obtain a second video, where the second video is an automatic editing result of the first video, includes:

[0133] According to the playback order of each candidate image in the first video, the candidate images are spliced ​​to obtain the second video; and manual editing of the second video is supported.

[0134] In one embodiment, the first model is used to perform the following operations:

[0135] For each preset image in the first video, identifying whether the preset image and its previous image belong to the same scene, and setting a mark for the preset image, the mark being used to indicate whether the corresponding preset image and its previous image belong to the same scene, and the preset image is not the first image of the first video;

[0136] The multiple video clips are obtained according to the marks corresponding to the preset images.

[0137] In one embodiment, the categories covered by the segment description text include: clarity category, contrast category, expression category and atmosphere category.

[0138] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0139] It should be understood that the "multiple" mentioned in this article refers to two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated with each other are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram, or a step is not executed. The embodiments of the present application do not limit this.

[0140] In addition, in the specific implementation of this application, account information and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, account permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0141] The above are merely exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video automatic editing method, characterized in that: The method comprises: Release the first video based on the first account; When the release duration of the first video reaches a first preset duration, segmenting the first video based on a first model to obtain a plurality of video segments; For each of the video clips, extract all bullet comments corresponding to the video clip to obtain a bullet comment set corresponding to the video clip; determine a clip description text corresponding to the video clip; and use a second model to extract a candidate image in the video clip based on the clip description text and the bullet comment set; fusing the candidate images corresponding to the video clips to obtain a second video, where the second video is an automatic editing result of the first video; The second model is used to perform the following operations: Intelligently classify the barrage set to obtain at least one first barrage set and at least one second barrage set, wherein the first barrage set is composed of barrages with similar general semantics, and the second barrage set is composed of barrages with similar semantics determined based on key information, wherein the key information is information related to the vertical category corresponding to the first video; Extracting a plurality of sub-segments from the video segment based on each of the first bullet comment sets and each of the second bullet comment sets; For each of the sub-segments, using a third model, obtaining a candidate image in the sub-segment based on the segment description text; The extracting a plurality of sub-segments from the video segment based on each of the first bullet comment sets and each of the second bullet comment sets includes: For each of the first bullet chat sets, according to the first target moments respectively corresponding to the bullet chats in the first bullet chat set, at least one first time period is obtained, and the number of first target moments included in each of the first time periods is greater than a third quantity threshold; in the video segment, a candidate sub-segment corresponding to each of the first time periods is extracted; For each of the second bullet chat sets, according to the second target moments respectively corresponding to the bullet chats in the second bullet chat set, at least one second time period is obtained, and the number of second target moments included in each of the second time periods is greater than the third number threshold; in the video segment, a candidate sub-segment corresponding to each of the second time periods is extracted; The candidate sub-segments that generate the intersection are merged to obtain a fused sub-segment; Based on the candidate sub-segments that do not generate an intersection and the fused sub-segment, a plurality of sub-segments in the video segment are obtained.

2. The method according to claim 1, characterized in that The method further includes publishing the second video based on a second account, where the second account is an associated account of the first account; Before the second video is released, the method further includes: deleting the first video released by the first account.

3. The method according to claim 2, characterized in that The general semantic distance between any two bullet comments in the first bullet comment set is less than a first distance threshold, the semantic distance between any two bullet comments in the second bullet comment set determined based on key information is less than a second distance threshold, and the first distance threshold is less than the second distance threshold; The number of barrages in the first barrage set is greater than a first number threshold, the number of barrages in the second barrage set is greater than a second number threshold, and the first number threshold is greater than the second number threshold.

4. The method according to claim 3, characterized in that: The segment description text includes a plurality of text attribute values, each of which indicates a target value of a corresponding category, and the third model performs the following operations: Obtain all images in the sub-segment, and perform the following operations on each of the images: Perform a cross-category and cross-modality comparison between the image and all the text attribute values ​​to obtain a horizontal fit degree of the image relative to each of the text attribute values, and the accumulation of all horizontal fit degrees output by the cross-category and cross-modality comparison is 1; For each of the text attribute values, the following operations are performed: determining all values ​​of the category corresponding to the text attribute value; performing a cross-modal comparison of the same category between all the values ​​and the image to obtain a first degree of fit of the image relative to each of the values; normalizing each of the first degrees of fit to obtain a second degree of fit corresponding to each of the first degrees of fit; taking the second degree of fit of the text attribute value as the vertical degree of fit of the image relative to the text attribute value; For each of the text attribute values ​​of the image, averaging the corresponding horizontal fit and the corresponding vertical fit to obtain the corresponding target fit; Averaging the target matching degrees of the image with respect to each of the text attribute values ​​to obtain the target matching degree of the image, wherein the target matching degree indicates the degree of matching between the image and the segment description text; Determining a video segment to be extracted within the sub-segment, wherein the video segment to be extracted includes an image with the largest target fitness; A candidate image is determined in the video segment to be extracted.

5. The method according to claim 4, characterized in that The step of fusing the candidate images corresponding to the video clips to obtain a second video, where the second video is an automatic editing result of the first video, includes: According to the playback order of each candidate image in the first video, the candidate images are spliced ​​to obtain the second video; and manual editing of the second video is supported.

6. The method according to claim 5, characterized in that The first model is used to perform the following operations: For each preset image in the first video, identifying whether the preset image and its previous image belong to the same scene, and setting a mark for the preset image, the mark being used to indicate whether the corresponding preset image and its previous image belong to the same scene, and the preset image is not the first image of the first video; The multiple video clips are obtained according to the marks corresponding to the preset images.

7. The method according to claim 6, characterized in that The categories covered by the segment description text include: clarity category, contrast category, expression category and atmosphere category.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the automatic video editing method as described in any one of claims 1 to 7.

Images