Video generation method and apparatus, computer device, and storage medium

Abstract

A video generation method and apparatus, a computer device, and a storage medium. The method comprises: acquiring a video to be processed; performing beat matching detection on a target video picture of said video on the basis of a detection model, to determine a first beat matching detection result corresponding to said video; determining target music matching the first beat matching detection result, and adjusting said video on the basis of a second beat matching detection result of the target music and the first beat matching detection result; and generating a target beat-matched video on the basis of said adjusted video and the target music. By means of the method, the video generation effect and the video generation efficiency can be improved.

Description

Video generation method, device, computer equipment and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202311277823.4 filed on September 28, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a video generation method, apparatus, computer equipment, and storage medium. Background Art

[0003] With the development of short videos, more and more people are posting various videos online. These videos often include background music. To enhance the video's quality, the background music often needs to be time-matched to the video content. For example, in a dance video, the dance moves need to be time-matched to the music. Related technologies often require manual editing and adjustment to achieve time-matched video and music, which is inefficient.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure at least provide a video generation method, apparatus, computer equipment, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a video generation method, comprising:

[0007] Get the video to be processed;

[0008] Performing card point detection on the target video screen of the video to be processed based on the detection model, and determining a first card point detection result corresponding to the video to be processed;

[0009] Determining target music that matches the first stick point detection result, and adjusting the to-be-processed video based on a second stick point detection result of the target music and the first stick point detection result;

[0010] A target card point video is generated based on the adjusted video to be processed and the target music.

[0011] In a possible implementation, the video to be processed is a video containing target object motion; the detection model includes an image feature extraction sub-model and a motion information extraction sub-model;

[0012] The performing card point detection on the target video screen of the to-be-processed video based on the detection model to determine a first card point detection result corresponding to the to-be-processed video includes:

[0013] Extracting target image features of the target video picture based on the image feature extraction sub-model; and extracting motion features in the target video picture for representing motion information of the target object based on the motion information extraction sub-model;

[0014] The first sticking point detection result is determined based on the target image feature and the motion feature.

[0015] In a possible implementation, determining the first sticky point detection result based on the target image feature and the motion feature includes:

[0016] Fusing the target image feature and the motion feature to obtain a fused feature;

[0017] Processing the fused features based on the attention module to obtain target encoding features;

[0018] The target coding feature is processed based on a fully connected layer to determine the first sticking point detection result.

[0019] In one possible implementation, performing card point detection on a target video screen of the video to be processed based on a detection model to determine a first card point detection result corresponding to the video to be processed includes:

[0020] Performing frame extraction processing on the video frames of the video to be processed to obtain multiple frame extracted video frames;

[0021] Inputting the extracted video frames into the detection model to determine detection results corresponding to each extracted video frame; wherein the detection results corresponding to the extracted video frames include a probability that the extracted video frame is a video frame corresponding to a card point position, and a card point strength at the video position where the extracted video frame is located;

[0022] Based on the pre-set screening conditions, the detection results corresponding to each extracted video frame are filtered to determine the first card point detection result corresponding to the video to be processed.

[0023] In one possible implementation, determining target music that matches the first sticking point detection result includes:

[0024] Determine the second card point detection result of each candidate music;

[0025] The first stick point detection result is matched with the second stick point detection result of each candidate music, and target music that matches the first stick point detection result is determined based on the matching result.

[0026] In a possible implementation, adjusting the to-be-processed video based on the second stick point detection result and the first stick point detection result of the target music includes:

[0027] Determining difference information between the first checkpoint detection result and the second checkpoint detection result;

[0028] Determining speed information of each video segment of the video to be processed based on the difference information;

[0029] The video segment of the video to be processed is adjusted according to the speed information; wherein the difference between the card point detection result of the video to be processed after adjustment and the second card point detection result of the target music is smaller than the difference before adjustment.

[0030] In a possible implementation, the first card point detection result corresponding to the to-be-processed video includes a first card point position of the target video screen and a card point strength of the first card point position;

[0031] The second sticking point detection result includes the second sticking point position of the candidate music and the sticking point strength of the second sticking point position of the candidate music;

[0032] The determining difference information between the first checkpoint detection result and the second checkpoint detection result includes:

[0033] Determining a first time interval between adjacent first checkpoint positions and a second time interval between adjacent second checkpoint positions;

[0034] The difference information between the first time interval and the second time interval is determined.

[0035] In one possible implementation, the detection model is adjusted based on the following method:

[0036] Obtaining a reference video; wherein the difference between the card point detection result of the video image and the card point detection result of the audio contained in the reference video meets a preset condition;

[0037] Separating the video image and the audio from the reference video, and performing card point detection on the audio to determine supervision card point information;

[0038] Performing card point detection on the video image based on the detection model to be adjusted to determine predicted card point information;

[0039] The detection model to be adjusted is adjusted based on the supervised checkpoint information and the predicted checkpoint information.

[0040] In a second aspect, an embodiment of the present disclosure further provides a video generation device, including:

[0041] An acquisition module is used to obtain the video to be processed;

[0042] a determination module, configured to perform card point detection on a target video screen of the video to be processed based on a detection model, and determine a first card point detection result corresponding to the video to be processed;

[0043] a matching module, configured to determine target music that matches the first stick point detection result, and adjust the to-be-processed video based on a second stick point detection result of the target music and the first stick point detection result;

[0044] A generation module is used to generate a target card point video based on the adjusted video to be processed and the target music.

[0045] In a possible implementation, the video to be processed is a video containing target object motion; the detection model includes an image feature extraction sub-model and a motion information extraction sub-model;

[0046] The determination module is configured to, when performing card point detection on the target video screen of the video to be processed based on the detection model and determining a first card point detection result corresponding to the video to be processed,:

[0047] Extracting target image features of the target video picture based on the image feature extraction sub-model; and extracting motion features in the target video picture for representing motion information of the target object based on the motion information extraction sub-model;

[0048] The first sticking point detection result is determined based on the target image feature and the motion feature.

[0049] In one possible implementation, when determining the first sticking point detection result based on the target image feature and the motion feature, the determination module is configured to:

[0050] Fusing the target image feature and the motion feature to obtain a fused feature;

[0051] Processing the fused features based on the attention module to obtain target encoding features;

[0052] The target coding feature is processed based on a fully connected layer to determine the first sticking point detection result.

[0053] In one possible implementation, the determination module, when performing card point detection on a target video screen of the video to be processed based on the detection model and determining a first card point detection result corresponding to the video to be processed, is configured to:

[0054] Performing frame extraction processing on the video frames of the video to be processed to obtain multiple frame extracted video frames;

[0055] Inputting the extracted video frames into the detection model to determine detection results corresponding to each extracted video frame; wherein the detection results corresponding to the extracted video frames include a probability that the extracted video frame is a video frame corresponding to a card point position, and a card point strength at the video position where the extracted video frame is located;

[0056] Based on the pre-set screening conditions, the detection results corresponding to each extracted video frame are filtered to determine the first card point detection result corresponding to the video to be processed.

[0057] In one possible implementation, the matching module, when determining target music that matches the first sticking point detection result, is configured to:

[0058] Determine the second card point detection result of each candidate music;

[0059] The first stick point detection result is matched with the second stick point detection result of each candidate music, and target music that matches the first stick point detection result is determined based on the matching result.

[0060] In one possible implementation, when adjusting the to-be-processed video based on the second stick point detection result and the first stick point detection result of the target music, the matching module is configured to:

[0061] Determining difference information between the first checkpoint detection result and the second checkpoint detection result;

[0062] Determining speed information of each video segment of the video to be processed based on the difference information;

[0063] The video segment of the video to be processed is adjusted according to the speed information; wherein the difference between the card point detection result of the video to be processed after adjustment and the second card point detection result of the target music is smaller than the difference before adjustment.

[0064] In a possible implementation, the first card point detection result corresponding to the to-be-processed video includes a first card point position of the target video screen and a card point strength of the first card point position;

[0065] The second sticking point detection result includes the second sticking point position of the candidate music and the sticking point strength of the second sticking point position of the candidate music;

[0066] The matching module is configured to, when determining difference information between the first checkpoint detection result and the second checkpoint detection result:

[0067] Determining a first time interval between adjacent first checkpoint positions and a second time interval between adjacent second checkpoint positions;

[0068] The difference information between the first time interval and the second time interval is determined.

[0069] In a possible implementation, the device further includes an adjustment module configured to:

[0070] Obtaining a reference video; wherein the difference between the card point detection result of the video image and the card point detection result of the audio contained in the reference video meets a preset condition;

[0071] Separating the video image and the audio from the reference video, and performing card point detection on the audio to determine supervision card point information;

[0072] Performing card point detection on the video image based on the detection model to be adjusted to determine predicted card point information;

[0073] The detection model to be adjusted is adjusted based on the supervised checkpoint information and the predicted checkpoint information.

[0074] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0075] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0076] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0078] FIG1 shows a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0079] FIG2 shows a flow chart of a method for adjusting a detection model provided by an embodiment of the present disclosure;

[0080] FIG3 shows a schematic diagram of a video generation device provided by an embodiment of the present disclosure;

[0081] FIG4 shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0083] With the development of short videos, more and more people are posting various videos online. These videos often include background music. To enhance the video's quality, the background music often needs to be time-matched to the video content. For example, in a dance video, the dance moves need to be time-matched to the music. Related technologies often require manual editing and adjustment to achieve time-matched video and music, which is inefficient.

[0084] Based on the above research, the present disclosure provides a video generation method, device, computer equipment and storage medium. After obtaining the video to be processed, the target video picture of the video to be processed can be detected by the detection model, and the target music matching the first card point detection result of the video to be processed can be determined, and the video to be processed is adjusted based on the second card point detection result of the target music and the first card point detection result, and then the target card point video is generated based on the adjusted video to be processed and the target music, wherein the target card point video is a video matching the card point of the target music. In this way, the target music closest to the card point can be automatically matched to the video to be processed, and by adjusting the video to be processed, the target video picture of the video to be processed and the target music can be synchronized at the card point, thereby improving the video effect of the generated target card point video and improving the video editing efficiency.

[0085] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0086] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0087] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0088] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0089] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0090] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0091] To facilitate understanding of this embodiment, a video generation method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the video generation method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. The computer device includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the video generation method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0092] 1 is a flowchart of a video generation method provided by an embodiment of the present disclosure. The method includes steps 101 to 104, wherein:

[0093] Step 101: Obtain the video to be processed.

[0094] Step 102: Perform card point detection on the target video frame of the video to be processed based on the detection model to determine a first card point detection result corresponding to the video to be processed.

[0095] Step 103: Determine target music that matches the first card point detection result, and adjust the video to be processed based on the second card point detection result of the target music and the first card point detection result.

[0096] Step 104: Generate a target card point video based on the adjusted video to be processed and the target music.

[0097] The following is a detailed description of the above steps.

[0098] For step 101,

[0099] The video to be processed may be a video to be subjected to card point adjustment. For example, the video to be processed may be a dance video, a sports video, or the like.

[0100] Optionally, the video to be processed may only have video images without audio, or the video to be processed may contain audio. After the video to be processed is acquired, the audio of the video to be processed may be separated and the separated audio may be deleted.

[0101] In one possible implementation, the process of obtaining the video to be processed may be obtaining a video captured in real time by the user, or receiving a video uploaded locally. The process of obtaining the video to be processed may be performed after authorization. For the specific authorization process, refer to the description of authorization above.

[0102] For step 102,

[0103] Optionally, the first card point detection result corresponding to the video to be processed may include the first card point position of the target video screen and the card point strength of the first card point position.

[0104] Among them, the first card point position can refer to the end time point of the execution action of the target object in the target video screen, where the execution action here can refer to a single action or a group of actions; the card point strength of the first card point position can be used to characterize the level of the card point, for example, it can include weak, medium, strong, etc.

[0105] In one possible implementation, to improve the efficiency of stuck point detection, the target video image may be a multi-frame extracted video frame obtained by extracting the video frames of the video to be processed. Thus, when performing stuck point detection, the multi-frame extracted video frame may be input into the detection model for stuck point detection.

[0106] In a possible scenario, the strength of the stuck point can be represented by the magnitude of the movement amplitude of the target object at the end of the movement / the speed of the movement execution, etc.

[0107] The detection model is a model for performing card point detection. Optionally, the detection model may include an image feature extraction sub-model, an action information extraction sub-model, and an attention module.

[0108] In one possible implementation, the video to be processed is a video containing a target object in motion. When performing card point detection on a target video frame of the video to be processed based on the detection model and determining a first card point detection result corresponding to the video to be processed, the following steps may be employed:

[0109] Step a1: extracting target image features of the target video picture based on the image feature extraction submodel; and extracting motion features in the target video picture for representing motion information of the target object based on the motion information extraction submodel.

[0110] Here, the target video picture may include multiple video frames. When extracting the target image features of the target video picture based on the image feature extraction sub-model, in addition to extracting the image features of each video frame, the inter-frame features between each video frame can also be extracted. Therefore, the extracted target image features contain temporal feature information.

[0111] When the target video screen contains a person, before extracting the motion features in the target video screen based on the action information extraction sub-model, the character action skeleton in the target video screen can also be extracted first, and then the motion features can be determined based on the extracted character action skeleton and the action information extraction sub-model.

[0112] For example, when extracting a character action skeleton, it can be extracted through a human posture estimation network, and the human posture and action can be estimated by analyzing the human key points in the video frame. The character action skeleton can contain the position information of the character key points in the video frame.

[0113] The motion information extraction sub-model may exemplarily be a convolution and mask image autoencoder, which can recognize the character action skeleton and extract the motion information.

[0114] Step a2: Determine the first card point detection result based on the target image feature and the motion feature.

[0115] In one possible implementation, when determining the first card point detection result based on the target image features and the motion features, the target image features and the motion features can be first fused to obtain a fused feature; then the fused feature is processed based on the attention module to obtain a target coding feature, and then the target coding feature is processed based on the fully connected layer to determine the first card point detection result.

[0116] Among them, when fusing the target image features and the motion features, the target image features and the motion features can be processed in series; before processing the fused features based on the attention module, the fused features can be pooled first, for example, the fused features can be average pooled to reduce the feature dimension of the fused features.

[0117] The main function of the self-attention module is to encode and decode the fused features to extract deeper features.

[0118] In one possible implementation, when determining the first card point detection result corresponding to the video to be processed, the above-mentioned frame-by-frame video frames can be input into the detection model to determine the detection results corresponding to each frame-by-frame video frame; wherein the detection results corresponding to the frame-by-frame video frames include the probability that the frame-by-frame video frame is the video frame corresponding to the card point position, and the card point strength of the video position where the frame-by-frame video frame is located; and then based on pre-set filtering conditions, the detection results corresponding to each frame-by-frame video frame are filtered to determine the first card point detection result corresponding to the video to be processed.

[0119] The detection result can be exemplarily represented by a ternary sequence. For example, the detection result corresponding to a certain video frame may be (non-stuck point, 0, 80%), which indicates that the video frame is not a video frame corresponding to a stuck point position, the stuck point strength is 0, and the confidence level of the result is 80%; if the detection result corresponding to a certain video frame is (stuck point, weak, 70%), it indicates that the video frame is a video frame corresponding to a stuck point position, the stuck point strength is weak, and the confidence level of the result is 70%.

[0120] The detection result probability corresponding to each sampled video frame based on the pre-set filtering condition can be a pre-set condition for filtering the strength of the stuck point. For example, the pre-set filtering condition can be: filtering video frames with a strong strength and a probability greater than 90%. For video frames that meet the filtering condition, the position of the video frame in the video to be processed can be determined as the stuck point location in the first stuck point detection result, and the stuck point strength of the video frame can be determined as the stuck point strength corresponding to the stuck point location. Video frames that do not meet the filtering condition can be uniformly set as non-stuck points.

[0121] In this way, there may be some video frames whose corresponding detection results are stuck points after being detected by the detection model, but after being filtered by the filtering conditions, the corresponding stuck point results are set to non-stuck points.

[0122] Optionally, the first card point detection result may be represented in the form of a sequence, where the sequence includes each card point position (ie, the position in the video to be processed) and the card point strength corresponding to the card point position.

[0123] For step 103 and step 104,

[0124] The target music that matches the first sticking point detection result may refer to music that relatively matches the first sticking point detection result in terms of both the sticking point position and the sticking point strength.

[0125] Exemplarily, when determining the target music that matches the first card point detection result, you can first determine the second card point detection result corresponding to each candidate music, then match the first card point detection result and the second card point detection result, and determine the target music that matches the first card point detection result based on the matching results.

[0126] The candidate music may refer to music stored in a database, and the second card point detection result of each candidate music may be predetermined.

[0127] Here, the music's card point detection is different from the video frame's card point detection. Exemplarily, when determining the second card point detection result of the candidate music, a spectrum analysis method can be used.

[0128] For example, a short-time Fourier transform algorithm can be used to convert the candidate music audio file from the time domain to the frequency domain, and the frequency domain spectrum can be analyzed to determine the location and strength of the stuck point. The stuck point location can be the time position where a peak or trough occurs, and the stuck point strength can be determined by the energy or amplitude of the frequency at the stuck point location. For example, the average value of the frequency spectrum row or column can be used as the stuck point strength.

[0129] When matching the first card point detection result with the second card point detection result of each candidate music, the first card point position in the first card point detection result can be matched with the second card point position in the second card point detection result, and the card point strength of the first card point position can be matched with the card point strength of the second card point position.

[0130] Optionally, the second card point detection result can also be represented by a sequence, and its representation method is similar to that of the first card point detection result; when matching the first card point detection result with the second card point detection result of each candidate music, the distance between the sequences of the two card point detection results can be detected by the dynamic time warping method. The smaller the distance, the more similar the two sequences are in shape and trend, that is, the higher the similarity.

[0131] When determining the target music that matches the first card point detection result based on the matching result, the music with the highest similarity may be selected as the target music, that is, the candidate music with the smallest distance may be selected as the target music.

[0132] In another possible implementation, since the amount of music stored in the database may be large, if the first card point detection result is matched with the second card point detection results of all candidate music in each matching process, the matching efficiency may be low.

[0133] Therefore, the video style of the video to be processed can be determined first, and then based on the style tags carried by each piece of music stored in the database, candidate music matching the video style can be screened, and then the target music can be determined based on the second card point detection results and the first card point detection results of the screened candidate music.

[0134] Exemplarily, the video style of the video to be processed may be detected based on a style detection network.

[0135] The adjusting of the video to be processed may refer to adjusting the speed of different video segments of the video to be processed, where the adjusted speed is related to the first stuck point detection result and the second stuck point detection result.

[0136] In a possible implementation, when adjusting the to-be-processed video based on the first and second sticky point detection results, the following steps may be referred to:

[0137] Step b1: Determine difference information between the first checkpoint detection result and the second checkpoint detection result.

[0138] Step b2: Determine the speed information of each video segment of the video to be processed based on the difference information.

[0139] Step b3: adjusting the video to be processed according to the speed information; wherein the difference between the card point detection result of the video to be processed after adjustment and the second card point detection result of the target music is smaller than the difference before adjustment.

[0140] Specifically, the first card point detection result corresponding to the video to be processed includes the first card point position of the target video screen and the card point strength of the first card point position; the second card point detection result includes the second card point position of the candidate music and the card point strength of the second card point position of the candidate music; when determining the difference information between the first card point detection result and the second card point detection result, a first time interval between adjacent first card point positions and a second time interval between adjacent second card point positions can be first determined; and then the difference information between the first time interval and the second time interval can be determined.

[0141] Specifically, for each card point position in the video to be processed, there is a matching card point position in the target music. Then, the first time interval between two adjacent card point positions in the video to be processed may be different from the time interval between two corresponding card point positions in the target music. The difference in time intervals can reflect the difference information between the first card point detection result and the second card point detection result.

[0142] For example, if the correspondence between the position of the checkpoint in the first checkpoint detection result and the position of the checkpoint in the second detection result is as shown in Table 1 below:

[0143] Table 1

[0144] The difference information between the first checkpoint detection result and the second checkpoint detection result can be represented by the difference between the first time interval between A1 and A2 and the second time interval between B1 and B2, the first time interval between A2 and A3 and the second time interval between B2 and B3, the first time interval between A3 and A4 and the second time interval between B3 and B4, etc.

[0145] Optionally, if the difference between the first time interval between any two first checkpoint positions and the second time interval between the corresponding two second checkpoint positions is too large (e.g., greater than a preset multiple), it may be due to the checkpoint detection accuracy. Therefore, this difference information may not be used as the difference information between the first checkpoint detection result and the second checkpoint detection result.

[0146] In a possible implementation, the difference information between the first time interval and the second time interval may refer to a multiple relationship between the first time interval and the second time interval, that is, a ratio between the second time interval and the first time interval.

[0147] After determining the difference information, when determining the speed information of each video segment of the video to be processed based on the difference information, the video segment may be a video segment between two first card point positions, and the speed information of the video segment may be the ratio between the second time interval corresponding to the two first card point positions and the first time interval.

[0148] For example, if the timestamp of the first card point position 1 (i.e., the time position in the video to be processed) is 1 minute and 0 seconds, the timestamp of the first card point position 2 is 1 minute and 2 seconds, the timestamp of the second card point position 1 corresponding to the first card point position 1 (i.e., the time position in the target music) is 3 minutes and 2 seconds, and the timestamp of the second card point position 2 corresponding to the first card point position 1 is 3 minutes and 8 seconds, then the first time interval is 1 minute and 0 seconds to 1 minute and 2 seconds = 2 seconds, and the second time interval is 3 minutes and 2 seconds to 3 minutes and 8 seconds = 6 seconds, then the speed information is 2 seconds / 6 seconds = 1 / 3.

[0149] When adjusting the video segments of the video to be processed according to the speed information, the speed of each video segment may be adjusted according to the speed information corresponding to each video segment to obtain an adjusted video to be processed.

[0150] Here, the purpose of adjusting the video clip of the video to be processed is to make the card point information of the adjusted video to be processed closer to the card point information of the target music, thereby achieving the card point of the video content and the music; if the video clip of the video to be processed is adjusted too much, it may affect the display effect of the video.

[0151] Therefore, the maximum adjustment speed and the minimum adjustment speed can be set. If the calculated speed information is between the maximum adjustment speed and the minimum adjustment speed, the corresponding video clip can be adjusted directly according to the calculated speed information. If the calculated speed information is not between the maximum adjustment speed and the minimum adjustment speed, the corresponding video clip can be adjusted directly according to the maximum adjustment speed or the minimum adjustment speed. Here, the specific selection of the maximum adjustment speed or the minimum adjustment speed can be related to the calculated speed information, and the adjustment speed closest to the calculated speed information can be taken.

[0152] After adjusting the video to be processed, the adjusted video to be processed and the target music may be merged to synthesize the target card point video.

[0153] Here, the target card point video is a video in which the video screen matches the music card points. The card point matching does not mean that the position of each card point is the same. Compared with the video to be processed and the target music, the card point matching degree of the target card point video is higher.

[0154] The following describes the adjustment process of the detection model. FIG2 shows a method for adjusting the detection model provided by an embodiment of the present disclosure, which includes the following steps:

[0155] Step 201: Obtain a reference video; wherein the difference between the card point detection result of the video image and the card point detection result of the audio contained in the reference video meets a preset condition.

[0156] Here, the reference videos can be manually edited videos for card point matching, where the target object's motion card points are relatively consistent with the music card points. It should be noted that due to the large number of reference videos, they may not be selected through automatic card point detection, but rather manually selected videos with good card point matching.

[0157] Step 202: Separate the video image and audio from the reference video, perform card point detection on the audio, and determine supervision card point information.

[0158] Here, the process of detecting audio card points is described with reference to the above embodiment and will not be repeated here.

[0159] Since the reference video itself has a better card point effect, after the audio is carded, the card point detection result of the video should theoretically be consistent with that of the audio. Therefore, the card point detection result of the audio can be used as supervision information.

[0160] In practical applications, the supervised checkpoint information can be a sequence of annotations linearly related to the duration of the reference video. Specifically, for each video, the video can be divided into N parts at preset time intervals. The length of the annotation sequence is N, and each element in the annotation sequence indicates whether the time node contains a checkpoint and the strength of the checkpoint.

[0161] For example, if the value is 0, it means that the time node is not a stuck point. If it is not 0, it means that the time node is a stuck point. The stuck point strength can be represented by S. The value of S can include (1, 2, 3). Different values ​​represent different stuck point strengths. The larger the value, the higher the strength.

[0162] Step 203: Perform card point detection on the video image based on the detection model to be adjusted to determine predicted card point information.

[0163] Step 204: Adjust the detection model to be adjusted based on the supervised checkpoint information and the predicted checkpoint information.

[0164] Specifically, the loss value may be calculated based on the supervised checkpoint information and the predicted checkpoint information, and the detection model to be adjusted may be adjusted based on the calculated loss value.

[0165] Here, the image feature extraction sub-model and the motion information extraction sub-model included in the card point detection model can be pre-trained models, and can be trained exemplarily by self-supervised pre-training. The specific training method is not limited in this disclosure.

[0166] In the video generation method provided by the embodiment of the present disclosure, after obtaining the video to be processed, the target video picture of the video to be processed can be detected by using a detection model, and the target music that matches the first card point detection result of the video to be processed can be determined, and the video to be processed is adjusted based on the second card point detection result of the target music and the first card point detection result, and then the target card point video is generated based on the adjusted video to be processed and the target music, wherein the target card point video is a video that matches the card point of the target music. In this way, the target music of the card point can be automatically matched for the video to be processed, and by adjusting the video to be processed, the target video picture of the video to be processed can be made to be more matched with the target music, thereby improving the video effect of the generated target card point video and improving the editing efficiency of the video.

[0167] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0168] Based on the same inventive concept, the embodiment of the present disclosure also provides a video generation device corresponding to the video generation method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned video generation method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0169] 3 , which is a schematic diagram of the architecture of a video generation device provided by an embodiment of the present disclosure, the device includes: an acquisition module 301, a determination module 302, a matching module 303, a generation module 304, and an adjustment module 305; wherein,

[0170] An acquisition module 301 is used to acquire a video to be processed;

[0171] A determination module 302 is configured to perform card point detection on a target video frame of the video to be processed based on a detection model, and determine a first card point detection result corresponding to the video to be processed;

[0172] A matching module 303 is configured to determine target music that matches the first stick point detection result, and adjust the video to be processed based on the second stick point detection result of the target music and the first stick point detection result;

[0173] The generating module 304 is configured to generate a target card point video based on the adjusted video to be processed and the target music.

[0174] In a possible implementation, the video to be processed is a video containing target object motion; the detection model includes an image feature extraction sub-model and a motion information extraction sub-model;

[0175] The determination module 302 is configured to, when performing card point detection on the target video frame of the video to be processed based on the detection model and determining a first card point detection result corresponding to the video to be processed:

[0176] Extracting target image features of the target video picture based on the image feature extraction sub-model; and extracting motion features in the target video picture for representing motion information of the target object based on the motion information extraction sub-model;

[0177] The first sticking point detection result is determined based on the target image feature and the motion feature.

[0178] In one possible implementation, when determining the first sticking point detection result based on the target image feature and the motion feature, the determining module 302 is configured to:

[0179] Fusing the target image feature and the motion feature to obtain a fused feature;

[0180] Processing the fused features based on the attention module to obtain target encoding features;

[0181] The target coding feature is processed based on a fully connected layer to determine the first sticking point detection result.

[0182] In one possible implementation, the determination module 302, when performing card point detection on the target video screen of the video to be processed based on the detection model and determining a first card point detection result corresponding to the video to be processed, is configured to:

[0183] Performing frame extraction processing on the video frames of the video to be processed to obtain multiple frame extracted video frames;

[0184] Inputting the extracted video frames into the detection model to determine detection results corresponding to each extracted video frame; wherein the detection results corresponding to the extracted video frames include a probability that the extracted video frame is a video frame corresponding to a card point position, and a card point strength at the video position where the extracted video frame is located;

[0185] Based on the pre-set screening conditions, the detection results corresponding to each extracted video frame are filtered to determine the first card point detection result corresponding to the video to be processed.

[0186] In one possible implementation, when determining target music that matches the first stick point detection result, the matching module 303 is configured to:

[0187] Determine the second card point detection result of each candidate music;

[0188] The first stick point detection result is matched with the second stick point detection result of each candidate music, and target music that matches the first stick point detection result is determined based on the matching result.

[0189] In one possible implementation, when adjusting the to-be-processed video based on the second stick point detection result and the first stick point detection result of the target music, the matching module 303 is configured to:

[0190] Determining difference information between the first checkpoint detection result and the second checkpoint detection result;

[0191] Determining speed information of each video segment of the video to be processed based on the difference information;

[0192] The video segment of the video to be processed is adjusted according to the speed information; wherein the difference between the card point detection result of the video to be processed after adjustment and the second card point detection result of the target music is smaller than the difference before adjustment.

[0193] In a possible implementation, the first card point detection result corresponding to the to-be-processed video includes a first card point position of the target video screen and a card point strength of the first card point position;

[0194] The second sticking point detection result includes the second sticking point position of the candidate music and the sticking point strength of the second sticking point position of the candidate music;

[0195] The matching module 303 is configured to, when determining the difference between the first checkpoint detection result and the second checkpoint detection result:

[0196] Determining a first time interval between adjacent first checkpoint positions and a second time interval between adjacent second checkpoint positions;

[0197] The difference information between the first time interval and the second time interval is determined.

[0198] In a possible implementation, the apparatus further includes an adjustment module 305 configured to:

[0199] Obtaining a reference video; wherein the difference between the card point detection result of the video image and the card point detection result of the audio contained in the reference video meets a preset condition;

[0200] Separating the video image and the audio from the reference video, and performing card point detection on the audio to determine supervision card point information;

[0201] Performing card point detection on the video image based on the detection model to be adjusted to determine predicted card point information;

[0202] The detection model to be adjusted is adjusted based on the supervised checkpoint information and the predicted checkpoint information.

[0203] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0204] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Referring to Figure 4, there is a schematic diagram of the structure of the computer device 400 provided in the embodiment of the present disclosure. The computer device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters. The computer device shown in Figure 4 is only an example and should not bring any limitations to the functions and scope of use of the embodiment of the present disclosure.

[0205] As shown in FIG4 , a computer device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory device (ROM) 402 or a program loaded from a storage device 408 into a random access memory device (RAM) 403. Various programs and data required for the operation of the computer device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0206] Typically, the following devices may be connected to I / O interface 405: input device 406 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output device 407 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 408 including, for example, a magnetic tape, hard disk, etc.; and communication device 409. Communication device 409 may allow computer device 400 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 illustrates computer device 400 with various devices, it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0207] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the video generation method. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0208] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the video generation method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0209] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the video generation method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0210] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0213] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0214] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0215] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A video generation method, comprising: Get the video to be processed; Performing card point detection on the target video screen of the video to be processed based on the detection model, and determining a first card point detection result corresponding to the video to be processed; Determining target music that matches the first card point detection result, and adjusting the to-be-processed video based on a second card point detection result of the target music and the first card point detection result; A target card point video is generated based on the adjusted video to be processed and the target music.

2. The video generation method according to claim 1, wherein: The video to be processed is a video containing the motion of the target object; the detection model includes an image feature extraction sub-model and a motion information extraction sub-model; The performing card point detection on the target video screen of the video to be processed based on the detection model to determine a first card point detection result corresponding to the video to be processed includes: Extracting target image features of the target video picture based on the image feature extraction submodel; and extracting motion features in the target video picture for characterizing motion information of the target object based on the motion information extraction submodel; The first sticking point detection result is determined based on the target image feature and the motion feature.

3. The video generation method according to claim 2, wherein: The determining the first sticking point detection result based on the target image feature and the motion feature includes: Fusing the target image feature and the motion feature to obtain a fused feature; Processing the fused features based on the attention module to obtain target encoding features; The target coding feature is processed based on a fully connected layer to determine the first card point detection result.

4. The video generation method according to claim 1, wherein: The performing card point detection on the target video screen of the video to be processed based on the detection model to determine a first card point detection result corresponding to the video to be processed includes: Performing frame extraction processing on the video frames of the video to be processed to obtain multiple frame extracted video frames; Input the extracted video frames into the detection model to determine the detection results corresponding to each extracted video frame; wherein the detection results corresponding to the extracted video frames include the probability that the extracted video frame is a video frame corresponding to a stuck point position, and the stuck point strength of the video position where the extracted video frame is located; Based on the pre-set screening conditions, the detection results corresponding to each extracted video frame are filtered to determine the first card point detection result corresponding to the video to be processed.

5. The video generation method according to any one of claims 1 to 4, wherein: The determining of the target music matching the first stick point detection result includes: Determine the second card point detection result of each candidate music; The first card point detection result is matched with the second card point detection result of each candidate music, and based on the matching result Determine target music that matches the first card point detection result.

6. The video generation method according to any one of claims 1 to 5, wherein: The adjusting the video to be processed based on the second card point detection result of the target music and the first card point detection result includes: Determine difference information between the first checkpoint detection result and the second checkpoint detection result; Based on the difference information, determining speed information of each video segment of the video to be processed; The video segment of the video to be processed is adjusted according to the speed information; wherein the difference between the card point detection result of the video to be processed after the adjustment and the second card point detection result of the target music is smaller than the difference before the adjustment.

7. The video generation method according to claim 6, wherein: The first card point detection result corresponding to the video to be processed includes the first card point position of the target video screen and the card point strength of the first card point position; The second sticking point detection result includes the second sticking point position of the candidate music and the sticking point strength of the second sticking point position of the candidate music; The determining difference information between the first checkpoint detection result and the second checkpoint detection result includes: Determine a first time interval between adjacent positions of the first checkpoints, and a second time interval between adjacent positions of the second checkpoints; The difference information between the first time interval and the second time interval is determined.

8. The video generation method according to any one of claims 1 to 7, wherein: The detection model is adjusted based on the following method: Acquire a reference video; wherein the difference between the card point detection result of the video screen and the card point detection result of the audio contained in the reference video meets a preset condition; Separating the video image and the audio from the reference video, and performing card point detection on the audio to determine the supervision card point information; Performing card point detection on the video screen based on the detection model to be adjusted to determine predicted card point information; The detection model to be adjusted is adjusted based on the supervised checkpoint information and the predicted checkpoint information.

9. A video generating device, comprising: An acquisition module, configured to acquire a video to be processed; A determination module is configured to perform a card point detection on a target video screen of the video to be processed based on a detection model, and determine a first card point detection result corresponding to the video to be processed; a matching module configured to determine a target music that matches the first card point detection result, and adjust the to-be-processed video based on a second card point detection result of the target music and the first card point detection result; The generation module is configured to generate a target card point video based on the adjusted video to be processed and the target music.

10. A computer device comprising: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video generation method according to any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 8 are executed.

Images