Video generation method and device, storage medium and electronic equipment

Abstract

The invention relates to a video generation method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a target video generated by using a video generation model; identifying a bounding box of each main body in each frame of image in the target video; determining the position of a bounding box and image features of an image in the bounding box; determining the total loss of the target video according to the position and the image features; and adjusting the video generation model according to the total loss, and regenerating the target video until the target video meeting the condition is generated. The technical problem that the accuracy of the video generated by the video generation model is low is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video generation, and particularly relates to a video generation method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the development of the Internet, in the prior art, a user can generate a video by using a video generation model. A common method can be to input a piece of text to constrain the video to be generated, and to let the video generation model generate the video according to the input text.

[0003] However, the video generated by the video generation model according to the text in the prior art does not necessarily meet the requirements of the text, resulting in low accuracy of the generated video. SUMMARY

[0004] The present application provides a video generation method and device, a storage medium and an electronic device to solve the technical problem of low accuracy of the video generated by the video generation model.

[0005] In a first aspect, the present application provides a video generation method, comprising: obtaining a target video generated by using a video generation model; identifying a bounding box of each subject in each frame of image in the target video; determining a position of the bounding box and an image feature of the image in the bounding box; determining a total loss of the target video according to the position and the image feature; adjusting the video generation model according to the total loss, and regenerating the target video until a target video meeting a condition is generated.

[0006] In a second aspect, the present application provides a video generation device, comprising: an obtaining module configured to obtain a target video generated by using a video generation model; an identifying module configured to identify a bounding box of each subject in each frame of image in the target video; a first determining module configured to determine a position of the bounding box and an image feature of the image in the bounding box; a second determining module configured to determine a total loss of the target video according to the position and the image feature; and an adjusting module configured to adjust the video generation model according to the total loss, and regenerate the target video until a target video meeting a condition is generated.

[0007] In a third aspect, the present application provides an electronic device, comprising: at least one communication interface; at least one bus connected with the at least one communication interface; at least one processor connected with the at least one bus; and at least one memory connected with the at least one bus, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the video generation method of any one of the above aspects.

[0008] In a fourth aspect, the present application also provides a computer storage medium storing computer executable instructions for executing the video generation method of any of the above aspects.

[0009] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: the solution provided by the embodiments of the present application acquires a target video generated by using a video generation model; identifies a bounding box of each subject in each frame of image in the target video; determines a position of the bounding box and an image feature of the image in the bounding box; determines a total loss of the target video according to the position and the image feature; adjusts the video generation model according to the total loss, and regenerates the target video until a target video meeting the condition is generated, so that after the video generation model generates the target video, the loss can be calculated according to the generated target video, and in the case that the total loss is unqualified, the video generation model is adjusted to let the video generation model regenerate the target video, and finally an accurate video generation model and a target video generated by the video generation model are obtained. The accuracy of the video generated by the video generation model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without any creative labor under the premise of the drawings.

[0012] One or more embodiments are illustrated by way of example in the drawings that are for illustrative purposes only, and these illustrative examples do not constitute a limitation on the embodiments, and elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified, and the drawings do not constitute a proportional limitation.

[0013] Figure 1 A flowchart of a video generation method provided by the embodiments of the present application is shown in FIG. 1;

[0014] Figure 2 A flowchart of another video generation method provided by the embodiments of the present application is shown in FIG. 2;

[0015] Figure 3 A flowchart of still another video generation method provided by the embodiments of the present application is shown in FIG. 3;

[0016] Figure 4 A flowchart of still another video generation method provided by the embodiments of the present application is shown in FIG. 4.

[0017] Figure 5 A structural schematic diagram of a video generation device provided by an embodiment of the present application is shown in the figure.

[0018] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] The following disclosure provides many different embodiments, or examples, for implementing different structures of the present application. For the purpose of simplicity, the components and arrangements of the examples that are described herein are shown and described separately. It is to be understood, however, that the implementations of the present application are not limited to those examples; indeed, many modifications are possible. In addition, the present application is well suited to achieving the purposes, objects and advantages thereof with devices, systems and / or methods of the examples that are disclosed here. Furthermore, the encompassed embodiments are applicable to other implementations and equipment of numerous types that fall within the scope of the present application.

[0021] To solve the technical problem of low accuracy of videos generated by a video generation model in the prior art, the present application provides a video generation method, which can improve the accuracy of videos generated by a video generation model.

[0022] Figure 1 A flowchart of a video generation method provided by an embodiment of the present application is shown in the figure. Figure 1 As shown in the figure, the video generation method comprises:

[0023] S101, obtaining a target video generated by using a video generation model;

[0024] S102, identifying a bounding box of each subject in each frame of image in the target video;

[0025] S103, determining a position of the bounding box and an image feature of the image in the bounding box;

[0026] S104, determining a total loss of the target video according to the position and the image feature;

[0027] S105, adjusting the video generation model according to the total loss, and regenerating the target video until a target video meeting the condition is generated.

[0028] The video generation method described above can be applied in the process of generating a target video by a video generation model. A user can interact with the video generation model, for example, by inputting voice or text to let the video generation model generate a predetermined target video. After the video generation model generates the corresponding target video according to the user's input, whether the target video generated by the video generation model meets the requirements is determined by calculating the loss of the target video. If it does not meet the requirements, the video generation model needs to be adjusted, and the target video is generated again according to the user's input described above until the target video that meets the requirements is generated. At this time, it means that the generation logic of the video generation model is accurate, and the generated target video also meets the user's requirements.

[0029] In this example, the target video can be generated by the video generation model first. After the target video is generated, each subject in each frame of image in the target video can be identified. For example, the image contains a person, an animal, etc. Each person and each animal needs to be identified. For all the identified subjects, each subject can be marked by a bounding box, which is used to mark the position and size of the subject. After identifying the subject, the image features of the marked subject, that is, the image features of the image framed by the bounding box, can be extracted. According to the position of the bounding box and the image features in the bounding box, the total loss of the target video can be determined.

[0030] When identifying the subject in each frame of image of the video, the video may contain multiple subjects, and multiple subjects may appear at the same time in the video. For example, two person subjects appear at the same time in multiple frames of the video. At this time, the two person subjects may be identified as one subject. Therefore, in order to improve the accuracy of subject identification, each identified subject can be split to obtain the sub-subjects of the subject. For example, taking a person subject as an example, the person subject is split to identify the head, torso, and limb parts of the person subject. If the identified sub-subjects meet the requirements, the multiple sub-subjects can be confirmed as the sub-subjects of the person subject. If the identified sub-subjects include parts other than the subject, for example, the heads of two persons are identified, it means that the identified sub-subjects do not all belong to the person subject. Further, it is determined that the identified subject has an error.

[0031] After identifying each subject in each frame of image, the bounding box can be an irregular bounding box or a regular bounding box when framing each subject by a bounding box. The regular bounding box can be a rectangular bounding box or a square bounding box. The boundary of the bounding box is coincided with the top, bottom, left, and right edges of the subject by framing the subject by the minimum bounding box.

[0032] In the embodiment, when the image features of the images in the bounding boxes are identified, the sizes of the subjects framed by all the bounding boxes can be standardized, so that the sizes of all the subjects are set to be the same. Then, the image features of the subjects in the images in the bounding boxes are extracted.

[0033] After the bounding boxes are determined and the image features of the images in the bounding boxes are extracted, the total loss of the target video can be calculated, and the video generation model is adjusted according to the calculated loss.

[0034] The scheme provided by the embodiment of the application comprises the following steps: a target video generated by using a video generation model is obtained; a bounding box of each subject in each frame image in the target video is identified; a position of the bounding box and an image feature of an image in the bounding box are determined; a total loss of the target video is determined according to the position and the image feature; the video generation model is adjusted according to the total loss, and the target video is regenerated until a target video meeting the condition is generated, so that after the target video is generated by using the video generation model, the loss can be calculated according to the generated target video, and in the case that the total loss is unqualified, the video generation model is adjusted, and the target video is regenerated by using the video generation model, so that an accurate video generation model and a target video generated by using the video generation model are finally obtained. The accuracy of the video generated by using the video generation model is improved.

[0035] As an optional example, as shown in Figure 2 The total loss of the target video determined according to the position and the image feature comprises the following steps:

[0036] S201, a type consistency loss of the subjects in the target video is determined according to the total number of frames of the target video, the number of subjects in the target video and the image features.

[0037] S202, a number consistency loss in the target video is determined according to the number of subjects in the target video.

[0038] S203, a spatial logic consistency loss of the subjects in the target video is determined according to the position of the bounding box.

[0039] S204, the total loss is determined according to the type consistency loss, the number consistency loss and the spatial logic consistency loss.

[0040] In the example, when the total loss is determined, the type consistency loss, the number consistency loss and the spatial logic consistency loss of the target video are determined respectively, and the total loss is determined by the three losses.

[0041] The type consistency loss is used to constrain the type of the subject in the video frames in the target video. The more complex the type of the subject, the more the types, and the faster the change, the greater the calculated type consistency loss. The quantity consistency loss is used to constrain the quantity of the subject in the video frames in the target video. The greater the quantity, the greater the calculated quantity consistency loss. The spatial logic consistency loss is used to constrain the spatial logic of the subject in the video frames in the target video. The spatial logic is the spatial position change of the subject in consecutive video frames. The worse the spatial logic, the more irregular the spatial position change of the subject in consecutive frames or the more irregular the speed, acceleration or moving direction, and the greater the spatial logic consistency loss.

[0042] The total loss is determined by the three losses, and the size of the total loss represents the quantity, types and spatial logic of the subject in the video frames in the target video. The video generation model is adjusted by the total loss.

[0043] In determining the total loss, as shown in step S301, the weighted sum of the type consistency loss, the quantity consistency loss and the spatial logic consistency loss can be determined as the total loss. The weight of the type consistency loss is related to the control degree of the type of the subject in the expected scene. The quantity consistency loss is related to the stability degree of the quantity of the subject in the expected scene. The weight of the spatial logic consistency loss is related to the attention degree of the motion smoothness in the scene. Figure 3

[0044] In this embodiment, the size of the weight can be set. The logic of setting is to determine the size of each weight according to the type, quantity and spatial logic of the subject in the target video concerned by the scene.

[0045] For example, the user inputs voice or text, in which the type of the subject in the generated target video is more focused, and the weight corresponding to the type of the subject is set to be larger. If the user's voice or text is more focused on the spatial logic, for example, the subject in the generated video cannot or less appear to appear instantaneous, and the weight corresponding to the spatial logic can be set to be larger.

[0046] In one example, the type consistency loss of the subject in the target video is determined according to the total frame number of the target video, the quantity of the subject in the target video and the image feature, which includes: calculating the type consistency loss by the following formula (1):

[0047]

[0048] Where N is the quantity of the subject in the target video, T is the total frame number of the target video, f t，i ​is the image feature of the ith subject in the t-th frame, and ||f|| represents the L2 norm of the image feature.

[0049] In this embodiment, the type consistency loss is calculated based on the image feature of each subject in each frame of the target video and the feature of the subject in the previous frame of the video, so that the change of the subject between the video frames is learned.

[0050] In one example, determining the quantity consistency loss in the target video according to the number of subjects in the target video includes calculating the quantity consistency loss according to the following formula (2):

[0051]

[0052] where T is the total number of frames of the target video, n t is the number of subjects in the t-th frame, is the expected number of subjects in the text prompt.

[0053] In this embodiment, the quantity consistency loss is constrained based on the relationship between the number of subjects in the video frames of the target video and the expected number, so that the influence of the number of subjects in the video frames is learned.

[0054] In one example, determining the spatial logical consistency loss of the subject in the target video according to the position of the bounding box includes calculating the spatial logical consistency loss according to the following formula (3):

[0055]

[0056] where B t，i is the position of the bounding box of the ith subject in the t-th frame. In this embodiment, the spatial logical consistency loss is constrained based on the change of the position of the bounding box in the target video, so that the influence of the position change of the subject on the loss is learned.

[0057] Through the above examples, the type consistency loss, the quantity consistency loss and the spatial logical consistency loss can be learned, and finally the total loss is obtained.

[0058] As an optional example, the method further includes determining whether the subject in the adjacent two frames has abnormal motion according to the moving trajectory of the subject in the adjacent frames and the moving speed, direction and acceleration of the subject in the adjacent two frames.

[0059] In this embodiment, in addition to calculating the total loss to adjust the video generation model, the moving trajectory of the subject in the continuous video frames and the moving speed, direction and acceleration of the subject in the adjacent two frames can be determined to determine whether the subject in the video frames has a significant shift or misplacement. If a significant shift or misplacement occurs, the video generation model can also be adjusted.

[0060] Let's illustrate with an example. For example... Figure 4 As shown, if a user wants to generate a video based on text or speech, for example, when generating a video of "two airplanes flying in the air," the number of airplanes might change from two to one; when generating a video of "an athlete running across the finish line," the athlete might disappear midway or suddenly change direction. These problems seriously affect the realism and usability of the generated videos.

[0061] To address this issue, this embodiment introduces a "supervisor" model that comprehensively evaluates the generated videos and provides improvement suggestions. This model assesses the logical consistency of the videos from three key dimensions:

[0062] 1. Subject Recognition and Type Determination: The discrimination model first identifies various subjects in the video (such as people, vehicles, animals, etc.) and determines their specific types. For example, it can not only identify "this is an airplane" but also further determine "this is a fighter jet." This ensures that the subjects appearing in the video match the requirements of the user's text description.

[0063] 2. Subject Quantity Tracking: The discriminative model tracks the changes in the quantity of various subjects throughout the video. For example, if a user requests a video of "two cats playing," the discriminative model will ensure that there are always two cats in the entire video, and that the number does not change to one or three at any point.

[0064] 3. Motion Trajectory Analysis: The discrimination model analyzes the motion trajectories of various subjects in the video to ensure that their movements conform to physical laws and common sense logic. For example, when a person walks, there should be continuous steps rather than sudden teleportation; when a car turns, there should be a reasonable turning radius rather than a right-angle turn.

[0065] To achieve the above three functions, three key technology modules are introduced:

[0066] 1. Subject Detection and Tracking Module: This module uses computer vision technology (YOLOv8 and DeepSORT algorithms) to identify various subjects in the video and track their position and movement throughout the video.

[0067] 2. Subject Classification Module: This module uses deep learning technology (based on a ResNet classifier) ​​to classify the detected subjects in detail, identifying their specific types and features.

[0068] 3. Motion State Estimation Module: This module uses optical flow technology to analyze the motion state of the subject, including motion direction, velocity, and acceleration.

[0069] These three modules work together to jointly assess the spatio-temporal logical consistency of the generated video and quantify the assessment results through specific mathematical formulas (loss functions). The generation model will adjust and optimize based on these quantitative indicators, continuously improving the quality and realism of the generated video.

[0070] The workflow of the entire system is similar to a "generate-evaluate-feedback-improve" cycle: first, the generation model generates an initial video based on the user's text description; then, the discriminator model conducts multi-dimensional evaluation on this video; next, the evaluation results are fed back to the generation model as feedback signals; finally, the generation model optimizes based on the feedback and generates a more logically consistent video. This cycle can be repeated multiple times until the generated video reaches a satisfactory quality level.

[0071] Specifically as follows:

[0072] A subject detection and tracking module is constructed, which is implemented by combining YOLOv8 and DeepSORT algorithms. YOLOv8 is used to quickly and accurately detect various objects in each frame of the video. It divides the image into grids, and each grid unit is responsible for predicting bounding boxes and class probabilities. These predictions include the center coordinates (x, y) of the bounding box, its width (w), height (h), and confidence. The detection results are then passed to the DeepSORT multi-target tracking algorithm, which combines Kalman filtering and deep learning features to establish the association between consecutive frames. Specifically, DeepSORT assigns a unique identifier (ID) to each detected target and maintains this ID in subsequent frames through feature matching and motion prediction. This allows the system to continuously track the position and movement of each subject throughout the video sequence, even after temporary occlusion or reappearance after moving out of view. The output of this module includes the bounding box coordinates, class labels, and unique IDs of all subjects in each frame, providing basic data for subsequent consistency analysis.

[0073] Second, construct the subject classification module, which is responsible for more fine-grained classification of detected subjects and extracting feature vectors. This module uses a pre-trained ResNet deep convolutional neural network as a feature extractor, which solves the gradient vanishing problem in deep network training by introducing residual connections and can extract richer visual features. The processing flow is as follows: First, according to the boundary box coordinates provided by the subject detection and tracking module, crop the image area of each subject from the original video frame; Then, adjust these cropped or stretched images to a standard size (such as 224x224 pixels) and pre-process them (such as normalization); Next, input the pre-processed images into the ResNet model to extract feature vectors (usually the output of the last pooling layer). These feature vectors can be used for subject classification (by connecting fully connected layers and Softmax functions), and will also be used for subsequent type consistency evaluation. This module can extract high-level semantic features of the subject, providing a basis for evaluating the consistency of subject types between different frames.

[0074] Third, construct the motion state estimation module, which uses the optical flow method to analyze the motion characteristics of the subject, and estimates the motion of pixel points in image sequences through the optical flow method. The processing flow of this module is as follows: First, pre-process the consecutive video frames, such as grayscale and Gaussian smoothing, to reduce the influence of noise; Then, use the algorithm or Lucas-Kanade algorithm to calculate the dense optical flow field between adjacent frames, obtaining the displacement vector of each pixel point in the horizontal and vertical directions; Next, according to the boundary box provided by the subject detection and tracking module, extract the optical flow vector within each subject area; Finally, statistically analyze these optical flow vectors to calculate the average motion direction, speed and acceleration of the subject. These features are used to evaluate the smoothness and reasonableness of the subject's motion, detect abnormal motion patterns such as sudden acceleration, unnatural jumps or motion trajectories that do not conform to physical laws.

[0075] Based on the output of the above three modules, three consistency loss functions are designed:

[0076] Type consistency loss (L type ) is used to evaluate the type feature consistency of the same subject between consecutive frames, using the cosine similarity method, as shown in formula (1) above. In formula (1), · represents the dot product of vectors.

[0077] Quantity consistency loss (L number ) is used to evaluate whether the change in the number of subjects in the video is reasonable, using formula (2) above.

[0078] Trajectory consistency loss (L track) for evaluating the smoothness of the subject trajectory, using the sum of squared differences in position changes, i.e., the above formula (3). In formula (3), ||.|| represents the Euclidean distance.

[0079] The three loss functions are combined with weights to form the total loss:

[0080] L consistency = a * L type + b * L number + g * L track (4)

[0081] where a, b, and g are adjustable weight parameters used to balance the importance of different types of loss. In practical applications, these weights can be adjusted according to specific scene requirements, for example, in scenarios where the type of subject needs to be strictly controlled, the value of a can be increased; in scenarios where the number of subjects needs to be kept stable, the value of b can be increased; in scenarios where motion smoothness needs to be paid special attention to, the value of g can be increased.

[0082] Finally, the spatiotemporal logic consistency discrimination model is integrated into the text-to-video generation system. The specific process is as follows: first, the generation model generates an initial video according to the input text prompt; then, the spatiotemporal logic consistency discrimination model evaluates the video and calculates the consistency loss; next, the consistency loss is combined with the original loss function of the generation model (such as reconstruction loss, adversarial loss, etc.) to form a new optimization objective; finally, the generation model is updated based on this optimization objective to generate videos that better conform to the spatiotemporal logic. This process can be iterated multiple times until the generated video reaches a satisfactory quality level. In actual deployment, the trained discrimination model can be used as an independent evaluation component to filter and optimize the output of the generation model, ensuring that the final video content presented to the user meets the requirements of spatiotemporal logic consistency.

[0083] Figure 5 A structural schematic diagram of a video generation apparatus provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the video generation apparatus includes: Figure 5

[0084] An acquisition module 501 is configured to acquire a target video generated by using a video generation model.

[0085] An identification module 502 is configured to identify a bounding box of each subject in each frame of the target video.

[0086] A first determination module 503 is configured to determine a position of the bounding box and an image feature of the image in the bounding box.

[0087] A second determination module 504 is configured to determine a total loss of the target video according to the position and the image feature.

[0088] ​The adjustment module 505 is used to adjust the video generation model based on the total loss and regenerate the target video until a target video that meets the conditions is generated.

[0089] The aforementioned video generation device can be applied to the process of generating a target video using a video generation model. Users can interact with the video generation model, for example, by inputting voice or text to instruct it to generate a predetermined target video. After the video generation model generates the corresponding target video based on the user's input, it calculates a loss on the target video to determine if the generated video meets the requirements. If it does not meet the requirements, the video generation model is adjusted, and the target video is regenerated based on the user's input until a satisfactory target video is generated. This indicates that the generation logic of the video generation model is accurate, and the generated target video meets the user's requirements.

[0090] In this example, the target video can first be generated by a video generation model. After generating the target video, each subject in each frame of the target video can be identified. For example, if the image contains people and animals, then each person and each animal needs to be identified. For all identified subjects, each subject can be marked with a bounding box, which is used to mark the position and size of the subject. After identifying the subjects, the image features of the marked subjects can be extracted, that is, the image features of the image enclosed by the bounding box. Based on the position of the extracted bounding box and the image features of the image within the bounding box, the total loss of the target video can be determined.

[0091] When identifying the subject in each frame of a video, the video may contain multiple subjects, and these subjects may appear simultaneously. For example, if two human subjects appear simultaneously in multiple frames of a video, they might be identified as a single subject. Therefore, to improve the accuracy of subject identification, each identified subject can be broken down into its sub-subjects. For instance, taking a human subject as an example, it can be broken down into its head, torso, limbs, etc. If the identified sub-subjects meet the requirements, then multiple sub-subjects can be confirmed as sub-subjects of the original human subject. If the identified sub-subjects include parts outside the original subject, such as identifying two heads, it indicates that not all identified sub-subjects belong to the original human subject. This further confirms that there is an error in the identified subject.

[0092] After identifying each subject in each frame of image, each subject is framed by a bounding box, which can be an irregular bounding box or a regular bounding box, and the regular bounding box can be a rectangular bounding box, a square bounding box, etc. The subject is framed by a minimum bounding box, so that the boundary of the bounding box coincides with the uppermost, lowermost, leftmost and rightmost positions of the subject.

[0093] In the embodiment, when identifying the image features of the images in the bounding boxes, the sizes of the bounding boxes can be different, so the subjects framed by all the bounding boxes can be standardized in size, and the sizes of all the subjects can be set to be the same. Then the image features of the subjects in the images in the bounding boxes are extracted.

[0094] After the bounding boxes are determined and the image features of the images in the bounding boxes are extracted, the total loss of the target video can be calculated, and the video generation model is adjusted according to the calculated loss.

[0095] The scheme provided by the embodiment of the application comprises the following steps: obtaining a target video generated by using a video generation model; identifying a bounding box of each subject in each frame of image in the target video; determining a position of the bounding box and an image feature of an image in the bounding box; determining a total loss of the target video according to the position and the image feature; adjusting the video generation model according to the total loss, and regenerating the target video until a target video meeting a condition is generated, so that after the target video is generated by the video generation model, the loss can be calculated according to the generated target video, and in the case that the total loss is unqualified, the video generation model is adjusted to make the video generation model regenerate the target video, and finally an accurate video generation model and a target video generated by the video generation model are obtained. The accuracy of the video generated by the video generation model is improved.

[0096] Other examples of the embodiment are described in the above examples, which will not be described here.

[0097] As shown in Figure 6 The embodiment of the application provides an electronic device, which comprises a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 complete mutual communication through the communication bus 114,

[0098] The memory 113 is used for storing a computer program.

[0099] In an embodiment of the application, the processor 111 is used for executing the program stored in the memory 113, and realizes the video generation method provided by any one of the preceding method embodiments.

[0100] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the video generation method provided by any one of the foregoing method embodiments.

[0101] The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0102] Through the description of the foregoing embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the foregoing technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0103] It should be understood that the terms used herein are for the purpose of describing particular example embodiments only and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order in which they are described, unless specifically identified as an order dependent step. It is also to be understood that additional or alternative steps can be employed.

[0104] The above description is merely illustrative of the application and the specific examples, so that those skilled in the art can understand or implement the application. Various modifications to these examples will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the examples shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that, include: Obtain the target video generated using the video generation model; Identify the bounding box of each subject in each frame of the target video; Determine the position of the bounding box and the image features of the image within the bounding box; The total loss of the target video is determined based on the location and the image features; The video generation model is adjusted based on the total loss, and the target video is regenerated until a target video that meets the conditions is generated.

2. The method according to claim 1, characterized in that, The step of determining the total loss of the target video based on the location and the image features includes: The type consistency loss of the subjects in the target video is determined based on the total number of frames in the target video, the number of subjects in the target video, and the image features. Based on the number of subjects in the target video, determine the quantity consistency loss in the target video; Based on the position of the bounding box, determine the spatial logical consistency loss of the subject in the target video; The total loss is determined based on the type consistency loss, the quantity consistency loss, and the spatial logical consistency loss.

3. The method according to claim 2, characterized in that, The step of determining the type consistency loss of the subjects in the target video based on the total number of frames of the target video, the number of subjects in the target video, and the image features includes: The type consistency loss is calculated using the following formula: Where N is the number of subjects in the target video, T is the total number of frames in the target video, and f t，i Let |f|| be the image feature of the i-th subject in the t-th frame, and |f|| represent the L2 norm of the image feature.

4. The method according to claim 2, characterized in that, The step of determining the quantity consistency loss in the target video based on the number of subjects in the target video includes calculating the quantity consistency loss using the following formula: Where T is the total number of frames in the target video, and n t It is the number of subjects in frame t. This is the number of main elements expected in the text prompt.

5. The method according to claim 2, characterized in that, The step of determining the spatial logical consistency loss of the subject in the target video based on the position of the bounding box includes calculating the spatial logical consistency loss using the following formula: Among them B t,i It is the bounding box position of the i-th subject in frame t.

6. The method according to claim 2, characterized in that, The determination of the total loss based on the type consistency loss, the quantity consistency loss, and the spatial logical consistency loss includes: The total loss is determined by the weighted sum of the type consistency loss, the quantity consistency loss, and the spatial logic consistency loss. The weight of the type consistency loss is related to the degree of control over the subject type in the desired scenario, the quantity consistency loss is related to the stability of the number of subjects in the desired scenario, and the weight of the spatial logic consistency loss is related to the attention paid to motion smoothness in the scenario.

7. The method according to claim 1, characterized in that, The method further includes: Based on the movement trajectory of the subject in multiple adjacent frames and the movement speed, direction and acceleration of the subject in two adjacent frames, it is determined whether the subject in two adjacent frames has produced abnormal movement.

8. A video generation apparatus, characterized in that, include: The acquisition module is used to acquire the target video generated using the video generation model; The recognition module is used to identify the bounding box of each subject in each frame of the target video; The first determining module is used to determine the position of the bounding box and the image features of the image within the bounding box; The second determining module is used to determine the total loss of the target video based on the location and the image features; An adjustment module is used to adjust the video generation model based on the total loss and regenerate the target video until a target video that meets the conditions is generated.

9. An electronic device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor executes the computer program to implement the video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions for performing the video generation method according to any one of claims 1 to 7 of this application.

Images