Video generation method and device, equipment and storage medium

Abstract

The invention provides a video generation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a target video, and determining a target video frame representing the motion track of a target object in the target video; identifying motion information of the target object in the target video frame, and screening out a confidence video frame from the target video frame based on the identified motion information; wherein the motion information is at least used for representing the posture change of the target object; and identifying a dynamic action category of the target object in the confidence video frame, determining a to-be-edited video frame in the confidence video frame based on the identified dynamic action category, and generating a video clip of the target object according to the to-be-edited video frame. According to the technical scheme provided by the invention, the video generation efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a video generation method, device, equipment and storage medium. Background Art

[0002] In some practical application scenarios, video clips related to the target object can be generated based on pre-filmed video content. For example, highlights of the target object can be selected from the pre-filmed video content, and the video clips of the target object can be generated based on the selected highlights.

[0003] However, existing video generation methods usually have the problem of low generation efficiency. In some specific scenarios, such as when the target object moves quickly, it is difficult to select accurate highlights from the pre-shot video content, resulting in low efficiency in the final video generation. Summary of the invention

[0004] In view of this, one or more embodiments of the present disclosure provide a video generation method, apparatus, device, and storage medium, which can improve the efficiency of video generation.

[0005] On the one hand, the present disclosure provides a video generation method, which includes: acquiring a target video, and determining a target video frame that represents a motion trajectory of a target object in the target video; identifying motion information of the target object in the target video frame, and based on the identified motion information, screening out a confident video frame in the target video frame; wherein the motion information is at least used to represent a posture change of the target object; identifying a dynamic action category of the target object in the confident video frame, and based on the identified dynamic action category, determining a video frame to be edited in the confident video frame, and generating a video clip of the target object according to the video frame to be edited.

[0006] On the other hand, the present disclosure also provides a video generating device, which includes: a trajectory recognition unit, used to obtain a target video, and determine a target video frame representing the motion trajectory of a target object in the target video; a video frame screening unit, used to identify the motion information of the target object in the target video frame, and based on the identified motion information, screen out a confident video frame in the target video frame; wherein the motion information is at least used to represent the posture change of the target object; a video generating unit, used to identify the dynamic action category of the target object in the confident video frame, and based on the identified dynamic action category, determine a video frame to be edited in the confident video frame, and generate a video clip of the target object according to the video frame to be edited.

[0007] On the other hand, the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned video generation method is implemented.

[0008] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0009] The technical solution provided by one or more embodiments of the present disclosure can identify the motion information of the target object from the target video frame representing the motion trajectory of the target object. The motion information can indicate the posture change of the target object during the motion process.

[0010] Generally speaking, the change of posture can indirectly reflect the intensity of the target object during the movement, and the intensity of the movement can also reflect the excitement of the video. In this way, by analyzing the above-mentioned movement information, it is possible to select a confident video frame from the target video frame. In the confident video frame, the target object is more likely to have an exciting scene.

[0011] Subsequently, by identifying the dynamic action category of the confident video frame, the current motion stage of the target object can be determined, and then according to the recognition result of the dynamic action category, a video clip reflecting the motion process of the target user can be edited.

[0012] It can be seen that through the technical solutions provided by one or more embodiments of the present disclosure, the confident video frames with wonderful scenes can be effectively determined from the target video frames according to the motion information, and the wonderful scenes can be further located in detail in the confident video frames by identifying the dynamic action categories in the confident video frames, and subsequently the video clips of the target object can be generated for the located wonderful scenes. In the above manner, not only can the video clips of the target object in motion be edited, but the motion state shown in the video clips can also reflect the intensity of the motion process, thereby improving the efficiency and quality of video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The features and advantages of the embodiments of the present disclosure may be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0014] Figure 1 A schematic diagram of the steps of a video generation method in one embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic diagram of screening a confident video frame in one embodiment of the present disclosure is shown;

[0016] Figure 3 A schematic diagram of functional modules of a video generating device in one embodiment of the present disclosure is shown;

[0017] Figure 4 A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0019] One embodiment of the present disclosure provides a video generation method. Figure 1 , the method may include the following steps.

[0020] S1: Acquire a target video, and determine a target video frame representing a motion trajectory of a target object in the target video.

[0021] In this embodiment, the target object may be a user who needs to generate a video. It is understandable that before using the technical solutions disclosed in each embodiment of the present disclosure, the type, scope of use, and usage scenarios of the personal information involved in the present disclosure should be informed to the user in an appropriate manner in accordance with relevant laws and regulations and the user's authorization should be obtained.

[0022] For example, before implementing the technical solutions of various embodiments of the present disclosure, a prompt message may be sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, the prompt information can be sent to the user in a pop-up window, in which the prompt information can be presented in text. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree".

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0026] It can be understood that in the specific implementation of this application, identity information, video data and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0027] In this embodiment, through the camera deployed in the offline scene, when the target object is active in the offline scene, the target video containing the target object can be collected. It should be noted that in practical applications, the number of target videos can be more than one. For example, multiple cameras can be deployed in the offline scene, and target videos of different perspectives can be captured by different cameras. Subsequently, each target video can be processed in the same or similar manner. Of course, the subsequent processing of the target video and the process of editing the content of the target video are all carried out under the condition that the authorization of the target object has been obtained. In practical applications, the target video can be analyzed frame by frame by an object detection algorithm. For any video frame in the target video, after processing with the object detection algorithm, the position information of one or more bounding boxes corresponding to the video frame can be output. Among them, each bounding box can represent a human body in the video frame, and the human body existing in the video frame can be identified and located through the position information of the bounding box. In addition, after passing through the object detection algorithm, each bounding box representing the human body will have its own bounding box identifier, which can be a globally unique number, for example. After object detection is performed on different video frames of the target video, the position information of the bounding box and the bounding box identifier can be output for each video frame. Bounding boxes with the same bounding box identifier can represent the same human body. In this way, after the target video is detected frame by frame, the movement trajectory of the same human body can be tracked in different video frames by identifying the bounding boxes with the same bounding box identifier.

[0028] In this embodiment, the target object can be any human body for which video generation is requested. In the target video, the target object may only appear in part of the video frames and be identified by the object detection algorithm, and then this part of the video frames can be used as the target video frames representing the motion trajectory of the target object. In this way, by performing frame-by-frame object detection on the target video, the target video frames representing the motion trajectory of the target object can be determined in the target video.

[0029] S3: Identify motion information of the target object in the target video frame, and screen out a confident video frame from the target video frame based on the identified motion information; wherein the motion information is at least used to characterize a posture change of the target object.

[0030] In this embodiment, in the target video frame representing the motion trajectory of the target object, more detailed feature recognition can be performed on the target object in each target video frame. Specifically, considering that the video clips obtained by the final editing usually need to reflect the wonderful pictures of the target object during the motion process, and the wonderful pictures during the motion process often correspond to the target object's rich body movements. In view of this, the motion information of the target object in the target video frame can be identified, and then based on the motion information, the target video frame can be further screened, so as to screen out the confidence video frame with a high probability of the occurrence of the wonderful picture in the target video frame.

[0031] In this embodiment, the motion information of the target object can be represented in a variety of ways. Among them, the motion information of the target object can at least represent the posture change of the target object. The posture of the target object can include the posture information of each designated part of the target object. The designated part can be, for example, the hand, leg, waist and other human body parts of the target object that can reflect the motion state.

[0032] In one embodiment, the posture information of each designated part of the target object can be identified by a posture recognition model that has completed training. Specifically, the training sample of the posture recognition model in the training stage can be an image sample containing a designated part of the human body. In addition, for each image sample, the posture information of the designated part contained in the image sample can be pre-marked. In practical applications, the posture information of the designated part can be represented by a pre-set state value. For example, for the hand, there are usually multiple postures such as holding high, holding horizontally, and hanging down. Then, different state values ​​can be used to represent these different postures. For example, holding high can correspond to state value 1, holding horizontally can correspond to state value 2, and hanging down can correspond to state value 3. In this way, the above state values ​​can be used as the posture information of the designated part. In an image sample, there may be multiple designated parts, and each designated part can use the corresponding state value to represent the posture information according to the current posture. The collection of posture information of each designated part can be used as the movement information of the human body in the image sample.

[0033] In practical applications, the order of the designated parts can be pre-specified, and then the posture information of the designated parts can be spliced ​​according to the order result to form the motion information of the human body in the image sample. For example, the order of the designated parts of the human body is: hand, waist, leg, then the motion information of the human body in the image sample can be represented by the vector (hand posture information, waist posture information, leg posture information). Assuming that in a certain image sample, the state value corresponding to the hand posture information is 1, the state value corresponding to the waist posture information is 1, and the state value corresponding to the leg posture information is 2, then the motion information of the human body in the image sample can be represented by the vector (1, 1, 2).

[0034] Through the above method, each image sample can have pre-labeled motion information. The posture recognition model can be implemented through a neural network. The image sample is input into the posture recognition model as training data, and then the result output by the posture recognition model is supervised by the pre-labeled motion information, so that the corresponding error information can be generated. The error information can adjust the parameters of the neurons in the posture recognition model so that the output result can be consistent with the labeled motion information. Through the training of a large number of image samples, a posture recognition model that can accurately recognize the motion information of the human body in the image can be obtained.

[0035] In this embodiment, the bounding box of the target object in the target video frame can be obtained by the object detection algorithm. According to the bounding box, the regional image of the target object can be cut out from the target video frame, and then the regional image can be input into the trained posture recognition model to obtain the posture information of each specified part of the target object.

[0036] In this embodiment, after the posture information of each designated part is identified from the target video frame, for any designated part, it can be determined whether the posture information of the designated part has changed compared with the previous target video frame. In practical applications, the posture information of the designated part can be represented in the form of a vector, so when judging whether the posture information of the designated part has changed, it is only necessary to subtract the state values ​​at the corresponding positions in the two adjacent vectors. For example, for the first target video frame, the posture information of each designated part of the target object can be represented as (1, 1, 2), and for the second target video frame, the posture information of each designated part of the target object can be represented as (3, 1, 2). Then, by comparing the two vectors, the final judgment result can be obtained: the hand of the target object has changed its posture, and the posture change value is 3-1=2.

[0037] In this way, by comparing the posture information of the designated part in the adjacent target video frames, the posture change information of the designated part can be obtained, and the posture change information can be represented by the above-mentioned posture change value. Finally, the combination of the posture change information of each designated part can be used as the motion information of the target object in the target video frame. For example, for the above-mentioned first target video frame and the second target video frame, the motion information of the target object can be represented as (2, 0, 0). It should be noted that each numerical value in the motion information represents the posture change value. The larger the posture change value, the greater the posture change produced by the designated part. The posture change value here is not the same as the state value representing the posture information.

[0038] In one embodiment, the motion information of the target object can also be represented by speed and acceleration. Specifically, the speed of the target object can be determined by the adjacent first target video frame and the second target video frame, wherein the first target video frame can be located before the second target video frame. Through the object detection algorithm, the first bounding box of the target object can be identified in the first target video frame, and the second bounding box of the target object can be identified in the second target video frame. Then, the position difference between the second bounding box and the first bounding box can be determined. Specifically, when determining the position difference, the first center position of the first bounding box and the second center position of the second bounding box can be determined first, and then the position difference between the first center position and the second center position is calculated along the direction of movement of the target object, and the position difference can be used as the position difference between the second bounding box and the first bounding box. Among them, the first center position and the second center position can be represented by the pixel coordinates in the video frame. For example, the first center position can be expressed as (100, 200), and the second center position can be expressed as (200, 210). It can be found that the target object has been displaced both horizontally and vertically. However, by analyzing the horizontal coordinates and the vertical coordinates, it can be determined that the target object is currently mainly displaced horizontally. At this time, the difference between the horizontal coordinates of the second center position and the first center position can be used as the position difference between the first center position and the second center position.

[0039] In this embodiment, the time difference between the first target video frame and the second target video frame is known, so the ratio of the position difference to the time difference can be used as the speed of the target object. Generally speaking, the speed of the target object determined based on two adjacent target video frames can be used as the speed of the target object identified from the later target video frame (i.e., the second target video frame) in terms of the temporal relationship.

[0040] Of course, if in some scenarios the target object has moved to a certain extent in both the horizontal and vertical directions, the position difference can be calculated for the horizontal and vertical directions respectively, and based on the horizontal and vertical position differences, the horizontal and vertical speeds can be calculated respectively.

[0041] It should be noted that the above-mentioned position difference can be expressed by the number of pixels. For example, the horizontal coordinate difference of 100 between the above-mentioned first center position and the second center position can refer to 100 pixels. In this way, the speed of the target object can be calculated in the image coordinate system. In practical applications, the first center position and the second center position can be converted into coordinate values ​​in the world coordinate system by converting the coordinate values ​​between the image coordinate system, the camera coordinate system and the world coordinate system. The speed calculated in this way can characterize the speed of the target object in the world coordinate system. The specific coordinate system used for speed calculation can be flexibly selected according to actual needs.

[0042] In one embodiment, after determining the speed of the target object, the acceleration of the target object can be further determined. Specifically, still taking the first target video frame and the second target video frame as an example, after determining the first speed of the target object in the first target video frame and the second speed of the target object in the second target video frame, the speed difference between the second speed and the first speed can be calculated, and the acceleration of the target object can be determined based on the speed difference. Specifically, the time difference between the first target video frame and the second target video frame is known, and the ratio of the above speed difference to the time difference can be used as the acceleration of the target object. Similarly, the acceleration of the target object determined based on two adjacent target video frames can be used as the acceleration of the target object identified from the target video frame that is later in the timing relationship (i.e., the second target video frame).

[0043] It should be noted that, with the change of application scenarios and the advancement of technology, the motion information of the target object identified from the target video frame can also have more other forms, which will not be elaborated here.

[0044] In this embodiment, the motion information of the target object can be identified from each target video frame in the above manner. Among them, the motion information may include multiple different types of motion parameters. For example, these motion parameters may be the above-mentioned posture change value, speed, acceleration, etc. Although the target video frame can reflect the motion trajectory of the target object, in some target video frames, the target object may be in a non-motion state or in a state with an unsightly posture, and these target video frames may not be used as the material for subsequent video editing. In view of this, the confidence of each target video frame can be determined based on the motion information of the target video frame, and the confidence can characterize the possibility of the existence of wonderful pictures in the target video frame. In this way, according to the confidence, the target video frame can be screened to obtain a confident video frame with a high possibility of the existence of wonderful pictures.

[0045] In the present embodiment, when determining the confidence of each target video frame, each action parameter can be weighted according to the preset weight of each action parameter, so as to generate the confidence of the target video frame. In practical applications, the preset weight of each action parameter can be flexibly set according to the requirements of video generation. In addition, for different designated parts, there may be different attitude change values. In the process of calculating the confidence, the attitude change values ​​of different designated parts can be used as an independent motion parameter, and independent preset weights can be set. Of course, according to the different requirements of video generation, the attitude change values ​​of each designated part can also be averaged or weighted first, and after obtaining an overall comprehensive change value, the comprehensive change value and other action parameters are weighted summed, and the present disclosure does not limit this.

[0046] It should be noted that different types of motion parameters may have different corresponding physical dimensions. In order to perform weighted processing on different types of motion parameters in the same numerical dimension, each different type of motion parameter can be normalized first, so that the motion parameters can be processed into dimensionless data. When weighted processing is performed on the dimensionless data later, the consistency of data processing can be guaranteed.

[0047] In this embodiment, the confidence level can represent the possibility of the existence of wonderful scenes. Therefore, after obtaining the confidence level of each target video frame, the target video frame with a confidence level greater than or equal to a specified threshold can be used as the screened confident video frame.

[0048] See also Figure 2 In a practical application example, the confidence of each target video frame can be expressed as Figure 2 shown. Figure 2 The dotted line in represents the specified threshold mentioned above. After filtering the target video frame by confidence, we can get Figure 2 Confidence video frames for the two intervals shown.

[0049] In one embodiment, when calculating the confidence of the target video frame, in addition to referring to the motion information, the quality information of the regional image where the target object is located can also be referred to. Specifically, according to the detection result of the object detection algorithm and the bounding box of the target object, the regional image where the target object is located can be intercepted from the target video frame, and the contrast and brightness of the regional image can be analyzed to generate the quality information of the regional image. Generally speaking, the quality information can be a numerical value representing the quality quality, and the larger the numerical value, the better the quality can be represented. Subsequently, the weight values ​​of the motion information and the quality information can be obtained respectively, and the motion information and the quality information can be weighted according to the obtained weight values ​​to obtain the confidence of the target video frame. Then, the target video frame whose confidence is greater than or equal to the specified threshold can be used as the screened out confidence video frame. The specific process can be referred to the description of the aforementioned embodiment, which will not be repeated here.

[0050] S5: identifying the dynamic action category of the target object in the confident video frame, determining the video frame to be edited in the confident video frame based on the identified dynamic action category, and generating a video clip of the target object according to the video frame to be edited.

[0051] In this embodiment, in order to automatically complete the final video generation according to the confidence video frame, the dynamic action category of the target object in the confidence video frame can be identified. In practical applications, the dynamic action category can include a series of preset categories such as waving, twisting, jumping, sliding, etc. After the dynamic action category of the target object is currently identified from the confidence video frame, the video frames corresponding to some dynamic action categories can be edited into the final video clip according to the preset video generation strategy.

[0052] For example, in the preset video generation strategy, it can be limited to edit at least two different dynamic action categories into the video clip. Then, after the dynamic action category of the target object is identified from the confidence video frame, the set of confidence video frames corresponding to the dynamic action category can be determined at the same time. For example, the dynamic action category representing waving can correspond to the confidence video frames of the 10th frame to the 50th frame. When editing the video clip later, for any dynamic action category, a specified number of confidence video frames can be selected from the set of confidence video frames corresponding to the dynamic action category. The specified number can be flexibly determined according to the duration of the video clip. The specified number of confidence video frames selected from each dynamic action category can be used as the video frames to be edited. Of course, the specified number of confidence video frames can also be pre-processed by deduplication, denoising, etc. to obtain the video frames to be edited. After the video frames to be edited are determined in the confidence video frames, the video clip of the target object can be generated according to the video frames to be edited. In the process of generating the video clip, the selected confidence video frames can be processed by sorting, adding special effects, changing filters, etc., which will not be repeated here.

[0053] Of course, in some application scenarios, when screening the confidence video frames, the number of confidence video frames may not be limited. For example, after identifying the dynamic action category of the target object, the confidence video frames belonging to the dynamic action category can be screened out, and the confidence video frames that do not belong to the dynamic action category can be eliminated. For another example, after identifying the dynamic action category of the target object, the correlation between the confidence video frame and the dynamic action category can be determined based on the content displayed in each confidence video frame. Subsequently, the confidence video frames with a correlation higher than a certain threshold can be screened out, and the confidence video frames with a correlation lower than the threshold can be eliminated.

[0054] In this embodiment, each dynamic action category may correspond to its own action interval, and the action interval may include multiple stage actions arranged in time sequence. For example, for a dynamic action category representing waving, the stage actions in the action interval may include the action of starting waving, the action during waving, the action of ending waving, etc. For another example, for a dynamic action category representing jumping, the stage actions in the action interval may include the action of taking off, the action of jumping to the highest point, the action of landing, etc. For different dynamic action categories, the stage actions included in the action interval may be pre-defined.

[0055] In this embodiment, a large number of action intervals of dynamic action categories can be used as training samples to train the action prediction model. Specifically, for the action interval of the dynamic action category, each stage action in the action interval can have an action label that characterizes the dynamic action category, and at the same time, it can also have a stage label that characterizes the stage of the stage action in the action interval. For example, for the dynamic action category of jumping, each stage action in the action interval has an action label that characterizes "jumping". In addition, the stage actions such as take-off action, jumping to the highest point, and landing action can also have their own stage labels (for example, corresponding to the stage labels of "take-off", "highest point", and "landing" respectively). In this way, by inputting each stage action in the action interval into the action prediction model for training, the predicted action label and stage label can be output. Subsequently, by comparing the predicted action label and stage label with the pre-marked action label and stage label, error information can be obtained, and the error information is used to correct the action prediction model.

[0056] In practical applications, since the action prediction model needs to learn the association of multiple stage actions within an action interval, a neural network with a temporal relationship can be used to communicate the action prediction model. For example, the action prediction model can be implemented using the LSTM (Long Short-Term Memory) neural network architecture, or it can be implemented through the Transformer architecture.

[0057] In this embodiment, after the action prediction model is trained, the confidence video frame can be input into the action prediction model, and the action prediction model can output the corresponding action label and stage label. In this way, the confidence video frames with the same action label can be classified into the same dynamic action category, and the confidence video frames classified into the same dynamic action category can be sorted according to the stage label of the confidence video frame. In this way, based on the action label, the confidence video frames can be classified to obtain a set of confidence video frames corresponding to the dynamic action category. The above-mentioned action label and stage label can jointly characterize the dynamic action category of the target object in the confidence video frame. When the video is generated later, the relatively exciting stage action can also be edited into the video clip based on the stage label of the confidence video frame.

[0058] The technical solution provided by one or more embodiments of the present disclosure can identify the motion information of the target object from the target video frame representing the motion trajectory of the target object. The motion information can indicate the posture change of the target object during the motion process.

[0059] Generally speaking, the change of posture can indirectly reflect the intensity of the target object during the movement, and the intensity of the movement can also reflect the excitement of the video. In this way, by analyzing the above-mentioned movement information, it is possible to select a confident video frame from the target video frame. In the confident video frame, the target object is more likely to have an exciting scene.

[0060] Subsequently, by identifying the dynamic action category of the confident video frame, the current motion stage of the target object can be determined, and then according to the recognition result of the dynamic action category, a video clip reflecting the motion process of the target user can be edited.

[0061] It can be seen that through the technical solutions provided by one or more embodiments of the present disclosure, the confident video frames with wonderful scenes can be effectively determined from the target video frames according to the motion information, and the wonderful scenes can be further located in detail in the confident video frames by identifying the dynamic action categories in the confident video frames, and subsequently the video clips of the target object can be generated for the located wonderful scenes. In the above manner, not only can the video clips of the target object in motion be edited, but the motion state shown in the video clips can also reflect the intensity of the motion process, thereby improving the efficiency and quality of video generation.

[0062] See also Figure 3 In one embodiment of the present disclosure, a video generating device is provided, the device comprising:

[0063] The trajectory recognition unit 100 is used to obtain a target video and determine a target video frame representing a motion trajectory of a target object in the target video;

[0064] The video frame screening unit 200 is used to identify the motion information of the target object in the target video frame, and screen out the confident video frame in the target video frame based on the identified motion information; wherein the motion information is at least used to characterize the posture change of the target object;

[0065] The video generation unit 300 is used to identify the dynamic action category of the target object in the confident video frame, determine the video frame to be edited in the confident video frame based on the identified dynamic action category, and generate a video clip of the target object according to the video frame to be edited.

[0066] Among them, the specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0067] Each unit described in the above embodiments may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0068] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0069] See also Figure 4 An embodiment of the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned video generation method is implemented.

[0070] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0071] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0072] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0073] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0075] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0076] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0077] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a target video, and determine a target video frame representing a motion trajectory of a target object in the target video; Identify motion information of the target object in the target video frame, and select a confident video frame from the target video frame based on the identified motion information; wherein the motion information is at least used to characterize a posture change of the target object; The dynamic action category of the target object is identified in the confident video frame, and based on the identified dynamic action category, a video frame to be edited is determined in the confident video frame, and a video clip of the target object is generated according to the video frame to be edited.

2. The method according to claim 1, characterized in that Identifying the motion information of the target object in the target video frame includes: Based on the target video frame, identifying the posture information of each designated part of the target object; For any designated part, determine whether the posture information of the designated part has changed compared with the previous target video frame, and generate the posture change information of the designated part according to the determination result; Based on the combination of the posture change information of each of the designated parts, the motion information of the target object in the target video frame is determined.

3. The method according to claim 1 or 2, characterized in that: The motion information is also used to characterize the speed of the target object; the speed of the target object is determined in the following manner: For a first target video frame and a second target video frame that are adjacent to each other, a first bounding box of the target object is identified in the first target video frame, and a second bounding box of the target object is identified in the second target video frame; wherein the first target video frame is located before the second target video frame; A position difference between the second bounding box and the first bounding box is determined, and a speed of the target object in the second target video frame is determined according to the position difference.

4. The method according to claim 1 or 2, characterized in that: The motion information is also used to characterize the acceleration of the target object; the acceleration of the target object is determined in the following manner: For a first target video frame and a second target video frame that are adjacent to each other, determining a first speed of the target object in the first target video frame, and determining a second speed of the target object in the second target video frame; wherein the first target video frame is located before the second target video frame; The acceleration of the target object in the second target video frame is determined according to a speed difference between the second speed and the first speed.

5. The method according to claim 1, characterized in that The motion information includes a plurality of different types of motion parameters; based on the identified motion information, selecting a confident video frame from the target video frame includes: According to the preset weights of the action parameters, weighted processing is performed on the action parameters to generate the confidence of the target video frame; Based on the target video frames whose confidence is greater than or equal to a specified threshold, the filtered confident video frames are determined.

6. The method according to claim 1, characterized in that After determining the target video frame in the target video, the method further includes: intercepting an area image where the target object is located from the target video frame, and generating quality information of the area image; Based on the identified motion information, screening out a confident video frame from the target video frame comprises: respectively obtaining weight values ​​of the motion information and the quality information, and performing weighted processing on the motion information and the quality information according to the obtained weight values ​​to obtain the confidence of the target video frame; Based on the target video frames whose confidence is greater than or equal to a specified threshold, the filtered confident video frames are determined.

7. The method according to claim 1, characterized in that Different dynamic action categories correspond to respective sets of confident video frames; based on the identified dynamic action categories, determining the video frames to be edited in the confident video frames includes: For any dynamic action category, a specified number of confident video frames are selected from a set of confident video frames corresponding to the dynamic action category, and video frames to be edited are determined based on the selected confident video frames.

8. The method according to claim 1 or 7, characterized in that: Each dynamic action category corresponds to its own action interval, and the action interval includes a plurality of stage actions arranged in time sequence; Identifying the dynamic action category of the target object in the confidence video frame includes: The confidence video frame is input into a trained action prediction model to output an action label and a stage label of the confidence video frame through the action prediction model; the action label and the stage label are used to characterize the dynamic action category of the target object in the confidence video frame.

9. The method according to claim 8, characterized in that The method further comprises: Confidence video frames with the same action label are classified into the same dynamic action category, and the confidence video frames classified into the same dynamic action category are sorted according to the stage labels of the confidence video frames.

10. A video generating device, characterized in that: The device comprises: A trajectory recognition unit, used to acquire a target video and determine a target video frame representing a motion trajectory of a target object in the target video; A video frame screening unit, configured to identify motion information of the target object in the target video frame, and screen out a confident video frame from the target video frame based on the identified motion information; wherein the motion information is at least used to characterize a posture change of the target object; A video generation unit is used to identify the dynamic action category of the target object in the confident video frame, determine the video frame to be edited in the confident video frame based on the identified dynamic action category, and generate a video clip of the target object according to the video frame to be edited.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images