Video editing method and device, processing equipment and storage medium

Abstract

The invention provides a video editing method and device, processing equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the video editing method comprises the steps: determining a target person region image contained in at least one video frame of a target video; respectively acquiring first information in a target person area image contained in the video frame, wherein the first information comprises target person key point information and / or target person action information; and according to the first information, editing a target person region image contained in the at least one video frame to obtain an action editing result, the action editing result comprising action movement track information of a target person and the first information corresponding to the action movement track information. According to the embodiment of the invention, the problem that the character action video material is relatively single in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video editing method, device, processing equipment and storage medium. Background Art

[0002] Currently, in the process of video production and editing, video clips of specific actions of characters in the video can be edited separately through video processing, but the action video materials obtained in this way are relatively single, and it is difficult to provide users with a personalized video editing experience. Summary of the invention

[0003] The present invention provides a video editing method, device, processing equipment and storage medium, which solves the problem that the character action video material in the prior art is relatively single.

[0004] In order to solve the above technical problems, in a first aspect, the present invention provides a video editing method, comprising:

[0005] Determine a target person area image included in at least one video frame of a target video;

[0006] Respectively obtaining first information in the target person area image contained in the video frame, wherein the first information includes target person key point information and / or target person action information;

[0007] According to the first information, the target person area image contained in the at least one video frame is clipped to obtain an action clipping result, wherein the action clipping result includes action motion trajectory information of the target person and first information corresponding to the action motion trajectory information.

[0008] Optionally, determining the target person area image included in at least one video frame of the target video includes:

[0009] Performing target detection on each of the video frames to obtain at least one bounding box containing a person corresponding to each of the video frames;

[0010] For each of the video frames, determining a bounding box corresponding to a subject person from at least one bounding box corresponding to the video frame;

[0011] For each of the video frames, an image of a target person region contained in the video frame is obtained according to a boundary box corresponding to the main person.

[0012] Optionally, respectively acquiring first information in the target person area image contained in the video frame includes:

[0013] Performing key point detection on the target person area image contained in the video frame to obtain key point information of the target person in the target person area image;

[0014] And / or, performing time-series motion detection on the target person area image included in the video frame to obtain target person motion information of the target person area image.

[0015] Optionally, performing time-series motion detection on the target person region image included in the video frame to obtain target person motion information of the target person region image includes:

[0016] Inputting the target person region image into the fast-slow network model to obtain the action features and time features of the target person corresponding to the target person region image;

[0017] Performing encoding and decoding processing on the action features and time features of the target person to obtain the action category information and the action time information of the target person;

[0018] The target person action category information and the target person action time information are determined as the target person action information.

[0019] Optionally, the target person action information includes: target person action category information and target person action time information;

[0020] The step of clipping the target person area image contained in the at least one video frame according to the first information to obtain an action clipping result includes:

[0021] For each target person action category information, determine the target person action time information corresponding to the target person action category information;

[0022] For each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information corresponding to the target person action category information;

[0023] For each target character action category information, the action motion trajectory information and the target character action information corresponding to the action motion trajectory information are determined as the action editing result.

[0024] Optionally, the target person action information includes: target person action category information and target person action time information;

[0025] The step of clipping the target person area image contained in the at least one video frame according to the first information to obtain an action clipping result includes:

[0026] For each target person action category information, respectively determine the target person action time information and the target person key point information corresponding to the target person action category information;

[0027] For each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information and target person key point information corresponding to the target person action category information;

[0028] For each target character action category information, the action motion trajectory information and the first information corresponding to the action motion trajectory information are determined as the action editing result.

[0029] Optionally, for each target person action category information, respectively determining the target person action time information and the target person key point information corresponding to the target person action category information includes:

[0030] For each target person action category information, determine initial target person key point information, wherein the initial target person key point information is all target person key point information corresponding to the target person action category information;

[0031] For each target person action category information, based on the target person key point information corresponding to the starting time of the target person action in the target person action category information and / or the target person key point information corresponding to the ending time of the target person action in the target person action category information, the target person key point information that does not belong to the target person action category information is excluded from the initial target person key point information.

[0032] In a second aspect, an embodiment of the present invention further provides a video editing device, the device comprising:

[0033] An image acquisition module, used to determine a target person area image contained in at least one video frame of a target video;

[0034] A first processing module, used for respectively acquiring first information in the target person area image contained in the video frame, wherein the first information includes target person key point information and / or target person action information;

[0035] The second processing module is used to edit the target person area image contained in the at least one video frame according to the first information to obtain an action editing result, wherein the action editing result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information.

[0036] Optionally, the image acquisition module includes:

[0037] The target detection submodule is used to perform target detection on each of the video frames to obtain at least one bounding box containing a person corresponding to each of the video frames;

[0038] A subject detection submodule, configured to determine, for each of the video frames, a bounding box corresponding to a subject person from at least one bounding box corresponding to the video frame;

[0039] The first processing submodule is used to obtain, for each of the video frames, a target person area image contained in the video frame according to a boundary box corresponding to the main person.

[0040] Optionally, the first processing module includes:

[0041] The second processing submodule is used to perform key point detection on the target person area image contained in the video frame to obtain the target person key point information of the target person area image;

[0042] The time sequence detection submodule is used to perform time sequence action detection on the target person area image contained in the video frame to obtain the target person action information of the target person area image.

[0043] Optionally, the timing detection submodule includes:

[0044] A feature extraction unit, used for inputting the target person region image into a fast-slow network model to obtain the action features and time features of the target person corresponding to the target person region image;

[0045] A first processing unit is used to perform encoding and decoding processing on the action features and time features of the target person to obtain the action category information and the action time information of the target person;

[0046] The second processing unit is used to determine the target person action category information and the target person action time information as the target person action information.

[0047] Optionally, the target person action information includes: target person action category information and target person action time information;

[0048] The second processing module comprises:

[0049] The third processing submodule is used to determine the target person action time information and the target person key point information corresponding to each target person action category information respectively.

[0050] The fourth processing submodule is used to determine the action motion trajectory information of the target person according to all the target person action time information and target person key point information corresponding to the target person action category information for each target person action category information;

[0051] The fifth processing submodule is used to determine the action motion trajectory information and the first information corresponding to the action motion trajectory information as the action editing result for each target character action category information.

[0052] Optionally, the third processing submodule includes:

[0053] A third processing unit is used to determine initial target person key point information for each target person action category information, wherein the initial target person key point information is all target person key point information corresponding to the target person action category information;

[0054] The fourth processing unit is used to exclude the target person key point information that does not belong to the target person action category information from the initial target person key point information based on the target person key point information corresponding to the starting time of the target person action in the target person action category information and / or the target person key point information corresponding to the ending time of the target person action in the target person action category information.

[0055] Optionally, the target person action information includes: target person action category information and target person action time information;

[0056] The second processing module comprises:

[0057] A sixth processing submodule, for determining, for each target person action category information, target person action time information corresponding to the target person action category information;

[0058] A seventh processing submodule is used to determine the motion trajectory information of the target person according to all target person motion time information corresponding to the target person motion category information for each target person motion category information;

[0059] The eighth processing submodule is used to determine the action motion trajectory information and the target character action information corresponding to the action motion trajectory information as the action editing result for each target character action category information.

[0060] In the third aspect, an embodiment of the present invention further provides a processing device, comprising: a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the video editing method described in the first aspect.

[0061] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the video editing method described in the first aspect above are implemented.

[0062] In a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the video editing method as described in the first aspect.

[0063] The beneficial effects of the above technical solution of the present invention are as follows:

[0064] According to an embodiment of the present invention, after obtaining the target person area image contained in each video frame of at least one video frame of the target video, the first information (including target person key point information and / or target person action information) in the target person area image contained in the video frame can be obtained for each video frame, and then the target person area image contained in at least one video frame can be edited according to the first information to obtain an action editing result, wherein the action editing result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information. In this way, more creative materials can be provided to users, and users can use the action editing results to add special effects or other secondary creation operations, thereby solving the problem of relatively single character action video materials in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0066] Figure 1 A schematic diagram showing a flow chart of a video editing method according to an embodiment of the present invention;

[0067] Figure 2 A flow chart showing a video editing method according to another embodiment of the present invention;

[0068] Figure 3 A schematic diagram showing a multi-target detection model according to an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of a feature extraction (backbone) backbone network of an embodiment of the present invention is shown;

[0070] Figure 5 A schematic diagram of a feature fusion (neck) network according to an embodiment of the present invention is shown;

[0071] Figure 6 A schematic diagram of a key point position prediction (head) network of an embodiment of the present invention is shown;

[0072] Figure 7 A schematic diagram showing a sequential motion detection model according to an embodiment of the present invention;

[0073] Figure 8 A schematic diagram showing a Backbone structure based on SlowFast according to an embodiment of the present invention;

[0074] Fig. 9 A schematic diagram showing a Transformer-based encoding and decoding structure according to an embodiment of the present invention;

[0075] Fig.10 A schematic diagram showing the structure of a video editing device according to an embodiment of the present invention;

[0076] Fig.11 A schematic diagram showing the structure of a processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present invention. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for clarity and brevity, the description of known functions and structures is omitted.

[0078] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0079] In various embodiments of the present invention, it should be understood that the order of execution of the following processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. In addition, the terms "system" and "network" are often used interchangeably in this article.

[0080] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0081] In the embodiment of the present invention, the user terminal may be a mobile phone (or cell phone), or other devices capable of sending or receiving wireless signals, including user equipment, personal digital assistants (PDAs), wireless modems, wireless communication devices, handheld devices, laptop computers, cordless phones, wireless local loop (WLL) stations, CPE (Customer Premise Equipment) capable of converting mobile signals into WiFi signals, or mobile smart hotspots, smart home appliances, or other devices that can spontaneously communicate with a mobile communication network without human operation.

[0082] Here, we first introduce the relevant technologies:

[0083] (1) The intelligent editing method of hand movements based on traditional manual annotation and editing is the most basic technical solution for hand movement editing. Manually observe the hand movements in the video, and then filter the hand movement clips in the video by defining a series of rules. According to pre-set rules, such as movement position, speed, shape and other features, manually extract the hand movement frames in the video, splice and edit these frames in a certain order, and then automatically filter out the clips that meet the conditions for editing or mark the clips of interest through the rules. Although this method is highly flexible and can freely select hand movement clips according to needs. However, the manual annotation and editing process requires a lot of time and manpower, so it is difficult to apply to large-scale and real-time application scenarios, and there are problems such as high labor costs and low efficiency.

[0084] (2) Key point detection-based methods use computer vision techniques, such as human key point detection algorithms, to identify key points or outlines of the hands in video sequences. By tracking the position and movement of these key points, the timing information of the hand movements can be obtained. This method can provide accurate outlines or key point position information of hand movements, and has the advantage of lower computing resource requirements compared to deep learning methods. However, this method may have positioning errors for complex hand movements and is easily affected by factors such as lighting changes and occlusion.

[0085] (3) Gesture recognition-based methods use machine learning or deep learning algorithms to match hand movements with specific gestures or actions. By training a classifier or neural network, gestures in video sequences can be identified and the timing of hand movements can be obtained. This method can identify and classify specific gestures or actions and has good applicability for applications that require specific gesture recognition. However, this method requires a large amount of gesture training data and may perform poorly for complex hand movements.

[0086] (4) Optical flow-based method: This method uses an optical flow algorithm to estimate the pixel motion of each frame in a video sequence. By analyzing the optical flow pattern in the hand area, the motion information of the hand movement can be extracted and time series detection can be performed. This method can capture the motion information of hand movements and is more suitable for continuous hand movement detection. However, for complex backgrounds or fast movements, this method may have large errors, and the optical flow calculation itself may be computationally intensive.

[0087] (5) Deep learning-based methods use deep learning models, such as Convolutional Neural Networks (CNN) or Recurrent Neural Networks (RNN) to detect and track the timing of hand movements. Through these models, video sequences can be processed at the frame level, the features of hand movements can be extracted, and the category or state of hand movements in each frame can be predicted. This method is able to learn and capture complex hand movement patterns, and can achieve good performance when trained on large-scale datasets. However, this method requires a large amount of labeled data for training, and labeling the timing of hand movements may take a lot of time and manpower. In addition, this method also requires high computing resources and storage space, especially when processing video sequences, and may overfit for small-scale datasets or specific scenarios.

[0088] like Figure 1 As shown, an embodiment of the present invention provides a video editing method, comprising the following steps:

[0089] Step 11: Determine a target person area image contained in at least one video frame of the target video.

[0090] Here, the target person region image is obtained by editing the video frames. The target person region image contains a person image, and information such as the motion trajectory of the person image can be obtained in subsequent steps.

[0091] Step 12: respectively obtain first information in the target person area image contained in the video frame, wherein the first information includes target person key point information and / or target person action information.

[0092] Here, the target character key point information includes the position coordinates of each key point. For example, the character action can be a hand action or other actions. When editing hand actions, the key points can specifically be shoulders, elbows, palms, etc.

[0093] Step 13: According to the first information, the target person area image contained in the at least one video frame is edited to obtain an action editing result, wherein the action editing result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information.

[0094] In this embodiment, after obtaining the target person area image contained in each video frame of at least one video frame of the target video, the first information (including target person key point information and / or target person action information) in the target person area image contained in the video frame can be obtained for each video frame, and then the target person area image contained in at least one video frame can be edited according to the first information to obtain an action editing result, wherein the action editing result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information. In this way, more creative materials can be provided to users, making it convenient for users to use the action editing results to add special effects or other secondary creation operations, solving the problem of relatively single character action video materials in the prior art.

[0095] It should be noted that the video editing method provided in the embodiment of the present invention can be used for editing character hand movements, as well as foot movements, etc. In the subsequent embodiments, hand movement editing is used as an example for explanation, but it can be understood that the embodiment of the present invention is not limited to this.

[0096] Optionally, in step 11, determining a target person area image included in at least one video frame of the target video includes:

[0097] Step 1101 , performing target detection on each of the video frames to obtain at least one bounding box containing a person corresponding to each of the video frames.

[0098] Here, the video frame refers to a video frame in the target video. In step 1101, by performing target detection on the person in each video frame, a target detection result, such as a bounding box containing the person, can be obtained. In addition, the hands and feet of the person can also be used as detection targets to obtain other bounding boxes such as a hand box containing the person's hands and a foot box containing the feet.

[0099] In a specific implementation, a target detection algorithm in deep learning, such as Faster R-CNN, YOLO (You Only Look Once) algorithm, Single Shot MultiBox Detector (SSD) algorithm, etc., can be used to obtain a bounding box in a video frame.

[0100] Step 1102: for each of the video frames, determine a bounding box corresponding to a main character from at least one bounding box corresponding to the video frame.

[0101] It should be noted that there may be multiple people in a video frame, so multiple bounding boxes containing people may be detected. In step 1102, for the bounding box in the target detection result, the main person can be determined based on the size of the bounding box or the distance from the center point. For example:

[0102] Method 1: Determine the subject person based on the size of the bounding box: determine whether the size of the bounding box is greater than or equal to a preset threshold, and if so, determine the bounding box as the bounding box corresponding to the subject person. The appropriate preset threshold can be set by observing the size distribution of the bounding boxes in each video frame.

[0103] Method 2: Determine the subject based on the distance between the bounding box and the center point: Usually, the center position of the subject in the image is closer to the center point of the image. Therefore, the distance between the center point of each bounding box in a video frame and the center point of the image can be calculated. If the distance between the center point of a bounding box and the center point of the image is less than or equal to the preset distance, the bounding box is determined as the bounding box corresponding to the subject.

[0104] It should be noted that in addition to determining the boundary box corresponding to the main character (that is, obtaining the position information of the main character) in step 1102, the relevant position information of the main character (such as the hand frame position information) can also be obtained, and this position information can also be output as the action editing result.

[0105] Step 1103: for each of the video frames, according to the boundary box corresponding to the main character, obtain the target character area image contained in the video frame.

[0106] In this step, for each video frame, the bounding box corresponding to the main character in the video frame can be used to crop the video frame (mainly to crop out the main character image area for subsequent editing). The cropped image size can be adjusted to 96x96 to obtain the target character area image contained in the video frame.

[0107] It should be noted that in an embodiment of the present invention, the lightweight target detection algorithm YOLOV7 (that is, the target detection model) can be optimized using a labeled human body and hand frame data set, and the target frame position and category output by the model can be set to three categories: background, palm and person, so that the model can detect the required person target and its hand area while distinguishing between the foreground and background, thereby obtaining the position of the human body and hand area.

[0108] It should also be noted that, before step 1101, that is, before performing target detection on each of the video frames to obtain at least one bounding box containing a person contained in each of the video frames, the method further includes:

[0109] The target video is preprocessed with video stream, which may specifically include: video decoding, frame sampling, frame cropping, frame scaling and other video image frame operations. Among them, in video frame sampling and frame rate control: for high frame rate video streams, downsampling operations can be performed to reduce the number of frames to reduce computational complexity and increase processing speed; for low frame rate video streams, interpolation operations can be performed to increase the number of frames to improve image smoothness and details. In video resolution adjustment: for high-resolution video streams, downsampling operations can be performed to reduce the number of pixels in the image, thereby reducing the computational burden; for low-resolution video streams, interpolation operations can be performed to increase the number of pixels in the image to improve image quality and details.

[0110] After preprocessing the target video, the image size of the obtained video frame can be adjusted to 320x320, and then the adjusted image is sent as input to the pre-trained target detection model. Figure 3 As shown in the figure, the network structure of the target detection model mainly includes: feature extraction backbone network (Backbone), feature fusion network (Neck) and detection head (Head). Through this target detection model, the offset and category probability of the human body frame and hand frame in the whole image area can be predicted, and after post-processing, it is converted into the positioning of the human body frame (that is, the bounding box containing the person) and the hand area (that is, the bounding box containing the person's hand) on the image.

[0111] Optionally, in step 12, respectively obtaining first information in the target person area image contained in the video frame includes:

[0112] Step 1201, key point detection is performed on the target person region image contained in the video frame to obtain the target person key point information of the target person region image. Here, a deep learning key point detection algorithm, such as OpenPose, HRNet SimDR, etc., can be used to perform key point (such as hand key point) detection in the target person region image to obtain the target person key point information (such as left and right hand key point position information).

[0113] Specifically, the palm classification can be converted into the detection of arm key points to obtain the category information of left and right points. For example, three key points of the shoulder, elbow, and palm of the left and right arm points can be selected as the hand key point detection targets. Since the classification of left and right hand key points is strictly distinguished during the key point data annotation process, for each frame, the trained arm key point model can directly predict the key point positions of the left and right hands (i.e., the coordinate positions of the hand key points), and can also distinguish the left and right hand information (i.e., categories: left hand, right hand). In addition, the motion vector or speed of the key point can be calculated based on the results of the key point detection.

[0114] And / or, step 1202, performing time-series motion detection on the target person area image included in the video frame to obtain target person motion information of the target person area image.

[0115] Taking the image size of the target person area image as 96x96 as an example, here the video frame sequence (that is, the target person area image contained in at least one video frame) of 1xNx3x96x96 can be used as the input of the temporal action detection TAD model to perform temporal action detection. Among them, N=128 represents the maximum video sequence length that the TAD model can process, and 3 represents the number of RGB three channels of the image.

[0116] It is understandable that the execution order of the above step 1201 and step 1202 is not specific, they can be executed simultaneously, or one can be executed first and then the other.

[0117] In this embodiment, by using time-series motion detection, the key point motion sequence of continuous frames can be combined for analysis, thereby making better use of context information and improving the accuracy and robustness of motion detection. Compared with the traditional hand motion detection method based on rigid models, the time-series analysis based on hand key point detection and motion state is more suitable for the recognition of non-rigid motions, such as gestures and dynamic hand motions.

[0118] like Figure 3 As shown in the figure, the key point detection model is divided into three parts: backbone network (Backbone module), feature fusion network (i.e. neck feature fusion module), and detection head (i.e. head model prediction module), as follows:

[0119] (I) The Backbone feature extraction module is mainly used to extract robust human features for use in the feature fusion layer. The Backbone module can use the currently popular lightweight networks, including but not limited to: yolo series, moblienet series, squeezenet series and other classic backbone network structures.

[0120] After experimental comparison, the embodiment of the present invention selects the backbone network module of pp-yolo as the feature extraction backbone network. Figure 4 As shown in the figure, the input human body area is adjusted to the scale of 1x3x256x192, and hierarchical feature extraction is performed through the backbone network. The feature map descent step sizes are (32, 16, 8) respectively, and the corresponding hierarchical feature maps F1: 1x128x8x6, F2: 1x128x16x12, F3: 1x64x32x24 hierarchical features are obtained for feature fusion in the subsequent neck module.

[0121] (ii) Figure 5 As shown in the figure, the neck feature fusion module mainly uses a 1x1 convolution kernel to adjust the number of channels and reduces the feature scale through the Max-pooling operator, thereby fusing the features of different scales extracted by the Backbone module and increasing the generalization ability of feature learning.

[0122] (III) Figure 6 As shown, the head model prediction module can obtain the fused features from the neck module Figure 1 x256x8x6, adjust the channel of the feature map, convert the feature map to 1x6x2048, pass through two fully connected networks with 256 and 192 hidden layer nodes, respectively predict the position of each key point in the row and column directions, and obtain 1x6x256 and 1x6x192 feature vectors, where the number of second-dimensional channels is 6, indicating the 6 key points that need to be predicted. In the prediction stage, the softmax function is used on the two feature vectors to obtain the probability value and obtain the position of the maximum probability value, so that the position of the key point on the input image (that is, the target person area image) can be directly inferred.

[0123] Optionally, in step 1202, performing time-series motion detection on the target person region image included in the video frame to obtain target person motion information of the target person region image includes:

[0124] (1) Inputting the target person region image into the fast-slow network model to obtain the action features and time features of the target person corresponding to the target person region image.

[0125] Here, the fast-slow network model is the SlowFast model, which is a dual-channel video recognition model. SlowFast uses a slow high-resolution CNN (Fast channel) to analyze the static content in the video, and a fast low-resolution CNN (Slow channel) to analyze the dynamic content in the video. In this way, SlowFast extracts effective information from the video at both slow and fast frame rates, improving the action classification and action recognition effects.

[0126] (2) Encoding and decoding the action features and time features of the target person to obtain the action category information and the action time information of the target person.

[0127] Here, the target character action category information refers to the action category of the subject character. Taking hand actions as an example, the action category may specifically include clapping, clapping, waving arms, waving hands, and raising hands, etc. The target character action time information includes the start and end time points (i.e., the starting time point and the end time point) and the intermediate time points.

[0128] In this step, intelligent detection and positioning of the start, middle, and end time points of the target person's hand movement state can be achieved.

[0129] (3) Determine the target person's action category information and the target person's action time information as the target person's action information.

[0130] In this embodiment, based on an end-to-end temporal action detection algorithm, the starting time points, middle time points, and ending time points of the actions of the characters (such as hand actions) in a video sequence (i.e., at least one video frame) can be detected to obtain the time information of the target character's actions, and the action category (i.e., the target character's action category information) and the prediction confidence level of each action can also be output.

[0131] like Figure 7 As shown, as a specific embodiment of the present invention, a temporal action detection (TAD) model including a Backbon structure and a Transformer structure can be used to perform temporal action detection to obtain the target person action information of the above-mentioned target person area image.

[0132] The Backbon part in the embodiment of the present invention adopts the SlowFast fast and slow network model branch architecture, and uses the fast and slow branches to extract features from the time and space dimensions respectively. Among them, the Slow branch samples a smaller number of frames and a larger number of channels to learn spatial semantic information, and uses a step size τ (τ is usually set to 16) to collect video frames. Therefore, for a video with a frame rate of 30, this means that about 2 frames can be collected in 1 second, that is, T=2, and the number of channels of the slow branch is D. The Fast branch uses a larger number of frames and a smaller number of channels to learn motion information. A smaller step size is used to collect video frames, and the step size is τ / α, where α is usually set to 8. Therefore, for a video with a frame rate of 30, 15 frames (αT) can be collected in 1 second. The Fast branch keeps lightweight by using a smaller number of channels (βD), and β is usually set to 1 / 8.

[0133] If the input to the fast and slow network model is a video subsequence (that is, a video including at least one video frame containing the target person area image in each video frame): 1x128x3x96x96, after slowfast, two feature maps of different branches can be obtained. Figure 8 As shown in the figure, the slow branch is 1x2048x64, and the fast branch is 1x256x64. Finally, the feature maps of the two branches are merged into 1x2304x64 by channel. The action features and time features of the target person obtained after merging (i.e., feature information X: 1x2304x64) can be used as the input of the Tranformer structure in the next stage.

[0134] The Transformer structure mainly encodes and decodes the feature information 1x2304x64 extracted by Backbone and predicts the relevant time point information. Its main structure is as follows Fig. 9 Here, the encoder Encoder and the decoder Decoder can adopt the commonly used Transformer encoding and decoding network structure.

[0135] The feature information X: 1x2304x64 output by the Backbon structure is input into the Transformer structure, and the channel is reduced to 256 through the Project mapping module in the Transformer structure to speed up the operation. The Project mapping module outputs the mapped feature map size X1: 1x256x64. After that, the common Transformer encoding method is used to send X1 and the preset position encoding information P to the encoder network structure for feature encoding, and output the intermediate encoding feature E: 1x64x256 for subsequent decoding. The intermediate encoded feature E is decoded by the decoder to obtain the required relevant information: action category C: 1x32x5, where 32 means the step size is 128. After encoding and decoding, up to 32 action anchor points can be predicted, and 5 means the preset 5 action categories; action start and end time points SE: 1x32x2, where 2 means the start and end time prediction points; action middle time point M: 1x128x1, where 128 means the original sampling frame step size, which is a binary classification here; action confidence S: 1x32x1, the output represents the confidence prediction of the action of each anchor point.

[0136] In some specific embodiments, the first information includes target person action information, and the target person action information includes: target person action category information and target person action time information.

[0137] In step 13, the target person area image included in the at least one video frame is clipped according to the first information to obtain an action clipping result, including:

[0138] For each target character action category information, determine the target character action time information corresponding to the target character action category information; for each target character action category information, determine the target character action motion trajectory information based on all target character action time information corresponding to the target character action category information; for each target character action category information, determine the action motion trajectory information and the target character action information corresponding to the action motion trajectory information as the action editing result.

[0139] Here, the target person's action time information includes: a starting time point, an intermediate time point and a termination time point. The action motion track information and the target person's action information corresponding to the action motion track information are determined as the action editing result. The action motion track of the target person can be output as the action editing result according to the target person's action category information. The starting time point, the intermediate time point and the termination time point of each action of the subject person can also be marked in the video sequence (for example, marked in the form of a timestamp). The target person's action category information (such as applauding, clapping, waving arms, waving hands and raising hands, etc.) can also be output as the action editing result. In addition, the left and right hand information (i.e., category: left hand, right hand) can also be output as the action editing result.

[0140] In this embodiment, the motion trajectory of the action (such as the motion trajectory of the hand) in the target video can be edited, thereby achieving accurate recognition and motion analysis of the hand action.

[0141] In some other specific embodiments, the first information includes: target person key point information and target person action information, and the target person action information includes: target person action category information and target person action time information; in step 13, the target person area image included in the at least one video frame is clipped according to the first information to obtain the action clipping result, including:

[0142] Step 1301: for each target person action category information, determine the target person action time information and target person key point information corresponding to the target person action category information.

[0143] Here, the target person's key point information includes the position coordinates of each key point, and the target person's action time information includes: a starting time point, an intermediate time point, and an ending time point.

[0144] Step 1302: for each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information and target person key point information corresponding to the target person action category information.

[0145] Step 1303: for each target person action category information, determine the action motion trajectory information and the first information corresponding to the action motion trajectory information as the action editing result.

[0146] Here, the action motion trajectory information and the first information corresponding to the action motion trajectory information are determined as the action editing result, and the starting time point, the middle time point and the ending time point of each action of the subject character in the video sequence can be marked (for example, marked in the form of a timestamp), and the target character action category information (such as clapping, clapping, waving arms, waving hands and raising hands, etc.) can also be output as the action editing result, and the left and right hand position information and its motion trajectory in each action sequence can also be detected and located. In addition, the left and right hand information (i.e., category: left hand, right hand) can also be output as the action editing result.

[0147] In this embodiment, the motion trajectory of the action (such as the hand motion trajectory) in the target video can be intelligently edited, thereby achieving accurate recognition and motion analysis of the hand motion.

[0148] In some other embodiments, the first information includes key point information of the target person. In step 13, the target person region image included in the at least one video frame is edited according to the first information to obtain an action editing result, including: determining the action motion trajectory information of the target person and the key point information of the target person corresponding to the action motion trajectory information as the action editing result.

[0149] In this embodiment, the target person's key point information includes the position coordinates of each key point, and the action editing result obtained by editing may include the target person's action motion track information and the position coordinates of the key points. In this way, the action motion track (such as the hand motion track) in the target video can be obtained by editing, thereby achieving accurate recognition and motion analysis of the hand action.

[0150] like Figure 2 As shown, in an optional embodiment of the present invention, a video stream (i.e., a target video) is first input, and then the video stream can be preprocessed, and the main person is detected on the preprocessed video frame to obtain the target person area image contained in the video frame, and then, hand key point detection and timing action detection are performed on the target person area image (to obtain information such as the start and end time points of the hand action), and finally, the various information obtained in the above process is integrated and optimized to output the result (i.e., the action editing result).

[0151] In the above embodiment, a filter or smoothing algorithm can be used to smooth the motion sequence of the key points of the hand to reduce the impact of noise and instability, and necessary parameter tuning is performed to improve the performance of target detection and key point detection of the hand. In addition, there will be a lot of redundant time information when detecting the start and end time points through the time sequence action model. Consider setting a reasonable threshold or using sequence analysis technology to ensure the accuracy and stability of the hand action time points. Finally, according to the frame rate or timestamp information, it is converted into the actual time point to obtain the final output of the subject character and the target character action time information of the hand action (starting time point, middle time point and end time point).

[0152] in addition, Figure 7 The time information output by the model is mainly the sampled video frame number. In the embodiment of the present invention, it is necessary to convert the frame number information into actual time point information according to the frame rate in the target video (that is, divide the detected video frame number by the frame rate, i.e. the time point information), and use the converted time information as the target character action time information.

[0153] Considering the jitter problem in the prediction process, the use of key point position information will have a great impact. For the key points of the video sequence, some classic smoothing filter algorithms can be used to smooth the key point positions, such as mean filter, median filter, Gaussian filter, etc. These algorithms can effectively remove noise and unnecessary fluctuations and obtain a relatively smooth motion trajectory.

[0154] It should be noted that, due to the limited single prediction length of the temporal action detection model, the uncropped long video will be split into multiple overlapping sub-video sequences, which may cause redundancy in the prediction results of different sub-video sequences. In the embodiment of the present invention, some low-confidence action anchors can be filtered out according to the action prediction confidence. In addition, the action segments with a high overlap rate of the start and end times can also be merged in the time dimension, and finally the action segments with high confidence and no overlap rate are output.

[0155] Optionally, in step 1301, for each target person action category information, determining the target person action time information and the target person key point information corresponding to the target person action category information respectively includes:

[0156] For each target person action category information, determine initial target person key point information, wherein the initial target person key point information is all target person key point information corresponding to the target person action category information;

[0157] For each target person action category information, based on the target person key point information corresponding to the starting time of the target person action in the target person action category information and / or the target person key point information corresponding to the ending time of the target person action in the target person action category information, the target person key point information that does not belong to the target person action category information is excluded from the initial target person key point information.

[0158] In this embodiment, when excluding the target person key point information that does not belong to the target person action category information, Figure 7 The time information output by the model is divided into the time domain of the action, and then the key point position information within the interval is intercepted. According to the pre-defined intermediate state forms of each action, the target person's key point information that does not belong to the target person's action category information is excluded by analyzing the action and motion trajectory. According to the project requirements, the pre-defined intermediate states of each action can be simply described as: clapping (the moment when the palms close and separate each time), waving arms (the moment when the hands are at the highest point each time they are waved back and forth), waving hands (the moment when the palms are at the highest point each time they are waved back and forth), clapping (the moment when the palms touch the object each time), and raising hands (the moment when the hands are raised from bottom to top each time to the highest point).

[0159] In a specific embodiment, assuming that in a video frame, the start and end time points of a waving action are [t0, t1], which are converted into start and end frame numbers [f1, f2] through the video frame rate fps, then all the hand key points in the key point detection results of the waving action can be obtained according to the start and end frame numbers as [p1, p2, p]. Based on the key point position information, the hand motion trajectory can be calculated, and the key point positions in the interval can be analyzed by sliding windows to infer the middle moment of the waving action, and then the key point information of the target person that does not belong to this waving action can be excluded.

[0160] It should be noted that the video editing method of the embodiment of the present invention can realize intelligent editing of hand movements in video streams by providing accurate hand movement positioning and motion analysis, and has rich commercial value. For example, it can provide creative inspiration and rich visual effects for content creators, and can also provide users with personalized video editing experience. It can bring innovation and business opportunities to content creators, individual users, virtual reality and augmented reality applications, as well as motion analysis and health monitoring. With the continuous development of artificial intelligence and computer vision technology, the commercial prospects of intelligent editing of hand movements in video streams will be broader. For example, it can also be applied to fields such as gesture control systems, and is also suitable for film production, advertising production, sports analysis and other fields.

[0161] In addition, the object detection and hand key point detection based on deep learning in the embodiments of the present invention have good scalability and adaptability, and the performance can be further improved by using larger data sets and optimizing the network structure.

[0162] In the embodiment of the present invention, by integrating the currently advanced deep learning technologies (such as time-series motion detection, multi-target detection, and key point detection), the position of the main character and the key points of the hand in the video scene can be automatically detected, and the categories of the main character's hand movements can be analyzed, thereby realizing the intelligent recognition of the start, middle, and end time points of the hand movements and the tracking of the hand movement trajectories of the left and right hands. In this way, comprehensive information about the main character and hand movements can be provided, reducing the workload of manual intervention and manual analysis, improving the efficiency of video production and editing, providing users with more creative materials, and facilitating users to use the action editing results to add special effects or other secondary creation operations, solving the problem of relatively single character action video materials in the prior art.

[0163] like Fig.10 As shown, an embodiment of the present invention further provides a video editing device, the device comprising:

[0164] An image acquisition module 101 is used to determine a target person area image included in at least one video frame of a target video;

[0165] A first processing module 102 is used to respectively obtain first information in the target person area image contained in the video frame, wherein the first information includes target person key point information and / or target person action information;

[0166] The second processing module 103 is used to clip the target person area image contained in the at least one video frame according to the first information to obtain an action clipping result, wherein the action clipping result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information.

[0167] In this embodiment, after obtaining the target person area image contained in each video frame of at least one video frame of the target video, the first information (including target person key point information and / or target person action information) in the target person area image contained in the video frame can be obtained for each video frame, and then the target person area image contained in at least one video frame can be edited according to the first information to obtain an action editing result, wherein the action editing result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information. In this way, more creative materials can be provided to users, making it convenient for users to use the action editing results to add special effects or other secondary creation operations, solving the problem of relatively single character action video materials in the prior art.

[0168] Optionally, the image acquisition module 101 includes:

[0169] The target detection submodule is used to perform target detection on each of the video frames to obtain at least one bounding box containing a person corresponding to each of the video frames;

[0170] A subject detection submodule, configured to determine, for each of the video frames, a bounding box corresponding to a subject person from at least one bounding box corresponding to the video frame;

[0171] The first processing submodule is used to obtain, for each of the video frames, a target person area image contained in the video frame according to a boundary box corresponding to the main person.

[0172] Optionally, the first processing module 102 includes:

[0173] The second processing submodule is used to perform key point detection on the target person area image contained in the video frame to obtain the target person key point information of the target person area image;

[0174] The time sequence detection submodule is used to perform time sequence action detection on the target person area image contained in the video frame to obtain the target person action information of the target person area image.

[0175] Optionally, the timing detection submodule includes:

[0176] A feature extraction unit, used for inputting the target person region image into a fast-slow network model to obtain the action features and time features of the target person corresponding to the target person region image;

[0177] A first processing unit is used to perform encoding and decoding processing on the action features and time features of the target person to obtain the action category information and the action time information of the target person;

[0178] The second processing unit is used to determine the target person action category information and the target person action time information as the target person action information.

[0179] Optionally, the target person action information includes: target person action category information and target person action time information;

[0180] The second processing module 103 includes:

[0181] The third processing submodule is used to determine the target person action time information and the target person key point information corresponding to each target person action category information respectively.

[0182] The fourth processing submodule is used to determine the action motion trajectory information of the target person according to all the target person action time information and target person key point information corresponding to the target person action category information for each target person action category information;

[0183] The fifth processing submodule is used to determine the action motion trajectory information and the first information corresponding to the action motion trajectory information as the action editing result for each target character action category information.

[0184] Optionally, the third processing submodule includes:

[0185] A third processing unit is used to determine initial target person key point information for each target person action category information, wherein the initial target person key point information is all target person key point information corresponding to the target person action category information;

[0186] The fourth processing unit is used to exclude the target person key point information that does not belong to the target person action category information from the initial target person key point information based on the target person key point information corresponding to the starting time of the target person action in the target person action category information and / or the target person key point information corresponding to the ending time of the target person action in the target person action category information.

[0187] Optionally, the target person action information includes: target person action category information and target person action time information;

[0188] The second processing module comprises:

[0189] A sixth processing submodule, for determining, for each target person action category information, target person action time information corresponding to the target person action category information;

[0190] A seventh processing submodule is used to determine the motion trajectory information of the target person according to all target person motion time information corresponding to the target person motion category information for each target person motion category information;

[0191] The eighth processing submodule is used to determine the action motion trajectory information and the target character action information corresponding to the action motion trajectory information as the action editing result for each target character action category information.

[0192] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0193] like Fig.11As shown, an embodiment of the present invention further provides a processing device, including a transceiver 113, a processor 111, a memory 112, and a computer program stored in the memory 112 and executable on the processor 111, wherein the processor 111 is configured to read the program in the memory 112 and execute the following process:

[0194] Determine a target person area image included in at least one video frame of a target video;

[0195] Respectively obtaining first information in the target person area image contained in the video frame, wherein the first information includes target person key point information and / or target person action information;

[0196] According to the first information, the target person area image contained in the at least one video frame is clipped to obtain an action clipping result, wherein the action clipping result includes action motion trajectory information of the target person and first information corresponding to the action motion trajectory information.

[0197] In this embodiment, after obtaining the target person area image contained in each video frame of at least one video frame of the target video, the first information (including target person key point information and / or target person action information) in the target person area image contained in the video frame can be obtained for each video frame, and then the target person area image contained in at least one video frame can be edited according to the first information to obtain an action editing result, wherein the action editing result includes the action motion trajectory information of the target person and the first information corresponding to the action motion trajectory information. In this way, more creative materials can be provided to users, making it convenient for users to use the action editing results to add special effects or other secondary creation operations, solving the problem of relatively single character action video materials in the prior art.

[0198] Optionally, when determining the target person area image included in at least one video frame of the target video, the processor 111 is further configured to:

[0199] Performing target detection on each of the video frames to obtain at least one bounding box containing a person corresponding to each of the video frames;

[0200] For each of the video frames, determining a bounding box corresponding to a subject person from at least one bounding box corresponding to the video frame;

[0201] For each of the video frames, an image of a target person region contained in the video frame is obtained according to a boundary box corresponding to the main person.

[0202] Optionally, when respectively acquiring the first information in the target person area image contained in the video frame, the processor 111 is further configured to:

[0203] Performing key point detection on the target person area image contained in the video frame to obtain key point information of the target person in the target person area image;

[0204] And / or, performing time-series motion detection on the target person area image included in the video frame to obtain target person motion information of the target person area image.

[0205] Optionally, when the processor 111 performs time-series motion detection on the target person region image included in the video frame to obtain the target person motion information of the target person region image, it is further configured to:

[0206] Inputting the target person region image into the fast-slow network model to obtain the action features and time features of the target person corresponding to the target person region image;

[0207] Performing encoding and decoding processing on the action features and time features of the target person to obtain the action category information and the action time information of the target person;

[0208] The target person action category information and the target person action time information are determined as the target person action information.

[0209] Optionally, the target person action information includes: target person action category information and target person action time information;

[0210] When the processor 111 clips the target person area image contained in the at least one video frame according to the first information to obtain the action clipping result, it is specifically used to:

[0211] For each target person action category information, determine the target person action time information corresponding to the target person action category information;

[0212] For each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information corresponding to the target person action category information;

[0213] For each target character action category information, the action motion trajectory information and the target character action information corresponding to the action motion trajectory information are determined as the action editing result.

[0214] Optionally, the target person action information includes: target person action category information and target person action time information;

[0215] When the processor 111 clips the target person area image contained in the at least one video frame according to the first information to obtain the action clipping result, it is specifically used to:

[0216] For each target person action category information, respectively determine the target person action time information and the target person key point information corresponding to the target person action category information;

[0217] For each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information and target person key point information corresponding to the target person action category information;

[0218] For each target character action category information, the action motion trajectory information and the first information corresponding to the action motion trajectory information are determined as the action editing result.

[0219] Optionally, when the processor 111 determines the target person action time information and the target person key point information corresponding to each target person action category information respectively, the processor 111 is further configured to:

[0220] For each target person action category information, determine initial target person key point information, wherein the initial target person key point information is all target person key point information corresponding to the target person action category information;

[0221] For each target person action category information, based on the target person key point information corresponding to the starting time of the target person action in the target person action category information and / or the target person key point information corresponding to the ending time of the target person action in the target person action category information, the target person key point information that does not belong to the target person action category information is excluded from the initial target person key point information.

[0222] The bus architecture may include any number of interconnected buses and bridges, specifically one or more processors 111 represented by processor 111 and various circuits of memory 112 represented by memory 112 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 113 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor 111 is responsible for managing the bus architecture and general processing, and the memory 112 may store data used by the processor when performing operations.

[0223] The processing device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0224] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing relevant hardware through a computer program, wherein the computer program includes instructions for executing part or all of the steps of the above method; and the computer program may be stored in a readable storage medium, and the storage medium may be any form of storage medium.

[0225] In addition, a specific embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned video editing method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0226] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0229] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0230] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles described in the present invention. These improvements and modifications are also within the scope of protection of the present invention.

Claims

1. A video editing method, characterized in that: include: Determine a target person area image included in at least one video frame of a target video; Respectively obtaining first information in the target person area image contained in the video frame, wherein the first information includes target person key point information and / or target person action information; According to the first information, the target person area image contained in the at least one video frame is clipped to obtain an action clipping result, wherein the action clipping result includes action motion trajectory information of the target person and first information corresponding to the action motion trajectory information.

2. The method according to claim 1, characterized in that The step of determining a target person area image contained in at least one video frame of a target video comprises: Performing target detection on each of the video frames to obtain at least one bounding box containing a person corresponding to each of the video frames; For each of the video frames, determining a bounding box corresponding to a subject person from at least one bounding box corresponding to the video frame; For each of the video frames, an image of a target person region contained in the video frame is obtained according to a boundary box corresponding to the main person.

3. The method according to claim 1, characterized in that The step of respectively obtaining first information in the target person area image contained in the video frame includes: Performing key point detection on the target person area image contained in the video frame to obtain key point information of the target person in the target person area image; And / or, performing time-series motion detection on the target person area image included in the video frame to obtain target person motion information of the target person area image.

4. The method according to claim 3, characterized in that The performing time-series motion detection on the target person region image contained in the video frame to obtain the target person motion information of the target person region image includes: Inputting the target person region image into the fast-slow network model to obtain the action features and time features of the target person corresponding to the target person region image; Performing encoding and decoding processing on the action features and time features of the target person to obtain the action category information and the action time information of the target person; The target person action category information and the target person action time information are determined as the target person action information.

5. The method according to claim 1, characterized in that The target person action information includes: target person action category information and target person action time information; The step of clipping the target person area image contained in the at least one video frame according to the first information to obtain an action clipping result includes: For each target person action category information, determine the target person action time information corresponding to the target person action category information; For each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information corresponding to the target person action category information; For each target character action category information, the action motion trajectory information and the target character action information corresponding to the action motion trajectory information are determined as the action editing result.

6. The method according to claim 1, characterized in that The target person action information includes: target person action category information and target person action time information; The step of clipping the target person area image contained in the at least one video frame according to the first information to obtain an action clipping result includes: For each target person action category information, respectively determine the target person action time information and the target person key point information corresponding to the target person action category information; For each target person action category information, determine the action motion trajectory information of the target person according to all target person action time information and target person key point information corresponding to the target person action category information; For each target character action category information, the action motion trajectory information and the first information corresponding to the action motion trajectory information are determined as the action editing result.

7. The method according to claim 6, characterized in that The step of determining the target person action time information and the target person key point information corresponding to each target person action category information includes: For each target person action category information, determine initial target person key point information, wherein the initial target person key point information is all target person key point information corresponding to the target person action category information; For each target person action category information, based on the target person key point information corresponding to the starting time of the target person action in the target person action category information and / or the target person key point information corresponding to the ending time of the target person action in the target person action category information, the target person key point information that does not belong to the target person action category information is excluded from the initial target person key point information.

8. A processing device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the video editing method according to any one of claims 1 to 7 when executing the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video editing method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the video editing method according to any one of claims 1 to 7.

Images