Video processing method and device, equipment and storage medium

Abstract

The application discloses a video processing method, device and equipment and a storage medium, and relates to the technical field of video processing. The method comprises the following steps: performing anchor point marking processing on a silent frame image in a to-be-processed silent video segment to obtain a marked silent frame image; performing video shooting based on human body key point information in the marked silent frame image to obtain a motion video segment; determining an image distance between the silent frame image and a current frame image corresponding to the motion video segment; determining a motion start image frame and a motion end image frame in the motion video segment based on the image distance; and performing image pruning on the motion video segment based on the motion start image frame and the motion end image frame to obtain a pruned motion video segment. By using the technical solution, the technical problem that the shooting and pruning of motion data of a 2D digital person are usually assisted by manual work in the prior art, which leads to large errors, high delay and high operation cost, is solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to video processing methods, devices, equipment, and storage media. Background Art

[0002] With the development of digital human technology, digital humans are increasingly being used in scenarios such as video conferencing, live streaming, virtual character generation, and animation video production. Currently, digital humans on the market can be divided into 2D and 3D types. Compared to 3D digital humans, 2D digital humans require no modeling or motion capture, resulting in lower costs and wider applications.

[0003] In practical applications, 2D digital humans often need to stitch together video clips of different actions. Current 2D digital humans require high similarity between the first and last frames of each clip when stitching these clips. To ensure this similarity during data acquisition, common methods for capturing and trimming data rely on manual intervention, such as visual observation and screen tracing. However, these methods often result in significant errors, latency, and high operational costs. Summary of the Invention

[0004] The main purpose of this application is to provide a video processing method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that the shooting and trimming of 2D digital human motion data are usually done with manual assistance, resulting in large errors, delays and high operating costs.

[0005] To achieve the above objectives, the present application proposes a video processing method, which includes:

[0006] Marking anchor points on the silent frame images in the silent video clip to be processed to obtain marked silent frame images;

[0007] Shooting a video based on the human body key point information in the marked silent frame image to obtain an action video clip;

[0008] determining an image distance between the still frame image and a current frame image corresponding to the action video clip;

[0009] determining an action start image frame and an action end image frame in the action video clip based on the image distance;

[0010] The action video segment is trimmed based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0011] In one embodiment, the step of performing anchor point marking processing on the silent frame image in the silent video segment to be processed to obtain the marked silent frame image includes:

[0012] Determining key points of a human body in a silent frame image in a silent video clip to be processed;

[0013] The human body key points are used as anchor points to perform anchor point marking processing on the silent frame image to obtain a marked silent frame image.

[0014] In one embodiment, the step of determining the image distance between the still frame image and the current frame image corresponding to the action video clip includes:

[0015] Reading a current frame image from the action video clip frame by frame;

[0016] Performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image;

[0017] Performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image;

[0018] performing optical flow calculation on the silent frame image and the current frame image to obtain an optical flow distance between the silent frame image and the current frame image;

[0019] An image distance between the silent frame image and the current frame image is determined based on the key point distance, the inter-frame difference distance, and the optical flow distance.

[0020] In one embodiment, the step of calculating the key point distance between the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image includes:

[0021] Performing human key point detection on the silent frame image and the current frame image using a preset human key point detection model to obtain silent frame human key points and current frame human key points respectively;

[0022] Determine first position information corresponding to the human body key point of the silent frame, and second position information corresponding to the human body key point of the current frame;

[0023] A key point distance calculation is performed on the silent frame image and the current frame image based on the first position information and the second position information to obtain a key point distance between the silent frame image and the current frame image.

[0024] In one embodiment, the step of performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image includes:

[0025] Performing image difference calculation on the silent frame image and the current frame image to obtain an image difference result;

[0026] The image difference result is averaged to obtain an inter-frame difference distance between the silent frame image and the current frame image.

[0027] In one embodiment, the step of performing optical flow calculation on the silent frame image and the current frame image to obtain the optical flow distance between the silent frame image and the current frame image includes:

[0028] Performing grayscale processing on the silent frame image and the current frame image to obtain a silent grayscale frame image and a current grayscale frame image;

[0029] Optical flow calculation is performed on the silent grayscale frame image and the current grayscale frame image to obtain an optical flow distance between the silent frame image and the current frame image.

[0030] In one embodiment, the step of determining the action start image frame and the action end image frame in the action video clip based on the image distance includes:

[0031] Obtaining a starting video segment and an ending video segment in the action video segment;

[0032] Determining a first frame image from the starting video segment based on the image distance, where the first frame image is a frame image in the starting video segment having the smallest image distance from the silent frame image;

[0033] determining a second frame image from the ending video segment based on the image distance, where the second frame image is a frame image in the ending video segment having the smallest image distance from the silent frame image;

[0034] The first frame image and the second frame image are respectively determined as an action start image frame and an action end image frame in the action video clip.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a video processing device, which includes:

[0036] An image marking module is used to mark anchor points of silent frame images in the silent video clip to be processed to obtain marked silent frame images;

[0037] A video shooting module is used to shoot a video based on the human body key point information in the marked silent frame image to obtain an action video clip;

[0038] an image distance determination module, configured to determine an image distance between the still frame image and a current frame image corresponding to the action video clip;

[0039] an image frame determining module, configured to determine an action start image frame and an action end image frame in the action video clip based on the image distance;

[0040] The image trimming module is used to trim the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a video processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the video processing method described above are implemented.

[0043] The present application provides a video processing method, which discloses the following steps: shooting a video based on the information of key human body points in a marked silent frame image to obtain an action video clip; determining the image distance between the silent frame image and the current frame image corresponding to the action video clip; determining the action start image frame and the action end image frame in the action video clip based on the image distance; and performing image trimming on the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip. Compared with the prior art in which the shooting and trimming of 2D digital human action data are usually performed through manual assistance such as human eye observation and screen tracing, which have the problems of large manual errors, high latency, and high operating costs, the present invention shoots a video based on the information of key human body points in the marked silent frame image after anchor point marking to obtain an action video clip, and performs image trimming based on the action start image frame and the action end image frame in the action video clip to obtain a trimmed action video clip, thereby solving the technical problem in the prior art in which the shooting and trimming of 2D digital human action data are usually performed through manual assistance, which leads to large errors, high latency, and high operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings are only for the purpose of illustrating the embodiments of the present application, and for the person skilled in the art, other drawings can also be obtained without creative labor.

[0046] Figure 1 The flowchart provided by the video processing method embodiment one of the present application;

[0047] Figure 2 The still frame image in the video processing method of the present application;

[0048] Figure 3 The still frame anchor point image in the video processing method of the present application;

[0049] Figure 4 The flowchart provided by the video processing method embodiment two of the present application;

[0050] Figure 5 The difference result schematic diagram in the video processing method of the present application;

[0051] Figure 6 The optical flow result schematic diagram in the video processing method of the present application;

[0052] Figure 7 The flowchart provided by the video processing method embodiment three of the present application;

[0053] Figure 8 The overall flowchart of the video processing method of the present application;

[0054] Figure 9 The module structure schematic diagram of the video processing device embodiment of the present application;

[0055] Figure 10 The device structure schematic diagram of the hardware running environment involved in the video processing method in the embodiment of the present application.

[0056] The purpose realization, functional features and advantages of the present application will be further described with reference to the accompanying drawings combined with the embodiments. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0058] In order to better understand the technical solutions of the present application, the following will be described in detail combined with the drawings in the specification and specific embodiments.

[0059] The main solution of the embodiment of the present application is: based on the human body key point information in the marked silent frame image, video shooting is performed to obtain an action video clip; the image distance between the silent frame image and the current frame image corresponding to the action video clip is determined; based on the image distance, the action start image frame and the action end image frame in the action video clip are determined; based on the action start image frame and the action end image frame, the action video clip is image trimmed to obtain a trimmed action video clip.

[0060] In existing technologies, when stitching video clips together using 2D digital humans, high similarity is required between the first and last frames of each clip. To ensure this similarity during data acquisition, common methods for capturing and trimming data rely on manual intervention, such as visual observation and screen tracing. However, these methods often result in large errors, high latency, and high operational costs.

[0061] The present application provides a solution, which can shoot videos based on the human body key point information in the marked silent frame image after anchor point marking to obtain action video clips, and perform image trimming based on the action start image frame and action end image frame in the action video clip to obtain trimmed action video clips, thereby solving the technical problem in the prior art that the shooting and trimming of 2D digital human action data are usually done with manual assistance, resulting in large errors, delays and high operating costs.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or video processing device capable of implementing the above functions. The following uses a video processing device as an example (hereinafter referred to as the device) to illustrate this embodiment and the following embodiments.

[0063] Based on this, the embodiment of the present application provides a video processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the video processing method of the present application.

[0064] In this embodiment, the video processing method includes steps S10 to S50:

[0065] Step S10: performing anchor point marking processing on the silent frame image in the silent video segment to be processed to obtain a marked silent frame image.

[0066] It should be noted that the above-mentioned silent video segment to be processed can be a silent video segment that needs to be processed. In this embodiment, any video shooting device can be used to shoot a silent video segment, and the silent state needs to be maintained for more than 30 seconds to obtain the silent video segment to be processed.

[0067] It should be understood that the above-mentioned silent frame image can be an image frame in the silent video clip to be processed. In this embodiment, an image frame with a neutral posture, regular facial features, and a calm and gentle expression can be selected from the silent video clip to be processed as the silent frame image F s , refer to Figure 2 , Figure 2 It is a silent frame image in the video processing method of this application. Figure 2 The model in the picture changes from other poses to the silent frame pose, which is called homing.

[0068] It can be understood that the marked silent frame image may be a silent frame image marked with an anchor point.

[0069] Specifically, the step S10 includes: determining human key points in a silent frame image in a silent video segment to be processed; and marking the silent frame image with the human key points as anchor points to obtain a marked silent frame image.

[0070] It should be noted that the above-mentioned key points of the human body are key points in the human body, such as the upper and lower edges of the head, shoulders, elbows, hands and other key points. Figure 3 As shown, Figure 3 Anchor images are marked for silent frames in the video processing method of this application. Figure 3 The blue dots in the image are anchor points, which are used to guide the user to position themselves according to the anchor points during the shooting phase. In this embodiment, key points such as the upper, lower, left, and right edges of the human head, shoulders, elbows, and hands can be used as anchor points. Based on these anchor points, the still frame image is marked with anchor points, ultimately obtaining a still frame image with the marked anchor points.

[0071] Step S20: performing video capture based on the human body key point information in the marked silent frame image to obtain an action video clip.

[0072] It can be understood that the above-mentioned human body key point information can be the position information of the human body key points.

[0073] It should be understood that the aforementioned action video clips can be video clips of a user performing an action. In this embodiment, the device can visualize the anchor points in the marked silent frame images and guide the user in the current frame to align with the anchor points. After alignment, the user can begin the action, and the device starts capturing the video. After the user completes the action, the action can end when the user in the current frame aligns with the anchor points, ultimately obtaining the action video clip.

[0074] Step S30: determining an image distance between the still frame image and the current frame image corresponding to the action video clip.

[0075] It can be understood that the above-mentioned current frame image can be a frame image currently read from the action video clip.

[0076] It should be noted that the aforementioned image distance can be a weighted combination of the distance between key points in the static frame image and the current frame image, the inter-frame difference distance, and the optical flow distance. The key point distance can be the distance between key points in the static frame image and the current frame image; the inter-frame difference distance can be a feature used to describe the difference in pixel values ​​between the static frame image and the current frame image; and the optical flow distance can be the distance moved between pixels in the static frame image and the current frame image.

[0077] Step S40: determining an action start image frame and an action end image frame in the action video clip based on the image distance.

[0078] It should be understood that the aforementioned action-start image frame may be the image frame in the action video clip when the user begins to perform the action; and the aforementioned action-end image frame may be the image frame in the action video clip when the user completes the action. In this embodiment, the action-start image frame closest to the silent frame (hereinafter referred to as the first frame) may be controlled to be within the first second of the action video clip, and the action-end image frame closest to the silent frame (hereinafter referred to as the last frame) may be controlled to be within the last second of the action video clip.

[0079] Step S540: performing image trimming on the action video segment based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0080] In actual applications, the device can use the first frame as the starting point of the action video clip, trim and remove the image frames before the first frame, and use the last frame as the end point of the action video clip, trim and remove the image frames after the last frame, and finally obtain the trimmed action video clip.

[0081] This embodiment provides a video processing method, which discloses capturing a video based on information about key human points in a marked silent frame image to obtain an action video clip; determining an image distance between the silent frame image and a current frame image corresponding to the action video clip; determining an action start image frame and an action end image frame in the action video clip based on the image distance; and performing image trimming on the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip. Compared with the prior art, in which the capture and trimming of 2D digital human action data is usually performed through manual assistance such as human eye observation and screen tracing, which has problems such as large manual errors, high latency, and high operating costs, this embodiment captures a video based on information about key human points in a marked silent frame image after anchor point marking to obtain an action video clip, and performs image trimming based on the action start image frame and the action end image frame in the action video clip to obtain a trimmed action video clip, thereby solving the technical problem in the prior art that the capture and trimming of 2D digital human action data is usually performed through manual assistance, which leads to large errors, high latency, and high operating costs.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 4 , Figure 4 A flowchart of the second embodiment of the video processing method of this application is provided.

[0083] In this embodiment, step S30 includes steps S301 to S305:

[0084] Step S301: reading the current frame image from the action video clip frame by frame.

[0085] It can be understood that, in this embodiment, the device can read image frames from the action video clip frame by frame, wherein the image frame currently read by the device is the current frame image.

[0086] Step S302: performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image.

[0087] Specifically, the step S302 comprises: performing human key point detection on the silent frame image and the current frame image by a preset human key point detection model to obtain a silent frame human key point and a current frame human key point respectively; determining first position information corresponding to the silent frame human key point and second position information corresponding to the current frame human key point; and performing key point distance calculation on the silent frame image and the current frame image based on the first position information and the second position information to obtain a key point distance between the silent frame image and the current frame image.

[0088] It should be understood that the above-mentioned silent frame human key point can be a human key point in the silent frame image; the above-mentioned current frame human key point can be a human key point in the current frame image. Correspondingly, the above-mentioned first position information is the position information of the silent frame human key point; and the above-mentioned second position information is the position information of the current frame human key point.

[0089] In actual application, the silent frame image F s and the current frame image F a After preprocessing, the image can be scaled to the size required for input of a human key point detection network for human key point detection. The human key point detection model used can be MediaPipe. After human key point detection on the silent frame image F s and the current frame image F a , the position information of the key points in the image can be obtained. For example, the human key points of the silent frame can include P s 0, P s 1, …, P s 22, and the corresponding position information can be [x s 0, y s 0], [x s 1, y s 1], …, [x s 22, y s 22]. The human key points of the current frame can include P a 0, P a 1, …, P a 22, and the corresponding position information can be [x a 0, y a 0], [x a 1, y a 1], …, [x a 22, y a 22]. Then, the key point distance between the silent frame image and the current frame image can be calculated according to the first position information corresponding to the silent frame human key point and the second position information corresponding to the current frame human key point. The calculation formula can be:

[0090]

[0091] Among them, (P a iP s i) can be further calculated as:

[0092]

[0093] Where D keypoints is the key point distance, and i represents the key point of the human body.

[0094] Step S303: performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image.

[0095] Specifically, the step S303 includes: performing image difference calculation on the silent frame image and the current frame image to obtain an image difference result; and performing averaging processing on the image difference result to obtain an inter-frame difference distance between the silent frame image and the current frame image.

[0096] In this embodiment, the device may first perform a silent frame image F s With the current frame image F a Perform grayscale preprocessing and then calculate the silent frame image F s With the current frame image F a The image difference between them is obtained to obtain the image difference result, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the difference results in the video processing method of this application. The closer the pixel is to black, the smaller the inter-frame difference at that location; the closer the pixel is to white, the larger the inter-frame difference at that location. Finally, the image difference results can be averaged to obtain the inter-frame difference distance of the current image frame. The calculation formula is as follows:

[0097]

[0098] Among them, Diff sa The calculation method is:

[0099] Diff sa =|F S -F a |;

[0100] Where D frame is the inter-frame difference distance, Diff sa is the silent frame image F s With the current frame image F a The image difference result between them is w, which represents the width of the image, and h, which represents the height of the image.

[0101] Step S304: performing optical flow calculation on the silent frame image and the current frame image to obtain an optical flow distance between the silent frame image and the current frame image.

[0102] Specifically, the step S304 includes: performing grayscale processing on the silent frame image and the current frame image to obtain a silent grayscale frame image and a current grayscale frame image; performing optical flow calculation on the silent grayscale frame image and the current grayscale frame image to obtain an optical flow distance between the silent frame image and the current frame image.

[0103] It can be understood that the above-mentioned silent grayscale frame image can be a grayscale image obtained by grayscale processing the silent frame image; the above-mentioned current grayscale frame image can be a grayscale image obtained by grayscale processing the current frame image.

[0104] In practical applications, the device can first process the silent frame image F s With the current frame image F a Perform grayscale preprocessing to obtain the grayscale silent grayscale frame image and the current grayscale frame image, and then calculate the optical flow value between the silent grayscale frame image and the current grayscale frame image to obtain the optical flow distance of the current frame image. In this embodiment, the optical flow algorithm used to calculate the optical flow value can be the Farneback algorithm. The calculation result is as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of the optical flow results in the video processing method of this application. Figure 6 The green arrows in the figure represent the movement distance and direction of the pixels in the image.

[0105] Step S305: determining the image distance between the silent frame image and the current frame image based on the key point distance, the inter-frame difference distance, and the optical flow distance.

[0106] In practical applications, after calculating the key point distance, inter-frame difference distance, and optical flow distance between the silent frame image and the current frame image, a weighted sum of the key point distance, inter-frame difference distance, and optical flow distance can be performed to finally obtain the image distance between the silent frame image and the current frame image. The calculation formula for the image distance in this embodiment can be:

[0107] D image =αD keypoints +βD frame +γD flow ;

[0108] Where D image is the image distance, D keypoints is the key point distance, D frame is the inter-frame difference distance, D flow is the optical flow distance, α, β and γ are weight coefficients.

[0109] in,

[0110] α+β+γ=1;

[0111] 0<α, β, γ<1.

[0112] In this embodiment, the present invention discloses reading the current frame image from the action video clip frame by frame; performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image; performing image difference calculation on the silent frame image and the current frame image to obtain the inter-frame difference distance between the silent frame image and the current frame image; performing optical flow calculation on the silent frame image and the current frame image to obtain the optical flow distance between the silent frame image and the current frame image; determining the image distance between the silent frame image and the current frame image based on the key point distance, the inter-frame difference distance and the optical flow distance, thereby realizing multi-dimensional distance calculation between the silent frame image and the current frame image, enhancing the reliability of the image distance calculation, and thus improving the accuracy of subsequent video clip trimming.

[0113] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction and will not be described in detail later. Figure 7 , Figure 7 This is a flow chart of the third embodiment of the video processing method of this application.

[0114] In this embodiment, step S40 includes steps S401 to S404:

[0115] Step S401: obtaining a starting video segment and an ending video segment in the action video segment.

[0116] It should be noted that the above-mentioned starting video clip can be a video clip within a period of time from the beginning of the action video clip, for example, a video clip within 1 second before the action video clip; the above-mentioned ending video clip can be a video clip within a period of time from the end of the action video clip, for example, a video clip within the last 1 second of the action video clip.

[0117] Step S402: determining a first frame image from the starting video segment based on the image distance, where the first frame image is a frame image in the starting video segment having the smallest image distance from the silent frame image.

[0118] It can be understood that the first frame image may be a frame image having the smallest image distance with the silent frame image among all image frames of the starting video segment.

[0119] Step S403: determining a second frame image from the end video segment based on the image distance, where the second frame image is a frame image in the end video segment having the smallest image distance with the silent frame image.

[0120] It can be understood that the second frame image may be a frame image having the smallest image distance with the silent frame image among all the image frames of the ending video segment.

[0121] Step S404: determining the first frame image and the second frame image as the action start image frame and the action end image frame in the action video clip respectively.

[0122] In actual applications, the device can calculate the image distance between the silent frame image and the image frame sequence before the action video clip based on the silent frame image and the action video clip, and determine the image frame with the smallest image distance as the action start image frame. At the same time, the device can calculate the image distance between the silent frame image and the image frame sequence before the action video clip, and determine the image frame with the smallest image distance as the action end image frame.

[0123] In the specific implementation, refer to Figure 8 , Figure 8 This is the overall flow chart of the video processing method of this application. Figure 8 As shown, in the shooting stage, a silent video clip of the user can be first shot, and then a frame can be selected from it as the silent frame. Key points such as the head, shoulders, elbows, and hands are annotated based on the silent frame. Then, an action video clip can be shot using the key points as anchor points. In the trimming stage, the image distance between the image frame sequence of the first 1s of the action video clip and the silent frame can be calculated (the weighted sum of the human body key point distance, the inter-frame difference distance, and the optical flow distance). The image frame with the smallest image distance is the first frame. The first frame can then be used as the starting point of the action video clip, and the image frames before the first frame can be trimmed away. At the same time, the image distance between the image frame sequence of the last 1s of the action video clip and the silent frame can be calculated. The image frame with the smallest image distance is the last frame. The last frame can then be used as the terminal point of the action video clip, and the image frames after the last frame can be trimmed away, ultimately obtaining the trimmed action video clip.

[0124] In this embodiment, the method of obtaining a starting video segment and an ending video segment in an action video segment is disclosed; determining a first frame image from the starting video segment based on image distance, the first frame image being the frame image with the smallest image distance from the starting video segment to the silent frame image; determining a second frame image from the ending video segment based on image distance, the second frame image being the frame image with the smallest image distance from the silent frame image in the ending video segment; and determining the first frame image and the second frame image as the action start image frame and the action end image frame in the action video segment, respectively, so that the first frame and the last frame of the action video segment can be accurately identified and trimmed, thereby improving the reliability and accuracy of video trimming.

[0125] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0126] This application also provides a video processing device, please refer to Figure 9 , the video processing device includes:

[0127] An image marking module 10 is used to mark anchor points on silent frame images in the silent video clip to be processed, so as to obtain marked silent frame images;

[0128] A video shooting module 20 is used to shoot a video based on the human body key point information in the marked silent frame image to obtain an action video clip;

[0129] An image distance determination module 30 is configured to determine an image distance between the still frame image and a current frame image corresponding to the action video clip;

[0130] An image frame determining module 40 is configured to determine an action start image frame and an action end image frame in the action video clip based on the image distance;

[0131] The image trimming module 50 is configured to trim the action video segment based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0132] The video processing device provided in this application, using the video processing method in the above-mentioned embodiments, can solve the technical problem in the prior art that the capture and trimming of 2D digital human motion data is usually manually assisted, resulting in large errors, latency, and high operating costs. Compared with the prior art, the beneficial effects of the video processing device provided in this application are the same as those of the video processing method provided in the above-mentioned embodiments, and the other technical features of the video processing device are the same as those disclosed in the above-mentioned embodiments and are not further described here.

[0133] The present application provides a video processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video processing method in the above-mentioned embodiment 1.

[0134] Reference below Figure 10 , which shows a schematic diagram of the structure of a video processing device suitable for implementing the embodiments of the present application. The video processing device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The video processing device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0135] like Figure 10 As shown, the video processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the video processing device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. The communication device 1009 can allow the video processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a video processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0136] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The video processing device provided in this application utilizes the video processing method of the above-described embodiment to solve the technical problems of video processing. Compared with the prior art, the beneficial effects of the video processing device provided in this application are the same as those of the video processing method provided in the above-described embodiment. Other technical features of the video processing device are the same as those disclosed in the above-described embodiment and are not further described here.

[0138] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the video processing method in the above embodiment.

[0141] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination thereof.

[0142] The computer readable storage medium described above may be contained in a video processing device, or may exist separately without being assembled into the video processing device.

[0143] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the video processing device, the video processing device is caused to: capture a video based on the human key point information in the marked silent frame image, and obtain a motion video segment; determine an image distance between the silent frame image and a current frame image corresponding to the motion video segment; determine a motion start image frame and a motion end image frame in the motion video segment based on the image distance; and perform image pruning on the motion video segment based on the motion start image frame and the motion end image frame, and obtain a pruned motion video segment.

[0144] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0145] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0146] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0147] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the video processing method described above, and can solve the technical problems that the shooting and trimming of the motion data of the 2D digital person in the prior art are usually assisted by manual operation, resulting in large error, delay and high operation cost. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the video processing method provided by the above-mentioned embodiments, and will not be described here.

[0148] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video processing method, characterized in that: The method includes: Marking anchor points on the silent frame images in the silent video clip to be processed to obtain marked silent frame images; Shooting a video based on the human body key point information in the marked silent frame image to obtain an action video clip; determining an image distance between the still frame image and a current frame image corresponding to the action video clip; determining an action start image frame and an action end image frame in the action video clip based on the image distance; The action video segment is trimmed based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

2. The method according to claim 1, wherein The step of performing anchor point marking processing on the silent frame image in the silent video clip to be processed to obtain the marked silent frame image comprises: Determining key points of a human body in a silent frame image in a silent video clip to be processed; The human body key points are used as anchor points to perform anchor point marking processing on the silent frame image to obtain a marked silent frame image.

3. The method according to claim 1, wherein The step of determining the image distance between the still frame image and the current frame image corresponding to the action video clip includes: Reading a current frame image from the action video clip frame by frame; Performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image; Performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image; performing optical flow calculation on the silent frame image and the current frame image to obtain an optical flow distance between the silent frame image and the current frame image; An image distance between the silent frame image and the current frame image is determined based on the key point distance, the inter-frame difference distance, and the optical flow distance.

4. The method according to claim 3, wherein The step of calculating the key point distance between the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image includes: Performing human key point detection on the silent frame image and the current frame image using a preset human key point detection model to obtain silent frame human key points and current frame human key points respectively; Determine first position information corresponding to the human body key point of the silent frame, and second position information corresponding to the human body key point of the current frame; A key point distance calculation is performed on the silent frame image and the current frame image based on the first position information and the second position information to obtain a key point distance between the silent frame image and the current frame image.

5. The method according to claim 3, wherein The step of performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image includes: Performing image difference calculation on the silent frame image and the current frame image to obtain an image difference result; The image difference result is averaged to obtain an inter-frame difference distance between the silent frame image and the current frame image.

6. The method according to claim 3, wherein The step of performing optical flow calculation on the silent frame image and the current frame image to obtain the optical flow distance between the silent frame image and the current frame image includes: Performing grayscale processing on the silent frame image and the current frame image to obtain a silent grayscale frame image and a current grayscale frame image; Optical flow calculation is performed on the silent grayscale frame image and the current grayscale frame image to obtain an optical flow distance between the silent frame image and the current frame image.

7. The method according to any one of claims 1 to 6, characterized in that The step of determining the action start image frame and the action end image frame in the action video clip based on the image distance includes: Obtaining a starting video segment and an ending video segment in the action video segment; Determining a first frame image from the starting video segment based on the image distance, where the first frame image is a frame image in the starting video segment having the smallest image distance from the silent frame image; determining a second frame image from the ending video segment based on the image distance, where the second frame image is a frame image in the ending video segment having the smallest image distance from the silent frame image; The first frame image and the second frame image are respectively determined as an action start image frame and an action end image frame in the action video clip.

8. A video processing device, characterized in that: The device comprises: An image marking module is used to mark anchor points of silent frame images in the silent video clip to be processed to obtain marked silent frame images; A video shooting module is used to shoot a video based on the human body key point information in the marked silent frame image to obtain an action video clip; an image distance determination module, configured to determine an image distance between the still frame image and a current frame image corresponding to the action video clip; an image frame determining module, configured to determine an action start image frame and an action end image frame in the action video clip based on the image distance; The image trimming module is used to trim the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip.

9. A video processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images