Video processing method and device, equipment and storage medium

Abstract

The invention discloses a video processing method and device, equipment and a storage medium, and relates to the technical field of video processing, and the method comprises the steps: carrying out the marking anchor point processing of a silent frame image in a to-be-processed silent video clip, and obtaining a marked silent frame image; performing video shooting based on the human body key point information in the marked silent frame image to obtain an action video clip; determining an image distance between the silent frame image and a current frame image corresponding to the action video clip; determining an action starting image frame and an action ending image frame in the action video clip based on the image distance; and performing image trimming on the action video clip based on the action starting image frame and the action ending image frame to obtain a trimmed action video clip. By applying the technical scheme, the technical problems of large error, delay and high operation cost caused by the fact that shooting and trimming of the action data of the 2D digital human are usually assisted manually in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, apparatus, device and storage medium. Background Art

[0002] With the development of digital human technology, digital humans are increasingly used in video conferencing, live broadcasting, virtual character generation, animation video production and other scenarios. Currently, the types of digital humans on the market can be divided into 2D digital humans and 3D digital humans. Compared with 3D digital humans, 2D digital humans do not require modeling, motion capture and other operations, are low-cost, and are more widely used.

[0003] In practical applications, 2D digital humans usually need to stitch together video clips of different actions. When current 2D digital humans stitch together video clips, they have high requirements for the similarity of the first and last frames of each clip. In order to ensure the similarity of the first and last frames during the data acquisition phase, the common data shooting and trimming scheme is to use manual assistance such as human eye observation and screen point drawing. However, video shooting and trimming in this way usually have large errors, high latency and high operating costs. Summary of the invention

[0004] The main purpose of the present application is to provide a video processing method, device, equipment and storage medium, aiming to solve the technical problem that the shooting and trimming of the motion data of 2D digital humans in the prior art are usually done with manual assistance, resulting in large errors, delays and high operating costs.

[0005] To achieve the above object, the present application proposes a video processing method, which includes:

[0006] Marking anchor points of silent frame images in the silent video clip to be processed to obtain marked silent frame images;

[0007] Shooting a video based on the human body key point information in the marked silent frame image to obtain an action video clip;

[0008] Determining an image distance between the still frame image and a current frame image corresponding to the action video clip;

[0009] Determine an action start image frame and an action end image frame in the action video clip based on the image distance;

[0010] The action video segment is image trimmed based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0011] In one embodiment, the step of performing anchor point marking processing on the silent frame image in the silent video segment to be processed to obtain the marked silent frame image includes:

[0012] Determine the key points of the human body in the silent frame image in the silent video clip to be processed;

[0013] The human body key points are used as anchor points to perform anchor point marking processing on the silent frame image to obtain a marked silent frame image.

[0014] In one embodiment, the step of determining the image distance between the still frame image and the current frame image corresponding to the action video clip includes:

[0015] Reading the current frame image from the action video clip frame by frame;

[0016] Performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image;

[0017] Performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image;

[0018] Performing optical flow calculation on the silent frame image and the current frame image to obtain an optical flow distance between the silent frame image and the current frame image;

[0019] An image distance between the silent frame image and the current frame image is determined based on the key point distance, the inter-frame difference distance, and the optical flow distance.

[0020] In one embodiment, the step of calculating the key point distance between the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image includes:

[0021] Performing human key point detection on the silent frame image and the current frame image by using a preset human key point detection model to obtain silent frame human key points and current frame human key points respectively;

[0022] Determine the first position information corresponding to the human body key point of the silent frame, and the second position information corresponding to the human body key point of the current frame;

[0023] A key point distance calculation is performed on the silent frame image and the current frame image based on the first position information and the second position information to obtain a key point distance between the silent frame image and the current frame image.

[0024] In one embodiment, the step of performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image includes:

[0025] Performing image difference calculation on the silent frame image and the current frame image to obtain an image difference result;

[0026] The image difference result is averaged to obtain an inter-frame difference distance between the silent frame image and the current frame image.

[0027] In one embodiment, the step of performing optical flow calculation on the silent frame image and the current frame image to obtain the optical flow distance between the silent frame image and the current frame image includes:

[0028] Performing grayscale processing on the silent frame image and the current frame image to obtain a silent grayscale frame image and a current grayscale frame image;

[0029] Optical flow calculation is performed on the silent grayscale frame image and the current grayscale frame image to obtain an optical flow distance between the silent frame image and the current frame image.

[0030] In one embodiment, the step of determining the action start image frame and the action end image frame in the action video clip based on the image distance includes:

[0031] Obtaining a starting video segment and an ending video segment in the action video segment;

[0032] Determine a first frame image from the starting video segment based on the image distance, where the first frame image is a frame image in the starting video segment with the smallest image distance from the silent frame image;

[0033] Determine a second frame image from the ending video segment based on the image distance, where the second frame image is a frame image in the ending video segment with the smallest image distance from the silent frame image;

[0034] The first frame image and the second frame image are respectively determined as an action start image frame and an action end image frame in the action video clip.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a video processing device, the device comprising:

[0036] An image marking module is used to mark anchor points of silent frame images in the silent video clip to be processed, so as to obtain marked silent frame images;

[0037] A video shooting module, used for shooting a video based on the human body key point information in the marked silent frame image to obtain an action video clip;

[0038] An image distance determination module, used to determine the image distance between the still frame image and the current frame image corresponding to the action video clip;

[0039] An image frame determination module, configured to determine an action start image frame and an action end image frame in the action video clip based on the image distance;

[0040] The image trimming module is used to perform image trimming on the action video segment based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0041] In addition, to achieve the above objectives, the present application also proposes a video processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method described above are implemented.

[0043] The present application provides a video processing method, which discloses the following steps: performing video shooting based on human body key point information in a marked silent frame image to obtain an action video clip; determining an image distance between the silent frame image and a current frame image corresponding to the action video clip; determining an action start image frame and an action end image frame in the action video clip based on the image distance; performing image trimming on the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip; compared with the prior art in which the shooting and trimming of 2D digital human action data are usually performed through manual assistance such as human eye observation and screen point tracing, which have the problems of large manual errors, high latency, and high operating costs, the present invention performs video shooting based on human body key point information in a marked silent frame image after anchor point marking to obtain an action video clip, and performs image trimming based on the action start image frame and the action end image frame in the action video clip to obtain a trimmed action video clip, thereby solving the technical problems in the prior art in which the shooting and trimming of 2D digital human action data are usually performed through manual assistance, which lead to large errors, high latency, and high operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the video processing method of the present application;

[0047] Figure 2 It is a silent frame image in the video processing method of this application;

[0048] Figure 3 Marking anchor images for silent frames in the video processing method of this application;

[0049] Figure 4 A flowchart of the second embodiment of the video processing method of the present application is provided;

[0050] Figure 5 This is a schematic diagram of the difference result in the video processing method of this application;

[0051] Figure 6 This is a schematic diagram of the optical flow results in the video processing method of this application;

[0052] Figure 7 A flowchart of the third embodiment of the video processing method of the present application is provided;

[0053] Figure 8 This is the overall flow chart of the video processing method of this application;

[0054] Fig. 9 This is a schematic diagram of the module structure of the video processing device according to an embodiment of the present application;

[0055] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the video processing method in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of the embodiment of the present application is: based on the human body key point information in the marked silent frame image, video shooting is performed to obtain an action video clip; the image distance between the silent frame image and the current frame image corresponding to the action video clip is determined; based on the image distance, the action start image frame and the action end image frame in the action video clip are determined; based on the action start image frame and the action end image frame, image trimming is performed on the action video clip to obtain a trimmed action video clip.

[0060] In the prior art, when 2D digital humans stitch video clips, they have high requirements for the similarity of the first and last frames of each clip. In order to ensure the similarity of the first and last frames during the data collection stage, the common data shooting and trimming scheme is to use manual auxiliary means such as human eye observation and screen point drawing, but video shooting and trimming in this way usually have large errors, high latency, and high operating costs.

[0061] The present application provides a solution, which can perform video shooting based on the human body key point information in the marked silent frame image after anchor point marking to obtain action video clips, and perform image trimming based on the action start image frame and the action end image frame in the action video clip to obtain the trimmed action video clip, thereby solving the technical problem in the prior art that the shooting and trimming of 2D digital human action data are usually performed with manual assistance, resulting in large errors, delays and high operating costs.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a video processing device, etc. The following takes a video processing device as an example (hereinafter referred to as a device) to illustrate this embodiment and the following embodiments.

[0063] Based on this, the present application embodiment provides a video processing method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the video processing method of the present application.

[0064] In this embodiment, the video processing method includes steps S10 to S50:

[0065] Step S10: marking anchor points of the silent frame images in the silent video clip to be processed to obtain marked silent frame images.

[0066] It should be noted that the above-mentioned silent video clip to be processed may be a silent video clip to be processed. In this embodiment, any video shooting device may be used to shoot a silent video clip, and the silent state needs to be maintained for more than 30 seconds, so as to obtain the silent video clip to be processed.

[0067] It should be understood that the above-mentioned silent frame image may be an image frame in the silent video clip to be processed. In this embodiment, an image frame with a neutral posture, regular facial features, and a calm and gentle expression may be selected from the silent video clip to be processed as the silent frame image F s , refer to Figure 2 , Figure 2 is a silent frame image in the video processing method of this application. Figure 2 The model in the picture changes from other poses to the silent frame pose, which is called homing.

[0068] It can be understood that the marked silent frame image may be a silent frame image marked with an anchor point.

[0069] Specifically, the step S10 includes: determining the key points of the human body in the silent frame image in the silent video segment to be processed; and marking the silent frame image with the key points of the human body as anchor points to obtain a marked silent frame image.

[0070] It should be noted that the above-mentioned key points of the human body are key points in the human body, such as the upper and lower edges of the head, shoulders, elbows, hands and other key points. Figure 3 As shown, Figure 3 Anchor images are marked for silent frames in the video processing method of this application. Figure 3 The blue dots in the figure are anchor points, which are used to guide the user to return to the position according to the anchor points during the shooting stage. In this embodiment, key points such as the upper and lower edges, left and right edges, shoulders, elbows, and hands of the human head can be used as anchor points, and the silent frame image is marked with anchor points based on these anchor points to finally obtain the silent frame image with marked anchor points.

[0071] Step S20: Shooting a video based on the human body key point information in the marked still frame image to obtain an action video clip.

[0072] It can be understood that the above-mentioned human body key point information may be the position information of the human body key points.

[0073] It should be understood that the above-mentioned action video clips can be video clips of users performing actions. In this embodiment, the device can visualize the anchor points in the marked silent frame image, and guide the user of the current screen to align with the anchor points. After alignment, the user can start the action, and the device starts shooting the video. After the user's action is completed, the action can be ended after the user of the current screen aligns with the anchor points, and finally the action video clip is obtained.

[0074] Step S30: determining an image distance between the still frame image and a current frame image corresponding to the action video clip.

[0075] It can be understood that the current frame image mentioned above may be a frame image currently read from the action video clip.

[0076] It should be noted that the above image distance may be a weighted distance of the human key point distance, the inter-frame difference distance, and the optical flow distance between the silent frame image and the current frame image. Among them, the human key point distance may be the distance between the key points in the silent frame image and the current frame image; the inter-frame difference distance may be a feature used to describe the difference in pixel values ​​between the silent frame image and the current frame image; and the optical flow distance may be the distance moved between the pixels in the silent frame image and the current frame image.

[0077] Step S40: determining an action start image frame and an action end image frame in the action video clip based on the image distance.

[0078] It should be understood that the above-mentioned action start image frame may be an image frame when the user starts to perform an action in the action video clip; and the above-mentioned action end image frame may be an image frame when the user completes the action in the action video clip. In this embodiment, the action start image frame (hereinafter referred to as the first frame) closest to the silent frame may be controlled within 1 second before the action video clip, and the action end image frame (hereinafter referred to as the last frame) closest to the silent frame may be controlled within the last second of the action video clip.

[0079] Step S540: performing image trimming on the action video segment based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0080] In actual applications, the device can use the first frame as the starting point of the action video clip, trim and remove the image frames before the first frame, and use the last frame as the end point of the action video clip, trim and remove the image frames after the last frame, and finally obtain the trimmed action video clip.

[0081] The present embodiment provides a video processing method, which discloses performing video shooting based on human body key point information in a marked silent frame image to obtain an action video clip; determining an image distance between the silent frame image and a current frame image corresponding to the action video clip; determining an action start image frame and an action end image frame in the action video clip based on the image distance; performing image trimming on the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip; compared with the prior art in which the shooting and trimming of 2D digital human action data are usually performed through manual assistance such as human eye observation and screen point tracing, which have the problems of large manual errors, high latency, and high operating costs, the present embodiment performs video shooting based on human body key point information in the marked silent frame image after anchor point marking to obtain an action video clip, and performs image trimming based on the action start image frame and the action end image frame in the action video clip to obtain a trimmed action video clip, thereby solving the technical problems in the prior art in which the shooting and trimming of 2D digital human action data are usually performed through manual assistance, which lead to large errors, high latency, and high operating costs.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , Figure 4 A flowchart diagram of the second embodiment of the video processing method of the present application is provided.

[0083] In this embodiment, step S30 includes steps S301 to S305:

[0084] Step S301: reading the current frame image from the action video clip frame by frame.

[0085] It can be understood that, in this embodiment, the device can read image frames from the action video clip frame by frame, wherein the image frame currently read by the device is the current frame image.

[0086] Step S302: performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image.

[0087] Specifically, the step S302 includes: performing human key point detection on the silent frame image and the current frame image through a preset human key point detection model to obtain the silent frame human key points and the current frame human key points, respectively; determining the first position information corresponding to the silent frame human key points, and the second position information corresponding to the current frame human key points; performing key point distance calculation on the silent frame image and the current frame image based on the first position information and the second position information to obtain the key point distance between the silent frame image and the current frame image.

[0088] It should be understood that the above-mentioned silent frame human body key points can be human body key points in the silent frame image; the above-mentioned current frame human body key points can be human body key points in the current frame image. Correspondingly, the above-mentioned first position information is the position information of the silent frame human body key points; the above-mentioned second position information is the position information of the current frame human body key points.

[0089] In practical applications, the silent frame image F s With the current frame image F a After preprocessing, the image can be scaled to the size required by the human key point detection network for human key point detection. The human key point detection model used can be MediaPipe. s With the current frame image F a After detecting the key points of the human body, the position information of the key points in the image can be obtained. For example, the key points of the human body in the silent frame may include: s 0,P s 1, ..., P s 22, the corresponding position information can be: [x s 0,y s 0],[x s 1, y s 1], ..., [x s 22, y s 22]; the human body key points of the current frame may include: P a 0,P a 1, ..., P a 22, the corresponding position information can be: [x a 0,y a 0],[x a 1, y a 1], ..., [x a 22, y a 22]. Then, the key point distance between the silent image frame and the current image frame can be calculated based on the first position information corresponding to the human key point of the silent frame and the second position information corresponding to the human key point of the current frame. The calculation formula can be:

[0090]

[0091] Among them, (P a iP s i) can be further calculated as:

[0092]

[0093] Where D keypoints is the key point distance, and i represents the key point of the human body.

[0094] Step S303: performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image.

[0095] Specifically, the step S303 includes: performing image difference calculation on the silent frame image and the current frame image to obtain an image difference result; performing averaging processing on the image difference result to obtain an inter-frame difference distance between the silent frame image and the current frame image.

[0096] In this embodiment, the device may first perform a silent frame image F s With the current frame image F a Perform grayscale preprocessing and then calculate the silent frame image F s With the current frame image F a The image difference between them is obtained to obtain the image difference result, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the difference result in the video processing method of this application. The closer the pixel is to black, the smaller the inter-frame difference at that position is; the closer the pixel is to white, the larger the inter-frame difference at that position is. Finally, the image difference result can be averaged to obtain the inter-frame difference distance of the current image frame. The calculation formula is as follows:

[0097]

[0098] Among them, Diff sa The calculation method is:

[0099] Diff sa =|F S -F a |;

[0100] Where D frame is the frame difference distance, Diff sa is the silent frame image F s With the current frame image F a The image difference result between them, w represents the width of the image, and h represents the height of the image.

[0101] Step S304: performing optical flow calculation on the silent frame image and the current frame image to obtain an optical flow distance between the silent frame image and the current frame image.

[0102] Specifically, the step S304 includes: performing grayscale processing on the silent frame image and the current frame image to obtain a silent grayscale frame image and a current grayscale frame image; performing optical flow calculation on the silent grayscale frame image and the current grayscale frame image to obtain an optical flow distance between the silent frame image and the current frame image.

[0103] It can be understood that the above-mentioned silent grayscale frame image can be a grayscale image obtained by performing grayscale processing on the silent frame image; and the above-mentioned current grayscale frame image can be a grayscale image obtained by performing grayscale processing on the current frame image.

[0104] In practical applications, the device can first perform a silent frame image F s With the current frame image F a Grayscale preprocessing is performed to obtain the grayscale silent grayscale frame image and the current grayscale frame image, and then the optical flow value between the silent grayscale frame image and the current grayscale frame image can be calculated to obtain the optical flow distance of the current frame image. In this embodiment, the optical flow algorithm used to calculate the optical flow value can be the Farneback algorithm. The calculation result is as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of the optical flow results in the video processing method of this application. Figure 6 The green arrows in the figure represent the movement distance and direction of the pixels in the image.

[0105] Step S305: determining the image distance between the still frame image and the current frame image based on the key point distance, the inter-frame difference distance and the optical flow distance.

[0106] In practical applications, after calculating the key point distance, frame difference distance and optical flow distance between the silent frame image and the current frame image, the key point distance, frame difference distance and optical flow distance can be weighted summed to finally obtain the image distance between the silent frame image and the current frame image. The calculation formula of the image distance in this embodiment can be:

[0107] D image =αD keypoints +βD frame +γD flow ;

[0108] Where D image is the image distance, D keypoints is the key point distance, D frame is the frame difference distance, D flow is the optical flow distance, α, β and γ are weight coefficients.

[0109] in,

[0110] α+β+γ=1;

[0111] 0<α, β, γ<1.

[0112] In the present embodiment, the present invention discloses reading the current frame image from the action video clip frame by frame; performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image; performing image difference calculation on the silent frame image and the current frame image to obtain the frame difference distance between the silent frame image and the current frame image; performing optical flow calculation on the silent frame image and the current frame image to obtain the optical flow distance between the silent frame image and the current frame image; determining the image distance between the silent frame image and the current frame image based on the key point distance, the frame difference distance and the optical flow distance, thereby realizing multi-dimensional distance calculation between the silent frame image and the current frame image, enhancing the reliability of the image distance calculation, and thus improving the accuracy of subsequent video clip trimming.

[0113] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 7 , Figure 7 A flowchart of the third embodiment of the video processing method of the present application is provided.

[0114] In this embodiment, step S40 includes steps S401 to S404:

[0115] Step S401: obtaining a starting video segment and an ending video segment in the action video segment.

[0116] It should be noted that the above-mentioned starting video clip can be a video clip within a period of time from the beginning of the action video clip, for example, a video clip within 1 second before the action video clip; the above-mentioned ending video clip can be a video clip within a period of time from the end of the action video clip, for example, a video clip within the last 1 second of the action video clip.

[0117] Step S402: determining a first frame image from the starting video segment based on the image distance, where the first frame image is a frame image in the starting video segment with the smallest image distance from the silent frame image.

[0118] It can be understood that the first frame image may be a frame image having the smallest image distance with the silent frame image among all the image frames of the starting video segment.

[0119] Step S403: determining a second frame image from the end video segment based on the image distance, where the second frame image is a frame image in the end video segment with the smallest image distance from the silent frame image.

[0120] It can be understood that the second frame image may be a frame image having the smallest image distance with the silent frame image among all the image frames of the ending video segment.

[0121] Step S404: determining the first frame image and the second frame image as the action start image frame and the action end image frame in the action video clip respectively.

[0122] In actual applications, the device can calculate the image distance between the silent frame image and the image frame sequence before the action video clip based on the silent frame image and the action video clip, and determine the image frame with the smallest image distance as the action start image frame. At the same time, the device can calculate the image distance between the silent frame image and the image frame sequence before the action video clip, and determine the image frame with the smallest image distance as the action end image frame.

[0123] In the specific implementation, refer to Figure 8 , Figure 8 This is the overall flow chart of the video processing method of this application. Figure 8 As shown, in the shooting stage, a silent video clip of the user can be shot first, and then a frame can be selected from it as a silent frame, and key points such as the head, shoulders, elbows, and hands can be annotated according to the silent frame, and then the action video clip can be shot with the key points as anchor points. In the trimming stage, the image distance between the image frame sequence of the first 1s of the action video clip and the silent frame can be calculated (the weighted sum of the human body key point distance, the inter-frame difference distance, and the optical flow distance), where the image frame with the smallest image distance is the first frame, and then the first frame can be used as the starting point of the action video clip to trim and remove the image frames before the first frame. At the same time, the image distance between the image frame sequence of the last 1s of the action video clip and the silent frame can be calculated, where the image frame with the smallest image distance is the tail frame, and then the tail frame can be used as the terminal point of the action video clip to trim and remove the image frames after the tail frame, and finally obtain the trimmed action video clip.

[0124] In the present embodiment, the method of obtaining a starting video segment and an ending video segment in an action video segment is disclosed; determining a first frame image from the starting video segment based on image distance, the first frame image being a frame image with the smallest image distance to the silent frame image in the starting video segment; determining a second frame image from the ending video segment based on image distance, the second frame image being a frame image with the smallest image distance to the silent frame image in the ending video segment; and determining the first frame image and the second frame image as an action start image frame and an action end image frame in the action video segment, respectively, so that the first frame and the last frame of the action video segment can be accurately identified and trimmed, thereby improving the reliability and accuracy of video trimming.

[0125] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0126] This application also provides a video processing device, please refer to Fig. 9 , the video processing device comprises:

[0127] An image marking module 10 is used to mark anchor points on the silent frame images in the silent video clip to be processed, so as to obtain marked silent frame images;

[0128] A video shooting module 20, configured to shoot a video based on the human body key point information in the marked silent frame image to obtain an action video clip;

[0129] An image distance determination module 30, configured to determine an image distance between the still frame image and a current frame image corresponding to the action video clip;

[0130] An image frame determination module 40, configured to determine an action start image frame and an action end image frame in the action video clip based on the image distance;

[0131] The image trimming module 50 is used to perform image trimming on the action video segment based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

[0132] The video processing device provided by the present application adopts the video processing method in the above embodiment, which can solve the technical problem that the shooting and trimming of the action data of 2D digital human in the prior art is usually assisted by manual labor, resulting in large errors, delays and high operating costs. Compared with the prior art, the beneficial effects of the video processing device provided by the present application are the same as the beneficial effects of the video processing method provided by the above embodiment, and the other technical features in the video processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0133] The present application provides a video processing device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video processing method in the above-mentioned embodiment 1.

[0134] Reference below Fig.10 , which shows a schematic diagram of the structure of a video processing device suitable for implementing the embodiment of the present application. The video processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The video processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Fig.10 As shown, the video processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the video processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video processing device to communicate with other devices wirelessly or by wire to exchange data. Although the video processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The video processing device provided by the present application adopts the video processing method in the above embodiment to solve the technical problem of video processing. Compared with the prior art, the beneficial effects of the video processing device provided by the present application are the same as the beneficial effects of the video processing method provided by the above embodiment, and other technical features in the video processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the video processing method in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in the video processing device; or may exist independently without being assembled into the video processing device.

[0143] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the video processing device, the video processing device: performs video shooting based on the human body key point information in the marked silent frame image to obtain an action video clip; determines the image distance between the silent frame image and the current frame image corresponding to the action video clip; determines the action start image frame and the action end image frame in the action video clip based on the image distance; performs image trimming on the action video clip based on the action start image frame and the action end image frame to obtain a trimmed action video clip.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video processing method, and can solve the technical problem that the shooting and trimming of the action data of 2D digital humans in the prior art are usually manually assisted, resulting in large errors, delays, and high operating costs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the video processing method provided by the above-mentioned embodiment, which will not be repeated here.

[0148] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video processing method, characterized in that: The method includes: Marking anchor points of silent frame images in the silent video clip to be processed to obtain marked silent frame images; Based on the human body key point information in the marked silent frame image, video is shot to obtain an action video clip; Determining an image distance between the still frame image and a current frame image corresponding to the action video clip; Determine an action start image frame and an action end image frame in the action video clip based on the image distance; The action video segment is image trimmed based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

2. The method according to claim 1, characterized in that The step of performing anchor point marking processing on the silent frame image in the silent video segment to be processed to obtain the marked silent frame image comprises: Determine the key points of the human body in the silent frame image in the silent video clip to be processed; The human body key points are used as anchor points to perform anchor point marking processing on the silent frame image to obtain a marked silent frame image.

3. The method according to claim 1, characterized in that The step of determining the image distance between the still frame image and the current frame image corresponding to the action video clip comprises: Reading the current frame image from the action video clip frame by frame; Performing key point distance calculation on the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image; Performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image; Performing optical flow calculation on the silent frame image and the current frame image to obtain an optical flow distance between the silent frame image and the current frame image; An image distance between the silent frame image and the current frame image is determined based on the key point distance, the inter-frame difference distance, and the optical flow distance.

4. The method according to claim 3, characterized in that The step of calculating the key point distance between the silent frame image and the current frame image to obtain the key point distance between the silent frame image and the current frame image comprises: Performing human key point detection on the silent frame image and the current frame image by using a preset human key point detection model to obtain silent frame human key points and current frame human key points respectively; Determine the first position information corresponding to the human body key point of the silent frame, and the second position information corresponding to the human body key point of the current frame; A key point distance calculation is performed on the silent frame image and the current frame image based on the first position information and the second position information to obtain a key point distance between the silent frame image and the current frame image.

5. The method according to claim 3, characterized in that The step of performing image difference calculation on the silent frame image and the current frame image to obtain an inter-frame difference distance between the silent frame image and the current frame image comprises: Performing image difference calculation on the silent frame image and the current frame image to obtain an image difference result; The image difference result is averaged to obtain an inter-frame difference distance between the silent frame image and the current frame image.

6. The method according to claim 3, characterized in that The step of performing optical flow calculation on the silent frame image and the current frame image to obtain the optical flow distance between the silent frame image and the current frame image comprises: Performing grayscale processing on the silent frame image and the current frame image to obtain a silent grayscale frame image and a current grayscale frame image; Optical flow calculation is performed on the silent grayscale frame image and the current grayscale frame image to obtain an optical flow distance between the silent frame image and the current frame image.

7. The method according to any one of claims 1 to 6, characterized in that The step of determining the action start image frame and the action end image frame in the action video clip based on the image distance comprises: Obtaining a starting video segment and an ending video segment in the action video segment; Determine a first frame image from the starting video segment based on the image distance, where the first frame image is a frame image in the starting video segment with the smallest image distance from the silent frame image; Determine a second frame image from the ending video segment based on the image distance, where the second frame image is a frame image in the ending video segment with the smallest image distance from the silent frame image; The first frame image and the second frame image are respectively determined as an action start image frame and an action end image frame in the action video clip.

8. A video processing device, characterized in that: The device comprises: An image marking module is used to mark anchor points of silent frame images in the silent video clip to be processed, so as to obtain marked silent frame images; A video shooting module, used for shooting a video based on the human body key point information in the marked silent frame image to obtain an action video clip; An image distance determination module, used to determine the image distance between the still frame image and the current frame image corresponding to the action video clip; An image frame determination module, configured to determine an action start image frame and an action end image frame in the action video clip based on the image distance; The image trimming module is used to perform image trimming on the action video segment based on the action start image frame and the action end image frame to obtain a trimmed action video segment.

9. A video processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images