Video processing method and device

Abstract

The invention discloses a video processing method, and the method comprises the steps: obtaining a first video which comprises a first video frame and a second video frame, the first video frame and the second video frame are two adjacent video frames in the first video, and each of the first video frame and the second video frame comprises a plurality of objects; and for each object in the plurality of objects, according to the positions of the object in the first video frame and the second video frame, determining a motion parameter corresponding to the object so as to obtain a motion parameter corresponding to the first video. After the motion parameter corresponding to the first video is determined, whether the first video meets the requirement of the target model for the training sample can be determined based on the motion parameter corresponding to the first video. Therefore, by utilizing the scheme provided by the invention, whether the first video meets the requirement of the target model on the training sample can be determined, so that the quality of the training sample participating in target model training can be ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a video processing method and device. Background Art

[0002] Model training requires a large number of training samples. For example, the training of video generation models requires a large number of high-quality videos as training samples. For example, in order to enable the trained video generation model to generate videos of different scenes, styles, and contents, the training samples for training the video generation model should be as diverse as possible, for example, covering multiple types, scenes, and emotions.

[0003] How to ensure the quality of training samples used to train video generation models is a problem that has yet to be solved. Summary of the invention

[0004] In order to solve or at least partially solve the above technical problems, the present application provides a video processing method and device.

[0005] In a first aspect, the present application provides a video processing method, the method comprising:

[0006] Acquire a first video, where the first video includes a first video frame and a second video frame, where the first video frame and the second video frame are two adjacent video frames in the first video, and both the first video frame and the second video frame include a plurality of objects;

[0007] For each of the multiple objects, determine a motion parameter corresponding to the object according to positions of the object in the first video frame and the second video frame, so as to obtain a motion parameter corresponding to the first video;

[0008] According to the motion parameters corresponding to the first video, it is determined whether the first video meets the requirements of the target model for training samples.

[0009] Optionally, determining the motion parameter corresponding to the object according to the positions of the object in the first video frame and the second video frame respectively, so as to obtain the motion parameter corresponding to the first video, includes:

[0010] Determine a sparse optical flow vector corresponding to the object according to positions of the object in the first video frame and the second video frame respectively;

[0011] Determine motion parameters corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects.

[0012] Optionally, the motion parameter corresponding to the first video includes a dynamic score and / or a target parameter corresponding to the first video, and determining the motion parameter corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects includes:

[0013] determining a dynamic score of the first video according to the sum of the lengths of the sparse optical flow vectors corresponding to the multiple objects, the size of the first video, and the number of video frames included in the first video; and / or,

[0014] The target parameters are determined according to the sparse optical flow vectors respectively corresponding to the multiple objects, and the target parameters include at least one of the following: the movement speed of the first video, the stability of the movement speed of the first video, the movement direction of the first video, and the stability of the movement direction of the first video.

[0015] Optionally, the sparse optical flow vectors corresponding to the multiple objects are sorted in descending order of length, and determining the target parameter according to the sparse optical flow vectors corresponding to the multiple objects includes:

[0016] The target parameter is determined according to a certain number of sparse optical flow vectors corresponding to the multiple objects, which are ranked lower in length.

[0017] Optionally, the target model is a video generation model, and determining, according to the motion parameters corresponding to the first video, whether the first video meets the requirements of the target model for training samples includes:

[0018] If the dynamic score of the first video is less than or equal to a preset score, or the motion speed of the first video is less than or equal to a preset speed, or the stability of the motion direction of the first video is less than or equal to a preset stability threshold, it is determined that the first video does not meet the requirements of the video generation model for training samples; or

[0019] If the dynamic score of the first video is greater than a preset score, the movement speed of the first video is greater than a preset speed, and the stability of the movement direction of the first video is greater than a preset stability threshold, it is determined that the first video meets the requirements of the video generation model for training samples.

[0020] Optionally, if the first video meets the requirements of the video generation model for training samples, the method further includes: training the video generation model using the first video and motion parameters corresponding to the first video.

[0021] Optionally, the method further includes:

[0022] Determining a subtitle area of ​​a second video;

[0023] The subtitle area of ​​the second video is cropped to obtain a third video.

[0024] Optionally, determining the subtitle area of ​​the second video includes:

[0025] Determine a subtitle area corresponding to each video frame in a plurality of video frames included in the second video;

[0026] The subtitle area of ​​the second video is determined according to the subtitle areas respectively corresponding to the multiple video frames.

[0027] Optionally, the multiple video frames include a third video frame, and the subtitle area corresponding to the third video frame is determined in the following manner:

[0028] Identify text in the third video frame, and if the text is located in a non-central area of ​​the third video frame, or the size of the text is less than or equal to a preset size, or the two pairs of parallel lines included in the rectangular text box corresponding to the text are respectively parallel to two vertical edge lines of the third video frame, determine that the area corresponding to the text belongs to the subtitle area, wherein a first pair of parallel lines of the two pairs of parallel lines are parallel to a first edge line of the third video frame, a second pair of parallel lines of the two pairs of parallel lines are parallel to a second edge line of the third video frame, and the first edge line is perpendicular to the second edge line.

[0029] Optionally, determining the subtitle area of ​​the second video according to the subtitle areas respectively corresponding to the multiple video frames includes:

[0030] For any pixel in the second video, if the number of video frames in which the pixel belongs to the subtitle area among the multiple video frames is greater than a preset number, it is determined that the pixel belongs to the subtitle area of ​​the second video.

[0031] Optionally, the step of cropping the subtitle area of ​​the second video to obtain a third video includes:

[0032] According to the principle of maximizing the area of ​​the reserved area or according to the principle of optimizing the aspect ratio of the reserved area, the subtitle area of ​​the second video is cropped to obtain the third video.

[0033] Optionally, the first video and the second video are the same video, and determining the subtitle area of ​​the second video includes:

[0034] In the case that the first video meets the requirements of the target model for training samples, the subtitle area of ​​the first video is determined, and the third video is used to train the target model.

[0035] Optionally, the multiple objects include: multiple image corner points.

[0036] In a second aspect, the present application provides a video processing device, the device comprising:

[0037] An acquisition unit, configured to acquire a first video, wherein the first video includes a first video frame and a second video frame, wherein the first video frame and the second video frame are two adjacent video frames in the first video, and each of the first video frame and the second video frame includes a plurality of objects;

[0038] a first determining unit, configured to determine, for each of the plurality of objects, a motion parameter corresponding to the object according to positions of the object in the first video frame and the second video frame, respectively, so as to obtain the motion parameter corresponding to the first video;

[0039] The second determination unit is used to determine whether the first video meets the requirements of the target model for training samples according to the motion parameters corresponding to the first video.

[0040] Optionally, the first determining unit is configured to:

[0041] Determine a sparse optical flow vector corresponding to the object according to positions of the object in the first video frame and the second video frame respectively;

[0042] Determine motion parameters corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects.

[0043] Optionally, the motion parameter corresponding to the first video includes a dynamic score and / or a target parameter corresponding to the first video, and determining the motion parameter corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects includes:

[0044] determining a dynamic score of the first video according to the sum of the lengths of the sparse optical flow vectors corresponding to the multiple objects, the size of the first video, and the number of video frames included in the first video; and / or,

[0045] The target parameters are determined according to the sparse optical flow vectors respectively corresponding to the multiple objects, and the target parameters include at least one of the following: the movement speed of the first video, the stability of the movement speed of the first video, the movement direction of the first video, and the stability of the movement direction of the first video.

[0046] Optionally, the sparse optical flow vectors corresponding to the multiple objects are sorted in descending order of length, and determining the target parameter according to the sparse optical flow vectors corresponding to the multiple objects includes:

[0047] The target parameter is determined according to a certain number of sparse optical flow vectors corresponding to the multiple objects, which are ranked lower in length.

[0048] Optionally, the target model is a video generation model, and the second determining unit is used to:

[0049] If the dynamic score of the first video is less than or equal to a preset score, or the motion speed of the first video is less than or equal to a preset speed, or the stability of the motion direction of the first video is less than or equal to a preset stability threshold, it is determined that the first video does not meet the requirements of the video generation model for training samples; or

[0050] If the dynamic score of the first video is greater than a preset score, the movement speed of the first video is greater than a preset speed, and the stability of the movement direction of the first video is greater than a preset stability threshold, it is determined that the first video meets the requirements of the video generation model for training samples.

[0051] Optionally, if the first video meets the requirements of the video generation model for training samples, the device further includes:

[0052] A training unit is used to train a video generation model using the first video and motion parameters corresponding to the first video.

[0053] Optionally, the device further comprises:

[0054] A third determining unit, configured to determine a subtitle area of ​​a second video;

[0055] The cropping unit is used to crop the subtitle area of ​​the second video to obtain a third video.

[0056] Optionally, the third determining unit is used to:

[0057] Determine a subtitle area corresponding to each video frame in a plurality of video frames included in the second video;

[0058] The subtitle area of ​​the second video is determined according to the subtitle areas respectively corresponding to the multiple video frames.

[0059] Optionally, the multiple video frames include a third video frame, and the subtitle area corresponding to the third video frame is determined in the following manner:

[0060] Identify text in the third video frame, and if the text is located in a non-central area of ​​the third video frame, or the size of the text is less than or equal to a preset size, or the two pairs of parallel lines included in the rectangular text box corresponding to the text are respectively parallel to two vertical edge lines of the third video frame, determine that the area corresponding to the text belongs to the subtitle area, wherein a first pair of parallel lines of the two pairs of parallel lines are parallel to a first edge line of the third video frame, a second pair of parallel lines of the two pairs of parallel lines are parallel to a second edge line of the third video frame, and the first edge line is perpendicular to the second edge line.

[0061] Optionally, determining the subtitle area of ​​the second video according to the subtitle areas respectively corresponding to the multiple video frames includes:

[0062] For any pixel in the second video, if the number of video frames in which the pixel belongs to the subtitle area among the multiple video frames is greater than a preset number, it is determined that the pixel belongs to the subtitle area of ​​the second video.

[0063] Optionally, the cutting unit is used to:

[0064] According to the principle of maximizing the area of ​​the reserved area or according to the principle of optimizing the aspect ratio of the reserved area, the subtitle area of ​​the second video is cropped to obtain the third video.

[0065] Optionally, the first video and the second video are the same video, and the third determining unit is configured to:

[0066] In the case that the first video meets the requirements of the target model for training samples, the subtitle area of ​​the first video is determined, and the third video is used to train the target model.

[0067] Optionally, the multiple objects include: multiple image corner points.

[0068] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory;

[0069] The processor is used to execute instructions stored in the memory so that the electronic device performs the method as described in any one of the first aspects above.

[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute a method as described in any one of the above first aspects.

[0071] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.

[0072] Compared with the prior art, the embodiments of the present application have the following advantages:

[0073] The present application provides a video processing method, which includes: obtaining a first video, wherein the first video includes a first video frame and a second video frame, wherein the first video frame and the second video frame are two adjacent video frames in the first video, and each of the first video frame and the second video frame includes a plurality of objects. For each of the plurality of objects, the motion parameters corresponding to the object can be determined according to the positions of the object in the first video frame and the second video frame, respectively, so as to obtain the motion parameters corresponding to the first video. Since the motion parameters corresponding to the first video can indicate the motion of the first video, and the motion corresponding to the first video is an important factor in determining whether the first video meets the requirements of the target model for training samples. Therefore, after determining the motion parameters corresponding to the first video, it can be determined whether the first video meets the requirements of the target model for training samples based on the motion parameters corresponding to the first video. It can be seen that, using this solution, it can be determined whether the first video meets the requirements of the target model for training samples to ensure the quality of the training samples participating in the training of the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0076] Figure 2 A flowchart of another video processing method provided in an embodiment of the present application;

[0077] Figure 3a to Figure 3c A schematic diagram showing cropping of the second video is shown;

[0078] Figure 4 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] Currently, a variety of processing measures can be used to process videos to provide high-quality training samples for video generation models. The following introduces two ways to process videos.

[0081] Solution 1: If the camera used to shoot the video shakes a lot when shooting the video, it will affect the stability of the video. Correspondingly, if the video is used as a training sample to train the video generation model, it will affect the video generation effect of the video generation model. In view of this, in general, when selecting training samples for the video generation model, video content shot by professional shooting equipment is usually selected as the training sample.

[0082] Processing method 2: Considering that if the video used as a training sample contains text, the video generated by the trained video generation model may contain strange text when generating a video. Therefore, in one example, before using a video as a training sample, it can be first detected whether the video contains text. If the video contains text, the video is discarded, that is, the video is no longer used as a training sample for training the video generation model.

[0083] However, for the above processing method 1: using videos shot by professional shooting equipment as training samples, then:

[0084] On the one hand, the scale of training samples is difficult to meet the requirements of the video generation model for the scale of training samples, because the number of videos shot by professional equipment is limited. On the other hand, if the requirements of the video generation model for the scale of training samples are to be met, it is necessary to use additional professional shooting equipment to shoot videos, resulting in a high cost for obtaining training samples.

[0085] The inventors of this application have discovered that, for existing video materials, although they may not be shot with professional filming equipment, their jitter (such as lens jitter) may be relatively small. If such videos can be used as training samples for video generation models, more training samples can be obtained for video generation data while reducing costs.

[0086] The above processing method 2 directly discards the video containing text, which will cause a lot of video waste. The inventor of this application found that the subtitles in the training samples are the main reason for the video generated by the video generation model to include strange text. If the subtitles in the video can be identified and removed from the video, the video without subtitles can also be used as a training sample for the video generation model, thereby effectively avoiding the waste of video.

[0087] In view of this, an embodiment of the present application provides a video processing method and device, which can provide high-quality training samples for a video generation model.

[0088] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0089] Exemplary Methods

[0090] See also Figure 1 , which is a flow chart of a video processing method provided in an embodiment of the present application.

[0091] Figure 1 The method shown can be executed by the client or by the server, and is not specifically limited in the embodiments of the present application.

[0092] In this embodiment, the method may include, for example, the following steps: S101 - S103 .

[0093] S101: Acquire a first video, where the first video includes a first video frame and a second video frame, where the first video frame and the second video frame are two adjacent video frames in the first video, and both the first video frame and the second video frame include a plurality of objects.

[0094] In one example, the first video may be a video obtained from a specific database. In another example, the first video may be a video obtained from a public network resource. As an example, the first video may be a single-lens video, which refers to a video captured by a camera under a certain lens. The embodiments of the present application do not specifically limit the video type of the first video, nor do they specifically limit the content included in the first video.

[0095] In one example, the first video frame and the second video frame may be any two adjacent video frames included in the first video, for example, the first video frame is the i-th video frame in the first video, and the second video frame is the i+1-th video frame in the first video, where i is an integer greater than or equal to 1.

[0096] In the present application, the first video frame and the second video frame both include multiple objects, and the embodiments of the present application do not specifically limit the multiple objects. In one example, the multiple objects may be, for example, the subject in the first video. For example, if the first video is a video including a human hand, the multiple objects may be, for example, multiple fingers of a human hand. In another example, the multiple objects may be multiple image corner points. For this case, in one example, a corner detection algorithm (such as the Shi-Tomasi corner detection algorithm) may be used to perform corner detection on the first video frame and the second video frame, thereby obtaining multiple corner points included in both the first video frame and the second video frame.

[0097] S102: For each object among the multiple objects, determine a motion parameter of the object according to positions of the object in the first video frame and the second video frame, so as to obtain a motion parameter corresponding to the first video.

[0098] In one example, considering that for a certain object, the position difference between two adjacent video frames can reflect the movement of the object, therefore, in the present application, for each of the aforementioned multiple objects, the motion parameters of the object can be determined according to the position of the object in the first video frame and the second video frame, respectively. The motion parameters of the object can be used to identify the movement of the object, for example, the motion parameters of the object can be used to identify the movement distance and / or movement direction of the object.

[0099] The embodiment of the present application does not specifically limit the specific implementation of determining the motion parameters of the object.

[0100] In an example, for the object, a vector pointing from the first coordinate to the second coordinate may be calculated according to the first coordinate of the object in the first video frame and the second coordinate of the object in the second video frame, and the vector may be used as the motion parameter of the object.

[0101] In another example, considering that the sparse optical flow vector of the object can characterize the movement distance and movement direction of the object, the sparse optical flow vector of the object can be determined based on the position of the object in the first video frame and the second video frame. As a specific example, the Lucas-Kanade method can be used to calculate the position of the object in the first video frame and the second video frame to obtain the sparse optical flow vector of the object.

[0102] In one example, for the object, the position of the object in any two adjacent video frames in the first video can be used to calculate the sparse optical flow vector of the object, thereby obtaining multiple sparse optical flow vectors of the object, and processing the multiple sparse optical flow vectors, for example, calculating the average value of the multiple sparse optical flow vectors, and using the average value as the final sparse optical flow vector of the object.

[0103] In the present application, considering that the motion of a single object may not be representative, the motion of a single object cannot identify the motion of a video. However, the motion of multiple objects can characterize the motion of a video to a certain extent. Therefore, after obtaining the motion parameters corresponding to each of the aforementioned multiple objects, the motion parameters corresponding to the first video can be determined based on the motion parameters corresponding to the multiple objects. In the present application, the motion parameters corresponding to the first video are used to represent the motion of the first video. Among them, the motion of the first video can be embodied in at least one aspect, the first aspect is used to identify whether the first video is a static video, and the second aspect is used to identify the jitter of the first video.

[0104] As a specific example, in a scene where the motion parameter of an object is a sparse optical flow vector, the motion parameter corresponding to the first video may be determined according to the sparse optical flow vectors respectively corresponding to the multiple objects.

[0105] The embodiment of the present application does not specifically limit the motion parameters corresponding to the first video. In a specific example, the motion parameters corresponding to the first video include: the dynamic score corresponding to the first video. The dynamic score is used to identify whether the first video is a static video. In one example, considering that for a certain object, the length of its sparse optical flow vector can be used to identify the movement distance of the object, therefore, the dynamic score of the first video can be determined according to the sum of the lengths of the sparse optical flow vectors corresponding to the multiple objects. In one example, the sum of the lengths of the sparse optical flow vectors corresponding to the multiple objects can be used as the dynamic score of the first video. In another example, considering that the duration and size of different videos are different, in order to make the calculated dynamic score applicable to videos of different durations and sizes. In another example, the sum of the aforementioned lengths can be normalized in both the time dimension and the video size dimension. Specifically, the dynamic score of the first video can be obtained according to the sum of the aforementioned lengths, the size of the first video, and the number of video frames included in the first video. For example, the sum of the lengths can be divided by the size and the number of video frames included in the first video, and the obtained value can be used as the dynamic score of the first video. Among them:

[0106] The size of the video may be, for example, the length and width of a video frame included in the video.

[0107] The number of video frames included in the first video can correspond to the aforementioned time dimension. The more video frames included in the first video, the longer the duration of the first video is, and the fewer video frames included in the first video, the shorter the duration of the first video is.

[0108] In another specific example, the motion parameter corresponding to the first video includes a target parameter, which can be used to identify the jitter of the first video. In one example, considering the motion speed of the first video, the stability of the motion speed of the first video, the motion direction of the first video, and the stability of the motion direction of the first video, the jitter of the first video can be identified to a certain extent. Therefore, the target parameter can include one of the motion speed of the first video, the stability of the motion speed of the first video, the motion direction of the first video, and the stability of the motion direction of the first video.

[0109] In one example, the motion speed of the first video can be determined, for example, based on the average of the lengths of the sparse optical flow vectors of the multiple objects, that is, the average of the lengths of the sparse optical flow vectors of the multiple objects can be calculated and used as the motion speed of the first video.

[0110] In one example, the stability of the motion speed of the first video can be determined according to the variance (or standard deviation) of the lengths of the sparse optical flow vectors of the multiple objects. That is, the variance (or standard deviation) of the lengths of the sparse optical flow vectors of the multiple objects can be calculated, and the variance (or standard deviation) is used as a parameter indicating the stability of the motion speed of the first video.

[0111] In one example, the motion direction of the first video can be determined, for example, based on the mean of the directions of the sparse optical flow vectors of the multiple objects, that is, the mean of the directions of the sparse optical flow vectors of the multiple objects can be calculated and used as the motion direction of the first video.

[0112] In one example, the stability of the motion direction of the first video can be determined based on the variance (or standard deviation) of the directions of the sparse optical flow vectors of the multiple objects. That is, the variance (or standard deviation) of the directions of the sparse optical flow vectors of the multiple objects can be calculated, and the variance (or standard deviation) is used as a parameter indicating the stability of the motion direction of the first video.

[0113] In one example, considering that a video usually contains a background and a foreground, the movement speed of the background is usually lower than that of the foreground object. The jitter of the video is mainly caused by the jitter of the lens. The movement of the background is an important factor in measuring whether the lens of the video is jittery. Therefore, in one example, the sparse optical flow vectors corresponding to the aforementioned multiple objects can be sorted in descending order of length. Among them, a certain number of sparse optical flow vectors (for example, 50% of the sparse optical flow vectors) with a higher order correspond to the foreground motion, and a certain number of sparse optical flow vectors (for example, 50% of the sparse optical flow vectors with a lower order correspond to the background motion. Accordingly, when calculating the aforementioned target parameters, the sparse optical flow vectors corresponding to the sparse optical flow vectors respectively corresponding to the multiple objects can be combined to determine the target parameters. That is, the target parameters are no longer determined based on the sparse optical flow vectors corresponding to all the objects in the multiple objects, but are determined based on the sparse optical flow vectors of the objects corresponding to the background motion, so that the determined target parameters can more accurately determine the jitter of the lens of the first video. In this scenario, the method for determining the target parameters can be understood in conjunction with the following Table 1, and will not be repeated here.

[0114] Table 1

[0115]

[0116] S103: Determine, based on motion parameters corresponding to the first video, whether the first video meets the requirements of the target model for training samples.

[0117] After determining the motion parameters corresponding to the first video, it can be determined whether the first video meets the requirements of the target model for training samples based on the motion parameters corresponding to the first video. The target model mentioned here can be, for example, a video generation model, that is, a model for generating videos. Of course, the target model can also be other models that use videos as training samples, which are not listed here one by one.

[0118] In one example, a video that is close to stillness is not suitable as a training sample for a video generation model. The dynamic score of the first video is less than or equal to a preset score, or the movement speed of the first video is less than or equal to a preset speed, both of which indicate that the first video is close to stillness. Therefore, if the dynamic score of the first video is less than or equal to a preset score, or the movement speed of the first video is less than or equal to a preset speed, it can be determined that the first video does not meet the requirements of the video generation model for training samples. The embodiments of the present application do not specifically limit the preset score and preset speed, and the preset score and preset speed can be determined based on actual conditions.

[0119] In another example, if the jitter of the first video (e.g., the jitter of the lens) is large, the first video is not suitable as a training sample for training the video generation model. The stability of the motion direction of the first video is less than or equal to a preset stability threshold. For example, if the variance (or standard deviation) of the directions of a certain number of sparse optical flow vectors ranked at the back is greater than or equal to a preset value, it means that the jitter of the first video (e.g., the jitter of the lens) is large. Therefore, if the stability of the motion direction of the first video is less than or equal to the preset stability threshold, it is determined that the first video does not meet the requirements of the video generation model for training samples.

[0120] Correspondingly, if the dynamic score of the first video is greater than the preset score, the movement speed of the first video is greater than the preset speed, and the stability of the movement direction of the first video is greater than the preset stability threshold, it means that the first video is a non-static video, and the jitter of the first video is also small. In this scenario, it can be determined that the first video meets the requirements of the video generation model for training samples.

[0121] In one example, if after executing S101-S103, it is determined that the first video meets the requirements of the video generation model for training samples, the first video can be used as a training sample of the video generation model for training. Correspondingly, in order to improve the training effect of the video generation model, the motion parameters of the first video can also be used as labels of the first video to participate in the training of the video generation model. In other words, if the first video meets the requirements of the video generation model for training samples, the first video and the motion parameters corresponding to the first video can be used to train the video generation model.

[0122] From the above description, it can be seen that by using the solution provided in the embodiment of the present application, the motion of the first video can be determined based on the positions of the aforementioned multiple objects in two adjacent video frames in the first video, thereby determining whether the first video meets the requirements of the target model (such as a video generation model) for training samples, so as to ensure the quality of the videos participating in the target model training.

[0123] In one example, the first video may not be shot by a professional shooting device, and the first video may be used as a training sample for the video generation model by using the above S101-S103, thereby providing more high-quality training samples for the video generation model while reducing costs.

[0124] The present application embodiment also provides a video processing method. Next, in combination with Figure 2 This paper introduces the video processing method.

[0125] See also Figure 2, which is a flow chart of another video processing method provided in an embodiment of the present application. Figure 2 The method shown may include the following S201-S202.

[0126] S201: Determine a subtitle area of ​​a second video.

[0127] In an example, the second video and the aforementioned first video may be the same video.

[0128] In another example, the second video and the first video may also be two different videos. In this scenario, as an example, the second video may be a video obtained from a specific database. As another example, the second video may be a video obtained from a public network resource.

[0129] In the present application, considering that using a video including subtitles to train a video generation model may affect the video generation effect of the trained video generation model, therefore, in the present application, the subtitle region of the second video can be identified, and the so-called subtitle region refers to the region including subtitles.

[0130] In an example, the subtitle area of ​​the second video may be determined by manual calibration. In other words, the subtitle area calibrated by a user may be obtained.

[0131] In another example, the following steps A1-A2 may be performed to determine the subtitle area of ​​the second video.

[0132] Step A1: Determine a subtitle area corresponding to each video frame in a plurality of video frames included in the second video.

[0133] In one example, a subtitle region recognition model may be pre-trained, and the multiple video frames may be input into the subtitle region recognition model to obtain the subtitle region corresponding to each of the multiple video frames.

[0134] In another example, the subtitle area in the video frame can be determined in combination with the features of the text in the video frame. The method for determining the subtitle area of ​​each video frame is the same. Next, the specific implementation of determining the subtitle area of ​​the video frame is described by taking the third video frame in the multiple video frames as an example. The third video frame can be any one of the multiple video frames.

[0135] In the present application, the text in the third video frame can be firstly identified. For example, optical character recognition (OCR) technology can be used to identify the text in the third video frame. Further, based on the features of the identified text, the subtitle area of ​​the third video frame is determined. Among them, the features of the text can include, for example, one or more features such as the size of the text, the location of the text, and the direction of the text. In the present application, for any text, the following operations can be performed to determine whether the text is a subtitle.

[0136] In one example, considering that the content text in the video frame is generally located in the central area of ​​the third video frame, for example, in the central 70% area of ​​the third video frame, and the subtitle area is generally located in the non-central area. Therefore, if the aforementioned text is located in the non-central area of ​​the third video frame, it can be determined that the text is a subtitle, and accordingly, it can be determined that the area corresponding to the text belongs to the subtitle area. Among them:

[0137] The central 70% area of ​​the third video frame refers to an area that occupies 70% of the area of ​​the third video frame and is centered on the center of the third video frame. The non-central area refers to an area other than the central area in the third video frame.

[0138] In another example, considering that the size of subtitles is generally not too large to avoid subtitles blocking other content in the video frame, for the aforementioned text, if the size of the text is less than or equal to the preset size, the text can be determined to be a subtitle, and accordingly, the area corresponding to the text can be determined to belong to the subtitle area.

[0139] In another example, considering that the direction of subtitles is generally relatively fixed, for example, horizontal subtitles or vertical subtitles. Whether it is a horizontal subtitle or a vertical subtitle, its direction is parallel to the edge line of the third video frame. For any subtitle, the rectangular text box corresponding to the subtitle includes two pairs of parallel lines. The two pairs of parallel lines should be parallel to the two vertical edge lines of the third video frame, respectively. For the convenience of description, the two vertical edge lines included in the third video frame are respectively referred to as the first edge line and the second edge line. The first edge line can be, for example, the upper edge line or the lower edge line of the third video frame, and the second edge line can be, for example, the left edge line or the right edge line of the third video frame. Then, of the two pairs of parallel lines, one pair of parallel lines is parallel to the first edge line, and the other pair of parallel lines is parallel to the second edge line. Therefore, if the two pairs of parallel lines included in the rectangular text box corresponding to the aforementioned text are parallel to the two vertical edge lines of the third video frame, respectively, it can be determined that the text is a subtitle, and accordingly, it can be determined that the area corresponding to the text belongs to the subtitle area.

[0140] After determining whether each character belongs to a subtitle, the subtitle area of ​​the third video frame can be determined. Specifically, the subtitle area of ​​the third video frame can include the area corresponding to the aforementioned characters belonging to the subtitle. For example, if characters 1 to 10 belong to subtitles, the subtitle area of ​​the third video frame includes the area corresponding to characters 1 to 10.

[0141] Step A2: Determine the subtitle area of ​​the second video according to the subtitle areas respectively corresponding to the multiple video frames.

[0142] In one example, considering that the subtitle area corresponding to each video frame may be different, and for the second video, the sizes of the multiple video frames included therein should be kept consistent. Therefore, after determining the subtitle areas corresponding to the multiple video frames, the subtitle area of ​​the second video can be determined according to the subtitle areas corresponding to the multiple video frames, so as to crop the second video based on the area of ​​the second subtitle.

[0143] In an example, a common area of ​​the subtitle areas of the aforementioned multiple video frames may be determined as the subtitle area of ​​the second video.

[0144] In another example, the subtitle areas of the multiple video frames can be combined to determine whether any pixel of the second video frame belongs to the subtitle area by voting. Specifically, for any pixel in the second video, if the number of video frames in which the pixel belongs to the subtitle area among the multiple video frames is greater than a preset number, it is determined that the pixel belongs to the subtitle of the second video. The preset number mentioned here may be related to the total number of video frames included in the second video. For example, the preset number may be greater than or equal to 70% of the total number. In other words, for the aforementioned pixel, it belongs to the subtitle area in most of the video frames, then it can be determined that the pixel belongs to the subtitle of the second video. For example: the second video includes 100 video frames, and the pixel belongs to the subtitle area in 75 video frames, then it can be determined that the pixel belongs to the subtitle area of ​​the second video. After performing the above judgment on any pixel, the subtitle area of ​​the second video can be obtained.

[0145] S202: Crop the subtitle area of ​​the second video to obtain a third video.

[0146] After determining the subtitle area of ​​the second video, the subtitle area of ​​the second video may be cropped to obtain a third video, wherein the cropped third video no longer includes the subtitle area of ​​the second video.

[0147] The embodiment of the present application does not specifically limit the method of cropping the subtitle area of ​​the second video, as long as the cropped third video no longer includes the subtitle area of ​​the second video.

[0148] In one example, the subtitle area of ​​the second video can be cropped according to the principle of retaining the largest area, thereby obtaining the third video. In this way, the obtained third video can retain more effective content.

[0149] In another example, the subtitle area of ​​the second video can be cropped according to the principle of optimal aspect ratio of the reserved area to obtain a third video. The optimal aspect ratio can be that the aspect ratio of the reserved area is closest to the aspect ratio of the video frame included in the second video, or that the aspect ratio of the reserved area is closest to the target aspect ratio. The embodiment of the present application does not make specific limitations. In this case, the cropped third video can maintain a good aspect ratio.

[0150] For reference Figure 3a to Figure 3c To understand, Figure 3a to Figure 3c A schematic diagram of cropping the second video is shown.

[0151] like Figure 3a As shown, 300 represents the total area of ​​the second video (specifically, the area corresponding to a certain video frame in the second video), and the subtitle area 310 of the second video.

[0152] If the subtitle area is cropped according to the principle of maximizing the area of ​​the reserved area, the reserved area is as follows: Figure 3b The shaded area 320 is shown. If the subtitle area is cropped according to the principle of optimal aspect ratio of the reserved area, the reserved area is as follows: Figure 3c The shaded area 330 is shown.

[0153] In one example, if the area of ​​the reserved region is small, for example, less than 40% of the area of ​​any video frame included in the second video, it means that the valid content included in the reserved region is limited, and the third video can be discarded. Correspondingly, if the area of ​​the reserved region is large, for example, greater than or equal to 40% of the area of ​​any video frame included in the second video, the third video can be retained.

[0154] In one example, the retained third video can be used as a training sample for the video generation model. In this scenario, using this solution, the second video containing text is no longer directly discarded, but the subtitles in the second video are identified and the subtitle area is cropped from the second video, so that the cropped third video can also be used as a training sample for the video generation model, thereby effectively avoiding video waste.

[0155] In one example, the aforementioned Figure 1 The method shown and Figure 2 The methods shown can be used in combination. Specifically, when the first video and the second video are the same video, Figure 1 The method shown and Figure 2 The methods shown can be used in combination. As a specific example, you can first execute Figure 1 The method shown in the following example is used to execute Figure 1 When the method shown in the figure determines that the first video meets the requirements of the target model for training samples, then execute Figure 2 The method shown in the figure cuts out the subtitle area in the first video to obtain a third video. Further, the third video is used to train the aforementioned target model. In this case, more high-quality training samples can be provided for the video generation model while reducing costs and avoiding video waste.

[0156] Exemplary Devices

[0157] Based on the method provided in the above embodiment, the embodiment of the present application further provides a device, which is described below in conjunction with the accompanying drawings.

[0158] See also Figure 4 , Figure 4 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application.

[0159] Figure 4 The video processing device shown is used to execute the video processing method provided in the above embodiment. Figure 4 As shown, the video processing device 400 includes: an acquisition unit 401 , a first determination unit 402 , and a second determination unit 403 .

[0160] The acquisition unit 401 is used to acquire a first video, where the first video includes a first video frame and a second video frame. The first video frame and the second video frame are two adjacent video frames in the first video, and both the first video frame and the second video frame include multiple objects.

[0161] The first determination unit 402 is used to determine, for each object among the multiple objects, a motion parameter corresponding to the object according to the position of the object in the first video frame and the second video frame, so as to obtain the motion parameter corresponding to the first video.

[0162] The second determination unit 403 is used to determine whether the first video meets the requirements of the target model for training samples according to the motion parameters corresponding to the first video.

[0163] Optionally, the first determining unit 402 is configured to:

[0164] Determine a sparse optical flow vector corresponding to the object according to positions of the object in the first video frame and the second video frame respectively;

[0165] Determine motion parameters corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects.

[0166] Optionally, the motion parameter corresponding to the first video includes a dynamic score and / or a target parameter corresponding to the first video, and determining the motion parameter corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects includes:

[0167] determining a dynamic score of the first video according to the sum of the lengths of the sparse optical flow vectors corresponding to the multiple objects, the size of the first video, and the number of video frames included in the first video; and / or,

[0168] The target parameters are determined according to the sparse optical flow vectors respectively corresponding to the multiple objects, and the target parameters include at least one of the following: the movement speed of the first video, the stability of the movement speed of the first video, the movement direction of the first video, and the stability of the movement direction of the first video.

[0169] Optionally, the sparse optical flow vectors corresponding to the multiple objects are sorted in descending order of length, and determining the target parameter according to the sparse optical flow vectors corresponding to the multiple objects includes:

[0170] The target parameter is determined according to a certain number of sparse optical flow vectors corresponding to the multiple objects, which are ranked lower in length.

[0171] Optionally, the target model is a video generation model, and the second determining unit is used to:

[0172] If the dynamic score of the first video is less than or equal to a preset score, or the motion speed of the first video is less than or equal to a preset speed, or the stability of the motion direction of the first video is less than or equal to a preset stability threshold, it is determined that the first video does not meet the requirements of the video generation model for training samples; or

[0173] If the dynamic score of the first video is greater than a preset score, the movement speed of the first video is greater than a preset speed, and the stability of the movement direction of the first video is greater than a preset stability threshold, it is determined that the first video meets the requirements of the video generation model for training samples.

[0174] Optionally, if the first video meets the requirements of the video generation model for training samples, the device further includes:

[0175] A training unit is used to train a video generation model using the first video and motion parameters corresponding to the first video.

[0176] Optionally, the device further comprises:

[0177] A third determining unit, configured to determine a subtitle area of ​​a second video;

[0178] The cropping unit is used to crop the subtitle area of ​​the second video to obtain a third video.

[0179] Optionally, the third determining unit is used to:

[0180] Determine a subtitle area corresponding to each video frame in a plurality of video frames included in the second video;

[0181] The subtitle area of ​​the second video is determined according to the subtitle areas respectively corresponding to the multiple video frames.

[0182] Optionally, the multiple video frames include a third video frame, and the subtitle area corresponding to the third video frame is determined in the following manner:

[0183] Identify text in the third video frame, and if the text is located in a non-central area of ​​the third video frame, or the size of the text is less than or equal to a preset size, or the two pairs of parallel lines included in the rectangular text box corresponding to the text are respectively parallel to two vertical edge lines of the third video frame, determine that the area corresponding to the text belongs to the subtitle area, wherein a first pair of parallel lines of the two pairs of parallel lines are parallel to a first edge line of the third video frame, a second pair of parallel lines of the two pairs of parallel lines are parallel to a second edge line of the third video frame, and the first edge line is perpendicular to the second edge line.

[0184] Optionally, determining the subtitle area of ​​the second video according to the subtitle areas respectively corresponding to the multiple video frames includes:

[0185] For any pixel in the second video, if the number of video frames in which the pixel belongs to the subtitle area among the multiple video frames is greater than a preset number, it is determined that the pixel belongs to the subtitle area of ​​the second video.

[0186] Optionally, the cutting unit is used to:

[0187] According to the principle of maximizing the area of ​​the reserved area or according to the principle of optimizing the aspect ratio of the reserved area, the subtitle area of ​​the second video is cropped to obtain the third video.

[0188] Optionally, the first video and the second video are the same video, and the third determining unit is configured to:

[0189] In the case that the first video meets the requirements of the target model for training samples, the subtitle area of ​​the first video is determined, and the third video is used to train the target model.

[0190] Optionally, the multiple objects include: multiple image corner points.

[0191] Since the device 400 is a device corresponding to the video processing method provided by the above method embodiment, the specific implementation of each unit of the device 400 is the same concept as the above method embodiment. Therefore, regarding the specific implementation of each unit of the device 400, you can refer to the relevant description part of the above method embodiment, which will not be repeated here.

[0192] The embodiment of the present application also provides an electronic device, the electronic device comprising a processor and a memory;

[0193] The processor is used to execute the instructions stored in the memory, so that the electronic device executes the video processing method provided by the above method embodiment.

[0194] An embodiment of the present application provides a computer-readable storage medium, including instructions, wherein the instructions instruct a device to execute the video processing method provided by the above method embodiment.

[0195] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes the video processing method provided by the above method embodiment.

[0196] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0197] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0198] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a first video, where the first video includes a first video frame and a second video frame, where the first video frame and the second video frame are two adjacent video frames in the first video, and both the first video frame and the second video frame include a plurality of objects; For each of the multiple objects, determine a motion parameter corresponding to the object according to positions of the object in the first video frame and the second video frame, so as to obtain a motion parameter corresponding to the first video; According to the motion parameters corresponding to the first video, it is determined whether the first video meets the requirements of the target model for training samples.

2. The method according to claim 1, characterized in that: The determining, according to the positions of the object in the first video frame and the second video frame respectively, the motion parameters corresponding to the object to obtain the motion parameters corresponding to the first video includes: Determine a sparse optical flow vector corresponding to the object according to positions of the object in the first video frame and the second video frame respectively; Determine motion parameters corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects.

3. The method according to claim 2, characterized in that The motion parameters corresponding to the first video include dynamic points and / or target parameters corresponding to the first video, and determining the motion parameters corresponding to the first video according to the sparse optical flow vectors respectively corresponding to the multiple objects includes: determining a dynamic score of the first video according to the sum of the lengths of the sparse optical flow vectors corresponding to the multiple objects, the size of the first video, and the number of video frames included in the first video; and / or, The target parameters are determined according to the sparse optical flow vectors respectively corresponding to the multiple objects, and the target parameters include at least one of the following: the movement speed of the first video, the stability of the movement speed of the first video, the movement direction of the first video, and the stability of the movement direction of the first video.

4. The method according to claim 3, characterized in that The sparse optical flow vectors corresponding to the multiple objects are sorted in descending order of length, and the target parameter is determined according to the sparse optical flow vectors corresponding to the multiple objects, including: The target parameter is determined according to a certain number of sparse optical flow vectors corresponding to the multiple objects, which are ranked lower in length.

5. The method according to claim 3 or 4, characterized in that: The target model is a video generation model, and determining whether the first video meets the target model's requirements for training samples according to the motion parameters corresponding to the first video includes: If the dynamic score of the first video is less than or equal to a preset score, or the motion speed of the first video is less than or equal to a preset speed, or the stability of the motion direction of the first video is less than or equal to a preset stability threshold, it is determined that the first video does not meet the requirements of the video generation model for training samples; or If the dynamic score of the first video is greater than a preset score, the movement speed of the first video is greater than a preset speed, and the stability of the movement direction of the first video is greater than a preset stability threshold, it is determined that the first video meets the requirements of the video generation model for training samples.

6. The method according to claim 5, characterized in that If the first video meets the requirements of the video generation model for training samples, the method further includes: training the video generation model using the first video and motion parameters corresponding to the first video.

7. The method according to claim 1, characterized in that The method further comprises: Determining a subtitle area of ​​a second video; The subtitle area of ​​the second video is cropped to obtain a third video.

8. The method according to claim 7, characterized in that The determining the subtitle area of ​​the second video includes: Determine a subtitle area corresponding to each video frame in a plurality of video frames included in the second video; The subtitle area of ​​the second video is determined according to the subtitle areas respectively corresponding to the multiple video frames.

9. The method according to claim 8, characterized in that The multiple video frames include a third video frame, and the subtitle area corresponding to the third video frame is determined in the following manner: Identify text in the third video frame, and if the text is located in a non-central area of ​​the third video frame, or the size of the text is less than or equal to a preset size, or the two pairs of parallel lines included in the rectangular text box corresponding to the text are respectively parallel to two vertical edge lines of the third video frame, determine that the area corresponding to the text belongs to the subtitle area, wherein a first pair of parallel lines of the two pairs of parallel lines are parallel to a first edge line of the third video frame, a second pair of parallel lines of the two pairs of parallel lines are parallel to a second edge line of the third video frame, and the first edge line is perpendicular to the second edge line.

10. The method according to claim 8, characterized in that The determining the subtitle area of ​​the second video according to the subtitle areas respectively corresponding to the plurality of video frames includes: For any pixel in the second video, if the number of video frames in which the pixel belongs to the subtitle area among the multiple video frames is greater than a preset number, it is determined that the pixel belongs to the subtitle area of ​​the second video.

11. The method according to claim 7, characterized in that The step of cropping the subtitle area of ​​the second video to obtain the third video includes: According to the principle of maximizing the area of ​​the reserved area or according to the principle of optimizing the aspect ratio of the reserved area, the subtitle area of ​​the second video is cropped to obtain the third video.

12. The method according to any one of claims 7 to 11, characterized in that: The first video and the second video are the same video, and determining the subtitle area of ​​the second video includes: In the case that the first video meets the requirements of the target model for training samples, the subtitle area of ​​the first video is determined, and the third video is used to train the target model.

13. The method according to claim 1, characterized in that The multiple objects include: multiple image corner points.

14. A video processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first video, wherein the first video includes a first video frame and a second video frame, wherein the first video frame and the second video frame are two adjacent video frames in the first video, and each of the first video frame and the second video frame includes a plurality of objects; a first determining unit, configured to determine, for each of the plurality of objects, a motion parameter corresponding to the object according to positions of the object in the first video frame and the second video frame, respectively, so as to obtain the motion parameter corresponding to the first video; The second determination unit is used to determine whether the first video meets the requirements of the target model for training samples according to the motion parameters corresponding to the first video.

15. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is used to execute instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct a device to execute the method according to any one of claims 1 to 13.

Images