Video special effects adding method and device

Abstract

The present invention discloses a method and device for adding special effects to a video, the method comprising: obtaining the optical flow information of any lens unit; determining the object that has been displaced in the lens unit according to the optical flow information, and obtaining the basic information and trajectory information of the object; determining the special effect parameters corresponding to the lens unit according to the basic information and trajectory information of the object, and preset special effect parameter corresponding rules, so as to pass the special effect parameters to the special effect engine to complete the special effect rendering of the video. The present invention can be used to automatically add special effects to the object presented in the video in real time, and the added special effects are related to the trajectory information and basic information of the object, so as to ensure that the added special effects meet the video presentation effect. The added special effects do not need to rely on professional tools, and the technical threshold for adding special effects is also reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of video special effects processing, and in particular to a method and device for adding video special effects. Background Art

[0002] Video effects can improve the viewing experience of users. Video effects are generally added in the following ways:

[0003] Using AE (After Effects, non-linear special effects production software) tools can directly add various types of special effects to videos, such as dynamic images, 2D, 3D effects, etc., but AE tools have a high threshold, a long production time, and a high level of professionalism.

[0004] When using textures to add special effects, the effects they present are all based on template solidification. Although the operation is simple, the effects produced on each object in the video are almost the same, and the characteristics of the objects cannot be highlighted.

[0005] When using AI to generate special effects, although the threshold for use is relatively low, its versatility is very poor, and it can generally only process special effects for a certain class of objects. When adding objects other than the specific class of objects, problems such as special effects drifting and disappearance will occur. Summary of the invention

[0006] In view of the above problems, embodiments of the present invention are proposed to provide a method and device for adding video special effects that overcome the above problems or at least partially solve the above problems.

[0007] According to one aspect of an embodiment of the present invention, a method for adding special effects to a video is provided, which includes:

[0008] Segment the video to obtain multiple shot units generated by shot switching;

[0009] For any lens unit, obtain the optical flow information of the lens unit;

[0010] According to the optical flow information, determine the object that has moved in the lens unit, and obtain the basic information and trajectory information of the object;

[0011] According to the basic information and trajectory information of the object, as well as the preset special effect parameter corresponding rules, the special effect parameters corresponding to the lens unit are determined, so that the special effect parameters can be passed to the special effect engine to complete the special effect rendering of the video.

[0012] According to another aspect of an embodiment of the present invention, a video special effect adding device is provided, comprising:

[0013] A segmentation module, adapted to segment the video shots to obtain multiple shot units generated by shot switching;

[0014] An optical flow acquisition module, adapted to acquire the optical flow information of any lens unit;

[0015] A trajectory acquisition module, adapted to determine the object that has been displaced in the lens unit according to the optical flow information, and to acquire basic information and trajectory information of the object;

[0016] The rendering module is suitable for determining the special effect parameters corresponding to the lens unit according to the basic information and trajectory information of the object and the preset special effect parameter corresponding rules, so as to pass the special effect parameters to the special effect engine to complete the special effect rendering of the video.

[0017] According to another aspect of an embodiment of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0018] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned video special effect adding method.

[0019] According to another aspect of the embodiments of the present invention, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for adding video special effects.

[0020] According to the video special effects adding method and device provided by the embodiment of the present invention, special effects can be automatically added to the objects presented in the video in real time, and the added special effects are related to the trajectory information and basic information of the objects, ensuring that the added special effects are consistent with the video presentation effect. The added special effects do not need to rely on professional tools, which also reduces the technical threshold for adding special effects.

[0021] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation method of the embodiment of the present invention is specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A flow chart of a method for adding special effects to a video according to an embodiment of the present invention is shown;

[0024] Figure 2 A schematic diagram showing the optical flow information display of a video frame is shown;

[0025] Figure 3 A schematic diagram showing object trajectory information in a video is shown;

[0026] Figure 4 A schematic diagram of the object recognition result in the video is shown;

[0027] Figure 5 A schematic diagram showing the correspondence between special effect parameters and basic information and trajectory information of the object;

[0028] Figure 6 A schematic diagram of a trajectory speed change value and a trajectory direction change value is shown;

[0029] Figure 7 A schematic diagram showing the structure of a video special effect adding device according to an embodiment of the present invention is shown;

[0030] Figure 8 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0032] Figure 1 FIG. 4 shows a flow chart of a method for adding special effects to a video according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0033] Step S101, segmenting the video into shots to obtain a plurality of shot units generated by shot switching.

[0034] In this embodiment, it is taken into consideration that there are shot switching in the video. For example, when the shot switches from one object a to another object b, the special effects added to the original object a cannot be directly applied to object b. Therefore, before adding special effects, the video is firstly divided into shots, and the adding of special effects to the video is changed to adding special effects to multiple shot units, so as to ensure that the added special effects are more suitable for each object in the video content.

[0035] Specifically, when performing shot segmentation on a video, the similarity of adjacent video frames in the video can be calculated to determine whether the similarity is less than a similarity threshold. The similarity threshold can be set according to the implementation situation and is not limited here. For example, the SIFT algorithm is used to extract key special effects in two adjacent video frames to calculate the similarity. If the similarity is less than the similarity threshold, it means that the shot has been switched. Shot segmentation is performed based on adjacent video frames to obtain multiple shot units generated according to the shot switching.

[0036] Step S102: For any lens unit, obtain the optical flow information of the lens unit.

[0037] For any lens unit among the multiple lens units generated by lens segmentation, the optical flow information of the lens unit is obtained. The optical flow information can be obtained based on an optical flow method, such as Horn-Schunck, Lucas-Kanade, Farneback, etc., which is not limited here.

[0038] Step S103: determining the object that has moved in the lens unit according to the optical flow information, and obtaining basic information and trajectory information of the object.

[0039] According to the optical flow information, the percentage of pixels with optical flow changes can be calculated. For example, the optical flow information of all pixels in the video frame is sensed, the coordinate change of the pixel (i.e., the optical flow change) is determined, and the pixel with optical flow change is compared with all pixels to obtain the percentage of pixels with optical flow change. It is determined whether the pixel percentage exceeds the preset percentage threshold. If so, it can be determined that the lens in the lens unit has shifted, and the lens displacement information in the lens unit is obtained accordingly. Here, the preset percentage threshold can be set according to the implementation situation, such as 99%, which is not limited here.

[0040] The lens shift information includes lens shift speed, lens shift direction, etc. When obtaining the lens shift information, the pixels in the video frame may be classified according to the pixel shift speed, and pixels with the same shift speed may be classified into one category. For example, the pixels may be sorted according to the size of the pixel shift speed, and pixels with the same shift speed may be sorted in the same rank, and the number of repeated pixels with the same shift speed may be obtained by counting. The pixel movement speed corresponding to the maximum value of the number of repeated pixels is taken as the lens shift speed, and at this time, the opposite direction of the pixel movement direction is taken as the lens shift direction.

[0041] by Figure 2 Take this as an example to illustrate: Figure 2Each arrow in the picture represents the direction of displacement, and the length of the arrow represents the displacement speed. The longer the arrow, the greater the displacement speed. The arrow can be determined based on the optical flow information. The arrows in the picture are determined by the optical flow information of each pixel. The light-colored long arrow at the position of the airship above the picture (hereinafter referred to as airship A) represents the displacement direction and displacement speed of airship A; the arrow diagonally above the position of the airship below the picture (hereinafter referred to as airship B) represents the displacement direction and displacement speed of airship B; the arrows to the right in the picture are the relative displacement direction and displacement speed of the static object caused by the lens displacement; the left arrow on the right side of the camera in the picture represents the displacement direction and displacement speed of the camera (lens). The acquisition of the left arrow specifically includes: the arrows in the picture can be sorted according to the arrow length (i.e., displacement speed), such as sorting from large to small, and the arrows with the same length are classified together to obtain the number of pixel repetitions with the same displacement speed. Figure 2 The displacement speeds are sorted for ease of understanding. Figure 2 Taking the number of arrows in the middle as an example, we can get 8 light-colored long arrows of airship A, 9 upward diagonal arrows of airship B, and 57 arrows of static objects to the right. The arrow with the largest number of repetitions is the arrow of the static object to the right. It can be determined that the displacement direction of the lens is in the opposite direction, which is the arrow to the left; the displacement speed of the lens is the same, that is, the length of the arrow of the lens is the same as the length of the arrow of the static object to the right. Furthermore, when classifying the displacement speed of pixels in the video frame, there are more pixels and more processing volume; for some scenes such as close-ups and close-ups, deviations and other problems are also prone to occur. Considering the above problems, the objects in the lens unit can be identified, and the displaced objects can be processed as a whole to Figure 2 For example, the displacement speed of airship A and the displacement speed of airship B can be obtained in turn, which simplifies the processing volume and improves the processing accuracy. Each object in the lens unit can be identified or segmented based on the optical flow information, and the basic information of the object can be obtained at the same time. The basic information includes object name, object color, object outline, object size, etc. Figure 2 In the image processing, the object airships can be identified, the object outline can be segmented, the size of the object in the lens unit, the color presented in the lens unit, etc. can be determined.

[0042] According to the lens displacement information and optical flow information, the object displacement information can be calculated. If the lens is stationary, that is, its displacement speed is 0, the obtained optical flow information is the optical flow information of the object, and the displacement information of the object can be directly determined; if the lens is moving, that is, it has a displacement speed, through Figure 2To calculate the displacement information of the lens in the example, it is necessary to add the displacement information determined based on the optical flow information to the displacement information of the lens, that is, the displacement information determined based on the optical flow information needs to be added with the deviation caused by the movement of the lens to obtain the displacement information of the object. Figure 2 For example, the displacement information of airship A needs to add its light-colored long arrow to the arrow pointing to the left of the lens, add the vectors, adjust the direction and length of the light-colored long arrow, and obtain the displacement information of airship A. Alternatively, the relative reference method of the optical flow picture can also be used, such as extracting the feature point A1 of the object position in the previous frame of the adjacent video frames, and the feature point A2 of the object position in the next frame; the feature point B1 of the non-object position in the previous frame of the adjacent video frames, and the corresponding feature point B2 of the non-object position in the next frame, through the matching of the non-object position feature points, that is, the feature point B1 matches the feature point B2, the coordinates of the adjacent video frames of the anchor point, the non-object position feature points match and coincide, and the direction and distance between A1 and A2 are the displacement direction and displacement distance of the object. The above is an example, and it can be set according to the implementation situation during specific implementation, and it is not limited here.

[0043] Determine the trajectory information of the object in the lens unit based on the object displacement information. The trajectory information includes trajectory type, trajectory length, trajectory speed, trajectory direction, etc. By statistically analyzing the object displacement information in each video frame in the lens unit, the trajectory length and trajectory direction of the object can be obtained, and the trajectory speed can be further calculated, and the trajectory in the video frame can be classified to obtain the trajectory type. Specifically, based on the displacement information of the object in all video frames under the lens unit, the motion trajectory of the object under the lens unit can be obtained. Figure 3 As shown in the figure, the left side is the video frame, and the lines on the right side are the motion trajectories of each vehicle, which are obtained by statistically analyzing the displacement information of all video frames under the lens unit. The motion trajectory can be classified into trajectory types using classification algorithms such as classification models, SVM, and machine learning. The trajectory types are shown in Table 1 below:

[0044] Track Type Scenario examples or descriptions Circle Ballet twirl Arc semicircle parabola volleyball straight line Boxing, sprinting S Line Wander Change of direction The ball hits and bounces back Changing direction back and forth Up and down

[0045] Table 1

[0046] As shown in Table 1, the first column is the trajectory type, and the second column is the corresponding scene example or description, that is, in what scene the corresponding trajectory type appears. Table 1 is only an example, and the trajectory type can be set according to the implementation situation, and is not limited here. In addition to the trajectory type, the trajectory information also includes trajectory length, trajectory speed, trajectory direction, etc. The trajectory length and trajectory speed are based on the size in the video screen. The trajectory direction is basically the same as the displacement direction. After obtaining the basic information and trajectory information of the object, the trajectory information of the object can be associated with the basic information. Figure 4In the above example, the object that is displaced is identified as a car. The basic information and trajectory information of the associated object can be obtained that the trajectory type of the car is a straight line, the trajectory length is 1920 pixels, the trajectory speed is 800 pixels / s, the trajectory direction is 0 degrees, the object name is car, the color is #2B323C (the average color value or local color value of the object can be taken here, or multiple different color values ​​can be retained according to local features, corresponding to the subsequent addition of different special effects, etc.), the contour is obtained by extracting the contour of the object segmentation, and the object size is 200*150 pixels, etc. Further, this embodiment mainly adds special effects to the displaced object, and does not need to be identified for the static object that is not displaced.

[0047] Step S104, determining the special effect parameters corresponding to the lens unit according to the basic information and trajectory information of the object and the preset special effect parameter corresponding rules, so as to pass the special effect parameters to the special effect engine to complete the special effect rendering of the video.

[0048] In this embodiment, preset special effect parameter corresponding rules can be pre-set, wherein a corresponding relationship between special effect parameters and basic information and trajectory information of an object can be established for each special effect style, thereby obtaining preset special effect parameter corresponding rules. According to the changes in the basic information and trajectory information of the object in the video, dynamic special effect adjustment can be achieved, and the current state of the moving object in the video can be adapted. Figure 5 As shown, Figure 5 It contains various special effects styles, such as streamer, lightning, flame, rain, cloud, etc. The dotted arrows represent the corresponding relationship between special effect styles and trajectory classification and special effect parameters ( Figure 5 The correspondence between the special effect adjustment parameters in the function and the basic information, trajectory information and other information. Figure 5 This is just an example. Figure 5 The rendering method (map following, particle rendering, filter, etc.), special effect color (same color as the object, opposite color to the object, random color, set to xxx color), special effect position (attached to the object, overlapping with the object, corresponding to the object coordinates: increase or decrease the x, y coordinate position), special effect transparency, special effect density, special effect intensity, special effect display frequency, special effect size, etc., can also include parameters such as brightness, contrast, vibration amplitude, saturation, repulsion, gravity, particle size, original particle shape, optical flow density, etc., which are set according to the implementation situation and are not limited here. Figure 5 In the description, objects are used as an example, but are not limited thereto. In this embodiment, objects may also include various objects such as portraits, animals and plants. The correspondence between each special effect parameter and the basic information and trajectory information of the object may be many-to-one, such as the rendering methods of particles and filters may be used for one object at the same time. The specific correspondence is set according to the implementation situation and is not limited here.

[0049] Furthermore, when the video contains text information and audio information, such as subtitles, BGM background music, interlude music, etc., the text information and audio information in the video can be obtained, and the emotional information of the video can be determined based on the text information and audio information. For example, the subtitles in the video can be identified using technologies such as OCR to obtain the corresponding text, and text-emotion classification algorithms such as NLP can be used to pre-train the text-emotion classification model, or to establish an emotional vocabulary, etc., to classify the text by emotion and obtain emotional information. Emotional information includes, for example, emotional classification and scores. Audio information can also be classified through music-emotion classification algorithms, such as using SVM, machine learning, deep learning and other classification algorithms to obtain emotional information corresponding to the audio information. According to the emotional information of the video and the preset special effect parameter correspondence rules, the special effect parameters corresponding to the lens unit are determined. For example Figure 5 As shown, the relevant parameters also include emotions, lines, etc., corresponding to emotional information.

[0050] The special effect styles are described in Table 2 below as an example:

[0051]

[0052] Table 2

[0053] For example, when the special effect style is streamer and the track type is straight line, if someone punches in the corresponding video, a bomb sticker 1101 special effect is displayed on the fist; when the video also includes strong anger, a flame sticker 1104 special effect appears on the full screen. For example, when the special effect style is streamer and the track type is circle, the rendering method can include two different methods: particles and filters. The special effect position can be different according to the size of the object, such as if the object size is larger than 200*200 pixels, it is attached to the object, and if the object size is smaller than 200*200 pixels, it overlaps with the object.

[0054] Furthermore, in order to facilitate the control of the coherence and integrity of the special effects of each lens unit of the video so that the video presents a global overall special effect, it is also possible to determine whether the preset special effect trigger conditions are met based on the basic information and trajectory information of the object. The preset special effect trigger conditions include, for example, the trajectory speed change value is greater than the preset speed change value, the trajectory direction change value is greater than the preset direction change value, the trajectory length is within the preset length special effect range value, the trajectory speed is within the preset speed special effect range value, the number of special effect types is within the preset type range value, the total number of special effects is within the preset special effect number range value, the object size is within the preset object size range value, etc. That is, different preset special effect trigger conditions are set for different special effects. When the basic information and trajectory information of the object meet the corresponding preset special effect trigger conditions, the corresponding preset variable special effect parameters are set for the object, the special effects are adjusted to different degrees, and then the preset variable special effect parameters and special effect parameters are passed to the special effect engine to complete the special effect rendering of the video. As shown in Table 3 below:

[0055]

[0056]

[0057] Table 3

[0058] Among them, different preset variable special effect parameters are corresponding to different trajectory lengths of the object, such as a trajectory length of 1250 pixels, corresponding to a general special effect, with a weight of 0.8; a trajectory length of 2000 pixels, corresponding to a small number of special effects, with a weight of 0.5, etc. There are 3 types of special effects in the same shot, corresponding to a general special effect, with a weight of 0.8; there are 2 types of special effects in the same shot, corresponding to a small number of special effects, with a weight of 0.5, etc. Similarly, the number of special effects can be further reduced by setting preset special effect trigger conditions, and excessive special effects can be further prevented by limiting the number and type. In the preset special effect trigger conditions, the preset length special effect range value, the preset speed special effect range value, the preset type range value, the preset special effect number range value, the preset object size range value, etc. are set according to the implementation situation, and are not limited here.

[0059] Furthermore, in this embodiment, the preset special effect triggering conditions also include a trajectory speed change value greater than a preset speed change value, a trajectory direction change value greater than a preset direction change value, etc. The trajectory speed change value and the trajectory direction change value can be obtained by obtaining the optical flow information of the object in video frames or seconds, and then determining the trajectory speed change value and the trajectory direction change value. Based on the time sequence (video frame time sequence, time sequence, etc.), the trajectory speed and trajectory direction of the object with large changes, such as Figure 6 As shown in the figure, the trajectory speed of the gray area in the left figure has changed significantly, and the trajectory direction of the gray area in the right figure has changed significantly. By presetting the special effect trigger conditions, it is determined that the trajectory speed change value is greater than the preset speed change value, or the trajectory direction change value is greater than the preset direction change value, and the preset change special effect parameters are set accordingly. The preset change special effect parameters can be set, such as adding particle special effects. Multiple conditions in the preset special effect trigger conditions can be judged at the same time, or one or more of them can be selected for judgment to ensure the continuity and overall coordination of special effects in the video.

[0060] The preset variable special effect parameters can be passed to the special effect engine together with the special effect parameters. They can be set to a format that meets the requirements of the special effect engine and passed through the interface provided by the special effect engine. The special effect engine will render the video with special effects based on the received preset variable special effect parameters and special effect parameters.

[0061] According to the video special effects adding method provided by the embodiment of the present invention, special effects can be automatically added to the objects presented in the video in real time, and the added special effects are related to the trajectory information and basic information of the objects, ensuring that the added special effects are consistent with the video presentation effect. The added special effects do not need to rely on professional tools, which also reduces the technical threshold for adding special effects.

[0062] Figure 7 FIG. 1 is a schematic diagram showing the structure of a video special effect adding device provided by an embodiment of the present invention. Figure 7 As shown, the video special effect adding device includes:

[0063] A segmentation module 710 is adapted to segment the video into shots to obtain a plurality of shot units generated by shot switching;

[0064] An optical flow acquisition module 720, adapted to acquire optical flow information of any lens unit;

[0065] A trajectory acquisition module 730, adapted to determine the object that has moved in the lens unit according to the optical flow information, and acquire basic information and trajectory information of the object;

[0066] The rendering module 740 is adapted to determine the special effect parameters corresponding to the lens unit according to the basic information and trajectory information of the object and preset special effect parameter corresponding rules, so as to pass the special effect parameters to the special effect engine to complete the special effect rendering of the video.

[0067] Optionally, the segmentation module 710 is further adapted to:

[0068] Calculating the similarity between adjacent video frames in the video, and determining whether the similarity is less than a similarity threshold;

[0069] If so, it is determined that a shot switch occurs in the video, and shot segmentation is performed based on adjacent video frames to obtain a plurality of shot units generated by the shot switch.

[0070] Optionally, the trajectory acquisition module 730 is further adapted to:

[0071] Calculate the pixel ratio of the optical flow change according to the optical flow information;

[0072] Determine whether the pixel ratio exceeds a preset ratio threshold;

[0073] If so, obtaining lens displacement information in the lens unit;

[0074] According to the optical flow information, the object in the lens unit is identified and basic information of the object is obtained; the basic information includes the object name, object color, object outline and / or object size;

[0075] Calculate object displacement information based on lens displacement information and optical flow information;

[0076] The trajectory information of the object in the shot unit is determined according to the object displacement information; the trajectory information includes a trajectory type, a trajectory length, a trajectory speed and / or a trajectory direction.

[0077] Optionally, the lens displacement information includes a lens displacement speed and a lens displacement direction;

[0078] The optical flow acquisition module 720 is further adapted to:

[0079] Classify according to pixel displacement speed and obtain the number of repeated pixels with the same displacement speed;

[0080] The pixel movement speed corresponding to the maximum value of the pixel repetition number is taken as the lens shift speed, and the opposite direction of the pixel movement direction is taken as the lens shift direction.

[0081] Optionally, the trajectory acquisition module 730 is further adapted to:

[0082] The displacement information of the object in each video frame in the shot unit is counted to obtain the trajectory length and / or trajectory direction of the object, the trajectory speed is calculated, and the trajectory in the video frame is classified to obtain the trajectory type.

[0083] Optionally, the rendering module 740 is further adapted to:

[0084] Determine whether a preset special effect triggering condition is met according to the basic information and / or trajectory information of the object; the preset special effect triggering condition includes: a trajectory speed change value greater than a preset speed change value, a trajectory direction change value greater than a preset direction change value, a trajectory length within a preset length special effect range value, a trajectory speed within a preset speed special effect range value, a special effect type number within a preset type range value, a total special effect number within a preset special effect number range value and / or an object size within a preset object size range value;

[0085] If so, corresponding preset variable special effect parameters are set for the object, so that the preset variable special effect parameters and the special effect parameters are passed to the special effect engine to complete the special effect rendering of the video.

[0086] Optionally, the device also includes: an emotion acquisition module 750, suitable for acquiring text information and / or audio information in the video, and determining the emotional information of the video based on the text information and / or audio information; determining the special effects parameters corresponding to the lens unit based on the emotional information of the video and preset special effects parameter correspondence rules, so as to pass the special effects parameters to the special effects engine to complete the special effects rendering of the video.

[0087] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0088] An embodiment of the present invention further provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute the video special effect adding method in any of the above method embodiments.

[0089] Figure 8 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown, and the specific implementation of the embodiment of the present invention does not limit the specific implementation of the computing device.

[0090] like Figure 8 As shown, the computing device may include: a processor (processor) 802 , a communication interface (Communications Interface) 804 , a memory (memory) 806 , and a communication bus 808 .

[0091] Features:

[0092] The processor 802 , the communication interface 804 , and the memory 806 communicate with each other via a communication bus 808 .

[0093] The communication interface 804 is used to communicate with other devices such as clients or other servers.

[0094] The processor 802 is used to execute the program 810, and specifically can execute the relevant steps in the above-mentioned video special effect adding method embodiment.

[0095] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0096] The processor 802 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0097] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0098] Program 810 can be specifically used to enable processor 802 to execute the video special effect adding method in any of the above method embodiments. The specific implementation of each step in program 810 can refer to the corresponding description in the corresponding steps and units in the above video special effect adding embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiments, which will not be repeated here.

[0099] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the embodiment of the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the embodiment of the present invention described herein, and the description of the above specific language is to disclose the preferred implementation of the embodiment of the present invention.

[0100] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0101] Similarly, it should be understood that in order to streamline the embodiments of the invention and to aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the invention, the various features of the embodiments of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: that the claimed embodiments of the invention require more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, characterized in that each claim itself serves as a separate embodiment of the invention.

[0102] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0103] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0104] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The embodiments of the present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing an embodiment of the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0105] It should be noted that the above embodiments illustrate rather than limit the present invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim listing a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A method for adding special effects to a video, characterized in that: It includes: Segment the video to obtain multiple shot units generated by shot switching; For any lens unit, obtain the optical flow information of the lens unit; Performing object recognition or object segmentation according to the optical flow information to determine the object that has been displaced in the lens unit, and obtaining basic information of the object; and obtaining lens displacement information in the lens unit according to the optical flow information, and calculating object displacement information according to the lens displacement information and the optical flow information to determine trajectory information of the object; According to the basic information and trajectory information of the object, and the preset special effect parameter corresponding rules, the special effect parameters corresponding to the lens unit are determined, so that the special effect parameters are passed to the special effect engine to complete the special effect rendering of the video.

2. The method according to claim 1, characterized in that: The step of dividing the video into shots to obtain a plurality of shot units generated by shot switching further includes: Calculating the similarity between adjacent video frames in the video, and determining whether the similarity is less than a similarity threshold; If so, it is determined that a shot switch occurs in the video, and shot segmentation is performed based on adjacent video frames to obtain a plurality of shot units generated by the shot switch.

3. The method according to claim 1, characterized in that The method of performing object recognition or object segmentation according to the optical flow information to determine the object that has been displaced in the lens unit, obtaining basic information of the object, and obtaining lens displacement information in the lens unit according to the optical flow information, calculating object displacement information according to the lens displacement information and the optical flow information, and determining the trajectory information of the object further includes: Calculate the pixel ratio of the optical flow change according to the optical flow information; Determine whether the pixel ratio exceeds a preset ratio threshold; If so, obtaining lens displacement information in the lens unit; According to the optical flow information, the object in the lens unit is identified and basic information of the object is obtained; the basic information includes the object name, object color, object outline and / or object size; Calculate object displacement information based on lens displacement information and optical flow information; The trajectory information of the object in the shot unit is determined according to the object displacement information; the trajectory information includes a trajectory type, a trajectory length, a trajectory speed and / or a trajectory direction.

4. The method according to claim 3, characterized in that The lens displacement information includes lens displacement speed and lens displacement direction; The obtaining of lens displacement information in the lens unit further comprises: Classify according to pixel displacement speed and obtain the number of repeated pixels with the same displacement speed; The pixel movement speed corresponding to the maximum value of the pixel repetition number is taken as the lens shift speed, and the opposite direction of the pixel movement direction is taken as the lens shift direction.

5. The method according to claim 3, characterized in that: Determining the trajectory information of the object in the shot unit according to the object displacement information further comprises: The displacement information of the object in each video frame in the shot unit is counted to obtain the trajectory length and / or trajectory direction of the object, the trajectory speed is calculated, and the trajectory in the video frame is classified to obtain the trajectory type.

6. The method according to claim 1, characterized in that The determining of the special effect parameters corresponding to the lens unit according to the basic information and trajectory information of the object and the preset special effect parameter corresponding rule, so as to pass the special effect parameters to the special effect engine to complete the special effect rendering of the video further includes: Determine whether a preset special effect triggering condition is met according to the basic information and / or trajectory information of the object; the preset special effect triggering condition includes: a trajectory speed change value greater than a preset speed change value, a trajectory direction change value greater than a preset direction change value, a trajectory length within a preset length special effect range value, a trajectory speed within a preset speed special effect range value, a special effect type number within a preset type range value, a total special effect number within a preset special effect number range value and / or an object size within a preset object size range value; If so, corresponding preset variable special effect parameters are set for the object, so that the preset variable special effect parameters and the special effect parameters are passed to the special effect engine to complete the special effect rendering of the video.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquiring text information and / or audio information in the video, and determining emotional information of the video according to the text information and / or audio information; According to the emotional information of the video and the preset special effect parameter corresponding rules, the special effect parameters corresponding to the lens unit are determined, so that the special effect parameters are passed to the special effect engine to complete the special effect rendering of the video.

8. A video special effects adding device, characterized in that: The device comprises: A segmentation module, adapted to segment the video shots to obtain multiple shot units generated by shot switching; An optical flow acquisition module, adapted to acquire the optical flow information of any lens unit; a trajectory acquisition module, adapted to perform object recognition or object segmentation according to the optical flow information to determine the object that has been displaced in the lens unit, acquire basic information of the object, and, according to the optical flow information, acquire lens displacement information in the lens unit, calculate object displacement information according to the lens displacement information and the optical flow information, and determine trajectory information of the object; The rendering module is adapted to determine the special effect parameters corresponding to the lens unit according to the basic information and trajectory information of the object and the preset special effect parameter corresponding rules, so as to pass the special effect parameters to the special effect engine to complete the special effect rendering of the video.

9. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video special effect adding method according to any one of claims 1-7.

10. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the video special effect adding method according to any one of claims 1 to 7.

Images