Video processing method, device and storage medium

Abstract

The present invention discloses a video processing method, device and storage medium, the method comprising: obtaining a first target video clip; performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set; merging the special effects corresponding to the target special effect tag set with the first target video clip to obtain a second target video clip.

Description

Technical Field

[0001] The present invention relates to the field of data services, and in particular to a video processing method, device and storage medium. Background Art

[0002] With the development and innovation of communication technology and the high popularity of intelligent terminal devices, online live broadcast and short video interactive applications have developed rapidly and become a new media method in people's work and life. How to enrich the application scenarios of video interaction and meet the diverse interactive needs of users is an effective strategy to attract users and improve user stickiness.

[0003] Currently, video interaction applications can add static image decorations to the user's face or head, or add corresponding video effects based on the user's movement status. The application scenarios are relatively simple, and it is impossible to add corresponding animation effects or music effects based on the user's expressions or voice information in the video. Summary of the invention

[0004] In view of this, the main purpose of the present invention is to provide a video processing method, device and storage medium.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a video processing method, the method comprising:

[0007] Obtaining a first target video clip;

[0008] Performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature;

[0009] Querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set;

[0010] The special effects corresponding to the target special effects tag set are merged with the first target video segment to obtain a second target video segment.

[0011] In the above scheme, the at least one feature includes: voice features, facial expression features, and human body movement features;

[0012] The performing feature recognition on the first target video segment to obtain a recognition result set includes:

[0013] At least one preset recognition model is used to recognize the first target video segment, and a recognition result corresponding to each of the features in the first target video segment is obtained.

[0014] In the above solution, the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence of each result;

[0015] The step of querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set includes:

[0016] Determine at least one result subset based on the confidence level of each result in the recognition results corresponding to each of the features from high to low; each of the result subsets includes: a result corresponding to at least one feature; and the confidence levels corresponding to the results included in each of the result subsets are the same;

[0017] According to the at least one result subset, query a preset special effect tag library to determine at least one target special effect tag;

[0018] The target special effect tag set is obtained according to the at least one target special effect tag.

[0019] In the above solution, the step of querying a preset special effect tag library according to the at least one result subset to determine at least one target special effect tag includes:

[0020] When a preset special effect label library is queried according to the first result subset and it is determined that at least one result in the first result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the first result subset is used as the target special effect label; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except the first result subset;

[0021] When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset;

[0022] When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

[0023] In the above solution, the step of merging the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment includes:

[0024] The special effects corresponding to the target special effects tags in the target special effects tag set are added to the first target video segment to obtain the first target video segment with the special effects added thereto as the second target video segment.

[0025] In the above solution, the obtaining of the first target video segment includes:

[0026] Identify the video data to determine at least a first video segment having a target object;

[0027] Extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment;

[0028] Correspondingly, merging the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment includes:

[0029] The special effects corresponding to the target special effects tags in the target special effects tag set are added to the first video segment to obtain the first video segment with the special effects added thereto as the second target video segment.

[0030] In the above scheme, the method further includes at least one of the following:

[0031] Training a first recognition model; the first recognition model is used to extract and recognize facial expression features;

[0032] Training a second recognition model; the second recognition model is used to extract and recognize human motion features;

[0033] A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

[0034] An embodiment of the present invention provides a video processing device, the device comprising:

[0035] An acquisition module, used for acquiring a first target video segment;

[0036] An identification module, configured to perform feature identification on the first target video segment to obtain an identification result set; the identification result set includes an identification result corresponding to at least one feature;

[0037] A determination module, used to query a preset special effect tag library according to the recognition result set to determine a target special effect tag set;

[0038] The processing module is used to merge the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment.

[0039] In the above scheme, the at least one feature includes: voice features, facial expression features, and human body movement features;

[0040] The recognition module is used to use at least one preset recognition model to recognize the first target video segment and obtain a recognition result corresponding to each feature in the first target video segment.

[0041] In the above solution, the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence of each result;

[0042] The determination module is used to determine at least one result subset based on the confidence level of each result in the recognition results corresponding to each of the features from high to low; each of the result subsets includes: a result corresponding to at least one feature; and the confidence levels corresponding to the results included in each of the result subsets are the same;

[0043] According to the at least one result subset, query a preset special effect tag library to determine at least one target special effect tag;

[0044] The target special effect tag set is obtained according to the at least one target special effect tag.

[0045] In the above scheme, the determination module is specifically used to query a preset special effect label library according to the first result subset, and when it is determined that at least one result in the first result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the first result subset is used as the target special effect label; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except itself;

[0046] When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset;

[0047] When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

[0048] In the above scheme, the processing module is used to add the special effects corresponding to each target special effect tag in the target special effect tag set to the first target video segment to obtain the first target video segment with the special effects added as the second target video segment.

[0049] In the above solution, the acquisition module is used to identify the video data and determine the first video segment having at least the target object;

[0050] Extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment;

[0051] Correspondingly, the processing module is used to add the special effects corresponding to each target special effect tag in the target special effect tag set to the first video segment to obtain the first video segment with the special effects added as the second target video segment.

[0052] In the above solution, the device further includes: a preprocessing module, configured to perform at least one of the following:

[0053] Training a first recognition model; the first recognition model is used to extract and recognize facial expression features;

[0054] Training a second recognition model; the second recognition model is used to extract and recognize human motion features;

[0055] A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

[0056] An embodiment of the present invention provides a video processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above video processing methods when executing the program.

[0057] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above video processing methods are implemented.

[0058] A video processing method, device and storage medium provided by an embodiment of the present invention include: obtaining a first target video clip; performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set; merging the special effects corresponding to the target special effect tag set with the first target video clip to obtain a second target video clip; in this way, by recognizing multiple features, special effects corresponding to recognition results of different features are obtained, multiple special effects are added to the video, the video fun is enhanced, and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the structure of a facial expression feature recognition model provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of the structure of a human motion feature recognition model provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of a speech recognition and keyword extraction model provided by an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of the structure of a video special effect adding device provided by an embodiment of the present invention;

[0065] Figure 7 A schematic diagram of the structure of another video processing device provided by an embodiment of the present invention;

[0066] Figure 8 A schematic diagram of the structure of yet another video processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described in detail with reference to the embodiments, and the related technology will be described first.

[0068] A method for adding video special effects provided in the related art obtains image frames in the video and identifies human joints to determine the motion state of the human joints, thereby obtaining video special effects that match the motion of the human joints. The above scheme only considers the human motion information in the video, and ignores other information such as the user's facial expressions and voice. For video content with small motion amplitude or no significant motion, the recognition accuracy of this method is low and the scope of application is limited.

[0069] In another video special effect adding technology method, by acquiring the image frame in the video and identifying the motion pixels of the target user, the user motion pixels that meet the preset position conditions are screened and a target pixel set is generated. When the target pixel set matched by the current image frame and the target pixel set matched by the previous image frame meet the special effect adding conditions, a matching video special effect is added. This method can identify the moving user and add matching dynamic special effects to the video, but the application scenarios are relatively fixed and single, and it is difficult to meet the needs of diversified interactive scenarios.

[0070] In summary, most of the current methods for automatically adding special effects to videos are based on human body movements and motion features for recognition and addition, and lack attention to the facial expression status and voice information of the user in the video. Both of the above methods only start from the target user's movements or motion status, and have low recognition accuracy for users with small movements or no significant motion features in the video, resulting in inaccurate special effects addition and relatively single scenes, which cannot meet the diverse scene requirements in video interactive applications.

[0071] Based on this, the method provided by an embodiment of the present invention obtains a first target video clip; performs feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; queries a preset special effect tag library based on the recognition result set to determine a target special effect tag set; merges the special effects corresponding to the target special effect tag set with the first target video clip to obtain a second target video clip.

[0072] Figure 1 A schematic diagram of a video processing method provided by an embodiment of the present invention; Figure 1 As shown, the method includes:

[0073] Step 101: Obtain a first target video clip;

[0074] Step 102: performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature;

[0075] Step 103: query a preset special effect tag library according to the recognition result set to determine a target special effect tag set;

[0076] Step 104: Merge the special effects corresponding to the target special effects tag set with the first target video clip to obtain a second target video clip.

[0077] In practical applications, considering that the amount of original video data is large and contains more noise, it is not conducive to feature extraction and model construction and training. Therefore, effective video clips can be extracted and special effects can be added to the effective video clips.

[0078] Based on this, the step of obtaining the first target video segment includes:

[0079] Identify the video data to determine at least a first video segment having a target object;

[0080] Interval frame extraction is performed on the first video segment to obtain a second video segment with a preset number of frames as the first target video segment.

[0081] Here, by preprocessing and segmenting the original video data, multiple valid shot segments (i.e., the first video segment) are obtained; then, interval frame extraction is performed on the valid shot segments to obtain a video segment consisting of N frames of images of each valid shot, which is used as the first target video segment.

[0082] The target object may be a human face, a human body, or a human voice. The N frames may be set by a developer based on requirements, for example, N may be 24, 26, 28, etc.

[0083] In this way, if the target object includes human speaking voices, it is possible to avoid the situation where there is music in the video data but no effective voice content; if the target object includes both human faces and human speaking voices, it is possible to avoid the situation where there is only human speaking voice, for example, the user is speaking but the person has not arrived and the live broadcast has not started; if the target object includes human faces, human bodies, and human speaking voices, accurate and effective video can be obtained, avoiding any of the above situations.

[0084] Here, the target object may be selected by the user through the device he / she holds, or may be set by the developer.

[0085] The method provided by the embodiment of the present invention can be applied to devices such as servers and terminals; the terminal can be any device with a collection device (such as a camera, which has both video collection and voice collection functions, for example, a camera with a video collection function and a microphone with an audio collection function, etc.) or a device connected to a collection device; for example, a tablet computer, a smart phone, a laptop computer, etc.

[0086] When the method is applied to a terminal, the terminal can collect video data through its own collection device.

[0087] (including images and sounds); and use Figure 1 The method shown performs corresponding processing and sends the processing result to the server, which sends the processing result to the relevant terminal (taking live broadcast as an example, the relevant terminal is the terminal for watching the live broadcast video) for presentation.

[0088] When the method is applied to a server, the terminal can collect video data (including images and sounds) through its own collection device; and send the collected video data to the server, which performs corresponding processing using the method shown in Figure 1 and sends the processing results to relevant terminals (taking live broadcast as an example, the relevant terminals are the sending terminal and other terminals watching the live broadcast video) for presentation.

[0089] In some embodiments, the at least one feature includes: a voice feature, a facial expression feature, a human body movement feature;

[0090] The performing feature recognition on the first target video segment to obtain at least one recognition result set includes:

[0091] At least one preset recognition model is used to recognize the first target video segment, and a recognition result corresponding to each feature in the first target video segment is obtained.

[0092] Here, the at least one preset recognition model may include:

[0093] A first recognition model is used to extract and recognize facial expression features;

[0094] The second recognition model is used to extract and recognize human motion features;

[0095] The third recognition model is used to extract voice data and extract keywords (the keywords are voice features) from the voice data.

[0096] That is, the using of at least one preset recognition model to recognize the first target video segment and obtain a recognition result corresponding to each feature in the first target video segment includes at least one of the following:

[0097] Using a first recognition model to recognize the first target video clip, obtaining a first recognition result corresponding to a face in the first target video clip; the first recognition result may include: a recognition result corresponding to a facial expression feature;

[0098] Using the second recognition model to recognize the first target video clip, obtaining a second recognition result corresponding to the limb in the first target video clip; the second recognition result may include: a recognition result corresponding to a human body motion feature;

[0099] The first target video segment is recognized using a third recognition model to obtain a third recognition result corresponding to the voice data in the first target video segment; the third recognition result may include: keywords.

[0100] In some embodiments, the method further comprises at least one of the following:

[0101] Training a first recognition model; the first recognition model is used to extract and recognize facial expression features;

[0102] Training a second recognition model; the second recognition model is used to extract and recognize human motion features;

[0103] A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

[0104] Among them, the first recognition model is obtained based on the training of Convolutional Neural Networks (CNN) and Extreme Learning Machine (ELM);

[0105] The second recognition model is obtained by training based on CNN and long short-term memory network (LSTM);

[0106] The third recognition model is trained based on a bidirectional long short-term memory network (BiLSTM) and a connectionist temporal classifier (CTC).

[0107] In some embodiments, the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence level of each result;

[0108] The step of querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set includes:

[0109] Determine at least one result subset based on the confidence level of each result in the recognition results corresponding to each of the features from high to low; each of the result subsets includes: a result corresponding to at least one feature; and the confidence levels corresponding to the results included in each of the result subsets are the same;

[0110] According to the at least one result subset, query a preset special effect tag library to determine at least one target special effect tag;

[0111] The target special effect tag set is obtained according to the at least one target special effect tag.

[0112] Here, the step of querying a preset special effect tag library according to the at least one result subset to determine at least one target special effect tag includes:

[0113] When a preset special effect label library is queried according to the first result subset and it is determined that at least one result in the first result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the first result subset is used as the target special effect label; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except the first result subset;

[0114] When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset;

[0115] When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

[0116] The above steps are described in detail with reference to the following examples.

[0117] Input the preprocessed video clip to be tested (i.e., the first target video clip) into the model; specifically, input the first recognition model, the second recognition model, and the third recognition model respectively;

[0118] The outputs of the three models are obtained, including: the recognition result of the first recognition model (for facial expression recognition) E , the recognition result of the second recognition model (used for action recognition) m , the recognition result of the third recognition model (used to extract the keyword set after voice recognition) w .

[0119] The result with the highest score (that is, the highest confidence or the highest confidence level) in the recognition results of each model is selected to generate the first-level special effect label candidate subset T1; T1 specifically includes: E The result of scoring first in them The result of scoring first in the w The result with the highest score; the first layer of special effect label candidate subset, that is, the first result subset mentioned above;

[0120] The second-highest-scoring result (i.e., the second-highest confidence level or the second-highest confidence level) is selected from the recognition results of each model to generate a second-layer special effect label candidate subset T2; T2 specifically includes: E The result of scoring second, O m The result of scoring second, O w The result with the second highest score; the second-layer special effect label candidate subset, that is, the second result subset mentioned above;

[0121] Similarly, in this way, n layers of special effect label candidate subsets are generated; the specific value of n can be set according to actual conditions;

[0122] The final generated special effect label candidate set T is {T1, T2, ..., T n}, where T i = {O Ei , O mi ,O wi}, i=1,2,...n.

[0123] Traverse the special effect label candidate set T generated above, loop through the elements in each layer Ti in order from high to low according to the confidence level, and match them with the pre-stored special effect label library. If a certain layer is matched successfully (here, at least one element in a certain layer is matched successfully), the loop ends; otherwise, continue to loop through the special effect label candidate set until the end. Finally, you can start adding the video special effect corresponding to the matched special effect label at the starting frame of the corresponding video segment, and the duration is the special effect duration.

[0124] The special effect tag library includes: at least one special effect and a tag corresponding to each special effect; the characteristics can be dynamic image special effects, for example, the tag: laugh, the corresponding special effect: laugh image; the tag: clapping, the corresponding special effect: clapping image. Each dynamic image special effect also corresponds to a duration, etc.; the characteristics can also be other special effects such as music special effects, which is only used to illustrate an example and is not limited.

[0125] In some embodiments, another method is provided for querying a preset special effect tag library based on the at least one recognition result set to determine a target special effect tag set.

[0126] Here, the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence of each result; the confidence level of each recognition result set in the at least one recognition result set is the same;

[0127] The querying of a preset special effect tag library according to the at least one recognition result set to determine a target special effect tag set may also include:

[0128] For each of the features, based on the confidence level of each result in the recognition results corresponding to each of the features from high to low, query the preset special effect label library according to the corresponding results in turn, and determine the special effect label corresponding to the corresponding result that meets the preset conditions as the target special effect label;

[0129] A target special effect label set is obtained according to the target special effect label corresponding to at least one feature.

[0130] Here, for each feature, the preset special effect tag library is queried according to the corresponding results in turn, and the special effect tag corresponding to the corresponding result that meets the preset conditions is determined as the target special effect tag, including:

[0131] When a preset special effect tag library is queried according to the first result and it is determined that the first result corresponds to a special effect tag, the special effect tag corresponding to the first result is used as the target special effect tag; the confidence level of the first result is higher than the confidence levels of other results except the first result in the recognition results of the feature corresponding to the first result;

[0132] When it is determined that the first result does not correspond to a special effect tag, a preset special effect tag library is queried according to the second result, and when it is determined that the second result corresponds to a special effect tag, the special effect tag corresponding to the second result is used as the target special effect tag; the confidence level of the second result is lower than the confidence level of the first result, and higher than the confidence level of other results in the recognition result of the feature corresponding to the second result, except for itself and the first result;

[0133] When it is determined that the second result does not correspond to a special effects tag, a preset special effects tag library is queried according to the third result; and so on, until the target special effects tag is determined; the confidence level of the third result is lower than the confidence level of the first result and the confidence level of the second result, and higher than the confidence level of other results in the recognition results of the features corresponding to the third result except itself, the first result and the second result.

[0134] In this way, the special effects corresponding to each feature are obtained according to the recognition results corresponding to each feature, so that multiple special effects can be obtained. In addition, the special effect labels corresponding to the query results according to the confidence level can avoid the problem that the special effects obtained by the query are not the most suitable special effects, and can also avoid the situation where no special effects can be found.

[0135] In some embodiments, merging the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment includes:

[0136] The special effects corresponding to the target special effects tags in the target special effects tag set are added to the first target video segment to obtain the first target video segment with the special effects added thereto as the second target video segment.

[0137] Here, each frame of the image in the video data corresponds to a corresponding time point, so each frame of the image in the first target video segment also corresponds to a time point;

[0138] When adding the special effects corresponding to each target special effect tag in the target special effect tag set to the first target video clip, the image to which the special effects need to be added can be determined based on the time point of the identified frame image, and the duration is the special effect duration.

[0139] In this way, the corresponding special effect is added to the corresponding first target video segment to obtain a second target video segment with the special effect added.

[0140] It should be noted that, if the first target video segment is a video segment composed of some frames extracted from the first video segment, the initial time point and the end time point of the first target video segment can be determined, and special effects can be added to the segment corresponding to the first video segment in the original video data between the initial time point and the end time point;

[0141] If the first target video segment is a video segment extracted from video data (without interval frame extraction), special effects can be directly added to the segment corresponding to the first target video segment in the original video data between the initial time point and the end time point.

[0142] In some embodiments, a deep learning algorithm is used to jointly construct a first recognition model, a second recognition model, and a third recognition model. The following is an example of training the first recognition model, the second recognition model, and the third recognition model.

[0143] The first recognition model is a facial expression recognition model, which is used to extract and recognize the facial expression features of the user.

[0144] For the construction of facial expression recognition model, convolutional neural network (CNN) and extreme learning machine (ELM) are jointly used to construct it. First, the CNN network is used to extract high-dimensional space expression features, and then the extracted expression features are input into the ELM network, and the ELM network is trained to obtain the trained ELM network as the facial expression recognition model. Here, ELM is a generalized single hidden layer feedforward neural network. The main feature is that the connection weights between the network input layer and the hidden layer can be randomly assigned without iterative adjustment. It is faster than traditional learning algorithms while ensuring learning accuracy. The construction process of the facial expression recognition model based on CNN-ELM is as follows: Figure 2 shown.

[0145] Figure 2 The facial expression features extracted by CNN are represented as X(x1,x2…x n ),Y(y1,y2…y n ) is the true label of the training data, and n is the number of training samples. In network training, a multiquadric function (a radial basis function) is used as the activation function of the entire network. Since the ELM network is a single hidden layer feedforward neural network, the loss function of the entire network can be represented by a linear system and solved by the least squares method. Compared with simply using CNN for feature extraction and model training, the method proposed in the embodiment of the present invention is faster and more efficient.

[0146] The second recognition model is an action recognition model, which is used to extract and recognize human action features. For the construction of the action recognition model, a deep learning model of convolutional neural network-long short-term memory network (CNN-LSTM) is adopted. Here, considering that the human actions in the video frames not only have changes in spatial position, but also have a certain connection with the context of the current frame, the deep learning model of CNN-LSTM is adopted. Among them, the CNN network is used to capture the spatial domain features of human actions, and the LSTM network is used to construct the temporal relationship of human actions in video frames. The joint learning of spatial domain information and temporal domain information is more efficient for the recognition of human actions in video frames. The specific model construction process is as follows: Figure 3 shown.

[0147] Figure 3 In LSTM, as a recursive neural network, it is mainly composed of some gate units, including core self-connection units, input gates, output gates, and forget gates. These gate units control the retention of valid information and the removal of invalid information respectively. Usually, the update method of an LSTM unit at time t is:

[0148] it =σ(W xi x t +W hi h t-1 +W ci c t-1 +b i )

[0149] f t =σ(W xf x t +W hf h t-1 +W cf c t-1 +b f )

[0150] o t =σ(W xo x t +W ho h t-1 +W co c t-1 +b o )

[0151] c t =f t c t-1 +i t tanh(W xc x t +W hc h t-1 +b c )

[0152] h t =o t tanh(c t )

[0153] Among them, σ(·) represents the sigmoid function, x t represents the input information at time t, i t represents the overall input information at time t, f t Indicates choosing to forget some information before time t, o t Indicates the information currently to be output, c t Indicates merging and updating the information between time t and the current information, h t represents all valuable information at time t and before, all W matrices represent the connection weights between two nodes, b i ,b f ,b o ,b c represents the bias vector.

[0154] In the overall model structure, 6 convolutional layers are used to extract the spatial features of different human body movements; 2 LSTM layers are used to establish associations between the feature vectors of action samples in the previous and next time frames; 3 fully connected layers are used to classify and identify the fused action sample feature vectors. The activation function of the last fully connected layer (FC, fully connected layers) is the SoftMax function, which predicts and outputs the probability values ​​of each category to which the current action belongs.

[0155] In the recognition results, different results correspond to different confidence levels, such as 0.8 (corresponding to "open"), 0.4 (corresponding to "jump"), ..., 0.3, 0.1, etc. The higher the confidence level, the higher the confidence level.

[0156] The third recognition model may actually include: a speech recognition model and a keyword extraction model; the speech recognition model is used to extract speech data; and the keyword extraction model is used to extract keywords from the speech data.

[0157] Regarding the construction of speech recognition and keyword extraction models, such as Figure 4 As shown, first, Mel frequency cepstral coefficients (MFCC) are used to extract audio features, and then, a BiLSTM-CTC neural network (i.e., a BiLSTM neural network and a connectionist temporal classifier (CTC)) are used to build an acoustic model. The bidirectional LSTM network can not only capture the audio features of historical frames, but also capture the information of future frames, which is more effective for modeling contextual problems such as speech. By linking the connectionist temporal classifier CTC, training can be performed without aligning labels at the frame level, and the CTC network output also includes a "Blank" node, which can effectively handle silence, pauses between words, and confusion between words. The model established by combining the BiLSTM network with CTC has a higher accuracy for speech recognition.

[0158] After audio recognition is converted into text, it is input into the keyword extraction (TextRank) model to obtain the main voice information expressed by the user in the current video. Voice information is also a reflection of the user's emotions and status. For example, the extracted keywords expressed by the user include useful, like, and heart, which can be used as candidate tags for adding video special effects, thereby enriching the video special effects adding scenes, making the video content more interactive and interesting, and better improving the user experience.

[0159] Figure 5 A schematic diagram of the structure of a video special effect adding device provided by an embodiment of the present invention; Figure 5As shown, the device includes: a video data processing module, a multimodal feature extraction module, a deep model construction module, a model prediction module, and a special effect adding module; wherein,

[0160] The video data processing module is used to identify the original video data and determine a first video segment having at least a target object; extract interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment.

[0161] The multimodal feature extraction module is used to perform feature recognition on the first target video segment to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature.

[0162] The multimodal feature extraction module is specifically used to extract and identify human motion features and facial expression features respectively using the first recognition model and the second recognition model for the first target video segment obtained by the video data processing module, and to extract speech features, i.e., keywords, using the third recognition model.

[0163] The deep model building module is used to build a model for feature recognition; the specific steps refer to Figure 2-4 The construction method corresponding to the model will not be repeated here.

[0164] The model prediction module is used to query the preset special effect label library according to the corresponding results for each of the features based on the confidence level of each result in the recognition results corresponding to each of the features from high to low, and determine the special effect label corresponding to the corresponding result that meets the preset conditions as the target special effect label; according to the target special effect label corresponding to at least one feature, a target special effect label set is obtained. For specific steps, refer to Figure 1 The instructions in the method shown will not be repeated here.

[0165] The special effect adding module is used to merge the special effects corresponding to the target special effect tag set with the first target video segment to obtain a second target video segment. Figure 1 The instructions in the method shown will not be repeated here.

[0166] When the video processing device provided in the above embodiment implements the corresponding video processing method, only the division of the above program modules is used as an example for explanation. In actual application, the above processing can be assigned to different program modules as needed.

[0167] Figure 6 A structural schematic diagram of another video special effect adding device provided by an embodiment of the present invention; as shown in FIG6 , the device may include:

[0168] A facial expression recognition module, used to extract and recognize facial expression features in a target video using a preset first recognition model;

[0169] A human action recognition module, used to extract and recognize human action features in a target video using a preset second recognition model;

[0170] A speech recognition and keyword extraction module, used to extract speech data of a target video using a preset third recognition model, and extract keywords from the speech data;

[0171] The special effects label candidate set generation module is used to query the preset special effects label library according to the recognition result set to determine the video special effects label candidate set; here, the recognition results include: recognition results corresponding to facial expression features, recognition results corresponding to human body movement features, and recognition results corresponding to keyword features.

[0172] A video special effect adding module, used for adding the special effects corresponding to the video special effect label candidate set into the target video to obtain the target video after adding the special effects;

[0173] The video special effects preview and save module is used to preview and / or save the target video after adding special effects.

[0174] Here, the emotions and behaviors of users in the target video are understood through the facial expression recognition module, human action recognition module, speech recognition and keyword extraction module to generate a corresponding set of special effect label candidates; the video special effect adding module will match the corresponding special effects from the preset special effect label library and add them to the video; finally, the preview module will preview and save the video content after the special effects are added.

[0175] Figure 7 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention; Figure 7 As shown, the device comprises:

[0176] An acquisition module, used for acquiring a first target video segment;

[0177] An identification module, configured to perform feature identification on the first target video segment to obtain an identification result set; the identification result set includes an identification result corresponding to at least one feature;

[0178] A determination module, used to query a preset special effect tag library according to the recognition result set to determine a target special effect tag set;

[0179] The processing module is used to merge the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment.

[0180] In some embodiments, the at least one feature includes: a voice feature, a facial expression feature, a human body movement feature;

[0181] The recognition module is used to use at least one preset recognition model to recognize the first target video segment and obtain a recognition result corresponding to each feature in the first target video segment.

[0182] In some embodiments, the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence level of each result;

[0183] The determination module is used to determine at least one result subset based on the confidence level of each result in the recognition results corresponding to each of the features from high to low; each of the result subsets includes: a result corresponding to at least one feature; and the confidence levels corresponding to the results included in each of the result subsets are the same;

[0184] According to the at least one result subset, query a preset special effect tag library to determine at least one target special effect tag;

[0185] The target special effect tag set is obtained according to the at least one target special effect tag.

[0186] In some embodiments, the determination module is specifically used to query a preset special effect tag library according to the first result subset, and when it is determined that at least one result in the first result subset corresponds to a special effect tag, use the special effect tag corresponding to at least one result in the first result subset as the target special effect tag; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except itself;

[0187] When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset;

[0188] When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

[0189] In some embodiments, the processing module is used to add the special effects corresponding to each target special effect tag in the target special effect tag set to the first target video segment to obtain the first target video segment with the special effects added as the second target video segment.

[0190] In some embodiments, the acquisition module is used to identify the video data and determine a first video segment having at least a target object;

[0191] Extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment;

[0192] Correspondingly, the processing module is used to add the special effects corresponding to each target special effect tag in the target special effect tag set to the first video segment to obtain the first video segment with the special effects added as the second target video segment.

[0193] In some embodiments, the apparatus further comprises: a preprocessing module, configured to perform at least one of the following:

[0194] Training a first recognition model; the first recognition model is used to extract and recognize facial expression features;

[0195] Training a second recognition model; the second recognition model is used to extract and recognize human motion features;

[0196] A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

[0197] It should be noted that: the video processing device provided in the above embodiment only uses the division of the above program modules as an example to illustrate when implementing the corresponding video processing method. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the server or terminal is divided into different program modules to complete all or part of the processing described above. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0198] Figure 8 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention; Figure 8As shown, the device 80 includes: a processor 801 and a memory 802 for storing a computer program that can be run on the processor; when the processor 801 is used to run the computer program, it executes: obtaining a first target video clip; performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set; merging the special effects corresponding to the target special effect tag set with the first target video clip to obtain a second target video clip. Wherein, the at least one feature includes: voice features, facial expression features, and human body movement features.

[0199] In one embodiment, the processor 801 is further configured to execute, when running the computer program: using at least one preset recognition model to recognize the first target video segment, and obtaining a recognition result corresponding to each of the features in the first target video segment. The recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence of each result.

[0200] In one embodiment, the processor 801 is also used to execute the following when running the computer program: determining at least one result subset based on the confidence level of each result in the recognition results corresponding to each of the features from high to low; each of the result subsets includes: a result corresponding to at least one feature; the confidence levels corresponding to the results included in each of the result subsets are the same; based on the at least one result subset, querying a preset special effect tag library to determine at least one target special effect tag; and obtaining the target special effect tag set based on the at least one target special effect tag.

[0201] In one embodiment, the processor 801 is further used to execute, when running the computer program: querying a preset special effect tag library according to the first result subset, and when determining that at least one result in the first result subset corresponds to a special effect tag, using the special effect tag corresponding to at least one result in the first result subset as the target special effect tag; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except itself;

[0202] When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset;

[0203] When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

[0204] In one embodiment, the processor 801 is also used to execute, when running the computer program: adding the special effects corresponding to each target special effects tag in the target special effects tag set to the first target video clip, to obtain the first target video clip with the special effects added as the second target video clip.

[0205] In one embodiment, the processor 801 is further configured to execute, when running the computer program: identifying the video data to determine a first video segment having at least a target object; extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment;

[0206] Correspondingly, in one embodiment, the processor 801 is also used to execute, when running the computer program: adding the special effects corresponding to each target special effects tag in the target special effects tag set to the first video clip, to obtain the first video clip with the special effects added, as the second target video clip.

[0207] In one embodiment, the processor 801 is further configured to execute, when running the computer program: training a first recognition model; the first recognition model is used to extract and recognize facial expression features;

[0208] Training a second recognition model; the second recognition model is used to extract and recognize human motion features;

[0209] A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

[0210] In actual application, the device 80 may further include: at least one network interface 803. The various components in the device 80 are coupled together via a bus system 804. It is understood that the bus system 804 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 In the figure, various buses are marked as bus system 804. There may be at least one processor 801. The network interface 803 is used for wired or wireless communication between the apparatus 80 and other devices.

[0211] The memory 802 in the embodiment of the present invention is used to store various types of data to support the operation of the device 80 .

[0212] The method disclosed in the above embodiment of the present invention can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 801 or the instruction in the form of software. The above processor 801 can be a general processor, a digital signal processor (DSP, DiGital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 802. The processor 801 reads the information in the memory 802 and completes the steps of the above method in combination with its hardware.

[0213] In an exemplary embodiment, the device 80 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0214] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored; when the computer program is executed by a processor, the computer program executes: obtaining a first target video clip; performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set; merging the special effects corresponding to the target special effect tag set with the first target video clip to obtain a second target video clip. The at least one feature includes: voice features, facial expression features, and human body movement features.

[0215] In one embodiment, when the computer program is executed by the processor, the computer program performs: using at least one preset recognition model to recognize the first target video segment, and obtaining a recognition result corresponding to each feature in the first target video segment. The recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence of each result.

[0216] In one embodiment, when the computer program is executed by the processor, the following is performed: determining at least one result subset based on the confidence level of each result in the recognition results corresponding to each of the features from high to low; each of the result subsets includes: a result corresponding to at least one feature; the confidence levels corresponding to the results included in each of the result subsets are the same; based on the at least one result subset, querying a preset special effect tag library to determine at least one target special effect tag; and obtaining the target special effect tag set based on the at least one target special effect tag.

[0217] In one embodiment, when the computer program is executed by a processor, the following is performed: querying a preset special effect tag library according to the first result subset, and when determining that at least one result in the first result subset corresponds to a special effect tag, using the special effect tag corresponding to at least one result in the first result subset as the target special effect tag; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except the first result subset;

[0218] When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset;

[0219] When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

[0220] In one embodiment, when the computer program is executed by the processor, the following is performed: adding the special effects corresponding to each target special effect tag in the target special effect tag set to the first target video segment to obtain the first target video segment with the special effects added as the second target video segment.

[0221] In one embodiment, when the computer program is executed by a processor, the computer program performs: identifying the video data to determine a first video segment having at least a target object;

[0222] Extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment;

[0223] Correspondingly, in one embodiment, when the computer program is executed by the processor, the following is performed: adding the special effects corresponding to each target special effect tag in the target special effect tag set to the first video clip, and obtaining the first video clip with the special effects added as the second target video clip.

[0224] In one embodiment, when the computer program is executed by a processor, the computer program performs: training a first recognition model; the first recognition model is used to extract and recognize facial expression features;

[0225] Training a second recognition model; the second recognition model is used to extract and recognize human motion features;

[0226] A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0230] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0231] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0232] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: The method comprises: Obtaining a first target video clip; Performing feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; the at least one feature includes: a voice feature, a facial expression feature, and a human body movement feature; the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence level of each result; Querying a preset special effect tag library according to the recognition result set to determine a target special effect tag set; The special effects corresponding to the target special effects tag set are merged with the first target video clip to obtain a second target video clip; wherein the step of querying a preset special effects tag library according to the recognition result set to determine the target special effects tag set includes: Determine at least one result subset based on the confidence level of each result in the recognition results corresponding to each feature from high to low; each of the result subsets includes: a result corresponding to at least one feature; and the confidence levels corresponding to the results included in each of the result subsets are the same; According to the at least one result subset, query a preset special effect tag library to determine at least one target special effect tag; According to the at least one target special effect tag, the target special effect tag set is obtained; wherein, according to the at least one result subset, a preset special effect tag library is queried to determine at least one target special effect tag, including: When a preset special effect label library is queried according to the first result subset and it is determined that at least one result in the first result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the first result subset is used as the target special effect label; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except the first result subset; When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset; When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

2. The method according to claim 1, characterized in that The performing feature recognition on the first target video segment to obtain a recognition result set includes: At least one preset recognition model is used to recognize the first target video segment, and a recognition result corresponding to each of the features in the first target video segment is obtained.

3. The method according to claim 1, characterized in that The step of merging the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment includes: The special effects corresponding to the target special effects tags in the target special effects tag set are added to the first target video segment to obtain the first target video segment with the special effects added thereto as the second target video segment.

4. The method according to claim 1, characterized in that: The obtaining of the first target video segment comprises: Identify the video data to determine at least a first video segment having a target object; Extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment; Accordingly, merging the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment includes: The special effects corresponding to the target special effects tags in the target special effects tag set are added to the first video segment to obtain the first video segment with the special effects added thereto as the second target video segment.

5. The method according to claim 2, characterized in that: The method further comprises at least one of the following: Training a first recognition model; the first recognition model is used to extract and recognize facial expression features; Training a second recognition model; the second recognition model is used to extract and recognize human motion features; A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

6. A video processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a first target video segment; A recognition module, configured to perform feature recognition on the first target video clip to obtain a recognition result set; the recognition result set includes a recognition result corresponding to at least one feature; the at least one feature includes: a speech feature, a facial expression feature, and a human action feature; the recognition result includes: at least one result and a confidence level corresponding to each result; the confidence level is determined based on the confidence level of each result; A determination module, used to query a preset special effect tag library according to the recognition result set to determine a target special effect tag set; The processing module is used to merge the special effects corresponding to the target special effects tag set with the first target video segment to obtain a second target video segment; wherein the determination module is used to determine at least one result subset based on the confidence level of each result in the recognition results corresponding to each feature from high to low; each of the result subsets includes: a result corresponding to at least one feature; and the confidence levels corresponding to the results included in each of the result subsets are the same; According to the at least one result subset, query a preset special effect tag library to determine at least one target special effect tag; According to the at least one target special effect tag, the target special effect tag set is obtained; wherein the determination module is specifically used to query a preset special effect tag library according to the first result subset, and when it is determined that at least one result in the first result subset corresponds to a special effect tag, the special effect tag corresponding to at least one result in the first result subset is used as the target special effect tag; the confidence level corresponding to the first result subset is higher than the confidence levels corresponding to other result subsets except itself; When it is determined that no result in the first result subset corresponds to a special effect label, a preset special effect label library is queried according to the second result subset, and when it is determined that at least one result in the second result subset corresponds to a special effect label, the special effect label corresponding to at least one result in the second result subset is used as the target special effect label; the confidence level corresponding to the second result subset is lower than the confidence level corresponding to the first result subset, and higher than the confidence levels corresponding to other result subsets except itself and the first result subset; When it is determined that the results in the second result subset do not correspond to any special effect labels, the preset special effect label library is queried according to the third result subset, and so on, until the target special effect label is determined; the confidence level corresponding to the third result subset is lower than the confidence level corresponding to the first result subset and the confidence level corresponding to the second result subset, and is higher than the confidence levels corresponding to other result subsets except itself, the first result subset and the second result subset.

7. The device according to claim 6, characterized in that The recognition module is used to use at least one preset recognition model to recognize the first target video segment and obtain a recognition result corresponding to each feature in the first target video segment.

8. The device according to claim 6, characterized in that The processing module is used to add the special effects corresponding to each target special effect tag in the target special effect tag set to the first target video segment to obtain the first target video segment with the special effects added as the second target video segment.

9. The device according to claim 6, characterized in that The acquisition module is used to identify the video data and determine a first video segment having at least a target object; Extracting interval frames from the first video segment to obtain a second video segment with a preset number of frames as the first target video segment; Correspondingly, the processing module is used to add the special effects corresponding to each target special effect tag in the target special effect tag set to the first video segment to obtain the first video segment with the special effects added as the second target video segment.

10. The device according to claim 7, characterized in that The device further comprises: a preprocessing module, configured to perform at least one of the following: Training a first recognition model; the first recognition model is used to extract and recognize facial expression features; Training a second recognition model; the second recognition model is used to extract and recognize human motion features; A third recognition model is trained; the third recognition model is used to extract voice data and extract keywords from the voice data.

11. A video processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images