Video Processing Method and Apparatus, Electronic Device, and Computer-Readable Storage Medium

Abstract

The present disclosure relates to a video processing method, apparatus, electronic device, and computer-readable storage medium. Among them, the method includes: inputting an object video into an emotion prediction model to obtain an emotion feature curve of the object video, where the emotion feature curve represents the changes in emotion type and emotion intensity of the object video over time; obtaining the difference degree between the emotion feature curve of the object video and a target emotion feature curve; and when the difference degree is less than a first threshold, determining the object video as a target video. The present invention solves the technical problem in the related art that the error is relatively large when predicting a target video manually based on personal experience.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular, to a video processing method and apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] Nowadays, with the development of network technology, when releasing various types of videos such as movies, TV dramas, and variety shows through the network, it is often necessary to predict the playback volume of the upcoming video, and based on the playback volume, predict whether the upcoming video is a target video (for example, a hit) and be loved by the audience. However, in the related art, the playback volume of the video is usually predicted manually. However, predicting the target video based on the experience of the video content editors themselves will result in a large error and inaccurate prediction results.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] The present disclosure provides a video processing method and apparatus, an electronic device, and a computer-readable storage medium to at least solve the technical problem of large errors in predicting the target video by manual means based on personal experience. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a video processing method, including: inputting an object video into an emotion prediction model to obtain an emotion feature curve of the object video, where the emotion feature curve characterizes the change of the object video in terms of emotion type and emotion intensity over time; obtaining a difference degree between the emotion feature curve of the object video and a target emotion feature curve; and determining the object video as a target video when the difference degree is less than a first threshold.

[0006] Optionally, the obtaining a difference degree between the emotion feature curve of the object video and a target emotion feature curve includes: using a method similar to the dynamic time warping (DTW) method to determine a distance between the emotion feature curve of the object video and the target emotion feature curve; and using the distance as the difference degree between the emotion feature curve of the object video and the target emotion feature curve.

[0007] Optionally, inputting the object video into the emotion prediction model to obtain the emotion feature curve of the object video includes: for the video content of multiple time units included in the object video, respectively extracting the image features, audio features, and text features in the video content of the multiple time units through the feature extraction module in the emotion prediction model; respectively fusing the image features, audio features, and text features corresponding to the video content of the multiple time units through the fully connected layer network in the emotion prediction model to obtain fused features, and respectively predicting the emotion type and emotion intensity of the video content of the multiple time units based on the fused features; establishing the emotion feature curve of the object video based on the emotion type and emotion intensity of the video content of the multiple time units, and the time sequence features of the multiple time units.

[0008] Optionally, for the video content of multiple time units included in the object video, respectively extracting the image features, audio features, and text features in the video content of the multiple time units through the feature extraction module in the emotion prediction model includes: for the video content of any target time unit in the video content of the multiple time units: extracting the target image feature corresponding to the video content of the target time unit through the image feature extraction module in the feature extraction module, and integrating the target image feature and the historical image feature corresponding to the video content of the historical time unit before the target time unit through the first long short-term memory (LSTM) network to obtain the image feature corresponding to the video content of the target time unit; extracting the target audio feature corresponding to the video content of the target time unit through the audio feature extraction module in the feature extraction module, and integrating the target audio feature and the historical audio feature corresponding to the video content of the historical time unit before the target time unit through the second LSTM network to obtain the audio feature corresponding to the video content of the target time unit; extracting the target text feature corresponding to the video content of the target time unit through the text feature extraction module in the feature extraction module, and integrating the target text feature and the historical text feature corresponding to the video content of the historical time unit before the target time unit through the third LSTM network to obtain the text feature corresponding to the video content of the target time unit.

[0009] Optionally, the image features include at least one of the following: the scene feature of the video content, the expression feature of the video content, and the action feature of the video content.

[0010] Optionally, before obtaining the difference degree between the emotional feature curve of the object video and the target emotional feature curve, it further includes: obtaining a plurality of first historical target videos within a first predetermined historical time period; inputting the plurality of first historical target videos into the emotion prediction model to respectively obtain the emotional feature curves of the plurality of first historical target videos; clustering the emotional feature curves of the plurality of first historical target videos to obtain a clustering center curve; and in the case where the number of training samples corresponding to the clustering center curve for training the emotion prediction model is greater than or equal to a second threshold, determining the target emotional feature curve according to the clustering center curve.

[0011] Optionally, determining the target emotional feature curve according to the clustering center curve includes: obtaining a plurality of second historical target videos within a second predetermined historical time period, where the second predetermined historical time period is later than the first predetermined historical time period; inputting the plurality of second historical target videos into the emotion prediction model to respectively obtain the emotional feature curves of the plurality of second historical target videos; determining whether the clustering center curve is outdated according to the emotional feature curves of the plurality of second historical target videos; and in the case where it is determined that the clustering center curve is outdated, obtaining a new clustering center curve and determining the new clustering center curve as the target emotional feature curve.

[0012] Optionally, determining whether the clustering center curve is outdated according to the emotional feature curves of the plurality of second historical target videos includes: using the DTW method to respectively determine the distances between the emotional feature curves of the plurality of second historical target videos and the clustering center curve; determining the average value of the distances between the emotional feature curves of the plurality of second historical target videos and the clustering center curve; and in the case where the average value is greater than a third threshold, determining that the clustering center curve is outdated.

[0013] Optionally, the method further includes: in the case where the number of training samples corresponding to the clustering center curve for training the emotion prediction model is less than the second threshold, supplementing the training samples for training the emotion prediction model and then retraining the emotion prediction model to obtain an updated emotion prediction model until the updated emotional feature curves of the plurality of first historical target videos are predicted by using the updated emotion prediction model, clustering the updated emotional feature curves of the obtained plurality of first historical target videos to obtain an updated clustering center curve, and in the case where the number of training samples corresponding to the updated clustering center curve for training the updated emotion prediction model is greater than or equal to the second threshold, determining the target emotional feature curve according to the updated clustering center curve, and replacing the emotion prediction model with the finally obtained updated emotion prediction model.

[0014] Optionally, the target video includes a popular video in which the amount of a predetermined behavioral operation exceeds a predetermined value.

[0015] According to a second aspect of the embodiments of the present disclosure, there is provided a video processing apparatus, including: a first processing module configured to input an object video into an emotion prediction model to obtain an emotion feature curve of the object video, where the emotion feature curve characterizes the changes in emotion type and emotion intensity of the object video over time; a first acquisition module configured to acquire a difference degree between the emotion feature curve of the object video and a target emotion feature curve; and a first determination module configured to determine the object video as a target video when the difference degree is less than a first threshold.

[0016] Optionally, the first acquisition module includes: a first determination unit configured to determine a distance between the emotion feature curve of the object video and the target emotion feature curve by using a method similar to the dynamic time warping (DTW) method; and a first processing unit configured to use the distance as the difference degree between the emotion feature curve of the object video and the target emotion feature curve.

[0017] Optionally, the first processing module includes: a first extraction unit configured to, for video content of a plurality of time units included in the object video, respectively extract image features, audio features, and text features in the video content of the plurality of time units through a feature extraction module in the emotion prediction model; a first fusion unit configured to, through a fully connected layer network in the emotion prediction model, respectively fuse the image features, the audio features, and the text features corresponding to the video content of the plurality of time units to obtain fusion features, and respectively predict the emotion type and emotion intensity of the video content of the plurality of time units based on the fusion features; and a first establishment unit configured to establish an emotion feature curve of the object video based on the emotion type and emotion intensity of the video content of the plurality of time units, and the timing features of the plurality of time units.

[0018] Optionally, the first extraction unit includes: a first processing subunit, configured to, for the video content of any target time unit in the video content of the multiple time units, extract the target image features corresponding to the video content of the target time unit through the image feature extraction module in the feature extraction module, and integrate the target image features and the historical image features corresponding to the video content of the historical time units before the target time unit through a first long short-term memory (LSTM) network to obtain the image features corresponding to the video content of the target time unit; a second processing subunit, configured to, for the video content of any target time unit in the video content of the multiple time units, extract the target audio features corresponding to the video content of the target time unit through the audio feature extraction module in the feature extraction module, and integrate the target audio features and the historical audio features corresponding to the video content of the historical time units before the target time unit through a second LSTM network to obtain the audio features corresponding to the video content of the target time unit; a third processing subunit, configured to, for the video content of any target time unit in the video content of the multiple time units, extract the target text features corresponding to the video content of the target time unit through the text feature extraction module in the feature extraction module, and integrate the target text features and the historical text features corresponding to the video content of the historical time units before the target time unit through a third LSTM network to obtain the text features corresponding to the video content of the target time unit.

[0019] Optionally, the image features include at least one of the following: the scene features of the video content, the expression features of the video content, and the action features of the video content.

[0020] Optionally, the apparatus further includes: a second acquisition module, configured to acquire a plurality of first historical target videos within a first predetermined historical time period before acquiring the difference degree between the emotional feature curve of the target video and the target emotional feature curve; a second processing module, configured to input the plurality of first historical target videos into the emotion prediction model to respectively obtain the emotional feature curves of the plurality of first historical target videos; a clustering module, configured to cluster the emotional feature curves of the plurality of first historical target videos to obtain a clustering center curve; a second determination module, configured to, when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is greater than or equal to a second threshold, determine the target emotional feature curve according to the clustering center curve.

[0021] Optionally, the second determination module includes: a first acquisition unit configured to acquire a plurality of second historical target videos within a second predetermined historical time period, where the second predetermined historical time period is later than the first predetermined historical time period; a second processing unit configured to input the plurality of second historical target videos into the emotion prediction model to respectively obtain emotion feature curves of the plurality of second historical target videos; a second determination unit configured to determine whether the clustering center curve is outdated according to the emotion feature curves of the plurality of second historical target videos; and a third determination unit configured to, when determining that the clustering center curve is outdated, acquire a new clustering center curve and determine the new clustering center curve as the target emotion feature curve.

[0022] Optionally, the second determination unit includes: a first determination subunit configured to respectively determine distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve by using the DTW method; a second determination subunit configured to determine an average value of the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve; and a third determination subunit configured to determine that the clustering center curve is outdated when the average value is greater than a third threshold.

[0023] Optionally, the apparatus further includes: a third determination module configured to, when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is less than the second threshold, supplement the training samples for training the emotion prediction model and then retrain the emotion prediction model to obtain an updated emotion prediction model until updated emotion feature curves of the plurality of first historical target videos are predicted by using the updated emotion prediction model, an updated clustering center curve is obtained after clustering the updated emotion feature curves of the obtained plurality of first historical target videos, and when the number of training samples for training the updated emotion prediction model corresponding to the updated clustering center curve is greater than or equal to the second threshold, determine the target emotion feature curve according to the updated clustering center curve, and replace the emotion prediction model with the finally obtained updated emotion prediction model.

[0024] Optionally, the target video includes a popular video with a predetermined behavior operation amount exceeding a predetermined value.

[0025] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the instructions to implement the video processing method as described in any one of the above.

[0026] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the video processing method as described in any one of the above.

[0027] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, such that when instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the video processing method as described in any one of the above.

[0028] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0029] Input an object video into an emotion prediction model to obtain the emotion feature curve of the object video, where the emotion feature curve characterizes the changes in emotion type and emotion intensity of the object video over time; obtain the difference degree between the emotion feature curve of the object video and the target emotion feature curve; in the case where the difference degree is less than a first threshold, determine the object video as the target video; obtain the emotion feature curve degree of the object video through the emotion prediction model, and by comparing the obtained emotion feature curve with the target emotion feature curve, it is possible to accurately predict whether the object video is the target video; thereby solving the technical problem of large errors in the related art due to manually predicting the target based on personal experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure and do not constitute an improper limitation of the present disclosure.

[0031] Figure 1 is a schematic diagram of an application environment of an optional video processing method according to an embodiment of the present invention;

[0032] Figure 2 is a flowchart of an optional video processing method according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of an emotion feature curve of an optional embodiment according to an embodiment of the present invention;

[0034] Figure 4 is a flowchart of another optional video processing method according to an embodiment of the present invention;

[0035] Figure 5 is a flowchart of updating a popular video in an optional video processing method according to an embodiment of the present invention;

[0036] Figure 6It is a flowchart for predicting popular videos in an optional video processing method according to an embodiment of the present invention;

[0037] Figure 7 It is a schematic structural diagram of an optional video processing device according to an embodiment of the present invention;

[0038] Figure 8 It is a schematic diagram of an optional terminal according to an embodiment of the present invention;

[0039] Figure 9 It is a schematic diagram of an optional server according to an embodiment of the present invention. Detailed implementation manners

[0040] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.

[0042] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] In the embodiments of the present disclosure, the following technical terms may be included but are not limited to:

[0044] User Generated Content, UGC: User-generated content;

[0045] Convolutional Neural Network, CNN: Convolutional neural network;

[0046] Dynamic Time Warping, DTW: Dynamic time warping;

[0047] Mel-scale Frequency Cepstral Coefficients, MFCC: Mel-frequency cepstral coefficients;

[0048] Long short-term memory, LSTM: Long short-term memory;

[0049] Optical Character Recognition, OCR: Optical character recognition;

[0050] According to one aspect of an embodiment of the present invention, a video processing method is provided. Optionally, as an alternative implementation, the above video processing method may, but is not limited to, be applied to a video processing system in a hardware environment as shown in Figure 1 the video processing system shown. Figure 1 is a schematic diagram of an optional video processing system according to an embodiment of the present invention. As shown in Figure 1 the figure, the video processing system may include, but is not limited to, a terminal device 102, a network 104, and a server 106. The number of terminal devices 102 here may include, but is not limited to, multiple hardware devices. A client logged in with a target user account runs on the terminal device 102 (as shown in Figure 1 the figure, the client is used to obtain the target video). The above terminal device 102 includes a human-computer interaction screen 1022, a processor 1024, and a memory 1026. The server 106 includes a database 1062 and a processing engine 1064.

[0051] Optionally, in this embodiment, the above terminal device 102 may be a terminal device configured with a target client, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android mobile phone, an iOS mobile phone, etc.), a laptop computer, a tablet computer, a handheld computer, a MID (Mobile Internet Devices, mobile Internet device), a PAD, a desktop computer, a smart TV, etc. The target client may be a video client, an instant messaging client, a browser client, an education client, etc., which are used to receive the virtual resources delivered. The above network 104 may include, but is not limited to: a wired network, a wireless network. Among them, the wired network includes: a local area network, a metropolitan area network, and a wide area network, and the wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication. The above server 106 may be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example, and this embodiment does not make any limitation thereto.

[0052] As an alternative embodiment, Figure 2 is a flowchart of an optional video processing method according to an embodiment of the present invention. As shown in Figure 2 the figure, the method includes the following processes:

[0053] Step S21, input the target video into the emotion prediction model to obtain the emotion feature curve of the target video, where the emotion feature curve characterizes the changes in the emotion type and emotion intensity of the target video over time;

[0054] Step S22, obtain the difference degree between the emotional feature curve of the object video and the target emotional feature curve;

[0055] Step S23, when the difference degree is less than the first threshold, determine the object video as the target video.

[0056] Through the above processing, input the object video into the emotion prediction model to obtain the emotional feature curve of the object video. Among them, the emotional feature curve characterizes the changes in the emotion type and emotion intensity of the object video over time; obtain the difference degree between the emotional feature curve of the object video and the target emotional feature curve; when the difference degree is less than the first threshold, determine the object video as the target video; obtain the emotional feature curve degree of the object video through the emotion prediction model, and compare the obtained emotional feature curve with the target emotional feature curve. Since the emotional feature curve degree can more accurately characterize the changes in the emotion type and emotion intensity of the object video over time, it can more accurately predict whether the object video is the target video; thus solving the technical problem of large errors in the related technology due to manual prediction based on personal experience, and thus achieving the effect of efficiently and accurately predicting the target video.

[0057] As an optional embodiment, the object video can be video files in different formats, such as MP4, RMVB, AVI format videos. The types of object videos can include movies, TV series, variety shows, animations or sports programs, etc., and no specific limitations are made here.

[0058] As an optional embodiment, the emotion type can include but is not limited to different emotion type elements such as anger, fear, happiness, love, sadness, surprise, etc.; the emotion intensity can include but is not limited to setting different emotion level elements. For example, different levels of emotion intensity are represented from -1 to 1. -1 represents the weakest emotion intensity, and 1 represents the strongest emotion intensity. Assume that the emotion type of the current object video is the happiness element and the corresponding emotion level is 1, then the current object video can be a comedy video with the highest emotion intensity.

[0059] As an optional embodiment, the above target video can include popular videos with a predetermined behavior operation amount exceeding a predetermined value, that is, videos with a relatively high user click volume or play volume. Compare the emotional feature curve corresponding to the target video with a relatively high click volume or play volume in the database with the emotional feature curve obtained after inputting the current object video into the emotion prediction model. When the difference degree between the two is less than the preset value, it indicates that the current object video may also become a video with a relatively high user click volume or play volume.

[0060] It should be noted that the above is only an example, and no specific limitations are made on the application scenarios and fields here.

[0061] As an alternative embodiment, when obtaining the difference degree between the emotional feature curve of the object video and the target emotional feature curve, various methods can be adopted. For example, a method similar to the dynamic time warping (DTW) algorithm can be used to determine the distance between the emotional feature curve of the object video and the target emotional feature curve; and the distance is used as the difference degree between the emotional feature curve of the object video and the target emotional feature curve. Among them, the DTW algorithm is a dynamic time warping algorithm, belonging to the dynamic warping algorithm, which is commonly used in speech recognition. It is an earlier and more classic algorithm in speech recognition and can identify whether different lengths of data match. Based on the idea of dynamic programming, it solves the problem of template matching with different pronunciation lengths. By borrowing the above idea of identifying data matching, specifically, when depicting the emotional feature curve in a predetermined coordinate system, the abscissa (time) can correspond to the time in DTW, and the ordinate (emotional type and emotional intensity) is equivalent to the speech signal amplitude in DTW. That is, the emotional type or emotional intensity of the object video is used as the row, and the target emotional type or emotional intensity of the target video is used as the column to form a distance matrix (where the value of the element in the distance matrix is the distance between the corresponding row and the corresponding column). According to the shortest continuous path from the starting vertex to the ending vertex obtained from the distance matrix, the distance between the emotional feature curve of the object video and the target emotional feature curve of the target video is determined (that is, the sum of the values of the matrix elements on the shortest continuous path obtained from the matrix). By using the above DTW-like algorithm to determine the distance between the emotional feature curve of the object video and the target emotional feature curve, and using the distance obtained according to the DTW-like method as the difference degree between the emotional feature curve of the object video and the target emotional feature curve, the obtained difference degree between the emotional feature curve of the object video and the target emotional feature curve can be made more accurate, providing a basis for subsequent determination of whether the object video is the target video.

[0062] In the embodiments of the present disclosure, as Figure 3As shown, the emotional features and the time axis of video playback can form a predetermined coordinate system: the direction of the time axis corresponds to different emotional intensities and emotional types of the frame images corresponding to the video at different times. The emotional feature set is mapped to any value within [0, 1], and the emotional intensity is any value within [-1, 1]. For example, the time is the abscissa, and the ordinate is the emotional intensity and emotional type. Curve Q1 is the emotional feature curve of the emotional type in the object video in the predetermined coordinate system based on the frame sequence (for example, it can include the emotional type and emotional intensity. For example, the emotional type is represented by the value on the ordinate, and for the emotional type corresponding to a certain time point, the corresponding emotional intensity is represented by the fluctuation above and below this value); Curve Q2 is the target emotional feature curve of the emotional intensity or emotional type in the target video in the predetermined coordinate system based on the frame sequence; the distance between the emotional type feature curve Q1 and the target emotional feature curve Q2 is used as the above-mentioned difference degree. By showing the difference degree in the way of the distance between the corresponding curves, the video of the target type can be obtained intuitively and accurately.

[0063] As an optional embodiment, input the object video into the emotion prediction model to obtain the emotional feature curve of the object video. The following processing method can be adopted: for the video content of multiple time units included in the object video, through the feature extraction module in the emotion prediction model, extract the image features, audio features and text features in the video content of multiple time units respectively; through the fully connected layer network in the emotion prediction model, fuse the image features, audio features and text features corresponding to the video content of multiple time units respectively to obtain the fused features, and based on the fused features, predict the emotional type and emotional intensity of the video content of multiple time units respectively; based on the emotional type and emotional intensity of the video content of multiple time units, and the time sequence features of multiple time units, establish the emotional feature curve of the object video.

[0064] Through the feature extraction module in the emotion prediction model, extract the image features, audio features and text features in the video content of multiple time units respectively; then, use the fully connected layer network to fuse the extracted image features, audio features and text features to obtain the fused features, and based on the time sequence features of multiple time units and the fused features of multiple time units, establish the emotional feature curve of the object video. Since the fully connected layer network is used to fuse the image features, audio features and text features of the object video, the obtained fused features consider the correlation relationship between the image features, audio features and text features. Therefore, the obtained fused features can more accurately reflect the emotional type and emotional intensity of the object video, and thus the emotional feature curve established based on the obtained emotional type and emotional intensity can also better represent the emotion of the object video.

[0065] It should be noted that in the embodiments of the present disclosure, Figure 4is a flowchart of another alternative video processing method according to an embodiment of the present invention. As shown in Figure 4 , it may include, but is not limited to, obtaining an image sequence, an audio sequence, and a text sequence corresponding to the target video through an ffmpeg tool (where ffmpeg is an open-source computer program that can be used to record, convert digital audio and video, and convert them into streams), and an OCR software. The image sequence is passed through a scene feature extraction module, an expression feature extraction module, and an action feature extraction module to obtain a multi-dimensional image vector matrix, and then the above multi-dimensional image vector matrix is reduced in dimension through an LSTM network to obtain a one-dimensional image vector. The audio is passed through MFCC and an audio feature extraction module to obtain a multi-dimensional audio vector matrix, and then the above multi-dimensional audio vector matrix is reduced in dimension through an LSTM network to obtain a one-dimensional audio vector. The text sequence is passed through a Word2Vector or word embedding algorithm tool and a text feature extraction module to obtain a multi-dimensional text vector matrix; then the above multi-dimensional text vector matrix is reduced in dimension through an LSTM network to obtain a one-dimensional text vector. Through the fully connected layer network in the emotion prediction model, the image features, audio features, and text features are fused to obtain fused features, and the emotion feature curve of the target video is predicted based on the fused features.

[0066] As an alternative embodiment, for the video content of multiple time units included in the target video, through the feature extraction module in the emotion prediction model, the image features, audio features, and text features in the video content of multiple time units are respectively extracted, including: for the video content of any target time unit in the video content of multiple time units: extracting the target image features corresponding to the video content of the target time unit through the image feature extraction module in the feature extraction module, and integrating the target image features and the historical image features corresponding to the video content of the historical time unit before the target time unit through the first long short-term memory (LSTM) network to obtain the image features corresponding to the video content of the target time unit; extracting the target audio features corresponding to the video content of the target time unit through the audio feature extraction module in the feature extraction module, and integrating the target audio features and the historical audio features corresponding to the video content of the historical time unit before the target time unit through the second LSTM network to obtain the audio features corresponding to the video content of the target time unit; extracting the target text features corresponding to the video content of the target time unit through the text feature extraction module in the feature extraction module, and integrating the target text features and the historical text features corresponding to the video content of the historical time unit before the target time unit through the third LSTM network to obtain the text features corresponding to the video content of the target time unit.

[0067] Through the above processing, for the video content of any target time unit in the video content of multiple time units, the target image features of the video content of the target time unit and the historical image features corresponding to the video content of the historical time units before the target time unit are integrated through the first long short-term memory (LSTM) network to obtain the image features corresponding to the video content of the target time unit. Therefore, the image features corresponding to the video content of the target time unit are not only the target image features of the video content of the target time unit, but also consider the historical image features corresponding to the video content of multiple historical time units before the target time unit. By considering the continuity and relevance of the image features between time units as described above, the obtained image features corresponding to the video content of the target time unit can be made more accurate. Similarly, the target audio features of the video content of the target time unit and the historical audio features corresponding to the video content of the historical time units before the target time unit are integrated through the second LSTM network to obtain the audio features corresponding to the video content of the target time unit. Similarly, the audio features corresponding to the video content of the target time unit are not only the target audio features of the video content of the target time unit, but also consider the historical audio features corresponding to the video content of multiple historical time units before the target time unit. By considering the continuity and relevance of the audio features between time units as described above, the obtained audio features corresponding to the video content of the target time unit can be made more accurate. Similarly, the target text features of the video content of the target time unit and the historical text features corresponding to the video content of the historical time units before the target time unit are integrated through the third LSTM network to obtain the text features corresponding to the video content of the target time unit. Similarly, the audio features corresponding to the video content of the target time unit are not only the target text features of the video content of the target time unit, but also consider the historical text features corresponding to the video content of multiple historical time units before the target time unit. By considering the continuity and relevance of the text features between time units as described above, the obtained text features corresponding to the video content of the target time unit can be made more accurate.

[0068] It should be noted that the above image features can include various types. For example, it can include at least one of the following: the scene feature of the video content, the expression feature of the video content, and the action feature of the video content. Among them, the scene feature is used to describe the scene where the video content is located. For example, it can be various life scenes, various game scenes, or various outdoor photo scenes, etc. The expression feature is used to describe the expressions of people or animals that appear in the video content. For example, it can be the expression of a certain protagonist, the expression of a certain specific animal, or the expression of a certain virtual character, etc. The action feature is used to describe the actions of people or animals that appear in the video content. It can also be the action of a certain protagonist or the action of a certain animal, etc. By using the above rich scene features, expression features, and action features, the obtained image features can be more comprehensive and accurate.

[0069] As one or more alternative embodiments, before obtaining the difference degree between the emotional feature curve of the target video and the target emotional feature curve, it may further include: obtaining a plurality of first historical target videos within a first predetermined historical time period; inputting the plurality of first historical target videos into the emotion prediction model to respectively obtain the emotional feature curves of the plurality of first historical target videos; clustering the emotional feature curves of the plurality of first historical target videos to obtain a clustering center curve; and when the number of training samples corresponding to the clustering center curve for training the emotion prediction model is greater than or equal to a second threshold, determining the target emotional feature curve based on the clustering center curve. Through the above processing, not only can the target emotional feature curve be obtained, but also the obtained target emotional feature curve is a curve that meets certain target conditions, that is, it can represent the emotional feature curve of the target video to a certain extent. At least in terms of the number of training samples for obtaining the target emotional feature curve, it is a curve that meets the target conditions. Since the emotional feature curves of the plurality of first historical target videos within the first predetermined historical time period are predicted by the emotion prediction model, after clustering the emotional feature curves of the plurality of first historical target videos, a clustering center curve is obtained. Since the number of training samples for training the emotion prediction model is greater than or equal to the predetermined threshold, that is, the above-mentioned second threshold, it can be concluded that the emotion prediction model is reliable. The emotional feature curves of the plurality of first historical target videos predicted by the reliable emotion prediction model should also be reliable. Furthermore, the clustering center curve obtained based on the emotional feature curves of the plurality of first historical target videos should also be reliable, thereby indirectly making the target emotional feature curve determined based on the clustering center curve relatively accurate and reliable.

[0070] As one or more alternative embodiments, determining the target emotion feature curve based on the clustering center curve includes: obtaining a plurality of second historical target videos within a second predetermined historical time period, where the second predetermined historical time period is later than the first predetermined historical time period; inputting the plurality of second historical target videos into an emotion prediction model to respectively obtain the emotion feature curves of the plurality of second historical target videos; determining whether the clustering center curve has become obsolete based on the emotion feature curves of the plurality of second historical target videos; in the case where it is determined that the clustering center curve has become obsolete, obtaining a new clustering center curve and determining the new clustering center curve as the target emotion feature curve. Through the above processing, based on the plurality of second historical target videos within the second predetermined historical time period that is later than the first predetermined historical time period, and based on the emotion feature curves corresponding to the plurality of second historical target videos, it is determined whether the clustering center curve determined based on the plurality of first historical target videos within the first predetermined historical time period has become obsolete, and in the case where it is determined that it has become obsolete, a new clustering center curve is obtained and the new clustering center curve is determined as the target emotion feature curve, thereby realizing the update of the clustering center curve, that is, realizing the update of the target emotion feature curve. Corresponding to popular videos, it realizes the automatic update of the target emotion feature curve corresponding to popular videos.

[0071] As one or more alternative embodiments, there are various ways to determine whether the clustering center curve has become obsolete based on the emotion feature curves of the plurality of second historical target videos. For example, the following processing method can be adopted: using the DTW method to respectively determine the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve; determining the average value of the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve; in the case where the average value is greater than a third threshold, determining that the clustering center curve has become obsolete. By comparing the distances between the newly obtained emotion feature curves and the existing clustering center curve, and taking the average of the distances corresponding to the emotion feature curves of the plurality of second historical target videos to obtain an average value, and when the average value is greater than a predetermined threshold, that is, the third threshold, it is determined that the existing clustering center curve has become obsolete, that is, the popular item predicted in the previous first predetermined historical time period is no longer popular. In addition, using the distance averaging processing method can avoid the situation of inaccurate individual distances, making the judgment result of whether the clustering center curve has become obsolete more accurate.

[0072] As one or more alternative embodiments, when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is less than the second threshold, after supplementing the training samples for training the emotion prediction model, the emotion prediction model is retrained to obtain an updated emotion prediction model until multiple updated emotion feature curves of the first historical target videos are predicted by using the updated emotion prediction model. After clustering the obtained multiple updated emotion feature curves of the first historical target videos, an updated clustering center curve is obtained. And when the number of training samples for training the updated emotion prediction model corresponding to the updated clustering center curve is greater than or equal to the second threshold, the target emotion feature curve is determined according to the updated clustering center curve, and the finally obtained updated emotion prediction model replaces the emotion prediction model. Through the above processing, when the number of training samples for training the emotion prediction model is less than the second threshold, that is, the obtained emotion prediction model is unreliable. Therefore, after increasing the number of training samples, the emotion prediction model can be retrained to obtain an updated emotion prediction model until an updated clustering center curve is obtained by using the updated emotion prediction model. When the number of training samples for training the updated emotion prediction model corresponding to the clustering center curve is greater than or equal to the second threshold, the target emotion feature curve is determined according to the updated clustering center curve. Even if the emotion prediction model obtained through training is reliable, the emotion feature curve obtained by using the emotion prediction model is also reliable, and it is also reliable and accurate to judge whether the object video is a popular video based on the emotion feature curve.

[0073] In the embodiments of the present disclosure, generally, videos with a large number of clicks or views often have the same emotion characteristics. Therefore, the number of target videos will remain at a certain level. For example, if the target video is a comedy movie, multiple comedy movies within the first predetermined time period (such as within one week) are obtained. Assuming the second threshold is 50, the number of multiple comedy movies is at least more than 50; by using the emotion prediction model, multiple comedy movies are predicted to obtain the emotion feature curves of multiple comedy movies; the emotion feature curves of multiple comedy movies are clustered to obtain a clustering center curve, and the target emotion feature curve is determined based on the clustering center curve.

[0074] If the clustering center curve indicates that the number of recently emerged popular videos is significantly less than the number of videos with a high historical view or click volume, the database of videos with a high historical view or click volume needs to be updated, and the target emotion feature curve is updated according to the new clustering center curve.

[0075] In one or more embodiments, the target type of video includes popular videos with the volume of predetermined user behavior operations exceeding a predetermined value. Here, the predetermined user behavior may include, but is not limited to, the number of clicks and plays of users, etc., and no specific limitation is made here.

[0076] As an alternative implementation, to illustrate the above video processing method, continuing with the above Figure 4 as an example, the video processing method may include:

[0077] Training stage of the emotion curve prediction model (i.e., the above-mentioned emotion prediction model):

[0078] 1. Model design stage:

[0079] The analysis of the emotion curve of video streaming refers to automatically analyzing the subjective feelings that video streaming may bring to the audience's emotions based on computer algorithms. For example, differentiating different emotional elements such as horror, intense excitement, and romance in the streaming media and the changes in their types and intensities over time. To apply it to the field of short videos / live broadcasts, in this alternative implementation, an emotion classification model and an emotion intensity model (the combination of the two is the above-mentioned emotion prediction model) are established for each frame in the video stream to judge the emotion type at a certain moment in the video and make a predefined setting. For example, anger, fear, happiness, love, sadness, surprise, etc. The emotion intensity model is used to predict the intensity of this feeling. For example, it can be represented by a continuous interval of (-1, 1), where -1 represents the weakest degree and 1 represents the strongest degree.

[0080] Video is a typical multi-modal data, and the pictures, sounds, and subtitles in it all play important roles in conveying emotions to users. At the same time, as the most important factor among them, the picture can be at least divided into three relatively independent dimensions, namely the scene, the facial expressions of the actors, and the actions of the actors, which affect the user's senses. Therefore, the emotion type model and the emotion intensity model extract the pictures, audio, and text in the input video stream respectively for processing, and extract the picture information through three feature extraction modules of scene, expression, and action.

[0081] As a streaming media, the emotion conveyed by a video to the audience at a certain moment usually undergoes a long period of foreshadowing before. In view of this, in this alternative implementation, the LSTM time series network is added to the emotion type model and the emotion intensity model to integrate historical information to predict the emotion type and intensity of the current frame. Then, two different fully connected layers are used to fuse the features of each dimension together and input them into the emotion type regressor and the emotion intensity regressor respectively to predict the emotion type and emotion intensity of the video.

[0082] 2. Training data preparation:

[0083] For the emotional feature curve prediction task of short videos / live broadcasts, in this optional implementation, relevant videos can be collected as training data, and the collection channels include but are not limited to short video platforms of various media. For a specific training video (a video with a relatively high click-through rate or playback volume), first label the emotional type and intensity of each frame of the video. To improve the labeling efficiency, the labeling granularity can be increased from each frame to a fixed duration, and the emotional type and intensity of each frame within this duration are the same.

[0084] 3. Preprocessing of training videos:

[0085] In this optional implementation, for example, the ffmpeg toolkit can be used to extract audio data from the training video, and the OCR algorithm can be used to extract text from the video frames frame by frame. The MFCC algorithm and the word2vec algorithm are respectively used to convert the extracted audio and text in the video into discrete feature vectors that can be used for deep learning.

[0086] 4. Model training:

[0087] First, perform frame-by-frame feature extraction on the video frames, audio, and text of the training video, then use the LSTM model to integrate the features at different times in the video, and finally use two fully connected layers to fuse the features of different dimensions to predict the emotional type and intensity of the video respectively.

[0088] The application stage of the emotional curve prediction model includes the following content:

[0089] 1. Predict the emotional feature curve of historical popular short videos / live broadcasts:

[0090] First, collect historical popular short videos / live broadcasts, and then use the trained emotional curve prediction model to predict the emotional feature curves of these video streams. The collection channels include but are not limited to short video platforms of various media.

[0091] 2. Calculation of emotional features of popular short videos / live broadcasts:

[0092] In this alternative embodiment, a clustering algorithm is used to cluster the emotional curves of historical popular short videos / live broadcasts in a high-dimensional space. When clustering, the sample distance is calculated by the DTW algorithm. After the emotional curve clustering is completed, if the number of samples corresponding to a certain clustering center is less than the set threshold, it can be considered that the prediction result of the emotional curve prediction model for this type of popular video does not meet the expectations. Therefore, after data augmentation of this type of popular video, it is added to the training set to retrain the emotional curve prediction model. This process is continuously carried out for several rounds until the number of samples corresponding to all clustering centers exceeds the threshold, and then the calculated clustering centers are stored in the database for management. The clustering centers in the database, as the emotional characteristics of popular short videos / live broadcasts, can be used to support several applications, such as guiding shooting, replicating popular models, and efficiently allocating exposures according to the probability of popularity, etc.

[0093] 3. Dynamic update of emotional characteristics of popular short videos / live broadcasts:

[0094] Popular short videos / live broadcasts usually have strong timeliness, so it is necessary to update the emotional characteristics of popular short videos / live broadcasts in the database regularly. Figure 5 It is a flowchart for updating popular videos in the optional video processing method according to an embodiment of the present invention. As Figure 5 shown, in this alternative embodiment, first set the emotional characteristic update time period. When there is an update requirement, a batch of recent popular and non-popular short videos / live broadcasts will be collected, and the trained emotional curve prediction model will be used to predict the emotional characteristic curves of these video streams. Then, the DTW algorithm is used to calculate the average distance between the emotional characteristics of recent popular and non-popular short videos / live broadcasts and the emotional characteristics in the database respectively, and the ratio between the average distances is calculated. If this value is greater than the given threshold, it is considered that the emotional characteristics of popular short videos / live broadcasts in the database are outdated, and it will jump back to the second step to recalculate the emotional characteristics of popular short videos / live broadcasts.

[0095] 4. Predicting popular short videos / live broadcasts using emotional characteristics:

[0096] Figure 6 It is a flowchart for predicting popular videos in the optional video processing method according to an embodiment of the present invention. As Figure 6 shown, when a user uploads a short video or conducts a live webcast, the emotional curve prediction model can be used to predict the emotional characteristic curve of the video stream first, and then the DTW algorithm is used to calculate the distance between this curve and the emotional characteristics of popular short videos / live broadcasts in the database. When the distance is less than the given threshold, it can be considered that there is a high probability that the uploaded short video / live broadcast will become popular, and corresponding processing can be performed on the uploaded short video / live broadcast. For example, more exposures can be given when distributing, which greatly improves the efficiency of traffic allocation.

[0097] Figure 7 is a schematic structural diagram of a video processing device shown according to an exemplary embodiment. Refer to Figure 7 , the device includes: a first processing module 71, a first acquisition module 72, and a first determination module 73. The device will be described below.

[0098] The first processing module 71 is configured to input an object video into an emotion prediction model to obtain an emotion feature curve of the object video, where the emotion feature curve represents the changes in emotion type and emotion intensity of the object video over time; the first acquisition module 72 is connected to the first processing module 71 and is configured to acquire the difference degree between the emotion feature curve of the object video and a target emotion feature curve; the first determination module 73 is connected to the first acquisition module 72 and is configured to determine the object video as a target video when the difference degree is less than a first threshold.

[0099] In one or more alternative embodiments, the first acquisition module includes: a first determination unit configured to determine the distance between the emotion feature curve of the object video and the target emotion feature curve by using a method similar to the dynamic time warping (DTW) method; a first processing unit configured to use the distance as the difference degree between the emotion feature curve of the object video and the target emotion feature curve.

[0100] In one or more alternative embodiments, the first processing module includes: a first extraction unit configured to, for the video content of multiple time units included in the object video, respectively extract image features, audio features, and text features in the video content of the multiple time units through a feature extraction module in the emotion prediction model; a first fusion unit configured to, through a fully connected layer network in the emotion prediction model, respectively fuse the image features, audio features, and text features corresponding to the video content of the multiple time units to obtain a fusion feature, and respectively predict the emotion type and emotion intensity of the video content of the multiple time units based on the fusion feature; a first establishment unit configured to establish an emotion feature curve of the object video based on the emotion type and emotion intensity of the video content of the multiple time units, and the timing features of the multiple time units.

[0101] In one or more alternative embodiments, the first extraction unit includes: a first processing subunit, configured to, for the video content of any target time unit among the video content of multiple time units, extract the target image features corresponding to the video content of the target time unit through the image feature extraction module in the feature extraction module, and integrate the target image features and the historical image features corresponding to the video content of the historical time units before the target time unit through the first long short-term memory (LSTM) network to obtain the image features corresponding to the video content of the target time unit; a second processing subunit, configured to, for the video content of any target time unit among the video content of multiple time units, extract the target audio features corresponding to the video content of the target time unit through the audio feature extraction module in the feature extraction module, and integrate the target audio features and the historical audio features corresponding to the video content of the historical time units before the target time unit through the second LSTM network to obtain the audio features corresponding to the video content of the target time unit; a third processing subunit, configured to, for the video content of any target time unit among the video content of multiple time units, extract the target text features corresponding to the video content of the target time unit through the text feature extraction module in the feature extraction module, and integrate the target text features and the historical text features corresponding to the video content of the historical time units before the target time unit through the third LSTM network to obtain the text features corresponding to the video content of the target time unit.

[0102] In one or more alternative embodiments, the image features include at least one of the following: the scene features of the video content, the expression features of the video content, and the action features of the video content.

[0103] In one or more alternative embodiments, the apparatus further includes: a second acquisition module, configured to acquire a plurality of first historical target videos within a first predetermined historical time period before obtaining the difference degree between the emotional feature curve of the target video and the target emotional feature curve; a second processing module, configured to input the plurality of first historical target videos into the emotion prediction model to respectively obtain the emotional feature curves of the plurality of first historical target videos; a clustering module, configured to cluster the emotional feature curves of the plurality of first historical target videos to obtain a clustering center curve; a second determination module, configured to determine the target emotional feature curve according to the clustering center curve when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is greater than or equal to a second threshold.

[0104] In one or more alternative embodiments, the second determination module includes: a first acquisition unit configured to acquire a plurality of second historical target videos within a second predetermined historical time period, where the second predetermined historical time period is later than the first predetermined historical time period; a second processing unit configured to input the plurality of second historical target videos into an emotion prediction model to respectively obtain emotion feature curves of the plurality of second historical target videos; a second determination unit configured to determine whether the clustering center curve has become obsolete based on the emotion feature curves of the plurality of second historical target videos; and a third determination unit configured to, when it is determined that the clustering center curve has become obsolete, acquire a new clustering center curve and determine the new clustering center curve as the target emotion feature curve.

[0105] In one or more alternative embodiments, the second determination unit includes: a first determination subunit configured to respectively determine the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve by using the DTW method; a second determination subunit configured to determine the average value of the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve; and a third determination subunit configured to determine that the clustering center curve has become obsolete when the average value is greater than a third threshold.

[0106] In one or more alternative embodiments, the apparatus further includes: a third determination module configured to, when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is less than a second threshold, supplement the training samples for training the emotion prediction model and then retrain the emotion prediction model to obtain an updated emotion prediction model until, by using the updated emotion prediction model, updated emotion feature curves of a plurality of first historical target videos are predicted, after clustering the obtained updated emotion feature curves of the plurality of first historical target videos, an updated clustering center curve is obtained, and when the number of training samples for training the updated emotion prediction model corresponding to the updated clustering center curve is greater than or equal to the second threshold, determine the target emotion feature curve based on the updated clustering center curve, and replace the emotion prediction model with the finally obtained updated emotion prediction model.

[0107] In one or more alternative embodiments, the target video includes a popular video with a predetermined behavior operation amount exceeding a predetermined value.

[0108] It should be noted that, in this embodiment, the embodiments of the above apparatus may refer to the embodiments of the above method, and no limitation is made here.

[0109] Embodiments of the present disclosure may provide an electronic device, for example, Figure 8FIG. 0 is a schematic diagram of an optional terminal according to an embodiment of the present invention. The electronic device may be a terminal 800. For example, the terminal 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0110] Referring to Figure 8 , the terminal 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0111] The processing component 802 generally controls the overall operation of the terminal 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0112] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of these data include instructions for any application or method operating on the terminal 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0113] The power component 806 provides power to various components of the terminal 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the terminal 800.

[0114] The multimedia component 808 includes a screen that provides an output interface between the terminal 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0115] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the terminal 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0116] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0117] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the terminal 800. For example, the sensor component 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and the keypad of the terminal 800. The sensor component 814 can also detect a change in the position of the terminal 800 or a component of the terminal 800, the presence or absence of user contact with the terminal 800, the orientation or acceleration / deceleration of the terminal 800, and the temperature change of the terminal 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0118] The communication component 816 is configured to facilitate communication between the terminal 800 and other devices in a wired or wireless manner. The terminal 800 can access a wireless network based on communication standards, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0119] In an exemplary embodiment, the terminal 800 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0120] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as the memory 804 including instructions, and the above instructions can be executed by the processor 820 of the terminal 800 to complete the above method. Optionally, the computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0121] Embodiments of the present disclosure can provide an electronic device. Figure 9 is a schematic diagram of an optional server according to an embodiment of the present invention. For example, the electronic device can be a server. Referring to Figure 9 , the server 900 includes a processing component 901, which further includes one or more processors, and memory resources represented by the memory 902 for storing instructions executable by the processing component 901, such as application programs. The application programs stored in the memory 902 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 901 is configured to execute instructions to perform the above method.

[0122] The server 900 may further include a power supply component 903 configured to perform power management of the server 900, a wired or wireless network interface 904 configured to connect the server 900 to a network, and an input / output (I / O) interface 905. The server 900 may operate based on an operating system stored in the memory 902, such as Windows ServerTM, MacOS XTM, UnixTM, LinuxTM, FreeBSDTM or similar systems.

[0123] In an exemplary embodiment, there is also provided a computer program product. When the computer program in the computer program product is executed by a processor of a terminal, the terminal is enabled to perform the video processing method of any one of the above.

[0124] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed herein. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0125] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A video processing method, characterized in that, Including: Inputting an object video into an emotion prediction model to obtain an emotion feature curve of the object video, where the emotion feature curve characterizes the changes in emotion type and emotion intensity of the object video over time; Obtaining the difference degree between the emotion feature curve of the object video and a target emotion feature curve; When the difference degree is less than a first threshold, determining the object video as a target video, where the target video is a popular video with a predetermined behavior operation amount exceeding a predetermined value, and the target emotion curve is the emotion feature curve corresponding to the popular video; Before obtaining the difference degree between the emotion feature curve of the object video and the target emotion feature curve, it further includes: obtaining a plurality of first historical target videos within a first predetermined historical time period; inputting the plurality of first historical target videos into the emotion prediction model to respectively obtain the emotion feature curves of the plurality of first historical target videos; clustering the emotion feature curves of the plurality of first historical target videos to obtain a clustering center curve; when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is greater than or equal to a second threshold, determining the target emotion feature curve according to the clustering center curve.

2. The method according to claim 1, wherein The obtaining the difference degree between the emotion feature curve of the object video and the target emotion feature curve includes: Using a method similar to the dynamic time warping (DTW) method to determine the distance between the emotion feature curve of the object video and the target emotion feature curve; Taking the distance as the difference degree between the emotion feature curve of the object video and the target emotion feature curve.

3. The method according to claim 1, characterized in that, The inputting an object video into an emotion prediction model to obtain an emotion feature curve of the object video includes: For the video content of multiple time units included in the object video, respectively extracting image features, audio features, and text features in the video content of the multiple time units through a feature extraction module in the emotion prediction model; Through a fully connected layer network in the emotion prediction model, respectively fusing the image features, the audio features, and the text features corresponding to the video content of the multiple time units to obtain a fused feature, and respectively predicting the emotion type and emotion intensity of the video content of the multiple time units based on the fused feature; Based on the emotion type and emotion intensity of the video content of the multiple time units, and the time sequence features of the multiple time units, establishing an emotion feature curve of the object video.

4. The method according to claim 3, wherein The respectively extracting image features, audio features, and text features in the video content of the multiple time units through a feature extraction module in the emotion prediction model for the video content of multiple time units included in the object video includes: For the video content of any target time unit among the video content of the multiple time units: Extract the target image features corresponding to the video content of the target time unit through the image feature extraction module in the feature extraction module, and integrate the target image features and the historical image features corresponding to the video content of the historical time units before the target time unit through the first long short-term memory (LSTM) network to obtain the image features corresponding to the video content of the target time unit; Extract the target audio features corresponding to the video content of the target time unit through the audio feature extraction module in the feature extraction module, and integrate the target audio features and the historical audio features corresponding to the video content of the historical time units before the target time unit through the second LSTM network to obtain the audio features corresponding to the video content of the target time unit; Extract the target text features corresponding to the video content of the target time unit through the text feature extraction module in the feature extraction module, and integrate the target text features and the historical text features corresponding to the video content of the historical time units before the target time unit through the third LSTM network to obtain the text features corresponding to the video content of the target time unit.

5. The method according to claim 3, characterized in that The image features include at least one of the following: the scene features of the video content, the expression features of the video content, and the action features of the video content.

6. The method according to claim 1, wherein The determining the target emotion feature curve according to the clustering center curve includes: Obtain a plurality of second historical target videos within a second predetermined historical time period, where the second predetermined historical time period is later than the first predetermined historical time period; Input the plurality of second historical target videos into the emotion prediction model to respectively obtain the emotion feature curves of the plurality of second historical target videos; Determine whether the clustering center curve is outdated according to the emotion feature curves of the plurality of second historical target videos; In the case where it is determined that the clustering center curve is outdated, obtain a new clustering center curve and determine the new clustering center curve as the target emotion feature curve.

7. The method according to claim 6, wherein The determining whether the clustering center curve is outdated according to the emotion feature curves of the plurality of second historical target videos includes: Adopt the dynamic time warping (DTW) method to respectively determine the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve; Determine the average value of the distances between the emotion feature curves of the plurality of second historical target videos and the clustering center curve; In the case where the average value is greater than a third threshold, determine that the clustering center curve is outdated.

8. The method according to claim 1, characterized in that, Further includes: In the case where the number of training samples for training the emotion prediction model corresponding to the clustering center curve is less than the second threshold, after supplementing the training samples for training the emotion prediction model, the emotion prediction model is retrained to obtain an updated emotion prediction model until the updated emotion feature curves of the multiple first historical target videos are predicted by using the updated emotion prediction model. After clustering the obtained updated emotion feature curves of the multiple first historical target videos, an updated clustering center curve is obtained. And in the case where the number of training samples for training the updated emotion prediction model corresponding to the updated clustering center curve is greater than or equal to the second threshold, the target emotion feature curve is determined based on the updated clustering center curve, and the finally obtained updated emotion prediction model replaces the emotion prediction model.

9. A video processing device, characterized in that, Including: A first processing module, configured to input an object video into an emotion prediction model to obtain an emotion feature curve of the object video, where the emotion feature curve represents the changes in emotion type and emotion intensity of the object video over time; A first acquisition module, configured to acquire the degree of difference between the emotion feature curve of the object video and the target emotion feature curve; A first determination module, configured to determine the object video as a target video when the degree of difference is less than a first threshold, where the target video is a popular video with a predetermined behavior operation amount exceeding a predetermined value, and the target emotion curve is the emotion feature curve corresponding to the popular video; Further including: a second acquisition module, configured to acquire a plurality of first historical target videos within a first predetermined historical time period before acquiring the degree of difference between the emotion feature curve of the object video and the target emotion feature curve; a second processing module, configured to input the plurality of first historical target videos into the emotion prediction model to respectively obtain emotion feature curves of the plurality of first historical target videos; a clustering module, configured to cluster the emotion feature curves of the plurality of first historical target videos to obtain a clustering center curve; a second determination module, configured to determine the target emotion feature curve based on the clustering center curve when the number of training samples for training the emotion prediction model corresponding to the clustering center curve is greater than or equal to a second threshold.

10. The device according to claim 9, characterized in that, The first acquisition module includes: A first determination unit, configured to use a method similar to the dynamic time warping (DTW) method to determine the distance between the emotion feature curve of the object video and the target emotion feature curve; A first processing unit, configured to use the distance as the degree of difference between the emotion feature curve of the object video and the target emotion feature curve.

11. The device according to claim 9, characterized in that, The first processing module includes: A first extraction unit, configured to, for the video content of a plurality of time units included in the object video, respectively extract image features, audio features, and text features in the video content of the plurality of time units through a feature extraction module in the emotion prediction model; A first fusion unit, configured to respectively fuse the image features, the audio features, and the text features corresponding to the video content of the multiple time units through a fully connected layer network in the emotion prediction model to obtain fused features, and respectively predict the emotion type and emotion intensity of the video content of the multiple time units based on the fused features; A first establishment unit, configured to establish an emotion feature curve of the target video based on the emotion type and emotion intensity of the video content of the multiple time units, and the temporal sequence features of the multiple time units.

12. The device according to claim 11, characterized in that, The first extraction unit includes: A first processing subunit, configured to, for the video content of any target time unit in the video content of the multiple time units, extract target image features corresponding to the video content of the target time unit through an image feature extraction module in the feature extraction module, and integrate the target image features and the historical image features corresponding to the video content of the historical time units before the target time unit through a first long short-term memory (LSTM) network to obtain the image features corresponding to the video content of the target time unit; A second processing subunit, configured to, for the video content of any target time unit in the video content of the multiple time units, extract target audio features corresponding to the video content of the target time unit through an audio feature extraction module in the feature extraction module, and integrate the target audio features and the historical audio features corresponding to the video content of the historical time units before the target time unit through a second LSTM network to obtain the audio features corresponding to the video content of the target time unit; A third processing subunit, configured to, for the video content of any target time unit in the video content of the multiple time units, extract target text features corresponding to the video content of the target time unit through a text feature extraction module in the feature extraction module, and integrate the target text features and the historical text features corresponding to the video content of the historical time units before the target time unit through a third LSTM network to obtain the text features corresponding to the video content of the target time unit.

13. The device according to claim 11, characterized in that, The image features include at least one of the following: scene features of video content, expression features of video content, and action features of video content.

14. The device according to claim 9, characterized in that, The second determination module includes: A first acquisition unit, configured to acquire multiple second historical target videos within a second predetermined historical time period, where the second predetermined historical time period is later than the first predetermined historical time period; A second processing unit, configured to input the multiple second historical target videos into the emotion prediction model to respectively obtain emotion feature curves of the multiple second historical target videos; A second determination unit, configured to determine whether the clustering center curve is outdated according to the emotion feature curves of the multiple second historical target videos; A third determination unit, configured to, when it is determined that the clustering center curve is outdated, acquire a new clustering center curve and determine the new clustering center curve as the target emotion feature curve.

15. The device according to claim 14, characterized in that The second determination unit includes: A first determination subunit, configured to respectively determine the distances between the emotional feature curves of the multiple second historical target videos and the cluster center curve by using the DTW method; A second determination subunit, configured to determine the average value of the distances between the emotional feature curves of the multiple second historical target videos and the cluster center curve; A third determination subunit, configured to determine that the cluster center curve is outdated when the average value is greater than a third threshold.

16. The device according to claim 9, characterized in that, It further includes: A third determination module, configured to, when the number of training samples for training the emotion prediction model corresponding to the cluster center curve is less than the second threshold, supplement the training samples for training the emotion prediction model and then retrain the emotion prediction model to obtain an updated emotion prediction model until the updated emotional feature curves of the multiple first historical target videos are predicted by using the updated emotion prediction model, the updated cluster center curve is obtained after clustering the updated emotional feature curves of the multiple first historical target videos, and when the number of training samples for training the updated emotion prediction model corresponding to the updated cluster center curve is greater than or equal to the second threshold, determine the target emotional feature curve according to the updated cluster center curve, and replace the emotion prediction model with the finally obtained updated emotion prediction model.

17. An electronic device, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the video processing method according to any one of claims 1 to 8.

18. A computer program product, including a computer program, characterized in that the computer program, when executed by a processor, implements the video processing method according to any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the video processing method according to any one of claims 1 to 8.

Images