Video category prediction method based on attention mechanism and related equipment

Abstract

The embodiment of the invention provides a video category prediction method based on an attention mechanism and related equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing feature extraction on target image data of a to-be-predicted video to obtain target image features and target global features; and processing the target image through the feature mapping layer and the local attention network to obtain target local features. And fusing the local and global features through a feature fusion layer to obtain a target composite feature. And obtaining a target composite feature of each piece of target image data to obtain a plurality of target composite features, and carrying out time sequence feature learning on the plurality of target composite features through a time sequence pooling module to obtain a composite time sequence feature. And performing category prediction on the target image data according to the composite time sequence features to obtain a predicted target category. And after the target local features are accurately extracted, prediction is performed based on the composite time sequence features, so that the accuracy of video category prediction is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video category prediction method based on an attention mechanism and related equipment. Background Art

[0002] Video is a widely used media form in modern society, containing a large amount of valuable information. However, since these media forms are highly interactive and dynamic, it is particularly important to understand and identify this information. It is of great significance to image analysis, human-computer interaction, and autonomous driving. In the field of image analysis, it can be used to classify videos, thereby improving the effects of video review, recommendation, and search.

[0003] Video classification is achieved by predicting the video category. Video category prediction is to predict the category of the video image. In the related art, the category prediction of the image can be performed by identifying and extracting local features through manually written rules, and the video category is predicted based on the local features. However, due to the limitations and errors of rule writing, the accuracy of feature extraction is low, which makes the accuracy of image category prediction low, and further makes the accuracy of video category prediction low. Therefore, how to improve the accuracy of video category prediction has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a video category prediction method and related equipment based on an attention mechanism, aiming to improve the accuracy of video category prediction.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a video category prediction method based on an attention mechanism, the method comprising:

[0006] The image of the video to be predicted is obtained according to a preset frame extraction strategy to obtain target image data;

[0007] Input each of the target image data into a preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module;

[0008] Performing feature extraction on each target image data through the feature extraction layer to obtain each target image feature;

[0009] Performing feature mapping on each of the target image features through the feature mapping layer to obtain each target mapping feature;

[0010] Performing local attention processing on each of the target mapping features through the local attention network to obtain each target local feature;

[0011] Performing global attention processing on each target image feature through the global attention network to obtain each target global feature;

[0012] Performing feature fusion on each of the target local features and the corresponding target global features through the feature fusion layer to obtain multiple target composite features;

[0013] Performing time series feature learning on the plurality of target composite features through the time series pooling module to obtain composite time series features;

[0014] The target image data is subjected to category prediction according to the composite temporal features to obtain a predicted target category.

[0015] In some embodiments, the local attention network includes a learning attention layer and a weighted summation layer, and the local attention network performs local attention processing on each of the target mapping features to obtain each target local feature, including:

[0016] The target mapping feature is normalized by the learning attention layer to obtain a local normalized feature; wherein the local normalized feature represents the weight of features at different positions;

[0017] The weighted summation layer performs weighted summation on the local normalized features and the target image features to obtain the target local features.

[0018] In some embodiments, the target category prediction model further includes a simple cycle unit, and performing category prediction on the target image data according to the composite temporal features to obtain a predicted target category includes:

[0019] Performing category probability prediction on the composite time series feature through the simple cycle unit to obtain first category prediction data; wherein the first category prediction data includes a probability value of each video category;

[0020] The target image data is subjected to category prediction according to the first category prediction data to obtain the predicted target category.

[0021] In some embodiments, the target category prediction model further includes a regularization layer, and performing category prediction on the target image data according to the composite temporal features to obtain the predicted target category further includes:

[0022] Performing a category probability prediction on the target global feature through the regularization layer to obtain second category prediction data; wherein the second category prediction data includes a probability value of each video category;

[0023] The target image data is subjected to category prediction according to the first category prediction data and the second category prediction data to obtain the predicted target category.

[0024] In some embodiments, the temporal pooling module includes a frame pooling layer and a relational reasoning layer, and the temporal feature learning is performed on the plurality of target composite features by the temporal pooling module to obtain the composite temporal feature, including:

[0025] Performing average pooling processing on the plurality of target composite features through the frame pooling layer to obtain a plurality of video clip features;

[0026] The relational reasoning layer performs temporal feature learning on each of the video clip features to obtain the composite temporal feature.

[0027] In some embodiments, before acquiring the image of the video to be predicted according to the preset frame extraction strategy to obtain the target image data, the method further includes:

[0028] Constructing the target category prediction model specifically includes:

[0029] Acquire image training data; wherein the image training data includes category labels;

[0030] Inputting the image training data into a preset original category prediction model for category prediction to obtain a predicted training category;

[0031] Performing loss calculation on the category label and the predicted training category using a preset classification loss function to obtain first loss data;

[0032] The parameters of the original category prediction model are adjusted based on the first loss data to obtain the target category prediction model.

[0033] In some embodiments, after calculating the category label and the predicted training category loss using a preset target loss function to obtain the first loss data, the method further includes:

[0034] The image training data is subjected to category probability prediction by the original category prediction model to obtain first category prediction training data and second category prediction training data; wherein the first category prediction training data is obtained by performing category probability prediction on the composite time series features corresponding to the image training data; and the second category prediction training data is obtained by performing category probability prediction on the target global features corresponding to the image training data;

[0035] Performing loss calculation on the first category prediction training data and the second category prediction training data using a preset regularized loss function to obtain second loss data;

[0036] The parameters of the original category prediction model are adjusted based on the first loss data and the second loss data to obtain the target category prediction model.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a video category prediction device based on an attention mechanism, the device comprising:

[0038] An acquisition module is used to acquire the image of the video to be predicted according to a preset frame extraction strategy to obtain target image data;

[0039] An input module, used to input each of the target image data into a preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module;

[0040] A feature extraction module, used for performing feature extraction on each target image data through the feature extraction layer to obtain features of each target image;

[0041] A feature mapping module, used for performing feature mapping on each of the target image features through the feature mapping layer to obtain each target mapping feature;

[0042] A local processing module, used for performing local attention processing on each of the target mapping features through the local attention network to obtain each target local feature;

[0043] A global processing module, used for performing global attention processing on each target image feature through the global attention network to obtain each target global feature;

[0044] A feature fusion module, used for fusing each of the target local features and the corresponding target global features through the feature fusion layer to obtain multiple target composite features;

[0045] A feature learning module, used for performing time series feature learning on the plurality of target composite features through the time series pooling module to obtain composite time series features;

[0046] The prediction module is used to perform category prediction on the target image data according to the composite temporal features to obtain a predicted target category.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The video category prediction method and related device based on the attention mechanism proposed in the present application obtain the image of the video to be predicted according to the preset frame extraction strategy to obtain the target image data, and input the target image data into the preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module. The feature extraction layer performs feature extraction on the target image data to obtain the target image feature. The feature mapping layer performs feature mapping on the target image feature to obtain the target mapping feature. The local attention network performs local attention processing on the target mapping feature to obtain the target local feature. The global attention network performs global attention processing on the target image feature to obtain the target global feature. The feature fusion layer performs feature fusion on the target local feature and the target global feature to obtain the target composite feature. The target composite feature of each target image data is obtained to obtain multiple target composite features, and the temporal feature learning is performed on the multiple target composite features through the temporal pooling module to obtain the composite temporal feature. The target image data is classified according to the composite temporal feature to obtain the predicted target category. In this embodiment, after accurately extracting the target local features, prediction is performed based on the composite temporal features, thereby improving the accuracy of video image category prediction and further improving the accuracy of video category prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is an optional flowchart of a video category prediction method based on an attention mechanism provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step S105 in FIG.

[0052] Figure 3 yes Figure 1 Flow chart of step S108 in FIG.

[0053] Figure 4 yes Figure 1 A first flow chart of step S109 in FIG.

[0054] Figure 5 yes Figure 1 The second flow chart of step S109 in FIG.

[0055] Figure 6is a flowchart of a video category prediction method based on an attention mechanism provided by another embodiment of the present application;

[0056] Figure 7 is a flowchart of a video category prediction method based on an attention mechanism provided in a third embodiment of the present application;

[0057] Figure 8 is a schematic diagram of the structure of a video category prediction device based on an attention mechanism provided in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] Video category prediction is to predict the category of a video image. In the related art, the category prediction of an image can be performed by identifying and extracting local features through manually written rules, and predicting the video category based on the local features. However, due to the limitations and errors of rule writing, the accuracy of feature extraction is low, which makes the accuracy of image category prediction low, and further makes the accuracy of video category prediction low.

[0063] Based on this, the embodiment of the present application provides a video category prediction method and related equipment based on an attention mechanism, which obtains the image of the video to be predicted, obtains the target image data, and then extracts the target image features from the target image data. Feature mapping is performed on the target image features to obtain the target mapping features, and then the target local features are obtained based on the target mapping features. Global attention processing is performed on the target image features to obtain the target global features. The target local features and the target global features are fused to obtain the target composite features, the target composite features of each target image data are obtained, and multiple target composite features are obtained, and then the multiple target composite features are subjected to time series feature learning to obtain composite time series features, and the target image data is predicted based on the composite time series features to obtain the predicted target category. After the target local features are accurately extracted, prediction is performed based on the composite time series features, which improves the accuracy of the category prediction of the video image, thereby improving the accuracy of the video category prediction.

[0064] The video category prediction method based on the attention mechanism and related devices provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the video category prediction method based on the attention mechanism in the embodiments of the present application is described.

[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0067] The video category prediction method based on the attention mechanism provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video category prediction method based on the attention mechanism, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] Please refer to Figure 1 , Figure 1 is an optional flowchart of a video category prediction method based on an attention mechanism provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S109.

[0070] Step S101, acquiring an image of a video to be predicted according to a preset frame extraction strategy to obtain target image data;

[0071] Step S102, inputting each target image data into a preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module;

[0072] Step S103, extracting features from each target image data through a feature extraction layer to obtain features of each target image;

[0073] Step S104, performing feature mapping on each target image feature through a feature mapping layer to obtain each target mapping feature;

[0074] Step S105, performing local attention processing on each target mapping feature through a local attention network to obtain each target local feature;

[0075] Step S106, performing global attention processing on each target image feature through a global attention network to obtain each target global feature;

[0076] Step S107, performing feature fusion on each target local feature and the corresponding target global feature through a feature fusion layer to obtain multiple target composite features;

[0077] Step S108, performing time series feature learning on multiple target composite features through a time series pooling module to obtain composite time series features;

[0078] Step S109, performing category prediction on the target image data according to the composite time series feature to obtain a predicted target category.

[0079] In step S101 of some embodiments, since the difference between adjacent frame images in the video to be predicted is small and may be of the same category, when predicting the video category of the video to be predicted, it is not necessary to predict the category of each frame image in the video to prevent waste of computing power and low category prediction speed. Therefore, it is necessary to obtain the image of the video to be predicted according to a preset frame extraction strategy, and then preprocess the obtained image to obtain the target image data. The frame extraction strategy can be to extract a preset number of video frames at intervals. For example, the video to be predicted has a total of 12 frames of images, and the preset number is 2, then the first frame, the fourth frame, the seventh frame, and the tenth frame of the video to be predicted are obtained. In one example, if the computing power is sufficient, the preset number can also be set to 0. When the preset number is 0, each frame image of the video to be predicted is obtained.

[0080] In one example, the frame extraction strategy may also be to extract frames according to the video frame type. For example, if the video frame type includes I frame, P frame and B frame, only the P frame image in the video to be predicted can be obtained to obtain the target image data. The frame extraction strategy may also be to extract frames according to a preset ratio value. For example, if the video to be predicted has a total of 12 frames of images and the preset ratio value is 0.5, then 6 frames of images of the video to be predicted are extracted, and the 6 frames of images should be as dispersed as possible, for example, the first frame, the third frame, the fifth frame, the seventh frame, the ninth frame and the eleventh frame of the video to be predicted.

[0081] In step S102 of some embodiments, a target category prediction model is pre-trained, wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer, and a temporal pooling module. Each target image data is input into the trained target category prediction model, so that the feature extraction layer, the feature mapping layer, the local attention network, the global attention network, the feature fusion layer, and the temporal pooling module in the target category prediction model process the target image data, thereby realizing category prediction of the target image data.

[0082] In step S103 of some embodiments, the feature extraction layer may be ResNet-50, which is a model containing 50 convolutional layers in a deep residual network. The target image data is input into ResNet-50 for feature extraction to obtain the target image features, which are shown in the following formula (1):

[0083]

[0084] Among them, H, W and C represent the height, width and number of channels of the target image features respectively. represents the target image features of the t-th frame image in the video to be predicted, Represents the feature vector of the nth spatial position in the target image feature of the tth frame image, R C represents a real number. * represents two branches, *∈{SA,GP}, SA represents the local attention branch, GP represents the global attention branch, Can be or Represents the target image features or Input to different layers.

[0085] In step S104 of some embodiments, the feature mapping layer uses a function Φ() to convert the target image features Mapping to target mapping features The target mapping feature is shown in the following formula (2):

[0086]

[0087] in, represents the target mapping feature, Represents the target image features input to the feature mapping layer The feature value at the (H, W, C) position in the image is obtained by summing the 3D features using formula (2).

[0088] See also Figure 2In some embodiments, the local attention network includes a learning attention layer and a weighted summation layer, and step S105 may include but is not limited to steps S201 to S202:

[0089] Step S201, normalizing the target mapping features by learning the attention layer to obtain local normalized features; wherein the local normalized features represent the weights of features at different positions;

[0090] Step S202, performing weighted summation on the local normalized features and the target image features through a weighted summation layer to obtain the target local features.

[0091] In step S201 of some embodiments, the learning attention layer uses a softmax function to normalize the target mapping features Get local normalized features As shown in the following formula (3):

[0092]

[0093] in, It represents the weight of the (H, W) position, indicating the importance of the corresponding spatial position feature. exp represents the exponential operation with the natural constant e as the base.

[0094] In step S202 of some embodiments, the weighted sum layer performs the local normalization feature and target image features Perform weighted summation to obtain the target local features containing detail information As shown in the following formula (4):

[0095]

[0096] in, is the local feature of the target containing detailed information, Represents the target image features input to the feature mapping layer The eigenvalue at the (H,W) position in Represents the weight of the (H,W) position.

[0097] Through the above steps S201 to S202, the target local features of each target image data can be obtained, so as to perform feature fusion based on the target local features and the corresponding target global features.

[0098] In step S106 of some embodiments, the global attention network uses average pooling to process the C-dimensional target image features at HW spatial positions Convert 3D features into 1D features of fixed length to obtain the target global features As shown in the following formula (5):

[0099]

[0100] in, represents the target global feature of the t-th frame image, Represents the target image features The eigenvalue at the nth position in .

[0101] In step S107 of some embodiments, after obtaining the target local features and the target global features, a feature fusion layer is introduced to connect the target local features and the target global features in series to obtain a target composite feature containing rich information, as shown in the following formula (6):

[0102]

[0103] in, represents the target composite feature, represents the global characteristics of the target, It should be noted that it is necessary to obtain the target composite feature of each target image data to obtain multiple target composite features.

[0104] See also Figure 3 In some embodiments, the temporal pooling module includes a frame pooling layer and a relational reasoning layer, and step S108 may include but is not limited to steps S301 to S302:

[0105] Step S301, performing average pooling processing on multiple target composite features through a frame pooling layer to obtain multiple video clip features;

[0106] Step S302, performing temporal feature learning on each video clip feature through the relational reasoning layer to obtain a composite temporal feature.

[0107] In step S301 of some embodiments, the temporal pooling module infers the temporal relationship of behaviors from the video segment level, firstly divides multiple target composite features into multiple groups of video segments with the same time range in an equally spaced manner, and uses frame pooling layers with different time intervals to process the features contained in each video segment to obtain multiple video segment features, as shown in the following formula (7):

[0108]

[0109] in, represents the target composite feature after the fusion of the target local feature and the target global feature of the t-th frame image, Φ K () represents K-frame pooling operation, Represents the features of the sth video clip after K-frame pooling operation.

[0110] Specifically, taking three frame pooling layers as an example, 12 frames of video images are input, and 12 target composite features are obtained after feature extraction. Then the three frame pooling layers divide the 12 target composite features into four non-overlapping groups of video clips, and averagely pool the features contained in each group of video clips to obtain four video clip features.

[0111] In step S302 of some embodiments, the relational reasoning layer performs temporal feature learning on each video clip feature to obtain a composite temporal feature, as shown in the following formulas (8)-(9):

[0112]

[0113] The symbol [] indicates a serial operation. and They represent the parameter matrix of the K-frame relationship reasoning layer, P K represents the result of concatenating multiple video clip features, R K Indicates the composite timing characteristics of K frames.

[0114] Through the above steps S301 to S302, a composite time series feature can be obtained to perform category prediction on the target image data based on the composite time series feature.

[0115] See also Figure 4 In some embodiments, the target category prediction model further includes a simple cycle unit, and step S109 may include but is not limited to steps S401 to S402:

[0116] Step S401, performing category probability prediction on the composite time series feature through a simple cycle unit to obtain first category prediction data; wherein the first category prediction data includes a probability value of each video category;

[0117] Step S402: performing category prediction on the target image data according to the first category prediction data to obtain a predicted target category.

[0118] In step S401 to step S402 of the embodiment of the present application, a simple recurrent unit (SRU) is used to perform category probability prediction on the composite temporal features to obtain first category prediction data, which includes probability values ​​for each video category. The video category with the highest probability value in the first category prediction data is used as the predicted target category of the target image data. The predicted target category of the image in the video to be predicted is obtained, thereby realizing the video category prediction of the video to be predicted.

[0119] In the steps S101 to S109 shown in the embodiment of the present application, the target image data is obtained by acquiring the image of the video to be predicted according to the preset frame extraction strategy, and the target image data is input into the preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module. The target image data is subjected to feature extraction by the feature extraction layer to obtain the target image feature. The target image feature is subjected to feature mapping by the feature mapping layer to obtain the target mapping feature. The target mapping feature is subjected to local attention processing by the local attention network to obtain the target local feature. The target image feature is subjected to global attention processing by the global attention network to obtain the target global feature. The target local feature and the target global feature are subjected to feature fusion by the feature fusion layer to obtain the target composite feature. The target composite feature of each target image data is obtained to obtain multiple target composite features, and the multiple target composite features are subjected to temporal feature learning by the temporal pooling module to obtain the composite temporal feature. The target image data is subjected to category prediction according to the composite temporal feature to obtain the predicted target category. In this embodiment, after accurately extracting the target local features, prediction is performed based on the composite temporal features, thereby improving the accuracy of video image category prediction and further improving the accuracy of video category prediction.

[0120] See also Figure 5 In some embodiments, the target category prediction model further includes a regularization layer, and step S109 may also include but is not limited to steps S501 to S502:

[0121] Step S501, performing category probability prediction on the target global feature through a regularization layer to obtain second category prediction data; wherein the second category prediction data includes a probability value of each video category;

[0122] Step S502 , performing category prediction on the target image data according to the first category prediction data and the second category prediction data to obtain a predicted target category.

[0123] In step S501 of some embodiments, in order to further improve the accuracy of video category prediction, the category probability prediction is performed on the target global feature through a regularization layer to obtain second category prediction data; wherein the second category prediction data also includes the probability value of each video category.

[0124] Specifically, the category probability prediction is performed through regularization, for example, the category probability prediction can be performed through the softmax function.

[0125] In step S502 of some embodiments, the probability values ​​of the same category in the first category prediction data and the second category prediction data are added to obtain a total probability value for each video category, and the video category with the highest total probability value is used as the predicted target category of the target image data.

[0126] In one example, the probability values ​​of the same category in the first category prediction data and the second category prediction data may be weighted and then added together to obtain a total probability value of each video category, as shown in the following formula (10):

[0127] P final =aP GS +bP GP , (10)

[0128] Among them, P final Represents the total probability value of each video category, P GS Represents the probability value of the corresponding video category in the first category prediction data, P GP represents the probability value of the corresponding video category in the second category prediction data, a and b represent weights, and a+b=1.

[0129] In steps S501 to S502 shown in this embodiment, the category probability of the target global feature is predicted through the regularization layer to obtain the second category prediction data, thereby performing category prediction on the target image data based on the first category prediction data and the second category prediction data, thereby further improving the accuracy of video category prediction.

[0130] See also Figure 6 In some embodiments, before step S101, the video category prediction method based on the attention mechanism may also include, but is not limited to, building a target category prediction model, specifically including steps S601 to S604:

[0131] Step S601, obtaining image training data; wherein the image training data includes category labels;

[0132] Step S602, inputting the image training data into a preset original category prediction model for category prediction to obtain a predicted training category;

[0133] Step S603, performing loss calculation on the category label and the predicted training category using a preset classification loss function to obtain first loss data;

[0134] Step S604: Adjust the parameters of the original category prediction model based on the first loss data to obtain a target category prediction model.

[0135] In steps S601 to S602 of some embodiments, images in the training video are obtained and preprocessed to obtain image training data. The image training data includes a category label of the corresponding category. After the image training data is input into a preset original category prediction model for category prediction, a predicted training category is obtained.

[0136] In steps S603 to S604 of some embodiments, the loss of the category label and the predicted training category is calculated by a preset classification loss function to obtain first loss data. The parameters of the original category prediction model are adjusted based on the first loss data to obtain a target category prediction model. The classification loss function is shown in the following formula (11):

[0137]

[0138] Among them, t represents the t-th frame image, x represents the number of images in the training video, and y i represents the predicted training category of the t-th frame image, Represents the category label of the t-th frame image.

[0139] In steps S601 to S604 shown in this embodiment, loss calculation is performed on the category label and the predicted training category through a classification loss function to obtain first loss data, so as to adjust the parameters of the original category prediction model based on the first loss data to obtain a target category prediction model.

[0140] See also Figure 7 In some embodiments, after step S603, the video category prediction method based on the attention mechanism may also include but is not limited to steps S701 to S703:

[0141] Step S701, performing category probability prediction on the image training data through the original category prediction model to obtain first category prediction training data and second category prediction training data; wherein the first category prediction training data is obtained by performing category probability prediction on the composite time series features corresponding to the image training data; and the second category prediction training data is obtained by performing category probability prediction on the target global features corresponding to the image training data;

[0142] Step S702, performing loss calculation on the first category prediction training data and the second category prediction training data using a preset regularized loss function to obtain second loss data;

[0143] Step S703: Adjust the parameters of the original category prediction model based on the first loss data and the second loss data to obtain a target category prediction model.

[0144] In step S601 of some embodiments, in order to further improve the prediction accuracy of the target category prediction model, it is also necessary to perform category probability prediction on the image training data through the original category prediction model to obtain first category prediction training data and second category prediction training data. The first category prediction training data is obtained by performing category probability prediction on the composite time series features corresponding to the image training data, and the second category prediction training data is obtained by performing category probability prediction on the target global features corresponding to the image training data.

[0145] In steps S602 to S603 of some embodiments, the loss of the first category prediction training data and the second category prediction training data is calculated by a preset regularized loss function to obtain second loss data. The parameters of the original category prediction model are adjusted based on the first loss data and the second loss data to obtain the target category prediction model. The regularized loss function is shown in the following formula (12):

[0146] loss GL =max(0,P GP1 -P GL1 +γ), (12)

[0147] Among them, P GL1 Represents the probability value of the first category prediction training data, P GP1 represents the probability value of the second category prediction training data, γ represents a hyperparameter, indicating that the difference between the probability values ​​of the global branch and the attention branch should be greater than γ, and γ is set to 0.1. The final loss function loss is shown in the following formula (13):

[0148] loss=loss ce +loss GL , (13)

[0149] In steps S801 to S803 shown in this embodiment, a global attention regularization loss function is introduced to use the global information of the behavior to guide the attention branch to model more effective features, thereby further improving the accuracy of the target category prediction model.

[0150] See also Figure 8 The embodiment of the present application further provides a video category prediction device based on an attention mechanism, which can implement the above-mentioned video category prediction method based on an attention mechanism, and the device includes:

[0151] The acquisition module 801 is used to acquire the image of the video to be predicted according to a preset frame extraction strategy to obtain target image data;

[0152] An input module 802 is used to input each target image data into a preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module;

[0153] A feature extraction module 803 is used to extract features from each target image data through a feature extraction layer to obtain features of each target image;

[0154] A feature mapping module 804 is used to perform feature mapping on each target image feature through a feature mapping layer to obtain each target mapping feature;

[0155] A local processing module 805 is used to perform local attention processing on each target mapping feature through a local attention network to obtain each target local feature;

[0156] A global processing module 806 is used to perform global attention processing on each target image feature through a global attention network to obtain a global feature of each target;

[0157] A feature fusion module 807 is used to fuse each target local feature with the corresponding target global feature through a feature fusion layer to obtain multiple target composite features;

[0158] A feature learning module 808 is used to perform time series feature learning on multiple target composite features through a time series pooling module to obtain composite time series features;

[0159] The prediction module 809 is used to perform category prediction on the target image data according to the composite time series feature to obtain a predicted target category.

[0160] The specific implementation of the video category prediction device based on the attention mechanism is basically the same as the specific implementation of the video category prediction method based on the attention mechanism mentioned above, and will not be repeated here.

[0161] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned video category prediction method based on the attention mechanism when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0162] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0163] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video category prediction method based on the attention mechanism of the embodiment of this application;

[0165] Input / output interface 909, used to implement information input and output;

[0166] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 909, and the communication interface 904);

[0168] The processor 901 , the memory 902 , the input / output interface 909 and the communication interface 904 are connected to each other in communication within the device via the bus 905 .

[0169] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video category prediction method based on the attention mechanism is implemented.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video category prediction method based on attention mechanism, characterized in that: The method comprises: The image of the video to be predicted is obtained according to a preset frame extraction strategy to obtain target image data; Input each of the target image data into a preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module; Performing feature extraction on each target image data through the feature extraction layer to obtain each target image feature; Performing feature mapping on each of the target image features through the feature mapping layer to obtain each target mapping feature; Performing local attention processing on each of the target mapping features through the local attention network to obtain each target local feature; Performing global attention processing on each target image feature through the global attention network to obtain each target global feature; Performing feature fusion on each of the target local features and the corresponding target global features through the feature fusion layer to obtain multiple target composite features; Performing time series feature learning on the plurality of target composite features through the time series pooling module to obtain composite time series features; The target image data is subjected to category prediction according to the composite temporal features to obtain a predicted target category.

2. The method according to claim 1, characterized in that The local attention network includes a learning attention layer and a weighted summation layer, and the local attention network performs local attention processing on each of the target mapping features to obtain each target local feature, including: The target mapping feature is normalized by the learning attention layer to obtain a local normalized feature; wherein the local normalized feature represents the weight of features at different positions; The weighted summation layer performs weighted summation on the local normalized features and the target image features to obtain the target local features.

3. The method according to claim 1, characterized in that: The target category prediction model further includes a simple cycle unit, and the category prediction of the target image data is performed according to the composite time series feature to obtain the predicted target category, including: Performing category probability prediction on the composite time series feature through the simple cycle unit to obtain first category prediction data; wherein the first category prediction data includes a probability value of each video category; The target image data is subjected to category prediction according to the first category prediction data to obtain the predicted target category.

4. The method according to claim 3, characterized in that The target category prediction model further includes a regularization layer, and the performing category prediction on the target image data according to the composite temporal features to obtain the predicted target category further includes: Performing a category probability prediction on the target global feature through the regularization layer to obtain second category prediction data; wherein the second category prediction data includes a probability value of each video category; The target image data is subjected to category prediction according to the first category prediction data and the second category prediction data to obtain the predicted target category.

5. The method according to claim 1, characterized in that The temporal pooling module includes a frame pooling layer and a relational reasoning layer. The temporal pooling module performs temporal feature learning on a plurality of target composite features to obtain composite temporal features, including: Performing average pooling processing on the plurality of target composite features through the frame pooling layer to obtain a plurality of video clip features; The relational reasoning layer performs temporal feature learning on each of the video clip features to obtain the composite temporal feature.

6. The method according to claim 1, characterized in that Before obtaining the image of the video to be predicted according to the preset frame extraction strategy to obtain the target image data, the method further includes: Constructing the target category prediction model specifically includes: Acquire image training data; wherein the image training data includes category labels; Inputting the image training data into a preset original category prediction model for category prediction to obtain a predicted training category; Performing loss calculation on the category label and the predicted training category using a preset classification loss function to obtain first loss data; The parameters of the original category prediction model are adjusted based on the first loss data to obtain the target category prediction model.

7. The method according to claim 6, characterized in that After calculating the category label and the predicted training category loss using a preset target loss function to obtain first loss data, the method further includes: The image training data is subjected to category probability prediction by the original category prediction model to obtain first category prediction training data and second category prediction training data; wherein the first category prediction training data is obtained by performing category probability prediction on the composite time series features corresponding to the image training data; and the second category prediction training data is obtained by performing category probability prediction on the target global features corresponding to the image training data; Performing loss calculation on the first category prediction training data and the second category prediction training data using a preset regularized loss function to obtain second loss data; The parameters of the original category prediction model are adjusted based on the first loss data and the second loss data to obtain the target category prediction model.

8. A video category prediction device based on attention mechanism, characterized in that: The device comprises: An acquisition module is used to acquire the image of the video to be predicted according to a preset frame extraction strategy to obtain target image data; An input module, used to input each of the target image data into a preset target category prediction model; wherein the target category prediction model includes a feature extraction layer, a feature mapping layer, a local attention network, a global attention network, a feature fusion layer and a temporal pooling module; A feature extraction module, used for performing feature extraction on each target image data through the feature extraction layer to obtain features of each target image; A feature mapping module, used for performing feature mapping on each of the target image features through the feature mapping layer to obtain each target mapping feature; A local processing module, used for performing local attention processing on each of the target mapping features through the local attention network to obtain each target local feature and a target global feature; A global processing module, used for performing global attention processing on each target image feature through the global attention network to obtain each target global feature; A feature fusion module, used for fusing each of the target local features and the corresponding target global features through the feature fusion layer to obtain multiple target composite features; A feature learning module, used for performing time series feature learning on the plurality of target composite features through the time series pooling module to obtain composite time series features; The prediction module is used to perform category prediction on the target image data according to the composite temporal features to obtain a predicted target category.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video category prediction method based on the attention mechanism as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video category prediction method based on the attention mechanism described in any one of claims 1 to 7 is implemented.

Images