Video data processing method and device, computer device and storage medium

Abstract

The application discloses a video data processing method and device, computer equipment and a storage medium. The method comprises the following steps: extracting a target image frame from video data; obtaining an image feature map corresponding to the target image frame; the image feature map contains a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map; obtaining a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used for indicating the correlation degree between the feature parameters at each feature position; performing fusion processing on the feature parameters and the position feature embedding parameters at each feature position respectively to obtain fusion feature information at each feature position; and determining a video type of the video data according to the fusion feature information at each feature position. The application can improve the accuracy of the determined video type of the video data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video data processing, and particularly relates to a video data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the emergence of a large number of video applications, means for processing video data are also becoming more and more abundant, which includes processing of classifying video data to push video data of a type of interest to a user through classified video data.

[0003] In existing applications, a type label is usually selected for uploaded video data by a user when uploading the video data, or a type label is manually added to the video data by a technical staff in the background after the user uploads the video data, which not only consumes a large amount of manpower, but also does not unify the standard for classifying the video, resulting in inaccurate classification of the video. SUMMARY

[0004] The present application provides a video data processing method and device, computer equipment and a storage medium, which can improve the accuracy of the determined video type of the video data.

[0005] In one aspect, the present application provides a video data processing method, which comprises:

[0006] extracting a target image frame from the video data;

[0007] obtaining an image feature map corresponding to the target image frame; the image feature map contains a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map;

[0008] obtaining a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used to indicate the degree of association between the feature parameters at each feature position;

[0009] respectively performing fusion processing on the feature parameters and the position feature embedding parameters at each feature position to obtain fusion feature information at each feature position;

[0010] determining a video type of the video data according to the fusion feature information at each feature position.

[0011] In one aspect, the present application provides a video data processing device, which comprises:

[0012] an extraction module configured to extract a target image frame from the video data;

[0013] The acquisition module is configured to acquire an image feature map corresponding to the target image frame; the image feature map comprises a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map;

[0014] The acquisition module is configured to acquire a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used to indicate a correlation degree between the feature parameters at each feature position;

[0015] The fusion module is configured to perform fusion processing on the feature parameters and the position feature embedding parameters at each feature position respectively to obtain fusion feature information at each feature position.

[0016] The determination module is configured to determine a video type of the video data according to the fusion feature information at each feature position.

[0017] Optionally, the image feature map is acquired by calling a prediction network; the prediction network comprises an image feature extraction network and a type prediction network.

[0018] The acquisition module acquires the image feature map corresponding to the target image frame in the following manner:

[0019] The image feature extraction network in the prediction network is called to extract image features of the target image frame to obtain the image feature map.

[0020] The determination module determines the video type of the video data according to the fusion feature information at each feature position in the following manner:

[0021] The type prediction network in the prediction network is called to determine the video type of the video data according to the fusion feature information at each feature position.

[0022] Optionally, the acquisition module acquires the position feature embedding parameter at each feature position in the image feature map in the following manner:

[0023] The position feature embedding matrix is acquired from the image feature extraction network.

[0024] The position feature embedding parameter at each feature position is extracted from the position feature embedding matrix.

[0025] Optionally, the prediction network further comprises a text feature extraction network; and the apparatus is further configured to:

[0026] Acquire associated text data of the video data.

[0027] Call the text feature extraction network to extract text features of the associated text data to obtain a text feature matrix of the associated text data.

[0028] The determining module calls a type prediction network in the prediction network to determine a video type of the video data according to the fused feature information at each feature position, and the manner includes:

[0029] generating a fused feature matrix according to the fused feature information at each feature position;

[0030] performing splicing processing on the text feature matrix and the fused feature matrix to obtain a spliced feature matrix;

[0031] calling the type prediction network to determine the video type of the video data according to the spliced feature matrix.

[0032] Optionally, the apparatus is further configured to:

[0033] obtain sample video data and sample associated text data of the sample video data; the sample video data carries a video type label;

[0034] extract a sample image frame from the sample video data, and call an image feature extraction network to be trained in the initial prediction network to extract image features of the sample image frame, to obtain a sample image feature map corresponding to the sample image frame;

[0035] call the image feature extraction network to be trained to generate a sample fused feature matrix for the sample image frame according to the sample image feature map and an initial position feature embedding matrix; the initial position feature embedding matrix is obtained according to each feature position;

[0036] call a text feature extraction network to be trained in the initial prediction network to extract text features of the sample associated text data, to obtain a sample text feature matrix of the sample associated text data;

[0037] predict the video type of the sample video data according to the sample fused feature matrix and the sample text feature matrix;

[0038] update network parameters of the initial prediction network according to the predicted video type of the sample video data and an actual video type of the sample video data indicated by the video type label, to obtain the prediction network; the initial position feature embedding matrix belongs to the network parameters of the initial prediction network;

[0039] The prediction network includes a position feature embedding matrix, the position feature embedding matrix is an updated initial position feature embedding matrix, and the position feature embedding matrix includes position feature embedding parameters at each feature position.

[0040] Optionally, the initial position feature embedding matrix includes initial position embedding parameters at each feature position; and the apparatus is further configured to:

[0041] perform row encoding on each feature position of the sample image feature map, to obtain a row encoding value at each feature position.

[0042] perform column encoding on each feature position of the sample image feature map to obtain a column encoding value at each feature position;

[0043] perform summation processing on the row encoding value and the column encoding value at each feature position respectively to obtain an initial position embedding parameter at each feature position.

[0044] Optionally, the fusion module performs fusion processing on the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0045] perform summation fusion processing on the feature parameter and the position feature embedding parameter at each feature position respectively to obtain a fusion feature parameter at each feature position;

[0046] generate the fusion feature information at each feature position according to the fusion feature parameter at each feature position.

[0047] Optionally, the number of image feature maps is N, N being a positive integer; any one of the plurality of feature positions is denoted as an i-th feature position, i being a positive integer less than or equal to the total number of positions of the plurality of feature positions;

[0048] The fusion module generates the fusion feature information at each feature position according to the fusion feature parameter at each feature position in the following manner:

[0049] perform flattening processing on the fusion feature parameter at the i-th feature position in each image feature map to obtain the fusion feature information at the i-th feature position;

[0050] wherein the fusion feature information at the i-th feature position comprises N fusion feature parameters at the i-th feature position.

[0051] Optionally, the number of target image frames is a plurality;

[0052] The acquisition module acquires the image feature map corresponding to the target image frame, comprising:

[0053] generate a feature map corresponding to each target image frame respectively;

[0054] perform average fusion processing on the feature map corresponding to each target image frame respectively to obtain the image feature map.

[0055] Optionally, the extraction module extracts the target image frame from the video data in the following manner:

[0056] segment the video data to obtain a plurality of segmented video data;

[0057] extract an image frame from each of the segmented video data respectively, and determine the extracted image frame from each of the segmented video data as a target image frame.

[0058] Optionally, the apparatus is further configured to:

[0059] obtain a video preference type of the target object;

[0060] if the predicted video type of the video data belongs to the video preference type, push the video data to the target object.

[0061] In an aspect of the present disclosure, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to execute the method in the aspect of the present disclosure.

[0062] In an aspect of the present disclosure, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program comprising program instructions, the program instructions being executed by a processor to cause the processor to execute the method in the aspect of the present disclosure.

[0063] According to an aspect of the present disclosure, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the method provided in the aspect and various optional manners.

[0064] The present disclosure can extract a target image frame from video data, obtain an image feature map corresponding to the target image frame, the image feature map containing a plurality of feature parameters of the target image frame, each feature parameter corresponding to a feature position in the image feature map, obtain a position feature embedding parameter at each feature position in the image feature map, the position feature embedding parameter at each feature position being used to indicate a correlation degree between the feature parameters at each feature position, perform fusion processing on the feature parameters and the position feature embedding parameters at each feature position respectively to obtain fusion feature information at each feature position, and determine a video type of the video data according to the fusion feature information at each feature position. As can be seen, the method provided by the present disclosure can determine the video type of the video data through the image frame in the video data, and the video type is determined through the fusion feature information at each feature position in the image feature map, so that the feature space information (reflected through each feature position) in the image feature map is retained, and thus the determined video type of the video data is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these accompanying drawings without creative effort.

[0066] Figure 1 is a structural schematic diagram of a network architecture provided by an embodiment of the application;

[0067] Figure 2 is a scene schematic diagram of video type prediction provided by the application;

[0068] Figure 3 is a flow schematic diagram of a video data processing method provided by the application;

[0069] Figure 4a is a scene schematic diagram of obtaining fusion feature parameters at each feature position provided by the application;

[0070] Figure 4b is a scene schematic diagram of generating a fusion feature matrix provided by the application;

[0071] Figure 5 is a scene schematic diagram of video pushing provided by the application;

[0072] Figure 6 is a scene schematic diagram of video type prediction provided by the application;

[0073] Figure 7 is a flow schematic diagram of a video data processing method provided by the application;

[0074] Figure 8 is a scene schematic diagram of generating an initial position feature embedding matrix provided by the application;

[0075] Figure 9 is a scene schematic diagram of model training provided by the application;

[0076] Figure 10 is a structural schematic diagram of a video data processing apparatus provided by the application;

[0077] Figure 11 is a structural schematic diagram of a computer device provided by the application. DETAILED DESCRIPTION

[0078] With reference to the drawings, the technical solutions in the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of the present application.

[0079] The present application relates to artificial intelligence related technologies. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0080] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0081] In the present application, machine learning in artificial intelligence is mainly involved. Among them, machine learning (Machine Learning, ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and example-based learning.

[0082] Machine learning involved in the present application mainly refers to how to train a prediction model to predict the video type of video data through the trained prediction model, which can be specifically referred to in the description of the corresponding embodiments below. Figure 3

[0083] Please refer to Figure 1 , Figure 1 ​is a structural schematic diagram of a network architecture provided by an embodiment of the present application. As shown in Figure 1 , the network architecture can include a server 200 and a terminal device cluster, which can include one or more terminal devices, and the number of terminal devices will not be limited here. As shown in Figure 1 , the plurality of terminal devices can specifically include a terminal device 100a, a terminal device 101a, a terminal device 102a, …, and a terminal device 103a; as shown in Figure 1 , the terminal device 100a, the terminal device 101a, the terminal device 102a, …, and the terminal device 103a can all be network connected with the server 200, so that each terminal device can perform data interaction between the terminal device and the server 200 through the network connection.

[0084] As shown in Figure 1 , the server 200 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device can be a smart terminal such as a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart television, a vehicle-mounted terminal, etc. The following will take the communication between the terminal device 100a and the server 200 as an example to describe the embodiments of the present application in detail.

[0085] Please refer to Figure 2 , Figure 2 is a scenario schematic diagram of video type prediction provided by the present application. Among them, the terminal device 100a can be a terminal device of a target object, the target object can be a user of a video client, the server 200 can be a background server of the video client, the server 200 can classify various video data in a database, and the server can also know the video preference type of the target object, which is the type of video that the target object prefers to watch. The video preference type can be set by the user himself, or can be calculated by the server 200 according to the user's video watching behavior. Therefore, the server 200 can push the video data classified as the video preference type of the target object to the target object, i.e., to the terminal device 100a, and the terminal device 100a can display the video data pushed by the server 200 in the client page of the video client for the target object to browse and watch.

[0086] Therefore, it can be known that the server 200 needs to determine the video type of each video data before classifying the various video data. Here, the process of the server 200 determining the video data s is taken as an example to specifically explain the principle of the server determining the video type of each video data.

[0087] like Figure 2 As shown, first, the server 200 can extract image frames from the video data s, and then input the extracted image frames into the prediction network, which is a trained network that can be used to predict the video type of the video data. The specific training process of the prediction network can be seen in the following Figure 7 The method of extracting image frames from the video data s can be referred to as follows: Figure 3 The description of extracting the target image frame from the video data in the corresponding embodiment.

[0088] Therefore, an image feature map corresponding to the input image frame can be generated in the prediction network. The image feature map may include multiple feature parameters learned by the prediction network from the input image frame. The multiple feature parameters belong to multiple feature values ​​(i.e., multiple feature elements) in the image feature map. Here, the multiple feature parameters may specifically include feature parameter 1, feature parameter 2, feature parameter 3, feature parameter 4, feature parameter 5, feature parameter 6, feature parameter 7, feature parameter 8, and feature parameter 9. The generation method of the image feature map can be referred to as follows Figure 3 Description of generating an image feature map corresponding to a target image frame in the corresponding embodiment.

[0089] Among them, one characteristic parameter can correspond to a characteristic position, such as characteristic parameter 1 can correspond to characteristic position 1, characteristic parameter 2 can correspond to characteristic position 2, characteristic parameter 3 can correspond to characteristic position 3, characteristic parameter 4 can correspond to characteristic position 4, characteristic parameter 5 can correspond to characteristic position 5, characteristic parameter 6 can correspond to characteristic position 6, characteristic parameter 7 can correspond to characteristic position 7, characteristic parameter 8 can correspond to characteristic position 8, and characteristic parameter 9 can correspond to characteristic position 9.

[0090] Therefore, the server 200 can obtain the image features of the image frame according to each feature position, which can retain the spatial information of the image features, such as obtaining the image features at feature position 1, obtaining the image features at feature position 2, obtaining the image features at feature position 3, obtaining the image features at feature position 4, obtaining the image features at feature position 5, obtaining the image features at feature position 6, obtaining the image features at feature position 7, obtaining the image features at feature position 8, and obtaining the image features at feature position 9. The process of the server 200 obtaining the image features at each feature position can be referred to as follows Figure 3According to the description in the corresponding embodiment, the image features at each feature position are the fused feature information at each feature position in the image feature map. Further, the server can determine the video type of the video data s by the image features at each feature position in the image feature map.

[0091] By using the method provided in the present application, the image features at each feature position in the image feature map are obtained, the spatial information between the image features is retained, and the video type of the video data can be more accurately predicted by the image features at each feature position.

[0092] Please refer to Figure 3 , Figure 3 is a flowchart of a video data processing method provided in the present application. The execution subject in the embodiment of the present application can be a computer device or a computer device cluster composed of multiple computer devices. The computer device can be a server, a terminal device, or the like. The following will be described by taking the execution subject in the embodiment of the present application as a computer device as an example. As shown in Figure 3 , the method can include:

[0093] Step S101, extracting a target image frame from the video data.

[0094] Optionally, the video data can be any video that needs to determine the video type, and the video data can be uploaded by a user or obtained from a webpage. For example, the video type of the video data can be health, sports, entertainment, game, or movie, etc.

[0095] The way of extracting the target image frame from the video data can be:

[0096] The computer device can segment the video data, and then divide the video data into multiple segments. Each segment obtained by the segmentation can be referred to as a segmented video data, and one segmented video data can include multiple image frames. Further, the computer device can extract image frames from each segmented video data, and then the extracted image frames from each segmented video data can be regarded as target image frames.

[0097] For example, the video data can be segmented to obtain 16 segmented video data, and then one image frame can be randomly extracted from each of the 16 segmented video data, that is, one image frame is extracted from one segmented video data. Further, the 16 image frames extracted from the segmented video data can be regarded as target image frames, and a total of 16 target image frames can be obtained.

[0098] In step S102, an image feature map corresponding to the target image frame is obtained; the image feature map contains a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map.

[0099] Optionally, a trained prediction network can be invoked to generate the image feature map corresponding to the target image frame, and the training process of the prediction network can refer to the following Figure 7 The specific description of the corresponding embodiments.

[0100] The prediction network can include an image feature extraction network (a trained image feature extraction network), which can be used to extract (i.e., learn) image features in image data.

[0101] Since there can be multiple target image frames, the computer device can invoke the image feature extraction network to extract image features of each target image frame, and then obtain a feature map of each target image frame. The feature map of a target image frame contains feature parameters (such as feature values) extracted from the target image frame by the image feature extraction network. The feature map of the target image frame is represented as a feature matrix in the prediction network. Each element (i.e., each numerical value) in the feature matrix can be referred to as a feature parameter in the feature map. One feature parameter can correspond to one feature position. For example, one feature position can be the position of the first row and the first column in the feature matrix. For another example, one feature position can be the position of the second row and the third column in the feature matrix.

[0102] Further, the computer device can perform average fusion processing on the feature map corresponding to each target image frame, i.e., obtain the image feature map corresponding to the multiple target image frames. The sizes (i.e., dimensions) of the feature maps of the target image frames are the same. The average fusion processing on the feature maps corresponding to the multiple target image frames can refer to averaging the feature parameters at the same feature positions in the feature maps corresponding to the multiple target image frames. The average values obtained at each feature position can be used to obtain the image feature map.

[0103] It needs to be noted that if there are multiple feature channels, each target image frame can correspond to a feature map under each feature channel, therefore, one feature channel can correspond to one image feature map, and the image feature map corresponding to one feature channel is obtained by performing average fusion processing on the feature maps of each target image frame under the feature channel. In other words, there can be multiple image feature maps, and the dimensions (i.e. size) of the image feature map and the feature map of each target image frame are the same. The feature channel is also the channel in deep learning, one feature channel can correspond to the extraction of the feature in one dimension of the image, therefore, the more the feature channels, the more the dimensions of the image features extracted when extracting the image features.

[0104] The image feature map can include multiple feature parameters for the target image frame, one feature parameter corresponding to one feature position in the image feature map, and the feature parameters at each feature position in the image feature map being obtained by performing average operation on the feature parameters at the same feature position in the feature maps of each target image frame (i.e. calculating the average value, which is the average parameter). For example, if the dimension of the image feature map is 3*3, the image feature map can include 9 feature parameters (i.e. 9 feature elements), and the 9 feature parameters correspond to 9 feature positions.

[0105] In step S103, a position feature embedding parameter at each feature position in the image feature map is obtained; the position feature embedding parameter at each feature position is used to indicate the degree of association between the feature parameters at each feature position.

[0106] Optionally, the position feature embedding parameter at each feature position in the image feature map is used to indicate the position of each feature parameter in the image feature map, and is also used to indicate the degree of association between the feature parameters at each feature position. For example, since the position of each feature parameter in the image feature map can be known through the position feature embedding parameter at each feature position, the distance between each feature parameter in the image feature map can be known through the position feature embedding parameter at each feature position. Therefore, the farther the distance between two feature parameters, the smaller the degree of association between them, and vice versa. Since the feature parameters at each feature position interact and predict each other (for example, when predicting the type of video), it can be understood that the interaction between feature parameters with a smaller degree of association has a smaller impact on each other, and the interaction between feature parameters with a larger degree of association has a larger impact on each other. Therefore, by incorporating the corresponding position feature embedding parameter into the feature parameters at each feature position in the image feature map, the spatial information (such as position information) between the feature parameters is considered during prediction. The degree of association between each feature parameter is considered through this spatial information. Therefore, by fusing the position feature embedding parameter and the image feature map, the prediction accuracy of the type of video data can be improved subsequently.

[0107] The computer device can obtain the position feature embedding parameter at each feature position in the image feature map in the following way: the computer device can obtain a position feature embedding matrix from the prediction network (such as from the image feature extraction network in the prediction network). The position feature embedding matrix belongs to the network parameters of the prediction network, and is updated during the training of the prediction network. The specific way of updating and obtaining the position feature embedding matrix can be found in the following Figure 7 The process of training the initial prediction network to obtain the prediction network in the corresponding embodiment. The position feature embedding matrix is a general embedding matrix determined when the prediction network is trained. That is, the position feature embedding matrix can be used for the prediction of the type of any video data.

[0108] The size of the position feature embedding matrix is the same as that of the image feature map (in fact, the image feature map can also be represented as a matrix). The feature positions in the position feature embedding matrix correspond one-to-one to the feature positions in the image feature map. Therefore, the computer device can extract the position feature embedding parameter at each feature position from the position feature embedding matrix. The position feature embedding parameter belongs to the feature parameter (i.e., the feature value) in the position feature embedding matrix.

[0109] Step S104, respectively, the feature parameters and position feature embedding parameters at each feature position are fused to obtain the fusion feature information at each feature position.

[0110] Optionally, the computer device can fuse the feature parameters and the position feature embedding parameters at each feature position in the image feature map, and then the fusion feature information at each feature position can be obtained. Wherein, the computer device fuses the feature parameters and the position feature embedding parameters at each feature position in the image feature map can mean that the computer device respectively adds and fuses the feature parameters and the position feature embedding parameters at each feature position in the image feature map, and then the fusion feature parameters at each feature position can be obtained, that is, the fusion feature parameters at a feature position can be the sum of the feature parameters at the feature position and the position feature embedding parameters at the feature position in the image feature map.

[0111] Further, the computer device can generate the fusion feature information at each feature position through the fusion feature parameters at each feature position, which is described as follows:

[0112] Wherein, the number of image feature maps can be N, N is a positive integer, and the specific value of N is determined according to the actual application scene. It can be known that the feature parameters at each feature position in each image feature map can be added and fused with the corresponding position feature embedding parameters to obtain the fusion feature parameters at the corresponding feature position in the corresponding image feature map. In other words, the corresponding fusion feature parameters can be calculated at each feature position in an image feature map.

[0113] The feature positions related to the image in the present application are applicable to any image feature map or position feature embedding matrix, the sizes of all image feature maps and position feature embedding matrices are the same, a position of an element in the image feature map or the position feature embedding can be a feature position, the number of feature positions in each image feature map or position feature embedding matrix is the same, and they are one-to-one. Any one of the above plurality of feature positions can be represented as the ith feature position, i is a positive integer less than or equal to the total number of positions of the plurality of feature positions.

[0114] The computer device can perform flattening processing on the fusion feature parameter at the i-th feature position in each image feature map to obtain fusion feature information at the i-th feature position. The flattening processing on the fusion feature parameter at the i-th feature position in each image feature map can mean generating a feature vector corresponding to the fusion feature parameter at the i-th feature position in each image feature map, and the feature vector is the fusion feature information at the i-th feature position. One image feature map can include one fusion feature parameter at the i-th feature position. Therefore, it can be understood that the fusion feature information at the i-th feature position includes N fusion feature parameters at the i-th feature position, and the N fusion feature parameters are distributed in N image feature maps.

[0115] In step S105, the video type of the video data is determined according to the fusion feature information at each feature position.

[0116] Optionally, the prediction network described above further includes a trained type prediction network, and the computer device can call the type prediction network to determine the video type of the video data according to the fusion feature information at each feature position. The training process of the type prediction network can also be referred to in the following Figure 7 The specific description in the corresponding embodiment.

[0117] The computer device can further generate a fusion feature matrix according to the fusion feature information at each feature position of the image feature map. Since the fusion feature information at one feature position of the image feature map can be a feature vector (horizontal), the fusion feature matrix is composed of the fusion feature information at each feature position of the image feature map. The number of rows of the fusion feature matrix is the total number of positions of the plurality of feature positions in the image feature map. For example, if the dimension of an image feature map is 3*3, the number of rows of the fusion feature matrix is equal to 9, and the number of columns of the fusion feature matrix is equal to the number of image feature maps (i.e., the number of channels). For example, if the number of channels is equal to 768, the number of columns of the fusion feature matrix is equal to 768, that is, the fusion feature information at one feature position of the image feature map is a feature vector with a dimension of 1*768.

[0118] The computer device can only predict the video type of the video data according to the fusion feature matrix generated according to the target image frame. Therefore, the computer device can input the fusion feature matrix into a type prediction network, and then call the type prediction network to predict the video type of the video data according to the fusion feature matrix. The type prediction network called here can be the type prediction network described below Figure 7The sample fusion feature matrix is obtained by training sample video data and sample associated text data together, and the type prediction network to be trained below is a sample splicing feature matrix obtained by splicing a sample fusion feature matrix and a sample text feature matrix, therefore, if only the fusion feature matrix is used to predict the video type of the video data here, a matrix (which can be referred to as a zero matrix, and the feature parameters in the zero matrix can all be 0) of the same size as the sample text feature matrix filled with 0 elements can be spliced with the fusion feature matrix, and then the spliced matrix (ensuring that the matrix has the same dimension size as the sample splicing feature matrix during training) is input into the type prediction network to predict the video type of the video data.

[0119] Please refer to Figure 4a - Figure 4b , Figure 4a is a scene diagram provided by the present application for obtaining fusion feature parameters at each feature position, Figure 4b is a scene diagram provided by the present application for generating a fusion feature matrix. As Figure 4a indicated, it is assumed that there are 3 image feature maps, including image feature map t1, image feature map t2 and image feature map t3. Figure 1

[0120] Among them, the dimension of each image feature map is 2*2, and image feature map t1, image feature map t2 and image feature map t3 each contain feature parameters at 4 feature positions (assuming that the 4 feature positions are feature position 1, feature position 2, feature position 3 and feature position 4, respectively). Specifically, the feature parameters at feature position 1 can include feature parameter 1 in image feature map t1, feature parameter 11 in image feature map t2 and feature parameter 111 in image feature map t3; the feature parameters at feature position 2 can include feature parameter 2 in image feature map t1, feature parameter 22 in image feature map t2 and feature parameter 222 in image feature map t3; the feature parameters at feature position 3 can include feature parameter 3 in image feature map t1, feature parameter 33 in image feature map t2 and feature parameter 333 in image feature map t3; and the feature parameters at feature position 4 can include feature parameter 4 in image feature map t1, feature parameter 44 in image feature map t2 and feature parameter 444 in image feature map t3.

[0121] The position feature embedding matrix of the image can also include position feature embedding parameters at the above-mentioned 4 feature positions, such as position feature embedding parameter 1 at feature position 1, position feature embedding parameter 2 at feature position 2, position feature embedding parameter 3 at feature position 3 and position feature embedding parameter 4 at feature position 4.

[0122] ​Therefore, the feature parameters in each image feature map can be added with the position feature embedding parameters of the position features in the same feature position in the position feature embedding matrix to obtain the fusion feature parameters in the corresponding feature position in the corresponding image feature map.

[0123] As shown in Figure 4a For the image feature map t1, the fusion feature parameter at the feature position 1 can be the sum of the feature parameter 1 and the position feature embedding parameter 1 (which can be denoted as fusion feature parameter 1), the fusion feature parameter at the feature position 2 can be the sum of the feature parameter 2 and the position feature embedding parameter 2 (which can be denoted as fusion feature parameter 2), the fusion feature parameter at the feature position 3 can be the sum of the feature parameter 3 and the position feature embedding parameter 3 (which can be denoted as fusion feature parameter 3), and the fusion feature parameter at the feature position 4 can be the sum of the feature parameter 4 and the position feature embedding parameter 4 (which can be denoted as fusion feature parameter 4).

[0124] For the image feature map t2, the fusion feature parameter at the feature position 1 can be the sum of the feature parameter 11 and the position feature embedding parameter 1 (which can be denoted as fusion feature parameter 11), the fusion feature parameter at the feature position 2 can be the sum of the feature parameter 22 and the position feature embedding parameter 2 (which can be denoted as fusion feature parameter 22), the fusion feature parameter at the feature position 3 can be the sum of the feature parameter 33 and the position feature embedding parameter 3 (which can be denoted as fusion feature parameter 33), and the fusion feature parameter at the feature position 4 can be the sum of the feature parameter 44 and the position feature embedding parameter 4 (which can be denoted as fusion feature parameter 44).

[0125] For the image feature map t3, the fusion feature parameter at the feature position 1 can be the sum of the feature parameter 111 and the position feature embedding parameter 1 (which can be denoted as fusion feature parameter 111), the fusion feature parameter at the feature position 2 can be the sum of the feature parameter 222 and the position feature embedding parameter 2 (which can be denoted as fusion feature parameter 222), the fusion feature parameter at the feature position 3 can be the sum of the feature parameter 333 and the position feature embedding parameter 3 (which can be denoted as fusion feature parameter 333), and the fusion feature parameter at the feature position 4 can be the sum of the feature parameter 444 and the position feature embedding parameter 4 (which can be denoted as fusion feature parameter 444).

[0126] Therefore, as shown in Figure 4bAs shown, the fusion feature information at the feature position 1 can be a feature vector composed of the fusion feature parameter 1, the fusion feature parameter 11 and the fusion feature parameter 111; the fusion feature information at the feature position 2 can be a feature vector composed of the fusion feature parameter 2, the fusion feature parameter 22 and the fusion feature parameter 222; the fusion feature information at the feature position 3 can be a feature vector composed of the fusion feature parameter 3, the fusion feature parameter 33 and the fusion feature parameter 333; and the fusion feature information at the feature position 4 can be a feature vector composed of the fusion feature parameter 4, the fusion feature parameter 44 and the fusion feature parameter 444. The fusion feature matrix can include the fusion feature parameter 1, the fusion feature parameter 11, the fusion feature parameter 111, the fusion feature parameter 2, the fusion feature parameter 22, the fusion feature parameter 222, the fusion feature parameter 3, the fusion feature parameter 33, the fusion feature parameter 333, the fusion feature parameter 4, the fusion feature parameter 44 and the fusion feature parameter 444.

[0127] Optionally, the computer device can also determine the video type of the video data in combination with associated text data of the video data. The associated text data can be title information of the video data, or the associated text data can also be text data obtained by recognizing audio in the video data, i.e., text obtained by text conversion of the audio in the video data, or the associated text data can also be part of the text obtained by text conversion of the audio in the video data, which can be text containing a video keyword, i.e., a word in a pre-constructed keyword library. Alternatively, the associated text data can also be text content recognized from a target image frame of the video data.

[0128] Therefore, the computer device can obtain the associated text data of the video data. The prediction network can also include a trained text feature extraction network, and the training process of the text feature extraction network can also refer to the following Figure 7 corresponding embodiments.

[0129] Therefore, the computer device can call the text feature extraction network to extract text features of the associated text data, and then obtain a text feature matrix of the associated text data, which can be:

[0130] The computer device can first call the text feature extraction network to extract text features of the associated text data, and obtain a feature matrix of the associated text data. The number of rows of the feature matrix can be a maximum number of segmented words to be obtained from the associated text data, and the number of columns of the feature matrix can be a channel number of text feature extraction, which is the same as the channel number of the image, i.e., equal to the number of image feature maps, such as 768.

[0131] Further, the computer device can also obtain a text position feature embedding matrix for the text from the prediction network (e.g., from the text feature extraction network), the text position feature embedding matrix also belongs to the network parameters of the prediction network, the text position feature embedding matrix is updated during the process of training the prediction network, and the specific updating manner of the text position feature embedding matrix can be referred to below Figure 7 The process of training the initial prediction network to obtain the prediction network in the corresponding embodiment. The text position feature embedding matrix is a general embedding matrix determined when the prediction network is trained, that is, for any scene of predicting the type of video data in combination with text data, the text position feature embedding matrix can be fused.

[0132] Specifically, the size of the text position feature embedding matrix is the same as that of the feature matrix of the associated text data, and the feature positions in the text position feature embedding matrix correspond one-to-one to the feature positions in the feature matrix of the associated text data. Therefore, the computer device can extract the text position feature embedding parameters at each feature position in the feature matrix of the associated text data from the text position feature embedding matrix, the text position feature embedding parameters belong to the feature parameters (i.e., feature values) in the text position feature embedding matrix, which are the same as the concept of the position feature embedding parameters at each feature position in the image feature map, and the text position feature embedding parameters at each feature position in the feature matrix of the associated text data can also be used to indicate the degree of association between the feature parameters in the feature matrix of the associated text data.

[0133] Therefore, the computer device can add the feature matrix of the associated text data and the text position feature embedding matrix, that is, the feature parameters (i.e., elements, i.e., feature values) at each feature position in the feature matrix of the associated text data and the corresponding text position feature embedding parameters are summed, respectively, to obtain the text feature matrix of the associated text data. The size of the text position feature embedding matrix can be different from the size of the above-mentioned position feature embedding matrix for the image, but the number of columns of the text position feature embedding matrix needs to be the same as that of the above-mentioned position feature embedding matrix for the image, that is, the dimension of the row can be different, but the dimension of the column is the same. That is, the number of image feature maps (i.e., the number of channels) is the same as the number of columns of the text feature matrix.

[0134] Further, the computer device can splice the text feature matrix and the fusion feature matrix, such as up-down splicing (because the column numbers are the same), and the spliced feature matrix can be referred to as a spliced feature matrix, and the row number of the spliced feature matrix is equal to the sum of the row number of the text feature matrix and the row number of the fusion feature matrix. Further, the computer device can input the spliced feature matrix into the type prediction network, and call the type prediction network to determine (i.e., predict) the video type of the video data according to the spliced feature matrix. The computer device can determine the video type of a large amount of video data through the above process, and then classify (such as categorize) the video data according to the determined video type of the video data or determine the strategy of pushing the video data.

[0135] For example, the computer device can be a background server of a video client, and the video client has a target object, which can be any user of the video client. Therefore, the computer device can obtain the video preference type of the target object for the video, that is, the type of the video that the target object prefers to watch, and if the video preference type contains the determined video type of the video data, the video data can be pushed to the target object, such as sending the video data to the video client on the terminal device of the target object.

[0136] Please refer to Figure 5 , Figure 5 is a scene schematic diagram of video pushing provided by the present application. As shown in Figure 5 , the fusion feature matrix of the video data can be generated by the image feature extraction network in the prediction network, the text feature matrix of the associated text data of the video data can be generated by the text feature extraction network in the prediction network, and then the text feature matrix and the fusion feature matrix are spliced and input into the type prediction network in the prediction network, that is, the video type of the video data can be predicted. If the video preference type of the target object contains the predicted video type of the video data, the video data can be pushed to the terminal device of the target object.

[0137] Please refer to Figure 6 , Figure 6 is a scene schematic diagram of video type prediction provided by the present application. As shown in Figure 6As shown, the text features of the associated text data can be extracted in the text feature extraction network 100c to obtain a text feature matrix 103c: the associated text data of the video data can be "excessive anxiety may cause insomnia, do you have insomnia symptoms", the associated text data "excessive anxiety may cause insomnia, do you have insomnia symptoms" can be encoded by the text feature extraction network (after encoding, CLS excessive… symptom SEP is obtained), and then the semantic learning of the encoded associated text data is performed, so that the token embedding of the associated text data (that is, the feature matrix of the associated text data) is obtained. The text position embedding here can be the text position feature embedding matrix described above, and then the token embedding and the text position embedding of the associated text data are fused, so that the text feature matrix 103c described above is obtained. The text label here is used to inform the prediction network that the input associated text data is the data type of text. CLS is an added encoding identification bit (representing the first bit), and SEP is a sentence separator.

[0138] The image feature extraction network 101c can be composed of resnet50 (a residual network). The computer device can input the target image frame extracted from the video data into the resnet50 network to extract image features, and then perform convolution and pooling on the learned image features (the convolution and pooling here are used to reduce the feature dimension), so that the image embedding of the target image frame is obtained. The image embedding can be the image feature map corresponding to the target image frame, the image position embedding here can be the position feature embedding matrix for the image, and the fusion feature matrix 104c described above can be obtained by performing convolution on the image embedding and the image position embedding of the target image frame. The video label here is used to inform the prediction network that the input video data is the data type of video.

[0139] Further, the computer device can splice the obtained fusion feature matrix 104c and the text feature matrix 103c to obtain a spliced feature matrix 105c, and input the spliced feature matrix 105c into a type prediction network 102c (which is composed of a Transformer network, and the Transformer is a language network), and the type prediction network 102c can predict the video type (i.e., the classification result here) of the video data according to the spliced feature matrix 105c. For example, the classification result can include the probabilities of the video data predicted by the type prediction network 102c as various video types, and the various video types can be a plurality of video types learned by the type prediction network 102c in advance, such as a health video type, an entertainment video type, an education video type, and a technology video type. Therefore, the classification result can include a probability of 0.7 that the video data predicted by the type prediction network 102c is a health video type, a probability of 0.1 that the video data is an entertainment video type, a probability of 0.1 that the video data is an education video type, and a probability of 0.1 that the video data is a technology video type. Therefore, the computer device can take the video type corresponding to the largest probability (i.e., the health video type) as the video type of the video data predicted finally.

[0140] It should be noted that in the prior art, after generating the plurality of image feature maps corresponding to the target image frame, the feature parameters in each image feature map are usually screened, and then one feature parameter is retained from one image feature map, for example, only the feature parameter with the largest value in each image feature map is retained, and then a corresponding feature vector can be generated through the feature parameter with the largest value in each image feature map, that is, the feature vector contains the feature parameter with the largest value in each image feature map, and then in the prior art, the video type of the video data is predicted through the feature vector, and the feature vector has a similar function to the fusion feature matrix in the embodiments of the present application.

[0141] It can be seen that in the prior application, a feature parameter is usually screened and reserved in an image feature map, which causes the obtained feature vector to miss the spatial information between the feature parameters at each feature position in the image feature map. In the present application, after obtaining the multiple image feature maps corresponding to the target image frame, the features at each feature position in the image feature map are obtained in units of each feature position in the image feature map. The feature at each feature position can be the fusion feature information at the feature position, and the fusion feature information can be a feature vector. Therefore, one feature position corresponds to one feature vector in the present application, and the fusion feature matrix in the present application contains the feature vectors at each feature position in the image feature map. It can be seen that the fusion feature matrix in the embodiment of the present application reserves the features at each feature position and considers the spatial information between the features (such as fusion feature information) at each feature position. Therefore, the accuracy of predicting the video type of the video data by using the fusion feature matrix in the embodiment of the present application is greatly improved compared with the accuracy of predicting the video type of the video data by using only one feature vector corresponding to all feature positions in the prior application.

[0142] In addition, in the embodiment of the present application, the fusion feature information at each feature position not only contains the feature parameters in the image feature map, but also contains the corresponding position feature embedding parameters fused with the feature parameters at each feature position, so that the positional relationship between the feature parameters at each feature position is more accurate, and the fusion feature information at each feature position is also more accurate. Therefore, predicting the video type of the video data by using the fusion feature information at each feature position is also more accurate.

[0143] The present application can extract a target image frame from video data, obtain an image feature map corresponding to the target image frame, the image feature map containing multiple feature parameters of the target image frame, each feature parameter corresponding to a feature position in the image feature map, obtain a position feature embedding parameter at each feature position in the image feature map, the position feature embedding parameter at each feature position being used to indicate the correlation degree between the feature parameters at each feature position, perform fusion processing on the feature parameters and the position feature embedding parameters at each feature position respectively to obtain fusion feature information at each feature position, and determine the video type of the video data according to the fusion feature information at each feature position. It can be seen that the method proposed in the present application can determine the video type of the video data by using the image frame in the video data, and the video type is determined by using the fusion feature information at each feature position in the image feature map, so that the feature spatial information (reflected by each feature position) in the image feature map is reserved, and therefore the determined video type of the video data is more accurate.

[0144] Please refer to Figure 7 ,Figure 7 is a flowchart of a video data processing method provided by the present application. The execution subject in the embodiment of the present application can also be a computer device, and the embodiment of the present application describes the process of training an initial prediction network to obtain a prediction network. As shown in Figure 7 , the method can include:

[0145] In step S201, sample video data and sample associated text data of the sample video data are obtained, and the sample video data carries a video type label.

[0146] Optionally, the computer device can obtain sample video data and sample associated text data of the sample video data, the sample video data and the sample associated text data being data for training an initial prediction network, the sample video data being any video data, and the sample video data carrying a video type label, the video type label indicating an actual type (i.e., an actual video type) of the sample video data, such as a health, entertainment, or finance type. The sample associated text data can be title information of the sample video data, or can be text data obtained by recognizing audio of the sample video data.

[0147] In step S202, a sample image frame is extracted from the sample video data, and an image feature extraction network to be trained in the initial prediction network is called to extract image features of the sample image frame, to obtain a sample image feature map corresponding to the sample image frame.

[0148] Optionally, the computer device can extract a sample image frame from the sample video data, where the computer device extracts the sample image frame from the sample video data in the same manner as the computer device extracts the target image frame from the video data, and details can be referred to the description in the above Figure 3 corresponding embodiment.

[0149] The initial prediction network includes an image feature extraction network to be trained, a text feature extraction network, and a type prediction network. The image feature extraction network can be a resnet50 network (a residual network), through which image features of each sample image frame can be extracted to obtain a feature map of each sample image frame. The image feature extraction network can include a plurality of convolution blocks (for extracting image features), and when extracting image features of each sample image frame through the image feature extraction network, the feature map of each sample image frame can be output through the fifth convolution block in the plurality of convolution blocks. In this way, the network depth during image feature extraction is moderate, and the output feature map is more accurate. The specific convolution block used to output the feature map of each sample image frame can be determined according to an actual application scenario, and is not limited in this regard. The text feature extraction network can be a BERT network (Bidirectional Encoder Representations from Transformers, a language network), and the type prediction network can be a transformer network (a graph neural network). The type prediction network in this application can be obtained by connecting three transformer network layers in series. By using three transformer network layers as the type prediction network, the network depth is not too large and is easy to train on the basis of comprehensive learning and prediction of features. The number of transformer networks constituting the type prediction network can also be determined according to an actual application scenario, and is not limited in this regard.

[0150] The computer device can call the image feature extraction network to be trained in the initial prediction network to extract image features of the sample image frame, and then obtain a sample image feature map corresponding to the sample image frame. The principle of obtaining the sample image feature map corresponding to the sample image frame is the same as that of obtaining the image feature map corresponding to the target image frame.

[0151] In step S203, the image feature extraction network to be trained is called to generate a sample fusion feature matrix for the sample image frame according to the sample image feature map and an initial position feature embedding matrix. The initial position feature embedding matrix is obtained according to each feature position.

[0152] Optionally, the initial position feature embedding matrix is a matrix to be updated for obtaining position feature embedding parameters at each feature position in the image feature map. The initial position feature embedding matrix also belongs to the network parameters of the initial prediction network and needs to be trained and updated.

[0153] The computer device can call the image feature extraction network to be trained to generate a sample fusion feature matrix for the sample image frame according to the sample image feature map and the initial position feature embedding matrix, where the initial position feature embedding matrix corresponds to the above-mentioned Figure 3 The position feature embedding matrix for the image in the corresponding embodiment (both have the same size), the sample image feature map corresponds to the image feature map in the above-mentioned corresponding embodiment of way 3 (both have the same size), and the sample fusion feature matrix corresponds to the above-mentioned Figure 3 The fusion feature matrix in the corresponding embodiment (both have the same size), so it can be understood that the principle of generating a sample fusion feature matrix for a sample image frame according to a sample image feature map and an initial position feature embedding matrix here is the same as the above-mentioned Figure 3 The principle of generating a fusion feature matrix for a target image frame according to an image feature map and a position feature embedding matrix in the corresponding embodiment is the same, which will not be described here. The plurality of feature positions in the sample image feature map are the same as the plurality of feature positions in the above-mentioned image feature map, one-to-one correspondence.

[0154] Wherein, the initial position feature embedding matrix is only used when the initial prediction network starts to be trained, and subsequently, the initial position feature embedding matrix is iteratively updated during the training of the initial prediction network. The above-mentioned initial position feature embedding matrix of the image can be obtained in the following way:

[0155] The computer device can perform row encoding on each feature position of the sample image feature map to obtain a row encoding value of the feature position, such as a row encoding value of 0 obtained by encoding the first row, a row encoding value of 1 obtained by encoding the second row, a row encoding value of 2 obtained by encoding the third row, and so on.

[0156] The computer device can also perform column encoding on each feature position of the sample image feature map to obtain a column encoding value of the feature position, such as a column encoding value of 0 obtained by encoding the first column, a column encoding value of 1 obtained by encoding the second column, a column encoding value of 2 obtained by encoding the third column, and so on.

[0157] Therefore, one feature position in the sample image feature map corresponds to one row encoding value and one column encoding value, and further, the computer device can sum the row encoding values and column encoding values at each feature position in the sample image feature map, and the summing results are used as the initial position embedding parameters at the feature position, and the initial position embedding matrix can be generated through the initial position embedding parameters at each feature position. The initial position embedding matrix contains the initial position embedding parameters at each feature position, and the updated initial position embedding parameters are the position feature embedding parameters, please refer to the following description.

[0158] Please refer to Figure 8 , Figure 8 is a scene schematic diagram for generating an initial position feature embedding matrix provided by the present application. As shown in Figure 8 , there can be 9 feature positions in the sample image feature map, including feature position 1, feature position 2, feature position 3, feature position 4, feature position 5, feature position 6, feature position 7, feature position 8, and feature position 9.

[0159] The row encoding value at the feature position in the first row of the sample image feature map can be 0, the row encoding value at the feature position in the second row of the sample image feature map can be 1, and the row encoding value at the feature position in the third row of the sample image feature map can be 2. Similarly, the column encoding value at the feature position in the first column of the sample image feature map can be 0, the column encoding value at the feature position in the second column of the sample image feature map can be 1, and the column encoding value at the feature position in the third column of the sample image feature map can be 2. Therefore, the initial position feature embedding parameter at the feature position 1 is equal to the sum of the row encoding value 0 and the column encoding value 0 at the feature position 1, the initial position feature embedding parameter at the feature position 2 is equal to the sum of the row encoding value 0 and the column encoding value 1 at the feature position 2, the initial position feature embedding parameter at the feature position 3 is equal to the sum of the row encoding value 0 and the column encoding value 2 at the feature position 3, the initial position feature embedding parameter at the feature position 4 is equal to the sum of the row encoding value 1 and the column encoding value 0 at the feature position 4, the initial position feature embedding parameter at the feature position 5 is equal to the sum of the row encoding value 1 and the column encoding value 1 at the feature position 5, the initial position feature embedding parameter at the feature position 6 is equal to the sum of the row encoding value 1 and the column encoding value 2 at the feature position 6, the initial position feature embedding parameter at the feature position 7 is equal to the sum of the row encoding value 2 and the column encoding value 0 at the feature position 7, the initial position feature embedding parameter at the feature position 8 is equal to the sum of the row encoding value 2 and the column encoding value 1 at the feature position 8, and the initial position feature embedding parameter at the feature position 9 is equal to the sum of the row encoding value 2 and the column encoding value 2 at the feature position 9.

[0160] Further, as shown in Figure 8 , the initial position feature embedding matrix can be generated by the initial position feature embedding parameters at each feature position in the sample image feature map.

[0161] Step S204, calling the text feature extraction network to be trained in the initial prediction network to extract the text features of the sample associated text data, and obtaining the sample text feature matrix of the sample associated text data.

[0162] Optionally, the computer device can input the word segmentation in the sample associated text data into the text feature extraction network to be trained after encoding, so as to call the text feature extraction network to be trained to extract the text features of the sample associated text data, and obtain a sample feature matrix. Then, the computer device can call the text feature extraction network to be trained to generate a sample text feature matrix for the sample associated text data according to the sample feature matrix and the initial text position feature embedding matrix. The sample associated text data corresponds to the associated text data described above, and the sample feature matrix of the sample associated text data corresponds to the feature matrix of the associated text data described above. Figure 3 In the corresponding embodiment, the feature matrix of the associated text data (both have the same size), and the initial text position feature embedding matrix corresponds to the text position feature embedding matrix described above (both have the same size). Therefore, it can be understood that the principle of generating the sample text feature matrix according to the sample feature matrix of the sample associated text data and the initial text position feature embedding matrix is the same as the principle of generating the text feature matrix according to the feature matrix of the associated text data and the text position feature embedding matrix described above. Here, it will not be described again. The plurality of feature positions in the sample text feature matrix are the same as the plurality of feature positions in the text feature matrix, one by one.

[0163] The initial text position feature embedding matrix can be obtained in the following manner: the number of rows of the initial text position feature embedding matrix can be the number of word segmentation in the sample associated text data used, and each word segmentation is arranged in order according to its position in the sample associated text data. One word segmentation corresponds to one row, so the position of each word segmentation can be encoded to obtain an initial text feature position matrix. Specifically, the first row of the initial text feature position matrix can be 0, the second row can be 1, the third row can be 2, and so on.

[0164] Step S205, predicting the video type of the sample video data according to the sample fusion feature matrix and the sample text feature matrix.

[0165] Optionally, the computer device can splice the sample fusion feature matrix and the sample text feature matrix to obtain a sample spliced feature matrix, which corresponds to the spliced feature matrix in the corresponding embodiment. The computer device can call the type prediction network to be trained in the initial prediction network to predict the video type of the sample video data according to the sample spliced feature matrix. Figure 3

[0166] ​In the type prediction network, the input of the first transformer layer can be the sample splicing feature matrix, the input of the second transformer layer can be the output of the first transformer layer, and the input of the third transformer layer can be the output of the second transformer layer.

[0167] In a transformer layer, first, self-attention can be used to learn the input, and then dropout can be performed on the learning result, that is, output 1 in the transformer layer can be obtained; then, a shortcut mechanism can be used to add output 1 and the input, and layer norm can be performed on the added result, that is, output 2 in the transformer layer can be obtained; then, output 2 can be forward propagated through two fully connected layers and then subjected to dropout, that is, output 3 in the transformer layer can be obtained; finally, the shortcut mechanism can be used to add output 2 and output 3, and layer norm can be performed on the added result, that is, the final output in the transformer layer can be obtained. It can be understood that the above series of operations can be performed in each transformer layer.

[0168] Finally, the result output by the last layer of the three transformer layers can be taken as the final feature for predicting the type of the video data. The type prediction network can map the prediction feature to various video types learned in advance, to obtain the probability of the sample video data for various video types (that is, the final output of the initial prediction network), and finally the video type corresponding to the maximum probability can be taken as the predicted video type of the sample video data.

[0169] For example, the type prediction network can identify three video types, which can include video type 1, video type 2, and video type 3. During the training of the type prediction network, the type prediction network can constantly update the learned video features of various video types. When the predicted features of the sample video data are obtained, that is, the features of the sample video data finally identified by the type prediction network, since the type prediction network already knows the video features of various video types, the type prediction network can predict the probabilities of the predicted features being the video features of each video type, such as the probability of being the video feature of video type 1 being 0.1, the probability of being the video feature of video type 2 being 0.4, and the probability of being the video feature of video type 3 being 0.5. It can be seen that the probability corresponding to video type 3 is the largest, which is 0.5. Therefore, video type 3 can be taken as the predicted video type of the sample video data.

[0170] In step S206, the network parameters of the initial prediction network are updated according to the predicted video type of the sample video data and the actual video type of the sample video data indicated by the video type label, to obtain a prediction network.

[0171] Optionally, the computer device can generate a prediction loss for type prediction of the sample video data according to the predicted video type of the sample video data and the actual video type of the sample video data indicated by the video type label. The prediction loss loss can be a cross-entropy loss, as shown in the following formulas (1) and (2):

[0172]

[0173]

[0174] wherein the sample video data can be multiple, M represents the total number of sample video data, i is a positive integer less than or equal to M, y i represents the actual video type of the i-th sample video data (y i can be represented by a vector, wherein the value at the actual video type of the i-th sample video data is 1, and the values at other video types are 0), y′ i can represent the prediction result of the i-th sample video data predicted by the initial prediction network (y′ i can be a vector that can include the predicted probabilities of the i-th sample video data being various video types), H represents the network prediction algorithm of the initial prediction network (an algorithm for predicting video types), represents the sample splicing feature matrix corresponding to the i-th sample video data.

[0175] The network parameters of the initial prediction network can be updated by the determined prediction loss, and the goal of the update is to make the prediction loss approach a minimum value. The initial prediction network with updated network parameters can be used as the prediction network in the corresponding embodiment. Figure 3 The prediction network in the corresponding embodiment.

[0176] The network parameters of the initial prediction network can include the network parameters of the image feature extraction network, the network parameters of the text feature extraction network, the network parameters of the type prediction network, the initial position feature embedding matrix for the image, and the initial text position feature embedding matrix for the text.

[0177] Therefore, the trained prediction network can include the updated initial position feature embedding matrix for the image, which is the position feature embedding matrix. The trained prediction network can also include the updated initial text position feature embedding matrix for the text, which is the text position feature embedding matrix. The position feature embedding matrix for the image includes position feature embedding parameters for each feature position of the image (e.g., each feature position of the image feature map). The text position feature embedding matrix for the text includes text position feature embedding parameters for each feature position of the text (e.g., each feature position of the feature matrix of the sample associated text data).

[0178] Please refer to Figure 9 , Figure 9 is a scene diagram of model training provided by the present application. As shown in Figure 9 , the computer device can extract a sample image frame from sample video data carrying a video type label, and then input the sample image frame and sample associated text data into an initial prediction network. The initial prediction network predicts the video type of the sample video data based on the sample image frame and sample associated text data. The prediction loss can be obtained by comparing the predicted video type with the actual video type indicated by the video type label. The network parameters of the initial prediction network can be updated (i.e., corrected) by backpropagating the prediction loss in the initial prediction network. The initial prediction network with updated network parameters can be used as the trained prediction network.

[0179] In the present application, when predicting the type of video data in multiple modalities (such as a modality that can contain text and a modality of image frames), multiple networks (including the above-mentioned text feature extraction network, image feature extraction network, and type prediction network) can be trained end-to-end, so that the features between the trained multiple networks can better interact, and thus better prediction of the type of video data can be achieved through the trained multiple networks.

[0180] Please refer to Table 1 below, which lists the accuracy of predicting the type of video data using the prediction model (i.e. prediction network) provided in the present application, and the accuracy of predicting the type of video data using other existing models. As shown in Table 1:

[0181]

[0182] Table 1

[0183] Among them, Resnet50 is a residual network, netvlad (is an image feature extraction network), from Table 1, it can be seen that the accuracy of predicting the type of video data using the prediction network provided in the present application is significantly higher than the accuracy of predicting the type of video data using other networks (such as using Late fusion Resnet50+netvlad, and using Resnet50+netvlad for feature fusion by summation average).

[0184] Please refer to Figure 10 , Figure 10 is a structural schematic diagram of a video data processing apparatus provided by the present application. The video data processing apparatus can be a computer program (including program code) running in a computer device, for example, the video data processing apparatus is an application software, and the video data processing apparatus can be used to execute the corresponding steps in the method provided by the embodiments of the present application. As shown in Figure 10 , the video data processing apparatus 1 can include an extraction module 11, an acquisition module 12, a fusion module 13, and a determination module 14;

[0185] The extraction module 11 is configured to extract a target image frame from the video data.

[0186] The acquisition module 12 is configured to acquire an image feature map corresponding to the target image frame; the image feature map contains a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map;

[0187] The acquisition module 12 is configured to acquire a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used to indicate the degree of association between the feature parameters at each feature position.

[0188] The fusion module 13 is configured to fuse the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position.

[0189] The determination module 14 is configured to determine the video type of the video data according to the fusion feature information at each feature position.

[0190] Optionally, the image feature map is obtained by calling a prediction network; the prediction network comprises an image feature extraction network and a type prediction network.

[0191] The manner in which the acquisition module 12 acquires the image feature map corresponding to the target image frame comprises:

[0192] The image feature extraction network in the prediction network is called to extract the image feature of the target image frame to obtain the image feature map.

[0193] The manner in which the determination module 14 determines the video type of the video data according to the fusion feature information at each feature position comprises:

[0194] The type prediction network in the prediction network is called to determine the video type of the video data according to the fusion feature information at each feature position.

[0195] Optionally, the manner in which the acquisition module 12 acquires the position feature embedding parameter at each feature position in the image feature map comprises:

[0196] The position feature embedding matrix is acquired from the image feature extraction network.

[0197] The position feature embedding parameter at each feature position is extracted from the position feature embedding matrix.

[0198] Optionally, the prediction network further comprises a text feature extraction network; the apparatus 1 is further configured to:

[0199] Acquire the associated text data of the video data.

[0200] The text feature extraction network is called to extract the text feature of the associated text data to obtain a text feature matrix of the associated text data.

[0201] The manner in which the determination module calls the type prediction network in the prediction network to determine the video type of the video data according to the fusion feature information at each feature position comprises:

[0202] A fusion feature matrix is generated according to the fusion feature information at each feature position.

[0203] The text feature matrix and the fusion feature matrix are spliced to obtain a spliced feature matrix.

[0204] The calling type prediction network determines the video type of the video data according to the spliced feature matrix.

[0205] Optionally, the device 1 is further configured to:

[0206] obtain sample video data and sample associated text data of the sample video data; the sample video data carries a video type label;

[0207] extract a sample image frame from the sample video data, and call an image feature extraction network to be trained in the initial prediction network to extract image features of the sample image frame, to obtain a sample image feature map corresponding to the sample image frame;

[0208] call the image feature extraction network to be trained to generate a sample fusion feature matrix for the sample image frame according to the sample image feature map and an initial position feature embedding matrix; the initial position feature embedding matrix is obtained according to each feature position;

[0209] call a text feature extraction network to be trained in the initial prediction network to extract text features of the sample associated text data, to obtain a sample text feature matrix of the sample associated text data;

[0210] predict the video type of the sample video data according to the sample fusion feature matrix and the sample text feature matrix;

[0211] update network parameters of the initial prediction network according to the predicted video type of the sample video data and an actual video type of the sample video data indicated by the video type label, to obtain a prediction network; the initial position feature embedding matrix belongs to the network parameters of the initial prediction network;

[0212] The prediction network includes a position feature embedding matrix, the position feature embedding matrix is an updated initial position feature embedding matrix, and the position feature embedding matrix includes position feature embedding parameters at each feature position.

[0213] Optionally, the initial position feature embedding matrix includes initial position embedding parameters at each feature position; the device 1 is further configured to:

[0214] perform row encoding on each feature position of the sample image feature map, to obtain a row encoding value at each feature position;

[0215] perform column encoding on each feature position of the sample image feature map, to obtain a column encoding value at each feature position;

[0216] perform summation processing on the row encoding value and the column encoding value at each feature position respectively, to obtain an initial position embedding parameter at each feature position.

[0217] Optionally, the fusion module 13 fuses the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0218] Optionally, the fusion module 13 fuses the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0219] Optionally, the fusion module 13 fuses the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0220] Optionally, the number of image feature maps is N, N is a positive integer; any one of the plurality of feature positions is represented as the i-th feature position, i is a positive integer less than or equal to the total number of positions of the plurality of feature positions;

[0221] Optionally, the fusion module 13 fuses the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0222] Optionally, the fusion module 13 fuses the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0223] Optionally, the fusion module 13 fuses the feature parameter and the position feature embedding parameter at each feature position respectively to obtain the fusion feature information at each feature position in the following manner:

[0224] Optionally, the number of target image frames is a plurality;

[0225] The acquisition module 12 acquires the image feature map corresponding to the target image frame, including:

[0226] Generating a feature map corresponding to each target image frame respectively;

[0227] Optionally, the acquisition module 12 acquires the image feature map corresponding to the target image frame in the following manner:

[0228] Optionally, the extraction module 11 extracts the target image frame from the video data in the following manner:

[0229] Segmenting the video data to obtain a plurality of segmented video data;

[0230] Extracting image frames from each segmented video data respectively, and determining the extracted image frames from each segmented video data as target image frames.

[0231] Optionally, the device 1 is further used for:

[0232] Acquiring a video preference type of the target object;

[0233] If the predicted video type of the video data belongs to the video preference type, the video data is pushed to the target object.

[0234] According to one embodiment of the present application, Figure 3 The steps involved in the video data processing method shown can be performed by Figure 10 The modules in the video data processing device 1 shown can be used to perform the steps. Figure 3 The step S101 shown in the method can be performed by Figure 10 the extraction module 11 in the device, Figure 3 The steps S102-S103 shown in the method can be performed by Figure 10 the acquisition module 12 in the device; Figure 3 The step S104 shown in the method can be performed by Figure 10 the fusion module 13 in the device, Figure 3 The step S105 shown in the method can be performed by Figure 10 the determination module 14 in the device.

[0235] The present application can extract target image frames from video data; acquire image feature maps corresponding to the target image frames; the image feature maps contain a plurality of feature parameters of the target image frames, and each feature parameter corresponds to a feature position in the image feature map; acquire position feature embedding parameters at each feature position in the image feature map; the position feature embedding parameters at each feature position are used to indicate the correlation degree between the feature parameters at each feature position; respectively perform fusion processing on the feature parameters and the position feature embedding parameters at each feature position to obtain fusion feature information at each feature position; and determine the video type of the video data according to the fusion feature information at each feature position. As can be seen, the device proposed in the present application can determine the video type of the video data through the image frames in the video data, and when determining the video type through the image frames, the fusion feature information at each feature position in the image feature map is used for determination, so that the feature space information (reflected through each feature position) in the image feature map is retained, and therefore the determined video type of the video data is more accurate.

[0236] According to one embodiment of the present application, Figure 10The modules in the video data processing apparatus 1 shown can be combined into one or several units respectively or all, or some of the units can be further split into multiple sub-units with smaller functions, and the same operations can be implemented without affecting the implementation of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions, and in actual applications, the functions of one module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the video data processing apparatus 1 can also include other units, and in actual applications, these functions can also be assisted by other units, and can be implemented by multiple units in cooperation.

[0237] According to an embodiment of the present application, the video data processing apparatus 1 shown in Figure 3 and the video data processing method of the embodiments of the present application can be constructed by running a computer program (including program codes) capable of executing the steps involved in the corresponding method shown in Figure 10 by a general-purpose computer device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), etc.

[0238] Please refer to Figure 11 , Figure 11 is a structural schematic diagram of a computer device provided by the present application. As shown in Figure 11 , the computer device 1000 can include a processor 1001, a network interface 1004 and a memory 1005, in addition, the computer device 1000 can also include a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection communication between these components. Among them, the user interface 1003 can include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a non-volatile memory, for example, at least one disk memory. The memory 1005 can optionally be at least one storage device located away from the aforementioned processor 1001. As shown in Figure 11 , the memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module and a device control application program.

[0239] InFigure 11 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an interface for user input; and the processor 1001 can be used to call a device control application program stored in the memory 1005 to implement:

[0240] extracting a target image frame from the video data;

[0241] obtaining an image feature map corresponding to the target image frame; the image feature map contains a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map;

[0242] obtaining a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used to indicate the correlation degree between the feature parameters at each feature position;

[0243] performing fusion processing on the feature parameters and the position feature embedding parameters at each feature position respectively to obtain fusion feature information at each feature position;

[0244] determining the video type of the video data according to the fusion feature information at each feature position.

[0245] The image feature map is obtained by calling a prediction network; the prediction network includes an image feature extraction network and a type prediction network;

[0246] In a possible implementation, the processor 1001 can also be used to call the device control application program stored in the memory 1005 to implement:

[0247] calling the image feature extraction network in the prediction network to extract the image feature of the target image frame to obtain the image feature map;

[0248] In a possible implementation, the processor 1001 can also be used to call the device control application program stored in the memory 1005 to implement:

[0249] calling the type prediction network in the prediction network to determine the video type of the video data according to the fusion feature information at each feature position.

[0250] In a possible implementation, the processor 1001 can also be used to call the device control application program stored in the memory 1005 to implement:

[0251] obtaining a position feature embedding matrix from the image feature extraction network;

[0252] extracting the position feature embedding parameter at each feature position from the position feature embedding matrix.

[0253] The prediction network further comprises a text feature extraction network;

[0254] In an implementation, the processor 1001 can be further configured to invoke a device control application stored in the memory 1005 to implement:

[0255] obtain associated text data of the video data;

[0256] invoke the text feature extraction network to extract text features of the associated text data, to obtain a text feature matrix of the associated text data;

[0257] In an implementation, the processor 1001 can be further configured to invoke a device control application stored in the memory 1005 to implement:

[0258] generate a fusion feature matrix according to the fusion feature information at each feature position;

[0259] perform splicing processing on the text feature matrix and the fusion feature matrix to obtain a spliced feature matrix;

[0260] invoke the type prediction network to determine the video type of the video data according to the spliced feature matrix.

[0261] In an implementation, the processor 1001 can be further configured to invoke a device control application stored in the memory 1005 to implement:

[0262] obtain sample video data and sample associated text data of the sample video data; the sample video data carries a video type label;

[0263] extract a sample image frame from the sample video data, and invoke the image feature extraction network to be trained in the initial prediction network to extract image features of the sample image frame, to obtain a sample image feature map corresponding to the sample image frame;

[0264] invoke the image feature extraction network to be trained to generate a sample fusion feature matrix for the sample image frame according to the sample image feature map and an initial position feature embedding matrix; the initial position feature embedding matrix is obtained according to each feature position;

[0265] invoke the text feature extraction network to be trained in the initial prediction network to extract text features of the sample associated text data, to obtain a sample text feature matrix of the sample associated text data;

[0266] predict the video type of the sample video data according to the sample fusion feature matrix and the sample text feature matrix;

[0267] updating the network parameters of the initial prediction network according to the predicted video type of the sample video data and the actual video type of the sample video data indicated by the video type label, to obtain a prediction network; the initial position feature embedding matrix belongs to the network parameters of the initial prediction network;

[0268] The prediction network includes a position feature embedding matrix, and the position feature embedding matrix is the updated initial position feature embedding matrix.

[0269] The initial position feature embedding matrix includes initial position embedding parameters at each feature position.

[0270] In a possible implementation, the processor 1001 can further be configured to invoke a device control application stored in the memory 1005 to implement:

[0271] performing row encoding on each feature position of the sample image feature map to obtain a row encoding value at each feature position;

[0272] performing column encoding on each feature position of the sample image feature map to obtain a column encoding value at each feature position;

[0273] performing summation processing on the row encoding value and the column encoding value at each feature position respectively to obtain an initial position embedding parameter at each feature position.

[0274] In a possible implementation, the processor 1001 can further be configured to invoke a device control application stored in the memory 1005 to implement:

[0275] performing summation fusion processing on the feature parameter and the position feature embedding parameter at each feature position respectively to obtain a fusion feature parameter at each feature position;

[0276] generating fusion feature information at each feature position according to the fusion feature parameter at each feature position.

[0277] The number of image feature maps is N, where N is a positive integer; any feature position in the plurality of feature positions is denoted as an i th feature position, where i is a positive integer less than or equal to the total number of positions of the plurality of feature positions.

[0278] In a possible implementation, the processor 1001 can further be configured to invoke a device control application stored in the memory 1005 to implement:

[0279] performing flattening processing on the fusion feature parameter at the i th feature position in each image feature map to obtain fusion feature information at the i th feature position;

[0280] The fusion feature information at the i-th feature position comprises N fusion feature parameters at the i-th feature position.

[0281] The number of target image frames is multiple.

[0282] In an implementation, the processor 1001 can further be configured to invoke a device control application stored in the memory 1005 to implement:

[0283] generate a feature map corresponding to each target image frame, respectively;

[0284] perform average fusion processing on the feature map corresponding to each target image frame, respectively, to obtain an image feature map.

[0285] In an implementation, the processor 1001 can further be configured to invoke a device control application stored in the memory 1005 to implement:

[0286] segment the video data to obtain multiple segmented video data;

[0287] extract image frames from each segmented video data, and determine the extracted image frames from each segmented video data as target image frames.

[0288] In an implementation, the processor 1001 can further be configured to invoke a device control application stored in the memory 1005 to implement:

[0289] obtain a video preference type of the target object;

[0290] if the predicted video type of the video data belongs to the video preference type, push the video data to the target object.

[0291] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the foregoing Figure 3 the foregoing description of the video data processing method in the corresponding embodiments, the foregoing description of the video data processing device 1 in the corresponding embodiments, and the foregoing description of the beneficial effects of using the same. Figure 10 the foregoing description of the video data processing method in the corresponding embodiments, the foregoing description of the video data processing device 1 in the corresponding embodiments, and the foregoing description of the beneficial effects of using the same.

[0292] In addition, it should be noted that the present application further provides a computer readable storage medium, and the computer readable storage medium stores the computer program executed by the video data processing device 1 mentioned above, and the computer program comprises program instructions, and when the processor executes the program instructions, the foregoing Figure 3The video data processing method is described in the corresponding embodiments, and thus, the description will not be repeated here. In addition, the beneficial effects of using the same method are also not repeated. For technical details not disclosed in the computer storage medium embodiments involved in the present application, please refer to the description of the method embodiments of the present application.

[0293] As an example, the above program instructions can be deployed on a computer device for execution, or deployed on multiple computer devices located in one place for execution, or on multiple computer devices distributed in multiple places and interconnected through a communication network for execution. The multiple computer devices distributed in multiple places and interconnected through a communication network can constitute a blockchain network.

[0294] The computer readable storage medium can be an internal storage unit of the video data processing apparatus provided by any of the preceding embodiments or the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the computer device. The computer readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0295] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the preceding Figure 3 The video data processing method is described in the corresponding embodiments, and thus, the description will not be repeated here. In addition, the beneficial effects of using the same method are also not repeated. For technical details not disclosed in the computer storage medium embodiments involved in the present application, please refer to the description of the method embodiments of the present application.

[0296] The terms "first", "second", etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, and are not used to describe a particular order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment including a series of steps or units is not limited to the steps or units listed, but can optionally further include steps or units not listed, or can optionally further include other steps units inherent to the process, method, device, product or equipment.

[0297] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0298] The method and related apparatus provided by the embodiments of the present application are described with reference to the method flowchart and / or structural schematic diagram provided by the embodiments of the present application. Each flow and / or block in the method flowchart and / or structural schematic diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device that realizes the functions specified in the flow Figure 1 The functions specified in one flow or multiple flows and / or structural schematic Figure 1 The functions specified in one flow or multiple flows and / or structural schematic Figure 1 The functions specified in one flow or multiple flows and / or structural schematic Figure 1 The functions specified in one flow or multiple flows and / or structural schematic Figure 1 The functions specified in one flow or multiple flows and / or structural schematic The functions specified in one flow or multiple flows and / or structural schematic

[0299] The above merely provides the preferred embodiment of the application, and cannot allude the scope of the application; therefore, any equivalent changes made according to the claims of the application shall still fall within the scope of the application.

Claims

1. A method of processing video data, the method comprising: The method comprises: extracting a target image frame from video data; obtaining an image feature map corresponding to the target image frame; the image feature map contains a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map; obtaining a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used to indicate the position of each feature parameter in the image feature map, and the closer the distance between the positions of two feature parameters in the image feature map, the greater the correlation between the two feature parameters, and the farther the distance between the positions of the two feature parameters, the smaller the correlation between the two feature parameters; respectively, the feature parameters and the position feature embedding parameters at each feature position are fused to obtain fusion feature information at each feature position; determining the video type of the video data according to the fusion feature information at each feature position.

2. The method of claim 1, wherein, The image feature map is obtained by calling a prediction network; the prediction network comprises an image feature extraction network and a type prediction network; The image feature map corresponding to the target image frame is obtained by: calling the image feature extraction network in the prediction network to extract the image features of the target image frame to obtain the image feature map; The video type of the video data is determined according to the fusion feature information at each feature position by calling the type prediction network in the prediction network. The position feature embedding parameter at each feature position in the image feature map is obtained by:

3. The method of claim 2, wherein, obtaining a position feature embedding matrix from the image feature extraction network; extracting the position feature embedding parameter at each feature position from the position feature embedding matrix. The prediction network further comprises a text feature extraction network; the method further comprises:

4. The method of claim 2, wherein, obtaining associated text data of the video data; calling the text feature extraction network to extract text features of the associated text data to obtain a text feature matrix of the associated text data; The video type of the video data is determined according to the fusion feature information at each feature position by calling the type prediction network in the prediction network. A fusion feature matrix is generated according to the fusion feature information at each feature position; the text feature matrix and the fusion feature matrix are spliced to obtain a spliced feature matrix; the type prediction network is called to determine the video type of the video data according to the spliced feature matrix. The method further comprises:

5. The method of claim 4, wherein, obtaining sample video data and sample associated text data of the sample video data; the sample video data carries a video type label; extracting a sample image frame from the sample video data, and calling the image feature extraction network to be trained in the initial prediction network to extract image features of the sample image frame to obtain a sample image feature map corresponding to the sample image frame; ​ The image feature extraction network to be trained is invoked to generate a sample fusion feature matrix for the sample image frame according to the sample image feature map and an initial position feature embedding matrix, the initial position feature embedding matrix being obtained according to the each feature position; The text feature extraction network to be trained in the initial prediction network is invoked to extract text features of the sample associated text data, to obtain a sample text feature matrix of the sample associated text data; The video type of the sample video data is predicted according to the sample fusion feature matrix and the sample text feature matrix; The network parameters of the initial prediction network are updated according to the predicted video type of the sample video data and the actual video type of the sample video data indicated by the video type label, to obtain the prediction network; the initial position feature embedding matrix belongs to the network parameters of the initial prediction network. The prediction network includes a position feature embedding matrix, the position feature embedding matrix being an updated initial position feature embedding matrix, and the position feature embedding matrix including position feature embedding parameters at the each feature position.

6. The method of claim 5, wherein, The initial position feature embedding matrix includes initial position embedding parameters at the each feature position; the method further includes: The each feature position of the sample image feature map is row-encoded to obtain a row-encoding value at the each feature position; The each feature position of the sample image feature map is column-encoded to obtain a column-encoding value at the each feature position; The row-encoding value and the column-encoding value at the each feature position are summed to obtain the initial position embedding parameter at the each feature position.

7. The method of claim 1, wherein, The fusion processing of the feature parameter and the position feature embedding parameter at the each feature position is performed to obtain the fusion feature information at the each feature position, including: The feature parameter and the position feature embedding parameter at the each feature position are summed to obtain a fusion feature parameter at the each feature position; The fusion feature information at the each feature position is generated according to the fusion feature parameter at the each feature position.

8. The method of claim 7, wherein, The number of the image feature maps is N, N being a positive integer; any feature position in the plurality of feature positions is represented as an i-th feature position, i being a positive integer less than or equal to the total number of positions of the plurality of feature positions; The fusion feature information at the i-th feature position is generated according to the fusion feature parameter at the i-th feature position, including: The fusion feature parameter at the i-th feature position in each image feature map is flattened to obtain the fusion feature information at the i-th feature position; The fusion feature information at the i-th feature position includes N fusion feature parameters at the i-th feature position.

9. The method of claim 1, wherein, The number of the target image frames is a plurality; The image feature map corresponding to the target image frame is obtained, including: A feature map corresponding to each target image frame is generated; The feature maps corresponding to the each target image frame are averaged and fused to obtain the image feature map.

10. The method of claim 1, wherein, The extracting the target image frame from the video data comprises: segmenting the video data to obtain a plurality of segmented video data; extracting an image frame from each of the segmented video data, and determining the extracted image frame from each of the segmented video data as the target image frame.

11. The method of claim 1, wherein, The method further comprises: obtaining a video preference type of a target object; if the predicted video type of the video data belongs to the video preference type, pushing the video data to the target object.

12. A video data processing apparatus, comprising: The device comprises: an extracting module configured to extract a target image frame from video data; an obtaining module configured to obtain an image feature map corresponding to the target image frame; the image feature map comprises a plurality of feature parameters of the target image frame, and each feature parameter corresponds to a feature position in the image feature map; an obtaining module configured to obtain a position feature embedding parameter at each feature position in the image feature map; the position feature embedding parameter at each feature position is used to indicate the position of each feature parameter in the image feature map, if the distance between the positions of two feature parameters in the image feature map is farther, the correlation between the two feature parameters is smaller, and if the distance between the positions of the two feature parameters is closer, the correlation between the two feature parameters is greater; a fusion module configured to perform fusion processing on the feature parameter and the position feature embedding parameter at each feature position to obtain fusion feature information at each feature position; a determining module configured to determine a video type of the video data according to the fusion feature information at each feature position.

13. A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the method of any one of claims 1-11.

14. A computer device, comprising: A device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to perform the steps of the method of any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program being adapted to be loaded and executed by a processor to implement the method of any one of claims 1-11.

Images