A video pushing method and device and a storage medium

Abstract

The application discloses a video pushing method and device and a storage medium, and the method comprises the following steps: acquiring a first title text corresponding to a first video being played; performing title feature extraction on the first title text to obtain a first title feature vector, and performing visual feature extraction on the first video to obtain a first visual feature vector; acquiring a title feature vector corresponding to each video in a preset video set, and selecting a plurality of videos from the preset video set in a descending order of similarity between the corresponding title feature vector and the first title feature vector; acquiring a visual feature vector corresponding to each video in the plurality of videos, and sorting the plurality of videos in a descending order of similarity between the corresponding visual feature vector and the first visual feature vector to obtain a video sequence; and sequentially pushing each video in the video sequence. Through the technical scheme, the diversity of video pushing is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video push method, apparatus and storage medium. Background Technology

[0002] With the promotion of short video applications, video platforms have increasingly higher requirements for the accuracy of video recommendations.

[0003] Currently, the focus of video recommendation is often on the number of views, likes and dislikes, comments, and user behaviors such as quickly scrolling through, watching to completion, liking, and sharing. Alternatively, it may learn the click probability of users by clicking on view logs and filter recommended content that is highly similar to the user's historical data characteristics. This results in highly similar recommended video content and overly monotonous video content. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of the present invention aim to provide a video push method, apparatus, and storage medium that, for a currently playing video, combines both title text and visual content to push videos with high similarity to the video title text but low similarity to the actual video content, thereby improving the diversity of video pushes.

[0005] The technical solution of this invention is implemented as follows:

[0006] This invention provides a video push method, the method comprising:

[0007] Get the first title text corresponding to the currently playing first video;

[0008] The first title text is subjected to title feature extraction to obtain a first title feature vector, and the first video is subjected to visual feature extraction to obtain a first visual feature vector.

[0009] Obtain the title feature vector corresponding to each video in the preset video set, and select multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector;

[0010] Obtain the visual feature vector corresponding to each of the multiple videos, and sort the multiple videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence;

[0011] Each video in the video sequence is pushed out sequentially.

[0012] In the above method, the step of extracting title features from the first title text to obtain a first title feature vector includes:

[0013] Split the first title text into multiple sub-texts;

[0014] Using a pre-defined text understanding model, semantic analysis is performed on each of the multiple sub-texts to obtain multiple feature vectors that correspond one-to-one with the multiple sub-texts;

[0015] The first title feature vector is obtained by performing average pooling on the multiple feature vectors.

[0016] In the above method, the step of extracting visual features from the first video to obtain a first visual feature vector includes:

[0017] The first video is subjected to video frame sampling processing to obtain multiple video images;

[0018] Feature extraction is performed on each of the multiple video frames to obtain multiple image feature vectors that correspond one-to-one with the multiple video frames.

[0019] The first visual feature vector is obtained by aggregating the multiple image feature vectors using a preset multimodal video classification model.

[0020] In the above method, the step of performing video frame sampling processing on the first video to obtain multiple video images includes:

[0021] Obtain the total number of video images in the first video, and determine the number of video frame samples for the first video based on the total number of video images;

[0022] The first video is evenly divided into video segments with a number equal to the number of video frame samples, and one video frame is selected from each video segment to obtain the multi-frame video images.

[0023] In the above method, determining the number of video frame samples for the first video based on the total number of video images includes:

[0024] If the total number of video images is less than the first preset number of frames, the number of video frame samples is determined to be the first preset value;

[0025] If the total number of video images is greater than the second preset number of frames, the number of video frame samples is determined to be the second preset value; wherein, the first preset number of frames is less than the second preset number of frames, and the first preset value is less than the second preset value;

[0026] When the total number of video images is not less than the first preset frame number and not greater than the second preset frame number, the difference between the first preset frame number and the second preset frame number is determined as the frame number difference, and the difference between the first preset value and the second preset value is determined as the value difference;

[0027] Calculate the difference between the total number of video images and the preset interval frame number, and determine the product of the calculated difference and the frame number difference as the first value;

[0028] The ratio of the first value to the difference between the two values ​​is determined as the second value, and the difference between the second preset value and the second value is determined as the number of video frame samples.

[0029] In the above method, the step of aggregating the multiple image feature vectors using a preset multimodal video classification model to obtain the first visual feature vector includes:

[0030] Multiple image feature vectors are selected from the plurality of image feature vectors, and each of the plurality of image feature vectors is determined as an initial cluster center to form an initial cluster center group;

[0031] One image feature vector is selected from the plurality of image feature vectors and determined as the first image feature vector;

[0032] For the initial cluster center group, the initial cluster center that is closest to the first image feature vector is updated using the first image feature vector to obtain the first cluster center group;

[0033] Construct a first orthogonal loss function using the first cluster center group;

[0034] If the first orthogonal loss function is not less than a preset threshold, a second image feature vector is selected from the plurality of image feature vectors, and the first cluster center group is updated using the second image feature vector until the kth orthogonal loss function constructed by the kth cluster center group after the kth cluster center update is less than the preset threshold; k is a natural number greater than 1.

[0035] The first visual feature vector is determined using the kth cluster center group and the plurality of image feature vectors.

[0036] In the above method, determining the first visual feature vector using the k-th cluster center group and the plurality of image feature vectors includes:

[0037] For each of the multiple image feature vectors, calculate the distance to each cluster center in the k-th cluster center group, generate the corresponding distance vector, and calculate the proportion of each cluster center in the k-th cluster center group, generate the corresponding proportion vector.

[0038] For each of the multiple image feature vectors, a nonlinear activation process is performed to obtain the corresponding feature weight, which is then determined as the corresponding weight vector.

[0039] For each of the multiple image feature vectors, the corresponding distance vector, scale vector, and weight vector are fused to generate a corresponding fused feature vector.

[0040] The sum of the fused feature vectors corresponding to different image feature vectors among the multiple image feature vectors is determined as the video image feature vector, and the first visual feature vector is determined based on the video image feature vector.

[0041] In the above method, determining the first visual feature vector based on the video image feature vector includes:

[0042] The video image feature vector is normalized, and the normalized feature vector is then reduced in dimensionality to obtain the reduced feature vector.

[0043] The reduced feature vectors are normalized and activated to obtain the corresponding enhanced feature weights;

[0044] Using the enhanced feature weights, feature enhancement is performed on the dimensionality-reduced feature vector to obtain the first visual feature vector.

[0045] This invention provides a video push device, comprising:

[0046] The acquisition module is used to retrieve the first title text corresponding to the currently playing first video;

[0047] The feature extraction module is used to extract title features from the first title text to obtain a first title feature vector, and to extract visual features from the first video to obtain a first visual feature vector.

[0048] The selection module is used to obtain the title feature vector corresponding to each video in the preset video set, and select multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector;

[0049] The sorting module is used to obtain the visual feature vector corresponding to each of the multiple videos, and sort the multiple videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence.

[0050] The push module is used to push each video in the video sequence sequentially.

[0051] This invention provides a video push device, comprising: a processor, a memory, and a communication bus;

[0052] The communication bus is used to realize the communication connection between the processor and the memory;

[0053] The processor is used to execute the computer program stored in the memory to implement the above-described video push method.

[0054] The present invention provides a computer-readable storage medium storing one or more computer programs, which can be executed by one or more processors to implement the above-described video push method.

[0055] This invention provides a video push method, apparatus, and storage medium. The method includes: obtaining a first title text corresponding to a currently playing first video; extracting title features from the first title text to obtain a first title feature vector, and extracting visual features from the first video to obtain a first visual feature vector; obtaining the title feature vector corresponding to each video in a preset video set, and selecting multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector; obtaining the visual feature vector corresponding to each of the multiple videos, and sorting the multiple videos in ascending order of similarity between the corresponding visual feature vector and the first visual feature vector to obtain a video sequence; and sequentially pushing each video in the video sequence. The technical solution provided by this invention, for a currently playing video, combines both title text and visual content to push videos with high similarity to the video title text but low similarity to the actual video content, thereby improving the diversity of video pushes. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating a video push method provided in an embodiment of the present invention;

[0057] Figure 2 A flowchart illustrating an exemplary video processing method provided in an embodiment of the present invention;

[0058] Figure 3 A flowchart illustrating an exemplary feature enhancement method provided in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram illustrating an exemplary process for generating a video sequence, provided as an embodiment of the present invention.

[0060] Figure 5 An exemplary frame of a video in a playing video provided as an embodiment of the present invention;

[0061] Figure 6 An exemplary video sequence for push notifications provided by existing technology;

[0062] Figure 7 An exemplary video sequence for push notifications is provided as an embodiment of the present invention;

[0063] Figure 8 A schematic diagram illustrating an exemplary video sequence push method provided in an embodiment of the present invention;

[0064] Figure 9 A schematic diagram of the structure of a video push device provided in an embodiment of the present invention. Figure 1 ;

[0065] Figure 10 A schematic diagram of the structure of a video push device provided in an embodiment of the present invention. Figure 2 . Detailed Implementation

[0066] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings of the embodiments of the present invention. It is to be understood that the specific embodiments described herein are merely for explaining the relevant application and are not intended to limit the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the relevant application are shown in the accompanying drawings.

[0067] This invention provides a video push method, implemented by a video push device. Figure 1 This is a flowchart illustrating a video push method provided in an embodiment of the present invention. Figure 1 As shown, the main steps include:

[0068] S101. Get the first title text corresponding to the first video currently playing.

[0069] In an embodiment of the present invention, the video push device obtains the first title text corresponding to the currently playing first video.

[0070] It should be noted that, in the embodiments of the present invention, the video push device can directly obtain the first title text corresponding to the currently playing first video. The first title text is the title text of the first video, wherein the first video can be a video about a certain scenic spot.

[0071] S102. Extract title features from the first title text to obtain the first title feature vector, and extract visual features from the first video to obtain the first visual feature vector.

[0072] In an embodiment of the present invention, the video push device extracts title features from the first title text to obtain a first title feature vector, and extracts visual features from the first video to obtain a first visual feature vector.

[0073] It should be noted that, in the embodiments of the present invention, after the video push device obtains the first title text corresponding to the first video, it can extract the title features from the first title text to obtain the first title feature vector.

[0074] Specifically, in an embodiment of the present invention, the video push device extracts title features from the first title text to obtain a first title feature vector, including: splitting the first title text into multiple sub-texts; using a preset text understanding model to perform semantic analysis on each of the multiple sub-texts to obtain multiple feature vectors that correspond one-to-one with the multiple sub-texts; and performing average pooling on the multiple feature vectors to obtain the first title feature vector.

[0075] It should be noted that, in the embodiments of the present invention, the video push device can split the first title text into multiple sub-texts, and then use a preset text understanding model to perform semantic analysis on each of the multiple sub-texts to obtain multiple feature vectors. The preset text understanding model can be a pre-trained language (BERT) model, and each sub-text corresponds to a feature vector.

[0076] It should be noted that, in the embodiments of the present invention, the video push device performs average pooling on multiple feature vectors obtained by using a preset text understanding model to obtain the first title feature vector corresponding to the first title text.

[0077] It should be noted that, in the embodiments of the present invention, after obtaining the first title feature vector, the video push device will extract visual features from the first video to obtain the first visual feature vector.

[0078] Specifically, in an embodiment of the present invention, the video push device extracts visual features from the first video to obtain a first visual feature vector, including: performing video frame sampling processing on the first video to obtain multiple video images; extracting features from each video image in the multiple video images to obtain multiple image feature vectors that correspond one-to-one with the multiple video images; and using a preset multimodal video classification model to aggregate the multiple image feature vectors to obtain the first visual feature vector.

[0079] It should be noted that, in the embodiments of the present invention, after obtaining the first video, the video push device first performs video frame sampling processing on the first video to obtain multiple video images. Then, it extracts features from each of the multiple video images to obtain multiple image feature vectors, wherein each video image corresponds to one image feature vector. Finally, it uses a preset multimodal video classification model to aggregate the multiple image feature vectors to obtain a first visual feature vector. The preset multimodal video classification model can be the NeXtVLAD model.

[0080] Figure 2 This is a schematic flowchart illustrating an exemplary video processing method provided in an embodiment of the present invention. Figure 2 As shown, the video push device can divide the first title text corresponding to the first video into multiple sub-texts: [w1, w2, ..., w r ], where w r The first title text is divided into r-th sub-texts. These sub-texts are then input into a preset text understanding model to obtain multiple feature vectors corresponding to each sub-text. A pooling layer is then used to perform average pooling on these feature vectors, resulting in the output first title feature vector corresponding to the title text. Next, using the title feature vector, similar videos are retrieved, and multiple videos are selected in descending order of similarity between the title feature vectors of the similar videos and the first title feature vector. The video push device performs video frame sampling processing on the first video to obtain multiple video frames: [h1, h2, ..., h...]. r ], where h r Let r be the r-th frame of the first video. Then, feature extraction is performed on each frame of the multi-frame video image to obtain multiple image feature vectors that correspond one-to-one with the multi-frame video images. Then, a preset multimodal video classification model is used to aggregate the multiple image feature vectors to obtain the first visual feature vector. Then, for multiple videos, they are sorted from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence, and the video sequence is pushed.

[0081] Specifically, in an embodiment of the present invention, the video push device performs video frame sampling processing on the first video to obtain multiple video images, including: obtaining the total number of video images of the first video, and determining the number of video frame samples for the first video based on the total number of video images; uniformly dividing the first video into video segments with a number equal to the number of video frame samples, and selecting one video image from each of the divided video segments to obtain multiple video images.

[0082] It should be noted that, in the embodiments of the present invention, since the video duration varies, ranging from a few minutes to tens of minutes, in order to avoid the problems of too many or too few sampled video images caused by the traditional method of randomly extracting fixed frames, for example, if 300 random frames are randomly extracted from each video to describe the video information, then for short videos, too many sampled frames will result in low efficiency and affect performance, while for long videos, too few sampled frames will result in insufficient description of the video information. Therefore, during the video frame sampling process of the first video, the video push device determines the number of video frame samples for the first video by obtaining the total number of video images of the first video. Then, the first video is evenly divided into video segments with the number of video frame samples, and one video image is selected from each of the divided video segments to obtain multiple video images, which can ensure that the first video is sampled evenly.

[0083] Specifically, in an embodiment of the present invention, the video push device determines the number of video frame samples for the first video based on the total number of video images, including: when the total number of video images is less than a first preset frame number, determining the number of video frame samples as a first preset value; when the total number of video images is greater than a second preset frame number, determining the number of video frame samples as a second preset value; wherein the first preset frame number is less than the second preset frame number, and the first preset value is less than the second preset value; when the total number of video images is not less than the first preset frame number and not greater than the second preset frame number, determining the difference between the first preset frame number and the second preset frame number as a frame number difference, and determining the difference between the first preset value and the second preset value as a numerical difference; calculating the difference between the total number of video images and a preset interval frame number, and determining the product of the calculated difference and the frame number difference as a first value; determining the ratio of the first value to the numerical difference as a second value, and determining the difference between the second preset value and the second value as the number of video frame samples.

[0084] It should be noted that, in the embodiments of the present invention, after obtaining the total number of video images, the video push device determines the number of video frame samples for the first video based on the total number of video images. For example, if the total number of video images is less than a first preset frame number, the number of video frame samples is determined to be a first preset value; if the total number of video images is greater than a second preset frame number, the number of video frame samples is determined to be a second preset value; wherein, the first preset frame number is less than the second preset frame number, and the first preset value is less than the second preset value; if the total number of video images is not less than the first preset frame number and not greater than the second preset frame number, the video push device can use the first preset frame number, the second preset frame number, the first preset value, and the second preset value to determine the number of video frame samples. The implementation method is: the difference between the first preset frame number and the second preset frame number is determined as the number of frames. The difference between the first preset value and the second preset value is determined as the numerical difference; the difference between the total number of video images and the preset interval frame number is calculated, and the product of the calculated difference and the frame number difference is determined as the first value; the ratio of the first value to the numerical difference is determined as the second value, and the difference between the second preset value and the second value is determined as the number of video frame samples. Among them, the first preset frame number, the second preset frame number, the first preset value, the second preset value, and the preset interval frame number are all preset values, which can be any values. Specifically, they can be set according to actual needs and application scenarios. For example, when the first preset frame number is 500 frames, the second preset frame number is 6000 frames, the first preset value is 24, the second preset value is 128, and the preset interval frame number is 500, the number of video frame samples can be expressed by formula (1):

[0085]

[0086] Where N is the number of video frame samples, N total This represents the total number of video images.

[0087] Specifically, in an embodiment of the present invention, the video push device uses a preset multimodal video classification model to aggregate multiple image feature vectors to obtain a first visual feature vector. This includes: selecting multiple image feature vectors from the multiple image feature vectors and determining each image feature vector as an initial cluster center to form an initial cluster center group; selecting one image feature vector from the multiple image feature vectors and determining it as a first image feature vector; updating the initial cluster center group containing the first image feature vector with the initial cluster center closest to it using the first image feature vector to obtain a first cluster center group; constructing a first orthogonal loss function using the first cluster center group; if the first orthogonal loss function is not less than a preset threshold, continuing to select a second image feature vector from the multiple image feature vectors and updating the first cluster center group using the second image feature vector, until the kth orthogonal loss function constructed using the kth cluster center group after the kth cluster center update is less than the preset threshold; k is a natural number greater than 1; and determining the first visual feature vector using the kth cluster center group and the multiple image feature vectors.

[0088] It should be noted that, in the embodiments of the present invention, the video push device first selects multiple image feature vectors from multiple image feature vectors, and determines each image feature vector as an initial cluster center to form an initial cluster center group. Then, it selects one image feature vector from multiple image feature vectors to determine the first image feature vector. Then, it calculates the distance between the first image feature vector and the cluster centers included in the initial cluster center group, and uses the first image feature vector to update the initial cluster center that is closest to the first image feature vector in the initial cluster center group to obtain the first cluster center group. Then, it uses the first cluster center group to construct a first orthogonal loss function. If the first orthogonal loss function is less than a preset threshold, it directly uses the first cluster center group and multiple image feature vectors to determine the first visual feature vector. The preset threshold can be a value close to 0, such as 0.01 or other values ​​close to 0. The specific preset threshold can be set according to the actual situation and application scenario. In the embodiments of the present invention, the ideal preset threshold is 0. The specific orthogonal loss function is shown in formula (2):

[0089]

[0090] Among them, C k (j) represents the j-th cluster center. T is the transpose of the j-th cluster center. r To calculate the sum of the diagonal matrices, Loss is the orthogonal loss function, k is the number of cluster centers, and α is a preset parameter.

[0091] It should be noted that, in the embodiments of the present invention, if the first orthogonal loss function is not less than a preset threshold, a second image feature vector is selected from multiple image feature vectors. Then, the first cluster center group is updated using the second image feature vector until the kth orthogonal loss function constructed by the kth cluster center group after the kth cluster center update is less than the preset threshold. Finally, the first visual feature vector is determined using the kth cluster center group and multiple image feature vectors, where k is a natural number greater than 1.

[0092] It should be noted that, in the embodiments of the present invention, the cluster centers of the kth cluster center group obtained by the video push device are spatially orthogonal. When the cluster centers are orthogonal, it means that the distance between the cluster centers is relatively large, and the expression is more generalized.

[0093] Specifically, in an embodiment of the present invention, the video push device uses the k-th cluster center group and multiple image feature vectors to determine a first visual feature vector, including: for each image feature vector in the multiple image feature vectors, calculating the distance to each cluster center in the k-th cluster center group, generating a corresponding distance vector, and calculating the proportion of each cluster center in the k-th cluster center group, generating a corresponding proportion vector; for each image feature vector in the multiple image feature vectors, performing nonlinear activation processing to obtain a corresponding feature weight, and determining it as a corresponding weight vector; for each image feature vector in the multiple image feature vectors, fusing the corresponding distance vector, proportion vector, and weight vector respectively to generate a corresponding fused feature vector; and determining the sum of the fused feature vectors corresponding to different image feature vectors in the multiple image feature vectors as a video image feature vector, and determining the first visual feature vector based on the video image feature vector.

[0094] It should be noted that, in the embodiments of the present invention, after obtaining the k-th cluster center group, the video push device calculates the distance to each cluster center in the k-th cluster center group for each image feature vector in the multiple image feature vectors, obtains the corresponding distance vector, and calculates the proportion of each cluster center in the k-th cluster center group, obtains the corresponding proportion vector; for each image feature vector in the multiple image feature vectors, the corresponding feature weight is determined, obtains the corresponding weight vector; then, for each image feature vector in the different multiple image feature vectors, the corresponding distance vector, proportion vector, and weight vector are fused to generate the corresponding fused feature vector, and then the sum of the fused feature vectors corresponding to different image feature vectors in the multiple image feature vectors is determined as the video image feature vector, so as to determine the first visual feature vector based on the video image feature vector.

[0095] It should be noted that, in the embodiments of the present invention, the process by which the video push device obtains video image feature vectors using the k-th cluster center group and multiple image feature vectors can be as follows: multiple image feature vectors x are expanded and upgraded through a fully connected layer, and then the expanded and upgraded image feature vectors are... Perform a grouping transformation to obtain the grouped feature vectors. Where x is a feature vector of dimension (M,N), then, Let the feature vectors be of dimension (M, λN). for Dimensional feature vectors, where λ is the width multiplier and G is... The number of groups to be split The distance vector is obtained by subtracting the distance from each cluster center in the k-th cluster center group. And The input is processed by a fully connected layer and then normalized by a normalization layer to obtain a scaling vector. in, Let g be the proportion of the g-th group in the i-th image feature vector at the n-th cluster center. The input to the fully connected layer is then subjected to non-linear activation using an activation function to obtain the weight vector. in, Let be the weight of the i-th image feature vector in the g-th group. Then, for each image feature vector in the multiple image feature vectors, calculate the corresponding distance vector. Scale vector and weight vector The feature vector is fused to generate the corresponding fused feature vector. The sum of the fused feature vectors corresponding to different image feature vectors from multiple image feature vectors is then determined as the video image feature vector y. jn The specific formula is shown in formula (3):

[0096]

[0097] Among them, y jn For video image feature vectors, Let g be the fused feature vector corresponding to the i-th image feature vector, where g∈{1,……,G}, i∈{1,……,M}. n∈{1,……,k}.

[0098] It should be noted that, in this embodiment of the invention, after obtaining the video image feature vector, the video push device determines the first visual feature vector based on the video image feature vector.

[0099] Specifically, in an embodiment of the present invention, the video push device determines a first visual feature vector based on the video image feature vector, including: normalizing the video image feature vector and reducing the dimensionality of the normalized feature vector to obtain a dimensionality-reduced feature vector; normalizing and activating the dimensionality-reduced feature vector to obtain corresponding enhanced feature weights; and using the enhanced feature weights to enhance the dimensionality-reduced feature vector to obtain the first visual feature vector.

[0100] It should be noted that, in the embodiments of the present invention, after obtaining the video image feature vector, the video push device can directly determine the video image feature vector as the first visual feature vector. Of course, it can also normalize and / or reduce the dimensionality of the video image feature vector to obtain the processed feature vector. In this case, the video push device can directly determine the processed feature vector as the first visual feature vector. Alternatively, it can further enhance the processed feature vector. The specific enhancement process can be as follows: normalize and activate the processed feature vector to obtain the corresponding enhanced feature weights. Then, use the enhanced feature weights to enhance the processed feature vector to obtain the first visual feature vector. The normalization, dimensionality reduction, and feature enhancement processes of the video image feature vector are all optional. Only one or more of them can be performed. The specific settings can be made according to actual needs and application scenarios. The present invention does not limit this.

[0101] Figure 3 This is a flowchart illustrating an exemplary feature enhancement method provided in an embodiment of the present invention. Figure 3 As shown, the video push device performs feature enhancement processing on the dimensionality-reduced feature vector through a fully connected layer (FC), a batch normalization layer, a ReLU activation layer, another fully connected layer, a normalization layer, and an activation function (sigmoid) in sequence to obtain enhanced feature weights. Then, it fuses these weights with the dimensionality-reduced feature vector to obtain the first visual feature vector. In the figure, B is the number of feature vectors (batch size), and F is the dimension of the feature vector.

[0102] S103. Obtain the title feature vector corresponding to each video in the preset video set, and select multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector.

[0103] In an embodiment of the present invention, the video push device obtains the title feature vector corresponding to each video in a preset video set, and selects multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector.

[0104] It should be noted that, in the embodiments of the present invention, the video push device obtains the title feature vector corresponding to each video in the preset video set, and selects multiple videos from the preset video set in descending order of the similarity between the corresponding title feature vector and the first title feature vector. Specifically, the video push device can directly obtain the title feature vector corresponding to each video in the preset video set, or it can obtain the videos in the preset video set, and then perform feature extraction on the title text corresponding to each video to obtain the title feature vector corresponding to each video in the preset video set, and then select multiple videos in descending order of the similarity between the corresponding title feature vector and the first title feature vector.

[0105] It should be noted that, in the embodiments of the present invention, after obtaining the first title feature vector corresponding to the first video and the title feature vector corresponding to each video in the preset video set, the video push device calculates the similarity between the title feature vector corresponding to each video in the preset video set and the first title feature vector corresponding to the first video. For example, the similarity between two title feature vectors can be calculated using Euclidean distance, cosine distance, Mahalanobis distance, Manhattan distance, etc. Then, from the preset video set, multiple videos are selected in descending order of similarity to the first title feature vector corresponding to the first video. Specifically, the number of selected videos can be set according to actual needs and application scenarios, and the present invention does not limit this.

[0106] S104. Obtain the visual feature vector corresponding to each video in multiple videos, and sort the multiple videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence.

[0107] In an embodiment of the present invention, the video push device obtains the visual feature vector corresponding to each of the multiple videos, and sorts the multiple videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain a video sequence.

[0108] It should be noted that, in the embodiments of the present invention, the video push device obtains the visual feature vector corresponding to each video in the multiple videos. Specifically, the video push device can directly obtain the visual feature vector corresponding to each video in the multiple videos, or it can obtain multiple videos, and then perform feature extraction on each video in the multiple videos to obtain the visual feature vector corresponding to each video in the multiple videos. Then, it sorts the videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence.

[0109] Figure 4 This is a schematic diagram illustrating an exemplary process for generating a video sequence, provided as an embodiment of the present invention. Figure 4As shown, the video push device inputs the first title text corresponding to the currently playing first video into a preset text understanding model, which then splits the first title text into multiple sub-texts: word embeddings, segment embeddings, and position vectors. After 12 layers of encoding (enbeddings), the 11th layer of encoding outputs multiple feature vectors corresponding to multiple sub-texts. Then, an average pooling layer is used to perform average pooling on the multiple feature vectors to obtain the first title feature vector. Then, the similarity between the first feature vector and the title feature vector corresponding to each video in the preset video set is calculated. From the preset video set, multiple videos are selected in descending order of similarity between the corresponding title feature vector and the first title feature vector. In the 12-layer encoding, each layer of encoding needs to pass through a multi-attention network layer, a fully connected layer, a fusion or normalization layer, a feedforward network layer, a fully connected layer, and a fusion or normalization layer for feature processing in a sequentially connected manner. The first video is first processed by video frame sampling to obtain multiple video images. Then, a residual network layer, such as ResNet-101, is used to extract features from the multiple video images to obtain the first title feature vector. Multiple image feature vectors corresponding to each frame of video image are generated. Then, a pre-defined multimodal video classification model is used to aggregate the multiple image feature vectors to obtain the first visual feature vector. Finally, the similarity between the first visual feature vector and the visual feature vectors corresponding to multiple videos is calculated, and the multiple videos are sorted from low to high according to the similarity to obtain a video sequence for push. The implementation process of the feature vector using the pre-defined multimodal video classification model is as follows: the dimension is expanded through a fully connected layer. Then, the vectors after feature extraction are input into three branches. One branch is grouped and transformed, another branch is input into a fully connected layer and then normalized using a normalization layer, and the third branch is input into a fully connected layer and then non-linearly activated using an activation function. Finally, the output vectors of the three branches are fused. After that, feature processing is performed through a regularization layer, a dropout layer, a fully connected layer, and an enhancement layer to obtain the first visual feature vector.

[0110] S105. Push each video in the video sequence sequentially.

[0111] In an embodiment of the present invention, the video push device pushes each video in the video sequence sequentially.

[0112] It should be noted that, in the embodiments of the present invention, after obtaining the video sequence, the video push device will push each video in the video sequence in sequence.

[0113] It should be noted that, in the embodiments of the present invention, for a video that is being played, the video push device first determines the videos that have a high similarity to the title text of the video, and then pushes the videos from the determined videos in order of increasing similarity to the actual content of the video, so that the pushed videos have a high degree of correlation with the title text of the video being played, but a low degree of correlation with the video content, thereby improving the diversity of video push.

[0114] Figure 5 This is an exemplary frame of a video in a playing video provided as an embodiment of the present invention. Figure 6 An exemplary video sequence for push notifications provided by existing technology. Figure 7 This is an exemplary video sequence provided as an embodiment of the present invention. From Figure 5 , Figure 6 and Figure 7 As can be seen, existing technologies tend to recommend videos that are the same as the attractions featured in the currently viewed video, such as the same attraction. This results in a single type of content being pushed to the device. In contrast, this invention recommends videos that are similar to the attractions featured in the currently viewed video, such as videos from the same city or surrounding attractions, but with substantially different content. Figure 8 This is a schematic diagram illustrating an exemplary video sequence push method provided in an embodiment of the present invention, such as... Figure 8 As shown, the left side of the screen displays the video currently playing, while the right side displays the pushed video sequence.

[0115] This invention provides a video recommendation method, comprising: obtaining a first title text corresponding to a currently playing first video; extracting title features from the first title text to obtain a first title feature vector, and extracting visual features from the first video to obtain a first visual feature vector; obtaining the title feature vector corresponding to each video in a preset video set, and selecting multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector; obtaining the visual feature vector corresponding to each of the multiple videos, and sorting the multiple videos in ascending order of similarity between the corresponding visual feature vector and the first visual feature vector to obtain a video sequence; and sequentially recommending each video in the video sequence. The video recommendation method provided by this invention, for a currently playing video, combines both title text and visual content to recommend videos with high similarity to the video title text but low similarity to the actual video content, thereby improving the diversity of video recommendations.

[0116] This invention provides a video recommendation device. Figure 9 A schematic diagram of the structure of a video recommendation device provided in an embodiment of the present invention. Figure 1 .like Figure 9 As shown, it includes:

[0117] Module 901 is used to obtain the first title text corresponding to the currently playing first video;

[0118] The feature extraction module 902 is used to extract title features from the first title text to obtain a first title feature vector, and to extract visual features from the first video to obtain a first visual feature vector.

[0119] The selection module 903 is used to obtain the title feature vector corresponding to each video in the preset video set, and select multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector;

[0120] The sorting module 904 is used to obtain the visual feature vector corresponding to each of the plurality of videos, and sort the plurality of videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain a video sequence.

[0121] The push module 905 is used to push each video in the video sequence sequentially.

[0122] In one embodiment of the present invention, the feature extraction module 902 is further configured to split the first title text into multiple sub-texts; use a preset text understanding model to perform semantic analysis on each of the multiple sub-texts to obtain multiple feature vectors that correspond one-to-one with the multiple sub-texts; and perform average pooling on the multiple feature vectors to obtain the first title feature vector.

[0123] In one embodiment of the present invention, the feature extraction module 902 is further configured to perform video frame sampling processing on the first video to obtain multiple video images; extract features from each of the multiple video images to obtain multiple image feature vectors that correspond one-to-one with the multiple video images; and aggregate the multiple image feature vectors using a preset multimodal video classification model to obtain the first visual feature vector.

[0124] In one embodiment of the present invention, the feature extraction module 902 is further configured to determine the total number of video images contained in the first video, and determine the number of video frame samples for the first video based on the total number of video images; divide the first video evenly into video segments with a number equal to the number of video frame samples, and select one video image from each of the divided video segments to obtain the multiple video images.

[0125] In one embodiment of the present invention, the feature extraction module 902 is further configured to: determine the number of video frame samples as a first preset value when the total number of video images is less than a first preset frame number; and determine the number of video frame samples as a second preset value when the total number of video images is greater than a second preset frame number; wherein the first preset frame number is less than the second preset frame number, and the first preset value is less than the second preset value; when the total number of video images is not less than the first preset frame number and not greater than the second preset frame number, determine the difference between the first preset frame number and the second preset frame number as a frame number difference, and determine the difference between the first preset value and the second preset value as a numerical difference; calculate the difference between the total number of video images and a preset interval frame number, and determine the product of the calculated difference and the frame number difference as a first value; determine the ratio of the first value to the numerical difference as a second value, and determine the difference between the second preset value and the second value as the number of video frame samples.

[0126] In one embodiment of the present invention, the feature extraction module 902 is further configured to: select multiple image feature vectors from the plurality of image feature vectors, and determine each image feature vector as an initial cluster center to form an initial cluster center group; select one image feature vector from the plurality of image feature vectors to determine a first image feature vector; for the initial cluster center group, update the initial cluster center closest to the first image feature vector using the first image feature vector to obtain a first cluster center group; construct a first orthogonal loss function using the first cluster center group; if the first orthogonal loss function is not less than a preset threshold, continue to select a second image feature vector from the plurality of image feature vectors, and update the first cluster center group using the second image feature vector, until the kth orthogonal loss function constructed by the kth cluster center group after the kth cluster center update is less than the preset threshold; k is a natural number greater than 1; and determine the first visual feature vector using the kth cluster center group and the plurality of image feature vectors.

[0127] In one embodiment of the present invention, the feature extraction module 902 is further configured to: calculate the distance to each cluster center in the k-th cluster center group for each of the plurality of image feature vectors, generate a corresponding distance vector, and calculate the proportion of each cluster center in the k-th cluster center group, generate a corresponding proportion vector; perform nonlinear activation processing on each of the plurality of image feature vectors to obtain a corresponding feature weight, which is determined as a corresponding weight vector; fuse the corresponding distance vector, proportion vector, and weight vector for each of the plurality of image feature vectors to generate a corresponding fused feature vector; and determine the sum of the fused feature vectors corresponding to different image feature vectors in the plurality of image feature vectors as a video image feature vector, and determine a first visual feature vector based on the video image feature vector.

[0128] In one embodiment of the present invention, the feature extraction module 902 is further configured to normalize the video image feature vector, and reduce the dimension of the normalized feature vector to obtain a dimension-reduced feature vector; normalize and activate the dimension-reduced feature vector to obtain corresponding enhanced feature weights; and use the enhanced feature weights to enhance the dimension-reduced feature vector to obtain the first visual feature vector.

[0129] This invention provides a video push device. Figure 10 A schematic diagram of the structure of a video push device provided in an embodiment of the present invention. Figure 2 .like Figure 10 As shown, the device includes: a processor 1001, a memory 1002, and a communication bus 1003;

[0130] The communication bus 1003 is used to realize the communication connection between the processor 1001 and the memory 1002;

[0131] The processor 1001 is used to execute the computer program stored in the memory 1002 to implement the above-mentioned video push method.

[0132] This invention provides a video push device that obtains the first title text corresponding to the currently playing first video; extracts title features from the first title text to obtain a first title feature vector, and extracts visual features from the first video to obtain a first visual feature vector; obtains the title feature vector corresponding to each video in a preset video set, and selects multiple videos from the preset video set according to the similarity between the corresponding title feature vector and the first title feature vector from high to low; obtains the visual feature vector corresponding to each of the multiple videos, and sorts the multiple videos according to the similarity between the corresponding visual feature vector and the first visual feature vector from low to high to obtain a video sequence; and pushes each video in the video sequence sequentially. The video push device provided by this invention, for the currently playing video, combines both title text and visual content to push videos with high similarity to the video title text but low similarity to the actual video content, thus improving the diversity of video pushes.

[0133] This invention provides a computer-readable storage medium storing one or more computer programs, which can be executed by one or more processors to implement the aforementioned video push method. The computer-readable storage medium can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be a device including one or any combination of the above-mentioned memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.

[0134] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A video push method, characterized in that, The method includes: Get the first title text corresponding to the currently playing first video; The first title text is subjected to title feature extraction to obtain the first title feature vector, and the first video is subjected to video frame sampling processing to obtain multiple video images; Feature extraction is performed on each of the multiple video frames to obtain multiple image feature vectors that correspond one-to-one with the multiple video frames. Multiple image feature vectors are selected from the plurality of image feature vectors, and each of the plurality of image feature vectors is determined as an initial cluster center to form an initial cluster center group; One image feature vector is selected from the plurality of image feature vectors and determined as the first image feature vector; For the initial cluster center group, the initial cluster center that is closest to the first image feature vector is updated using the first image feature vector to obtain the first cluster center group; Construct a first orthogonal loss function using the first cluster center group; If the first orthogonal loss function is not less than a preset threshold, a second image feature vector is selected from the plurality of image feature vectors, and the first cluster center group is updated using the second image feature vector until the kth orthogonal loss function constructed by the kth cluster center group after the kth cluster center update is less than the preset threshold; k is a natural number greater than 1. The first visual feature vector is determined using the k-th cluster center group and the plurality of image feature vectors; Obtain the title feature vector corresponding to each video in the preset video set, and select multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector; Obtain the visual feature vector corresponding to each of the multiple videos, and sort the multiple videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence; Each video in the video sequence is pushed out sequentially.

2. The method according to claim 1, characterized in that, The step of extracting title features from the first title text to obtain a first title feature vector includes: Split the first title text into multiple sub-texts; Using a pre-defined text understanding model, semantic analysis is performed on each of the multiple sub-texts to obtain multiple feature vectors that correspond one-to-one with the multiple sub-texts; The first title feature vector is obtained by performing average pooling on the multiple feature vectors.

3. The method according to claim 1, characterized in that, The step of performing video frame sampling processing on the first video to obtain multiple video images includes: Obtain the total number of video images in the first video, and determine the number of video frame samples for the first video based on the total number of video images; The first video is evenly divided into video segments with a number equal to the number of video frame samples, and one video frame is selected from each video segment to obtain the multi-frame video images.

4. The method according to claim 3, characterized in that, Determining the number of video frame samples for the first video based on the total number of video images includes: If the total number of video images is less than the first preset number of frames, the number of video frame samples is determined to be the first preset value; If the total number of video images is greater than the second preset number of frames, the number of video frame samples is determined to be the second preset value; wherein, the first preset number of frames is less than the second preset number of frames, and the first preset value is less than the second preset value; When the total number of video images is not less than the first preset frame number and not greater than the second preset frame number, the difference between the first preset frame number and the second preset frame number is determined as the frame number difference, and the difference between the first preset value and the second preset value is determined as the value difference; Calculate the difference between the total number of video images and the preset interval frame number, and determine the product of the calculated difference and the frame number difference as the first value; The ratio of the first value to the difference between the two values ​​is determined as the second value, and the difference between the second preset value and the second value is determined as the number of video frame samples.

5. The method according to claim 1, characterized in that, The step of determining the first visual feature vector using the k-th cluster center group and the plurality of image feature vectors includes: For each of the multiple image feature vectors, calculate the distance to each cluster center in the k-th cluster center group, generate the corresponding distance vector, and calculate the proportion of each cluster center in the k-th cluster center group, generate the corresponding proportion vector. For each of the multiple image feature vectors, a nonlinear activation process is performed to obtain the corresponding feature weight, which is then determined as the corresponding weight vector. For each of the multiple image feature vectors, the corresponding distance vector, scale vector, and weight vector are fused to generate a corresponding fused feature vector. The sum of the fused feature vectors corresponding to different image feature vectors among the multiple image feature vectors is determined as the video image feature vector, and the first visual feature vector is determined based on the video image feature vector.

6. The method according to claim 5, characterized in that, The determination of the first visual feature vector based on the video image feature vector includes: The video image feature vector is normalized, and the normalized feature vector is then reduced in dimensionality to obtain the reduced feature vector. The reduced feature vectors are normalized and activated to obtain the corresponding enhanced feature weights; Using the enhanced feature weights, feature enhancement is performed on the dimensionality-reduced feature vector to obtain the first visual feature vector.

7. A video push device, characterized in that, include: The acquisition module is used to retrieve the first title text corresponding to the currently playing first video; The feature extraction module is used to extract title features from the first title text to obtain a first title feature vector, and to perform video frame sampling processing on the first video to obtain multiple video images; Feature extraction is performed on each frame of the multi-frame video images to obtain multiple image feature vectors corresponding one-to-one with the multi-frame video images. Multiple image feature vectors are selected from these multiple image feature vectors, and each of these multiple image feature vectors is determined as an initial cluster center, forming an initial cluster center group. One image feature vector is selected from the multiple image feature vectors and determined as a first image feature vector. For the initial cluster center group, the initial cluster center closest to the first image feature vector is updated using the first image feature vector, resulting in a first cluster center group. A first orthogonal loss function is constructed using the first cluster center group. If the first orthogonal loss function is not less than a preset threshold, a second image feature vector is selected from the multiple image feature vectors, and the first cluster center group is updated using the second image feature vector, until the k-th orthogonal loss function constructed using the k-th cluster center group after the k-th cluster center update is less than the preset threshold; k is a natural number greater than 1. A first visual feature vector is determined using the k-th cluster center group and the multiple image feature vectors. The selection module is used to obtain the title feature vector corresponding to each video in the preset video set, and select multiple videos from the preset video set in descending order of similarity between the corresponding title feature vector and the first title feature vector; The sorting module is used to obtain the visual feature vector corresponding to each of the multiple videos, and sort the multiple videos from low to high according to the similarity between the corresponding visual feature vector and the first visual feature vector to obtain the video sequence. The push module is used to push each video in the video sequence sequentially.

8. A video push device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute a computer program stored in the memory to implement the video push method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more computer programs, which can be executed by one or more processors to implement the video push method according to any one of claims 1-6.

Images