Method and apparatus for performing video sorting on basis of video sorting model, and device

Abstract

A method and apparatus for performing video sorting on the basis of a video sorting model, and a device, which belong to the technical field of artificial intelligence. The method comprises: acquiring N videos; generating respective video features of the N videos by means of a video encoder, so as to obtain N video features; by means of a collection context encoder and on the basis of the N video features, generating video context features between the N videos; by means of a position decoder and on the basis of the video context features, determining the probabilities of the N videos respectively corresponding to N candidate video positions; and on the basis of the probabilities, allocating the N videos to the N candidate video positions, so as to obtain a video sequence. In the method, video content association between a plurality of videos is taken into full consideration in the video context features, such that an obtained sorting result of a video sequence is more accurate. The method for determining the video sequence by means of taking into comprehensive consideration the probabilities of N videos respectively corresponding to N candidate video positions effectively improves the accuracy of the sorting result of the video sequence.

Description

Video ranking method, apparatus and equipment based on video ranking model

[0001] This application claims priority to Chinese Patent Application No. 202410911025.0, filed on July 8, 2024, entitled "Video Sorting Method, Apparatus and Device Based on Video Sorting Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a video sorting method, apparatus and device based on a video sorting model. Background Technology

[0003] On various online video platforms, video collections (setups) are a way for users to browse continuous content, where a video collection includes multiple videos. These collections cover a wide range of content, including online courses, TV series, movies, and animations. However, due to various reasons such as revisions, re-creation, deletion, and re-publishing, video creators sometimes publish these multiple videos in a disorganized manner, which often leads to a poor user browsing experience. Therefore, it is necessary to reorder the videos within the same video collection uploaded by creators to the video platform to improve the user browsing experience.

[0004] In related technologies, peer-to-peer ranking can be used to rank multiple videos. The peer-to-peer ranking method involves determining a ranking score for each video based on its extracted feature representation, and then using these scores to obtain the final ranking of all videos.

[0005] The above point-to-point ranking method learns the feature representation of each video separately to determine the ranking result of multiple videos. Due to the lack of global information of multiple videos, the generated ranking result will be inaccurate. Summary of the Invention

[0006] This application provides a video sorting method, apparatus, and device based on a video sorting model. The technical solution provided by this application is as follows:

[0007] According to one aspect of the embodiments of this application, a video ranking method based on a video ranking model is provided, the video ranking model including a video encoder, a set context encoder, and a position decoder, the method comprising:

[0008] Get N videos, where N is an integer greater than 1;

[0009] The video encoder generates video features for each of the N videos, resulting in N video features;

[0010] The collection context encoder generates video context features among the N videos based on the N video features.

[0011] The location decoder determines the probability that each of the N videos corresponds to one of the N candidate video locations based on the video context features.

[0012] Based on the probability, the N videos are assigned to the N candidate video positions to obtain a video sequence.

[0013] According to one aspect of the embodiments of this application, a training method for a video ranking model is provided, the video ranking model including a video encoder, a set context encoder, and a position decoder, the method comprising:

[0014] Obtain N sample videos, where N is an integer greater than 1;

[0015] The video encoder generates video features for each of the N sample videos, resulting in N sample video features;

[0016] The collection context encoder generates video context features among the N sample videos based on the features of the N sample videos;

[0017] The location decoder determines the probability that each of the N sample videos corresponds to one of the N candidate video locations based on the video context features.

[0018] With the goal of minimizing the probability error of the N sample videos, the parameters of the video ranking model are adjusted to obtain the trained video ranking model.

[0019] According to one aspect of the embodiments of this application, a video sorting apparatus based on a video sorting model is provided, the video sorting model including a video encoder, a set context encoder, and a position decoder, the apparatus comprising:

[0020] The acquisition module is used to acquire N videos, where N is an integer greater than 1;

[0021] The first generation module is used to generate video features for each of the N videos through the video encoder, thereby obtaining N video features;

[0022] The second generation module is used to generate video context features between the N videos based on the N video features using the collection context encoder;

[0023] The determination module is used to determine the probability that the N videos correspond to the N candidate video positions based on the video context features using the position decoder;

[0024] The module is used to assign the N videos to the N candidate video positions according to the probability, thereby obtaining a video sequence.

[0025] In some embodiments, the determining module is configured to generate video features with added position embeddings for each of the N videos based on the video context features and the position embeddings of each of the N candidate video locations, wherein the position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video locations; and the position decoder determines the probability that each of the N videos corresponds to one of the N candidate video locations based on the video features with added position embeddings of each of the N videos.

[0026] In some embodiments, the determining module is configured to determine the probability that the N videos correspond to the first candidate video position based on the video features of the added position embeddings of the N videos and the initial position embeddings, using the position decoder; and to determine the probability that the N videos correspond to the i-th candidate video position based on the video features of the added position embeddings of the N videos, the initial position embeddings, and the position embeddings of the first to (i-1)-th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

[0027] In some embodiments, the determining module is used to take the video features of the added position embedding of each of the N videos as key vectors and value vectors, take the initial position embedding as query vectors, and use the position decoder to determine the probability that the N videos correspond to the first candidate video position based on the key vectors, the value vectors and the query vectors.

[0028] In some embodiments, the determining module is used to take the video features of the added position embedding of each of the N videos as key vectors and value vectors, and take the initial position embedding and the position embedding of each of the 1st to i-1th candidate video positions as query vectors. The position decoder determines the probability that the N videos correspond to the i-th candidate video position based on the key vector, the value vector and the query vector.

[0029] In some embodiments, the obtaining module is used to assign the video with the highest probability at each of the N candidate video locations to the candidate video location to obtain the video sequence.

[0030] In some embodiments, the obtaining module is used to obtain multiple candidate bundles and the cumulative score of each candidate bundle according to the probability, each candidate bundle corresponding to a ranking result; take the ranking result corresponding to the candidate bundle with the highest cumulative score as the ranking result of the N videos; and assign the N videos to the N candidate video positions according to the ranking result of the N videos to obtain the video sequence.

[0031] In some embodiments, the obtaining module is configured to, when i equals 1, for the first candidate video position, determine the probability score of each of the N videos based on the probability of the N videos corresponding to the first candidate video position; select k candidate videos from the N videos based on the probability scores of the N videos, and generate a candidate bundle for the first candidate video position, where k is a positive integer less than or equal to N; when i is an integer from 2 to N, for the i-th candidate video position, determine the probability of the N videos corresponding to the i-th candidate video position; for each candidate bundle in the candidate bundles of the first i-1 candidate video positions, ... Based on the probability of the N videos corresponding to the i-th candidate video position, determine the probability score of the video among the N videos that is not included in the candidate bundle; based on the probability scores of the video among the N videos that is not included in the candidate bundle, select k candidate videos and add the k candidate videos to the candidate bundle respectively, update the candidate bundles of the first i-1 candidate video positions, and obtain the candidate bundles of the first i candidate video positions; for each candidate bundle in the candidate bundles of the first i candidate video positions, obtain the cumulative score of the candidate bundle based on the probability scores of the i videos included in the candidate bundle.

[0032] In some embodiments, the video ranking model further includes a successor predictor; the obtaining module is used to generate, through the successor predictor, a coherence score for the video pair corresponding to the i-th candidate video position in each of the N videos based on the video context features, wherein for each of the N videos, the video pair consists of the video and the video corresponding to the (i-1)-th candidate video position, where i is an integer greater than or equal to 2 and less than or equal to N; based on the probability and the coherence score for the video pair corresponding to the i-th candidate video position in each of the N videos, multiple candidate bundles and the cumulative score of each candidate bundle are obtained.

[0033] In some embodiments, the obtaining module is configured to, when i equals 1, for the first candidate video position, determine the probability score of each of the N videos based on the probability of the N videos corresponding to the first candidate video position; select k candidate videos from the N videos based on the probability scores of the N videos, and generate a candidate bundle for the first candidate video position, where k is a positive integer less than or equal to N; for each candidate bundle in the candidate bundle for the first candidate video position, determine the probability score of one video included in the candidate bundle as the combined score of the first video; when i is an integer from 2 to N, for the first i-1 candidate video positions... For each candidate bundle in the bundle, for each video among the N videos that is not included in the candidate bundle, the combined score of the video is determined based on the probability score of the video and the coherence score of the video pair. Based on the combined score, k candidate videos are selected from the N videos and added to the candidate bundle respectively. The candidate bundles at the first i-1 candidate video positions are updated to obtain the candidate bundles at the first i candidate video positions. For each candidate bundle in the candidate bundles at the first i candidate video positions, the cumulative score of the candidate bundle is obtained based on the combined scores of the i videos included in the candidate bundle.

[0034] In some embodiments, the first generation module is used to obtain the video frame set and text content of each of the N videos, wherein the text content is obtained based on at least one of the audio and text included in the video; for each of the N videos, the video encoder generates video features of the video based on the video frame set and text content of the video.

[0035] In some embodiments, the video encoder includes a visual encoder, a text encoder, and a multimodal feature fusion network; the first generation module is configured to generate visual features of the video based on a set of video frames using the visual encoder; generate text features of the video based on the text content using the text encoder; and generate video features of the video based on the visual features and the text features using the multimodal feature fusion network.

[0036] In some embodiments, the first generation module is used to take the visual features as key vectors and value vectors, and the initial video features and the text features as query vectors, and generate video features of the video through the multimodal feature fusion network based on the key vectors, the value vectors and the query vectors. The initial video features are learnable parameters in the training process of the video encoder, and the initial video features are used to learn the features of the video frames included in the video frame set and the features of the text content.

[0037] According to one aspect of the embodiments of this application, a training apparatus for a video ranking model is provided, the video ranking model including a video encoder, a set context encoder, and a position decoder, the apparatus comprising:

[0038] The acquisition module is used to acquire N sample videos, where N is an integer greater than 1;

[0039] The first generation module is used to generate video features for each of the N sample videos through the video encoder, thereby obtaining N sample video features;

[0040] The second generation module is used to generate video context features between the N sample videos based on the features of the N sample videos through the collection context encoder;

[0041] The determination module is used to determine the probability that the N sample videos correspond to the N candidate video positions based on the video context features using the position decoder;

[0042] The first adjustment module is used to adjust the parameters of the video ranking model with the goal of minimizing the error of the probability of the N sample videos, so as to obtain the trained video ranking model.

[0043] In some embodiments, the determining module is configured to generate video features with added position embeddings for each of the N sample videos based on the video context features and the position embeddings of each of the N candidate video locations, wherein the position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video locations; and the position decoder determines the probability that each of the N sample videos corresponds to one of the N candidate video locations based on the video features with added position embeddings of each of the N sample videos.

[0044] In some embodiments, the determining module is configured to determine the probability that the N sample videos correspond to the first candidate video position by means of the position decoder based on the video features of the added position embeddings of the N sample videos and the initial position embeddings; and to determine the probability that the N sample videos correspond to the i-th candidate video position by means of the position decoder based on the video features of the added position embeddings of the N sample videos, the initial position embeddings, and the position embeddings of the first to (i-1)-th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

[0045] In some embodiments, the video ranking model further includes a successor predictor, and the apparatus further includes a second adjustment module.

[0046] The second adjustment module is used to obtain video pairs for each of the N sample videos. For each of the N sample videos, the video pair consists of the sample video and any one of the N sample videos. The successor predictor generates a coherence score for the video pair based on the video context features of the video pair and the two sample videos included in the video pair. The parameters of the video ranking model are adjusted with the goal of minimizing the error of the coherence score of the positive video pairs in the video pair, to obtain the trained video ranking model. The positive video pair refers to a video pair consisting of two adjacent sample videos from the N sample videos.

[0047] In some embodiments, the first generation module is used to obtain the video frame set and text content of each of the N sample videos, wherein the text content is obtained based on at least one of the speech and text included in the sample video; for each of the N sample videos, the video encoder generates video features of the sample video based on the video frame set and text content of the sample video.

[0048] In some embodiments, the video encoder includes a visual encoder, a text encoder, and a multimodal feature fusion network; the first generation module is configured to generate visual features of the sample video based on the set of video frames of the sample video using the visual encoder; generate text features of the sample video based on the text content of the sample video using the text encoder; and generate video features of the sample video based on the visual features and the text features using the multimodal feature fusion network.

[0049] In some embodiments, the first generation module is used to take the visual features as key vectors and value vectors, and the initial video features and the text features as query vectors, and generate video features of the sample video through the multimodal feature fusion network based on the key vectors, the value vectors and the query vectors. The initial video features are learnable parameters in the training process of the video encoder, and the initial video features are used to learn the features of the video frames included in the video frame set and the features of the text content.

[0050] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described video ranking method based on a video ranking model or the above-described video ranking model training method.

[0051] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described video ranking method based on a video ranking model or the above-described video ranking model training method.

[0052] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described video ranking method based on a video ranking model or the above-described video ranking model training method.

[0053] The technical solutions provided in this application have at least the following beneficial effects:

[0054] On the one hand, the video encoder can extract the video features of each video, and the ensemble context encoder can extract the video context features of each video based on the video features. These video context features fully consider the video content associations between multiple videos, making the sorting results of the obtained video sequence more accurate. On the other hand, the position decoder can determine the probability of N videos corresponding to each candidate video position based on the video context features, that is, based on the learned video feature representations. This probability is used to guide the allocation of each video. This method of comprehensively considering the probability of N videos corresponding to N candidate video positions to determine the video sequence effectively improves the accuracy of the sorting results of the video sequence. Attached Figure Description

[0055] Figure 1 is a schematic diagram of a flowchart of a point-to-point sorting, pairwise sorting, and global list sorting method provided in an embodiment of this application;

[0056] Figure 2 is a schematic diagram of the implementation environment of a solution provided in an embodiment of this application;

[0057] Figure 3 is a flowchart of a video sorting method based on a video sorting model provided in an embodiment of this application;

[0058] Figure 4 is a schematic diagram of the components of a video ranking model provided in an embodiment of this application;

[0059] Figure 5 is a flowchart of a video sorting method based on a video sorting model provided in another embodiment of this application;

[0060] Figure 6 is a flowchart of a video sorting method based on a video sorting model provided in another embodiment of this application;

[0061] Figure 7 is a flowchart of a training method for a video ranking model provided in an embodiment of this application;

[0062] Figure 8 is a flowchart of a training method for a video ranking model provided in another embodiment of this application;

[0063] Figure 9 is a block diagram of a video sorting device based on a video sorting model provided in an embodiment of this application;

[0064] Figure 10 is a block diagram of a training device for a video ranking model provided in one embodiment of this application;

[0065] Figure 11 is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0067] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0068] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0069] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0070] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AIGC (Artificial Intelligence Generated Content), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0071] The solutions provided in this application relate to machine learning and deep learning technologies in artificial intelligence, and are specifically illustrated through the following embodiments.

[0072] Before introducing and explaining the technical solution of this application, some concepts involved in this application will be defined and explained.

[0073] 1. Sentence Ordering

[0074] Sentence ordering is a task that involves scrambling the order of sentences in a text, analyzing the semantic relationships and logical information between the sentences, and then reordering them to form a coherent and fluent text. For example, assuming the text consists of sentences 1, 2, and 3 in sequence, sentence ordering refers to analyzing the semantic relationships between the scrambled input sentences, such as sentences 2, 1, and 3, and then reordering them to obtain a coherent and fluent output text [sentence 1, sentence 2, sentence 3].

[0075] In related technologies, the proposed solutions for sentence ranking tasks can be divided into the following three categories:

[0076] (1) Pointwise Ordering: In this method, each sentence is considered individually, without regard to other sentences. Each sentence is assigned a score, and then the sentences are ordered according to their scores. The advantage of this method is its simplicity and ease of understanding, but it ignores the relationships between sentences, which may lead to unsatisfactory ordering results.

[0077] (2) Pairwise Ordering: This method considers the relationships between sentence pairs. For each pair of sentences, the model predicts which one should come first. These predictions are then used to generate the final ranking. This method is more complex than point-to-point ranking, but it takes into account the relationships between sentences and may therefore produce better ranking results.

[0078] (3) Listwise Ordering: This method considers global information about the entire list of sentences. The model directly predicts the optimal order for the entire list. This method is the most complex, but it takes into account the relationships between all sentences, and therefore may generate the best ordering result.

[0079] In summary, the main difference between these three methods lies in how they consider sentence relationships. Point-to-point sorting methods consider only individual sentences, pairwise comparison sorting methods consider sentence pairs, while list-wide sorting methods consider the entire list of sentences. As the information considered becomes more comprehensive, the complexity of these methods increases, but the resulting sorting results may also improve.

[0080] 2. Video sorting

[0081] Based on the three types of solutions proposed above for sentence sorting tasks, and considering that videos contain both image frames and text content, video sorting tasks can be regarded as a multimodal sorting problem, and can also be solved using the three sorting paradigms mentioned above, namely point-to-point sorting, pairwise comparison sorting, and list global sorting.

[0082] As shown in sub-Figure 1 of Figure 1, a point-to-point ranking model outputs ranking scores (S1, S2, and S3) for each of the three videos (Video 1, Video 2, and Video 3). Based on these ranking scores, a ranking sequence of the three videos is obtained. For example, assuming S3 > S1 > S2, the ranking sequence of the three videos is Video 3, Video 1, and Video 2. This point-to-point ranking method predicts a ranking score for each video individually. Due to the lack of global information from other videos, this can lead to inaccurate video ranking sequences.

[0083] As shown in Sub - figure 2 of Figure 1, pairwise comparison sorting refers to evaluating all possible video pairs and predicting their partial order to obtain a sorting sequence of multiple videos. Exemplarily, for videos Video 1 (V1), Video 2 (V2), and Video 3 (V3), all possible video pairs include (V1, V2), (V1, V3), and (V2, V3). By comparing the order of the two videos in each video pair respectively, a sorting sequence of the 3 videos is obtained. For example, by comparing the order of V1 and V2 in (V1, V2), assuming V1 > V2, that is, V1 is before V2 in the sequence. Similarly, by comparing the order of V1 and V3 in (V1, V3) and the order of V2 and V3 in (V2, V3), assuming V1 < V2, V2 < V3. Then the final sorting sequence is V3 > V1 > V2, that is, the sorting sequence is Video 3, Video 1, and Video 2. In this method, on the one hand, the number of video pairs grows quadratically (Quadratic Growth). Quadratic growth is also known as quadratic time complexity and is usually used to describe an algorithm whose running time or required resources are proportional to the square of the size of the input data. When comparing every pair of elements in a set of data, the amount of computation required grows quadratically as the size of the data set increases. Exemplarily, assume there is a video set containing n videos. To compare every video pair, the number of comparisons required will be n 2 , which is a quadratic function. As n increases, the number of comparisons required will increase rapidly, which is the so - called quadratic growth, where n is a positive integer. On the other hand, it will lead to the problem of cascading conflict. The cascading conflict problem refers to a phenomenon that when a pair of elements in the input data is wrongly compared or sorted, it may cause chain conflicts and errors among other related elements. This situation usually occurs when dealing with data with interdependent relationships. For example, in a sorting task, assume there is a data set {A, B, C, D} containing four elements that needs to be sorted according to a certain rule. The correct sorting order can be A, B, C, D. If when comparing A and B, the wrong conclusion A > B is drawn (actually A < B), then this may lead to cascading conflicts among other elements. For example, next, the conclusions B > C (actually B < C) and C > D (actually C < D) may be drawn, resulting in an incorrect sorting order for the whole.

[0084] As shown in Sub - figure 3 of Figure 1, the list global sorting method may include the following steps: First, extract the video features of each video through network E; then through network E cBased on the video features of each video, a sorted sequence of multiple videos is generated through sequence labeling or position decoding. Although the list paradigm can avoid the problems of point-to-point and pairwise sorting methods, existing global list sorting models are complex to model and are easily affected by the order of the input video sequences.

[0085] This application improves upon the global sorting method for lists by proposing a video sorting method based on position decoding and successor prediction. Specifically, as shown in sub-Figure 4 of Figure 1, where D represents position decoding and successor prediction. Position decoding utilizes learnable position embeddings for progressive position decoding to obtain the probabilities of N videos corresponding to N candidate video positions. Successor prediction is based on a pairwise sorting method, used to infer the successor video (the next video) of the previous video during the position decoding process.

[0086] Please refer to Figure 2, which shows a schematic diagram of an implementation environment provided in one embodiment of this application. This implementation environment may include a model training device 210 and a model usage device 220.

[0087] The model training device 210 can be an electronic device such as a mobile phone, desktop computer, tablet computer, laptop computer, vehicle terminal, server, intelligent robot, smart TV, multimedia playback device, or other electronic devices with strong computing power; this application does not limit this. The model training device 210 is used to train the video ranking model.

[0088] In this embodiment, the video ranking model is a deep neural network model. Optionally, the model training device 210 can train the video ranking model using machine learning to achieve better performance. Optionally, the training process of the video ranking model is as follows (this is only a brief description; the specific training process is described in the following embodiments): acquire N sample videos; generate video features for each of the N sample videos using a video encoder, resulting in N sample video features; generate video context features between the N sample videos using a set context encoder based on the N sample video features; determine the probability of each candidate video position corresponding to the N sample videos using a position decoder based on the video context features; adjust the parameters of the video ranking model with the goal of minimizing the error of the probabilities of the N sample videos, to obtain the trained video ranking model.

[0089] The device 220 using the model can be an electronic device such as a mobile phone, desktop computer, tablet computer, laptop computer, vehicle terminal, server, intelligent robot, smart TV, multimedia playback device, or other electronic devices with strong computing power; this application does not limit this. The device 220 using the model can employ a trained video ranking model to generate ranking results for multiple videos to be ranked.

[0090] The model training device 210 and the model usage device 220 can be two separate devices or the same device.

[0091] The method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. When the computer device is a server, it can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The computer device can be the model training device 210 in Figure 2, or the model usage device 220.

[0092] Please refer to Figure 3, which shows a flowchart of a video ranking method based on a video ranking model according to an embodiment of this application. The execution entity for each step of this method can be a computer device, for example, the computer device can be the model-using device 220 in the implementation environment shown in Figure 2. The method may include at least one of the following steps 310 to 350.

[0093] Step 310: Obtain N videos, where N is an integer greater than 1.

[0094] In some embodiments, N videos may belong to the same video set, also known as a video collection, which refers to a set of multiple content-related videos. That is, the N videos are content-related N videos. In this application, a video ranking model can be used to learn the semantic and logical relationships between the N videos to reorder the N videos from a random input, making them form a coherent and logical video ranking sequence.

[0095] Step 320: Generate video features for each of the N videos using the video encoder, thus obtaining N video features.

[0096] In some embodiments, a video may simultaneously contain multiple modalities of information, including visual information, audio information, and text information. Visual information refers to image frames in the video, elements that can be directly perceived visually; audio information refers to sound data in the video, including all audible sound elements such as dialogue, music, and sound effects; and text information refers to the text content appearing in the video, such as titles, subtitles, bullet comments, or text extracted from dialogue using technologies such as speech recognition.

[0097] In some embodiments, video features refer to vector representations of a video in visual, audio, and textual modalities. These vector representations can comprehensively capture visual, audio, and textual information, as well as other semantic features, in the video, enabling effective numerical representation of the video in multiple aspects, thereby supporting the analysis and processing of subsequent video ranking tasks.

[0098] Step 330: Generate video context features between N videos based on N video features using a set context encoder.

[0099] In this context, each of the N videos corresponds to a video context feature. For each of the N videos, the context video feature is used to characterize the video content association between that video and the other videos in the N videos.

[0100] Understandably, for any one of the N videos, its video features are used to represent the content of that single video across multiple modalities, including visual, audio, and text. The video context features of that video are learned based on the individual video features of the N videos and are used to represent the video content relationships between that video and the other videos in the N videos. Since video ranking tasks need to consider the semantic relationships between videos, i.e., video content relationships or video content associations, extracting the global video context features of each of the N videos enables the model to better understand and capture the video content associations between the N videos, thereby improving the accuracy and performance of subsequent ranking tasks.

[0101] In some embodiments, step 330 above can be implemented based on the following steps, assuming that the unordered set of video features composed of the video features of N videos is denoted as . in, This represents the video feature corresponding to the first candidate video position in a video sequence. This represents the video feature corresponding to the nth candidate video position in the video sequence. It is achieved through the context encoder E. c Generate a video context feature set H, as shown in Formula 1 below:

[0102] Where F is an unordered set of video features, E c It is a context encoder that receives F as input and generates a set of video context features H. Here, the set of video context features H refers to the set of video context features for each of the N videos, and its dimension can be represented as... Here, R refers to the set of real numbers. It can be understood that each row vector in the video context feature set H has a one-to-one correspondence with N videos; that is, each row vector represents the video context feature of one of the N videos.

[0103] In some embodiments, the collection context encoder can be built on a transformer network. The transformer network generates video context features for each of the N videos based on their respective video features. In some embodiments, the video features of the N videos can be used as key vectors, value vectors, and query vectors, and the transformer network generates the video context features for each of the N videos based on these vectors.

[0104] In some embodiments, the aggregate context encoder can be a graph neural network. First, before generating video context features for each of the N videos based on their respective video features using the graph neural network, the video features of the N videos can be used as graph nodes to calculate the similarity between the N videos, with the similarity used as edges of the graph. Further, the graph neural network can aggregate neighborhood information based on message passing to generate video features containing global context.

[0105] It should be understood that the embodiments of this application do not limit the method of generating video context features.

[0106] Step 340: Based on the video context features, the location decoder determines the probability that each of the N videos corresponds to one of the N candidate video locations.

[0107] It should be understood that for each of the N videos, there is a probability corresponding to each candidate video position. Based on this, the probabilities of the N videos corresponding to the N candidate video positions include a total of N*N probabilities.

[0108] In some embodiments, for each candidate video location, the probabilities of N videos corresponding to that candidate video location reflect the likelihood of that video appearing at that candidate video location. It is understood that a higher probability indicates a greater likelihood of the video appearing at that candidate video location, and a lower probability indicates a less likely likelihood of the video appearing at that candidate video location.

[0109] Step 350: Based on the probability, assign the N videos to the N candidate video positions to obtain the video sequence.

[0110] In some embodiments, the video at each candidate video position can be determined based on the probability that N videos correspond to N candidate video positions, thereby obtaining a video sequence.

[0111] In some embodiments, for each of the N candidate video locations, the video with the highest probability at that candidate video location is assigned to that candidate video location to obtain the video sequence.

[0112] For example, suppose the probability of N videos corresponding to the position of the i-th candidate video is denoted as P(o i )={p1,…,p n Let p1 be the highest probability among the probabilities of N videos corresponding to the i-th candidate video position. Then, the video corresponding to p1 is determined as the video at the i-th candidate video position.

[0113] It is understandable that for each candidate video position, the number of probabilities of the N videos determined is N. However, for the N candidate video positions, the corresponding videos should be N different videos. For each of the N candidate video positions, assigning the video with the highest probability at that position to that candidate video position to obtain the video sequence can include: for the i-th candidate video position, removing the videos corresponding to the first i-1 candidate video positions, and assigning the video with the highest probability among the remaining N videos to the i-th candidate video position. Similarly, the videos corresponding to each of the N candidate video positions can be determined, thus obtaining the video sequence.

[0114] The technical solution provided in this application has two aspects. On the one hand, the video encoder can extract the video features of each video, and the set context encoder can extract the video context features of each video based on the video features. The video context features fully consider the video content association between multiple videos, making the sorting result of the obtained video sequence more accurate. On the other hand, the position decoder can determine the probability of N videos corresponding to each candidate video position based on the video context features, that is, based on the learned video feature representation. This probability is used to guide the allocation of each video. This method of comprehensively considering the probability of N videos corresponding to N candidate video positions to determine the video sequence effectively improves the accuracy of the sorting result of the video sequence.

[0115] Step 320 above, which is to generate video features for each of the N videos by the video encoder, can be implemented according to steps 321A to 322A below, but is not limited thereto.

[0116] Step 321A: Obtain the video frame set and text content for each of the N videos. The text content is obtained based on at least one of the audio and text included in the video.

[0117] A video frame set refers to a collection of multiple video frames, which are used to characterize the visual features of the video. For example, a video frame set may include M video frames, where M is an integer greater than 1. Text content is extracted based on the speech and text included in the video and is used to characterize the audio and text features of the video. For example, text content may include the video's title, subtitles, and text extracted from the video's speech.

[0118] In some embodiments, the text content of the audio included in the video can be extracted based on ASR (Automatic Speech Recognition) technology.

[0119] In some embodiments, text content of text included in a video can be extracted based on OCR (Optical Character Recognition) technology. This application does not limit the method of extracting text content based on speech and text included in a video.

[0120] Step 322A: For each of the N videos, the video encoder generates video features based on the video frame set and text content of the video.

[0121] In some embodiments, since the video features are generated based on a set of video frames and text content, the generated video features are vector representations that fuse the visual and textual modalities of the video.

[0122] In some embodiments, the video encoder includes a visual encoder, a text encoder, and a multimodal feature fusion network; the visual encoder generates visual features of the video based on a set of video frames; the text encoder generates text features of the video based on the text content of the video; and the multimodal feature fusion network generates video features of the video based on the visual features and the text features.

[0123] In some embodiments, considering that the average length of the text exceeds the maximum length supported by many language models (i.e., 512 input units (tokens)), a block-based long text encoding strategy can be designed to extract the text content. This method preserves all semantic information while reducing computational complexity. Specifically, as shown in sub-Figure 1 of Figure 4, the text content obtained from the video using ASR technology is first divided into L blocks, and the CLIP (Contrastive Language-Image Pretraining) text encoder is used to encode the title and each block respectively, generating text features T∈R. (L+1)×d For visual feature extraction, the CLIP visual encoder can be used to extract visual features V∈R from M video frames.M×d .

[0124] In some embodiments, to fuse visual and textual features, cross-modal feature fusion based on an attention mechanism can be employed. Specifically, as shown in sub-Figure 1 of Figure 4, where K refers to the key vector, V to the value vector, and Q to the query vector. The multimodal feature fusion network can be constructed based on a transformer network. Visual features are used as the key and value vectors, and initial video features and textual features are used as the query vector. The multimodal feature fusion network generates video features based on the key, value, and query vectors. The initial video features are learnable parameters acquired during the training of the video encoder, used to learn the features of the video frames included in the video frame set, as well as the features of the text content.

[0125] The initial video features mentioned above can be denoted as [CLS] input units, and their corresponding outputs can be used as the video features F of the video, as shown in Formula 2 below:

[0126] Where Concat represents the concatenation operation, Softmax is the activation function, LN is the layer normalization operation, and FFN is a fully connected feedforward network, where F∈R d R is a real number, d is the dimension of the video feature, T is the text feature, and V is the visual feature.

[0127] The method described above generates video features based on the video frame set and text content. This fusion strategy ensures that the semantic information from both the visual and textual modalities of the video can be effectively combined, resulting in more comprehensive video features.

[0128] For step 320 above, that is, generating video features of N videos by the video encoder to obtain N video features, it can be implemented according to steps 321B to 322B below, but is not limited to therein.

[0129] Step 321B: Obtain the video frame set, audio content, and text content for each of the N videos;

[0130] A video frame set refers to a collection of multiple video frames, which are used to characterize the visual features of a video. For example, a video frame set may include M video frames, where M is an integer greater than 1. Audio content refers to the audio included in the video. Text content may include the video's title, subtitles, and text extracted from the video's audio.

[0131] Step 322B: For each of the N videos, the video encoder generates video features based on the video frame set, audio content, and text content of the video.

[0132] In some embodiments, since video features are generated based on a set of video frames, audio content, and text content, the generated video features are vector representations that fuse the visual, audio, and text modalities of the video.

[0133] In some embodiments, the video encoder includes a visual encoder, a speech encoder, a text encoder, and a multimodal feature fusion network; the visual encoder generates visual features of the video based on a set of video frames; the speech encoder generates speech features of the video based on the speech content of the video; the text encoder generates text features of the video based on the text content of the video; and the multimodal feature fusion network generates video features of the video based on the visual features, speech features, and text features.

[0134] In some embodiments, a multimodal feature fusion network can be used to concatenate video features, speech features, and text features of a video to obtain the video features of the video.

[0135] In some embodiments, a multimodal feature fusion network employs an attention mechanism to concatenate video features, speech features, and text features of a video to obtain the video features of the video.

[0136] In summary, the embodiments of this application do not limit the method of generating video features.

[0137] The method described above generates video features based on the video frame set, audio content, and text content. This fusion strategy ensures that the semantic information of the video's visual, audio, and text modalities can be effectively combined, resulting in more comprehensive video features.

[0138] Step 340 above, which is to determine the probability that the N videos correspond to the positions of the N candidate videos respectively by the position decoder based on the video context features of the N videos, can be implemented according to the following steps 341A to 342A, but is not limited thereto.

[0139] Step 341A: Based on the video context features and the position embeddings of the N candidate video locations, generate video features with added position embeddings for each of the N videos. The position embeddings are learnable parameters during the training process of the position decoder and are used to represent the positions of the candidate videos.

[0140] Location embedding, also known as learnable location embedding, is introduced in this application to decouple video feature learning from location decoding. It is understood that the video feature learning here is based on the video encoder and aggregate context encoder proposed above. In some embodiments, location decoding is used to determine the video corresponding to each candidate video location. Location embedding refers to the feature vector used to represent each candidate video location. In some embodiments, the location embeddings of N candidate video locations are N different identifiers used to distinguish different candidate video locations. In some embodiments, the location embedding is an adjustable parameter of the model during the training phase and can be obtained from the trained location decoder during the usage phase.

[0141] In some embodiments, the learnable position embedding can be denoted as in This refers to embedding at n locations. Then, video features H′ with added location embeddings are generated for each of the N videos, as shown in Formula 3 below:

[0142] H′=H+R (Formula 3)

[0143] Where H is the set of video context features.

[0144] Step 342A: The location decoder determines the probability that each of the N videos corresponds to one of the N candidate video positions based on the video features embedded at the added positions of each of the N videos.

[0145] In some embodiments, the probability that the N videos correspond to the position of the first candidate video is determined by the position decoder based on the video features of the N videos with added position embeddings and the initial position embeddings.

[0146] The initial position embedding is a learnable parameter during the training process of the position decoder. As shown in sub-Figure 2 of Figure 4, the position embedding s is the initial position embedding. The position decoder generates the probabilities of N videos corresponding to the first candidate video position based on the video features H′ of each of the N videos with added position embeddings and the initial position embedding s. Based on the probabilities, the decoding result of the first candidate video position is position embedding 3. Since position embedding 3 corresponds to video context feature 41, the video of the first candidate video position is the video corresponding to video context feature 41.

[0147] In some embodiments, a location decoder determines the probability that the N videos correspond to the i-th candidate video position based on the video features of the added location embeddings of each of the N videos, the initial location embedding, and the location embeddings of the 1st to (i-1th)th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N. For example, for the second candidate video position, the location decoder determines the probability that the N videos correspond to the second candidate video position based on the video features of the added location embeddings of each of the N videos, the initial location embedding, and the location embedding of the 1st candidate video position. For example, for the third candidate video position, the location decoder determines the probability that the N videos correspond to the third candidate video position based on the video features of the added location embeddings of each of the N videos, the initial location embedding, and the location embeddings of the 1st and 2nd candidate video positions.

[0148] The position embedding of the i-th candidate video position refers to the position embedding determined by decoding the i-th candidate video position.

[0149] For example, as shown in sub-Figure 2 of Figure 4, for the second candidate video position, since the decoding result of the first candidate video position is position embedding 3, the position embedding of the first candidate video position is also position embedding 3. Therefore, based on the initial position embedding s and position embedding 3, N probabilities of videos corresponding to the second candidate video position are generated. Based on the probabilities of the N videos, the decoding result of the second candidate video position is obtained as position embedding 1. Since position embedding 1 corresponds to video context feature 42, the video at the second candidate video position is the video corresponding to video context feature 42.

[0150] For example, as shown in sub-Figure 2 of Figure 4, for the third candidate video position, since the decoding result of the second candidate video position is position embedding 1, the position embedding of the second candidate video position is also position embedding 1. Therefore, based on the initial position embedding s, position embedding 3, and position embedding 1, N probabilities of videos corresponding to the third candidate video position are generated. Based on the probabilities of the N videos, the decoding result of the third candidate video position is obtained as position embedding 2. Since position embedding 2 corresponds to video context feature 43, the video at the third candidate video position is the video corresponding to video context feature 43.

[0151] In some embodiments, location decoding can be constructed based on a transformer network. In some embodiments, the video features with added location embeddings from each of the N videos are used as key and value vectors, and the initial location embeddings are... As a query vector, the location decoder determines the probability that the N videos correspond to the position of the first candidate video based on the key vector, value vector, and query vector.

[0152] In some embodiments, the video features H′ embedded at the added positions of each of the N videos are used as key vectors and value vectors, and the initial position embedding and the position embeddings of the 1st to i-1th candidate video positions are used. As a query vector, the location decoder determines the probability that the N videos correspond to the position of the i-th candidate video based on the key vector, value vector, and query vector. This can include the following steps: first, the location decoder calculates the output U′ based on the key vector, value vector, and query vector; then, based on the output U′, it determines the probability that the N videos correspond to the position of the i-th candidate video.

[0153] In some embodiments, the output U′ is calculated by the location decoder based on the key vector, value vector, and query vector, and can be determined as shown in Formula 4 below:

[0154] Where d is the dimension of the video context features for each video. Let H be the transpose of H′. Softmax is the activation function, LN is the layer normalization operation, and FFN is a fully connected feedforward network.

[0155] In some embodiments, after calculating U′, the probability of N videos corresponding to the position of the i-th candidate video is determined based on the output U′, as shown in Formula 5 below:

[0156] P(o i |o * <i ) = softmax(U′ i W o +b o ) (Formula 5)

[0157] Wherein, P(o i |o * <i ) refers to the sorted sequence o of the first i-1 candidate video positions. * <i Calculate the position o of the i-th candidate video under the following conditions i The probability of W. Softmax is the activation function. o and b o U′ represents the parameters of the linear layer. i It is the i-th element of U′ output by the position decoder.

[0158] In some embodiments, the position with the highest probability can be selected as the i-th candidate video bit in the output video sequence. Embedding allows us to obtain the probability of N videos corresponding to each candidate video position. This method decouples video feature learning from position decoding.

[0159] Step 340 above, which is to determine the probability that each of the N videos corresponds to one of the N candidate video positions based on the video context features of each of the N videos by using the position decoder, can be implemented according to step 341B below, but is not limited thereto.

[0160] Step 341B: Input the video context features, the initial position embedding, and the positions of the N candidate video positions into the position decoder to obtain the probabilities of the N videos corresponding to the N candidate video positions.

[0161] It should be understood that the explanation of the initial position embedding and the position embedding of each of the N candidate video positions can be found above, and will not be repeated in the embodiments of this application.

[0162] In some embodiments, the probability that each of the N videos corresponds to the position of the first candidate video in the video sequence is determined by the position decoder based on the video context features of each of the N videos and the initial position embedding.

[0163] In some embodiments, the probability that N videos correspond to the i-th candidate video position is determined by the position decoder based on the video context features of each of the N videos, the initial position embedding, and the position embedding of each of the 1st to i-1th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

[0164] In some embodiments, location decoding can be built upon a transformer network. In some embodiments, the video context features of each of the N videos are used as key and value vectors to embed the initial location. As a query vector, the location decoder determines the probability that N videos correspond to the position of the first candidate video based on the key vector, value vector, and query vector.

[0165] In some embodiments, the video context features of each of the N videos are used as key vectors and value vectors, and the initial position embedding and the position embeddings of the 1st to i-1th candidate video positions are used as query vectors. The position decoder determines the probability of the N videos corresponding to the i-th candidate video position based on the key vectors, value vectors and query vectors.

[0166] The method described above, by using the video context features of each of the N videos and the position embeddings of each of the N candidate video positions, can obtain the probability of each candidate video position corresponding to the N videos. This method can decouple video feature learning and position decoding.

[0167] Step 350 above, which involves assigning N videos to the N candidate video positions based on probability to obtain a video sequence, can be executed through steps 351 to 353.

[0168] Step 351: Based on the probability, obtain multiple candidate bundles and the cumulative score of each candidate bundle. Each candidate bundle corresponds to a ranking result.

[0169] Each candidate bundle represents a possible ranking result; that is, each candidate bundle corresponds to a different ranking result. These ranking results are determined based on the probability of N videos corresponding to each candidate video position. Specifically, for each candidate bundle, the video for each candidate video position is determined based on the probability of the N videos, and a score is calculated. By calculating the video for each of the N candidate video positions and the cumulative score, the optimal candidate bundle can be selected from multiple candidates, thus obtaining the final ranking result.

[0170] This application proposes two methods to obtain multiple candidate bundles and the cumulative score of each candidate bundle.

[0171] Method 1

[0172] In some embodiments, when i equals 1, for the first candidate video position, the probability scores of the N videos are determined according to the probabilities of the N videos corresponding to the first candidate video position; based on the probability scores of the N videos, k candidate videos are selected from the N videos, and a candidate bundle for the first candidate video position is generated, where k is a positive integer less than or equal to N.

[0173] In some embodiments, the videos corresponding to the k largest probability scores among the probability scores of the N videos are determined as k candidate videos.

[0174] For example, assuming N is 4, the probabilities of the N videos can be denoted as {p1, p2, p3, p4}, and assuming p1 > p2 > p3 > p4, and k is 2, then videos 1 and 2 corresponding to p1 and p2 are considered as two candidate videos. The candidate bundle for the first candidate video position includes candidate bundle 1 and candidate bundle 2. Candidate bundle 1 can include video 1 and can be denoted as {V1}, and candidate bundle 2 can include video 2 and can be denoted as {V2}.

[0175] In some embodiments, when i is an integer from 2 to N, for the i-th candidate video position, the probabilities of N videos corresponding to the i-th candidate video position are determined; for each candidate bundle in the candidate bundles of the first i-1 candidate video positions, the probability score of the video not included in the candidate bundle among the N videos is determined based on the probabilities of the N videos corresponding to the i-th candidate video position; based on the probability scores of the video not included in the candidate bundle among the N videos, k candidate videos are selected and added to the candidate bundles respectively, and the candidate bundles of the first i-1 candidate video positions are updated to obtain the candidate bundles of the first i candidate video positions; for each candidate bundle in the candidate bundles of the first i candidate video positions, the cumulative score of the candidate bundle is obtained based on the probability scores of the i videos included in the candidate bundle.

[0176] For each candidate bundle in the candidate bundles for the first i-1 candidate video positions, the videos not included in the candidate bundle among the N videos refer to the remaining videos among the N videos excluding those included in the candidate bundle. For example, for candidate bundle 1, since candidate bundle 1 includes video 1, the videos not included in candidate bundle 1 among the N videos are video 2, video 3, and video 4. For example, for candidate bundle 2, since candidate bundle 2 includes video 2, the videos not included in candidate bundle 1 among the N videos are video 1, video 3, and video 4.

[0177] For example, following the above example, for candidate bundle 1, assuming the probability order of the N videos corresponding to the i-th candidate video position is p2 > p1 > p3 > p4, then videos 2 and 3 corresponding to p2 and p3 are considered as two candidate videos. For candidate bundle 2, assuming the probability order of the N videos corresponding to the i-th candidate video position is also p2 > p1 > p3 > p4, then videos 1 and 3 corresponding to p1 and p3 are considered as two candidate videos.

[0178] The candidate bundles for the first two candidate video positions include candidate bundle 1, candidate bundle 2, candidate bundle 3, and candidate bundle 4. Candidate bundle 1 can include video 1 and video 2, and can be denoted as {V1, V2}; candidate bundle 2 can include video 1 and video 3, and can be denoted as {V1, V3}; candidate bundle 3 can include video 2 and video 1, and can be denoted as {V2, V1}; and candidate bundle 4 can include video 2 and video 3, and can be denoted as {V2, V3}.

[0179] The cumulative score of each candidate bundle is the sum of the probability scores of the videos included in that candidate bundle. For example, for candidate bundle 1 above, its cumulative score is the sum of the probability scores of video 1 and video 2, i.e., p1 + p2.

[0180] Similarly, according to the above method, when i is N, the above steps end, resulting in candidate bundles of N candidate video locations, and the cumulative score of each candidate bundle is calculated. Wherein, δ(o) i () refers to the probability score of the video corresponding to each candidate video position, i.e. Where j represents the decoding step, and j can be equal to i. S′ represents the set of videos already included in the candidate bundle. This is an indicator function that returns 1 if the condition is true, and 0 otherwise, used to exclude videos that already exist in the candidate bundle. The summation is performed N times to obtain the final cumulative score for each candidate bundle.

[0181] Method 2

[0182] In some embodiments, the video ranking model further includes a successor predictor; the successor predictor generates a coherence score for each video pair of N videos corresponding to the i-th candidate video position based on video context features. For each of the N videos, the video pair consists of the video and the video corresponding to the (i-1)-th candidate video position, where i is an integer greater than or equal to 2 and less than or equal to N; based on the probability and the coherence score of each video pair of N videos corresponding to the i-th candidate video position, multiple candidate bundles and the cumulative score of each candidate bundle are obtained.

[0183] In some embodiments, the candidate bundle of the first i candidate video locations is determined based on the probability and the coherence score of the video pairs corresponding to the i-th candidate video location in each of the N videos, as shown in Formula 6 below:

[0184] Wherein, δ(o) i S' refers to the combined score of the videos corresponding to each candidate video position, and S' refers to the set of videos already included in the candidate bundle. `ρ(·)` is an indicator function that returns 1 if the condition is true and 0 otherwise, used to exclude videos already present in the candidate bundle. `β` is a temperature hyperparameter, and `ρ(·)` can be a two-layer MLP (Multilayer Perceptron), using the video context features of two videos as input to calculate the coherence score between the two videos. This refers to the video context features of the video corresponding to the (j-1)th (i.e., i-1)th candidate video position in the candidate bundle. It refers to the video context feature of the video corresponding to the i-th candidate video position in the candidate bundle.

[0185] For example, assuming N is 4, the probabilities of the N videos can be denoted as {p1, p2, p3, p4}, and assuming p1 > p2 > p3 > p4, and k is 2, then videos 1 and 2 corresponding to p1 and p2 are considered as two candidate videos. The candidate bundle for the first candidate video position includes candidate bundle 1 and candidate bundle 2, where candidate bundle 1 can include video 1 and can be denoted as {V1}, and candidate bundle 2 can include video 2 and can be denoted as {V2}. For example, for candidate bundle 1, since the video corresponding to the (i-1)th candidate video position is video 1, the video pairs for each of the N videos can be denoted as (V1, V2), (V1, V3), (V1, V4), and (V1, V5). The coherence score for each video pair can be obtained using ρ(·) based on the video context features of each pair.

[0186] In some embodiments, when i equals 1, for the first candidate video position, the probability scores of the N videos are determined according to the probabilities of the N videos corresponding to the first candidate video position; based on the probability scores of the N videos, k candidate videos are selected from the N videos, and a candidate bundle for the first candidate video position is generated, where k is a positive integer less than or equal to N; for each candidate bundle in the candidate bundle for the first candidate video position, the probability score of one video included in the candidate bundle is determined as the combined score of the first video.

[0187] Since the prediction is made for the video at the first candidate video position, which does not include any videos before it, the coherence score can be disregarded here. Following method 1 above, the probability of the video is used as the combined score.

[0188] In some embodiments, when i is an integer from 2 to N, for each candidate bundle in the candidate bundles of the first i-1 candidate video positions, for each video in the N videos that is not included in the candidate bundle, the combined score of the video is determined based on the probability score of the video and the coherence score of the video pair; based on the combined score, k candidate videos are selected from the N videos and added to the candidate bundles respectively, and the candidate bundles of the first i-1 candidate video positions are updated to obtain the candidate bundles of the first i candidate video positions; for each candidate bundle in the candidate bundles of the first i candidate video positions, the cumulative score of the candidate bundle is obtained based on the combined scores of the i videos included in the candidate bundle.

[0189] In some embodiments, for each candidate bundle in the candidate bundles of the first i-1 candidate video locations, for each video in the N videos that is not included in the candidate bundle, the combined score of the video is determined by summing the probability score of the video and the coherence score of the video pairs.

[0190] In some embodiments, the videos corresponding to the k combined scores with the largest combined scores are determined as k candidate videos, and the k candidate videos are added to the candidate bundle respectively. The candidate bundles at the positions of the first i-1 candidate videos are updated to obtain the candidate bundles at the positions of the first i candidate videos.

[0191] As shown in Figure 5, a video set is first constructed, consisting of N videos. Then, video features are extracted from each of the N videos using a video encoder, and contextual features between the N videos are extracted using a set context encoder. For decoding step i, the probability of the N videos corresponding to the i-th candidate video position is calculated, as well as the coherence score of the video pair corresponding to the i-th candidate video position. The combined score of these two scores is calculated, and the k candidate videos with the highest combined scores are obtained, updating the candidate bundle. The decoding step ends when i equals n. At this point, the sum of the combined scores of the N videos included in each candidate bundle is calculated as the cumulative score of that candidate bundle. The candidate bundle with the highest cumulative score is used as the ranking result of the N videos.

[0192] Step 352: The ranking result corresponding to the candidate bundle with the highest cumulative score is used as the ranking result of the N videos.

[0193] Step 353: According to the sorting results of the N videos, assign the N videos to the N candidate video positions to obtain the video sequence.

[0194] In some embodiments, the candidate bundle with the highest cumulative score When calculating the cumulative score using method 1 above, δ(o) i ) refers to the probability score of the video at each candidate video position; when calculating the cumulative score using method 2 above, δ(o) i () refers to the combined score of videos at each candidate video position.

[0195] The above method can determine the final ranking result by selecting the optimal candidate bundle (i.e. the candidate bundle with the highest cumulative score) from multiple candidate bundles.

[0196] Furthermore, this application proposes two flexible methods for determining the cumulative score. Specifically, Method 1 uses the probability of N videos corresponding to each candidate video position to determine the video sequence. Method 2 not only considers the probability of N videos corresponding to each candidate video position, but also the coherence score of video pairs corresponding to each candidate video position and N videos, making the ranking results of adjacent videos more accurate, thereby further improving the accuracy of the ranking results of the video sequence.

[0197] Figure 6 shows a flowchart of another video ranking method based on a video ranking model, provided in this embodiment of the application. Assuming that videos 1, 2, and 3 are to be ranked, firstly, a video encoder obtains the video features F of each of videos 1, 2, and 3 based on the video frame set and text content. Then, a set context encoder obtains the video context features of each of videos 1, 2, and 3 based on their respective video features F. For the first candidate video position, a position decoder obtains the probability of N videos corresponding to the first candidate video position, and a successor predictor obtains the coherence score of the video pair corresponding to the first candidate video position. The first candidate video position is jointly decoded using the probability and coherence score to output video 3. Similarly, the second and third candidate video positions are jointly decoded to output video 1 and video 2 respectively. Therefore, the ranking result of these three videos is video 3, video 1, and video 2.

[0198] In some embodiments, the method proposed in this application is compared with methods of related technologies. This application collected at least one video collection uploaded by over 500 users from platform A and compiled all these video collections, totaling 5,792 video collections. These collections cover a wide range of topics, including film reviews, news broadcasts, and short documentaries, and showcase various scenes such as cityscapes, natural environments, and historical contexts. The order of the videos was manually checked and adjusted based on the titles, covers, and content of the videos in each video collection. For collections with explicit ordering information in their titles or covers (e.g., "Episode 1, Episode 2, Episode 3" or "First, Second, Third"), the videos were reordered accordingly. For collections without explicit ordering information, the entire video was watched and reordered according to their narrative relationships. The order of the videos in each video collection was then randomly shuffled to avoid the influence of the original video order, thus enabling a fair experimental comparison.

[0199] The dataset comprises a total of 5,792 video sets, covering 23,591 videos. The average video length is 142 seconds, with an average of 591.91 characters transcribed and 14.03 sentences per video. This application defines sets containing 3 videos as short sets and sets containing more than 3 videos as long sets. The dataset contains 3,628 short sets and 2,164 long sets. The dataset is divided into three subsets: training, validation, and testing. The training set is used for model training, the validation set is used for fine-tuning hyperparameters and selecting the best-performing model, and the testing set is used to evaluate the final performance of the model. The dataset is divided so that 70% of the video sets are allocated to the training set, 10% to the validation set, and the remaining 20% ​​to the testing set. This distribution ensures a comprehensive evaluation of model performance and facilitates effective fine-tuning during training.

[0200] The table below provides a performance comparison of the proposed method with related technologies on various evaluation metrics. To comprehensively evaluate model performance, this application sets up test models on three test sets, and the experimental results are shown in Table 1 below:

[0201] Table 1

[0202] Here, Acc (Accuracy) refers to the accuracy rate, which is the ratio of the number of correctly ranked videos to the total number of videos. τ (Kendall's τ) is the Kendall correlation coefficient, a rank correlation coefficient that measures the similarity between two rankings. It is calculated by comparing the number of consistent pairs (pairs with the same relative order in two rankings) with the number of inconsistent pairs (pairs with different relative orders). PMR (Perfect Matching Ration) measures the proportion of video sequences perfectly ranked by the model in the test set, i.e., the proportion of video sets where the predicted ranking result is exactly the same as the actual ranking result. The relevant methods include Listwise-SL (Listwise Sequence Labeling), Listwise-PTN (Listwise Potential Topic Network), ATTorderNet-MM (deep attentive sentence ordering network for multi-modalities), and BERSON-MM (BERT-enhanced relational sentence ordering network for multi-modalities).

[0203] As shown in Table 1 above, the proposed method consistently outperforms the baseline method across all evaluation metrics and test set settings. In particular, the proposed method achieves the highest Acc (86.99%), τ (89.16%), and PMR (85.47%) scores on the test set, demonstrating its effectiveness in organizing video sets into coherent sequences. This superior performance can be attributed to the multimodal feature extraction, location decoding, and successor prediction components of the proposed method. Furthermore, the proposed method exhibits robust performance on both short and long sets, consistently achieving higher scores than the baseline method. For example, the proposed method achieves Acc (92.33%), τ (89.72%), and PMR (89.39%) scores on short sets and Acc (82.49%), τ (82.49%), and PMR (78.95%) on long sets. Notably, the proposed method significantly outperforms the baseline method on long sets compared to short sets. This demonstrates that the proposed method can effectively handle assemblies of varying sizes and complexities, making it a promising solution for video ranking tasks in real-world scenarios. In summary, the results presented in the table highlight the effectiveness of the proposed method in organizing video assemblies into coherent and meaningful sequences, outperforming baseline methods across all evaluation metrics and test set settings.

[0204] Please refer to Figure 7, which shows a flowchart of a video sorting method provided in one embodiment of this application. The execution entity for each step of this method can be a computer device, such as the model training device 210 in the implementation environment shown in Figure 2. The method may include at least one of the following steps 710 to 750.

[0205] Step 710: Obtain N sample videos, where N is an integer greater than 1.

[0206] In some embodiments, N sample videos and the actual sorting results of the N sample videos are obtained.

[0207] Step 720: Generate video features for each of the N sample videos using the video encoder, thus obtaining N sample video features.

[0208] In some embodiments, a set of video frames and text content for each of the N sample videos are obtained, wherein the text content is obtained based on at least one of the speech and text included in the sample video; for each of the N sample videos, a video encoder generates video features of the sample video based on the set of video frames and text content of the sample video.

[0209] In some embodiments, a visual encoder generates visual features of a sample video based on a set of video frames; a text encoder generates text features of a sample video based on the text content of the sample video; and a multimodal feature fusion network generates video features of the sample video based on both visual and text features.

[0210] In some embodiments, the visual encoder and text encoder are initialized based on the CLIP model. The CLIP model is pre-trained on a large-scale vision-language dataset to reduce the semantic gap between vision and language.

[0211] In some embodiments, visual features are used as key vectors and value vectors, and initial video features and text features are used as query vectors. A multimodal feature fusion network generates video features of the sample video based on the key vectors, value vectors, and query vectors. The initial video features are learnable parameters during the training process of the video encoder. The initial video features are used to learn the features of the video frames included in the video frame set and the features of the text content.

[0212] Step 730: Generate video context features between N sample videos based on the features of N sample videos using a set context encoder;

[0213] Each of the N sample videos corresponds to a video context feature. For each of the N sample videos, the context video feature is used to characterize the video content association between the sample video and other sample videos in the N sample videos.

[0214] For detailed instructions, please refer to the usage guide.

[0215] Step 740: Based on the video context features, the location decoder determines the probability that each of the N sample videos corresponds to one of the N candidate video locations.

[0216] In some embodiments, video features with added position embeddings for each of the N sample videos are generated based on video context features and the position embeddings of each of the N candidate video locations. The position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video locations. The probability that each of the N sample videos corresponds to one of the N candidate video locations is determined by the position decoder based on the video features with added position embeddings of each of the N sample videos.

[0217] In some embodiments, the probability that the N sample videos correspond to the first candidate video position is determined by the position decoder based on the video features of the added position embeddings of the N sample videos and the initial position embeddings; the probability that the N sample videos correspond to the i-th candidate video position is determined by the position decoder based on the video features of the added position embeddings of the N sample videos, the initial position embeddings, and the position embeddings of the first to (i-1)th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

[0218] In some embodiments, the video features with added location embeddings of each of the N sample videos are used as key vectors and value vectors, and the initial location embedding is used as query vector. The location decoder determines the probability that the N sample videos correspond to the first candidate video position based on the key input, value input, and query input.

[0219] In some embodiments, the video features with added position embeddings for each of the N sample videos are used as key vectors and value vectors, and the initial position embeddings and the position embeddings corresponding to the 1st to i-1th candidate video positions are used as query vectors. The position decoder determines the probability of the N sample videos corresponding to the i-th candidate video position based on the key input, value input and query input.

[0220] Step 750: With the goal of minimizing the probability error of N sample videos, adjust the parameters of the video ranking model to obtain the trained video ranking model.

[0221] In some embodiments, minimizing the error of the probability of N sample videos means optimizing the model parameters with the goal of maximizing the probability of the correct video at each candidate video position, given the probability of N videos corresponding to each candidate video position.

[0222] In some embodiments, the model parameters can be optimized by minimizing the negative log-likelihood of the probability of a correct video, as shown in Equation 7 below:

[0223] in, This is the location decoder loss value. The probability that the i-th candidate video position is the correct video.

[0224] In some embodiments, this application also proposes another method for adjusting the parameters of a video ranking model. The implementation method may include the following steps 1 to 3:

[0225] Step 1: Obtain video pairs for each of the N sample videos. For each of the N sample videos, the video pair consists of the sample video and any one of the N sample videos.

[0226] In some embodiments, based on the video context features of each of the N sample videos, for each of the N videos, a video pair for that video is obtained. The set of video context features for each of the N videos is then used. For each It can be matched with all other video context features (including itself) to form n pairs. If video j is in the correct order o * The subsequent videos of the video series "i" will be... A video pair is considered a positive video pair. Otherwise, it is considered a negative video pair. For example, as shown in subfigure 3 of Figure 4, for a target video video 1st, the candidate successor video video 2nd is the successor video of video 1st. Therefore, the video pair consisting of video 1st and video 2nd is a positive video pair, and the video pairs consisting of video 1st and video 3rd, as well as video 1st and video 1st, are all negative video pairs.

[0227] Step 2: The successor predictor generates a coherence score for the video pair based on the video context features of the video pair and the two sample videos included in the video pair.

[0228] In some embodiments, a coherence score for a video pair is generated based on the video context features of the two videos included in the video pair using a network ρ(·). In some embodiments, the network ρ(·) may be constructed based on an MLP.

[0229] Step 3: With the goal of minimizing the error in the coherence score of positive video pairs in video pairs, adjust the parameters of the video ranking model to obtain the trained video ranking model. A positive video pair refers to a video pair consisting of two adjacent sample videos in N sample videos.

[0230] A video pair consisting of two adjacent video samples in N sample videos refers to two semantically coherent adjacent video pairs in the actual video sequence. Minimizing the error in the coherence score of positive video pairs refers to adjusting the model parameters with the objective of maximizing the coherence score of positive video pairs and minimizing the coherence score of negative video pairs, as shown in Equation 8 below:

[0231] in, β represents the loss value of the subsequent predictor, and β is a temperature hyperparameter. This refers to the score for the continuity of the video pair. The coherence score between the video at the i-th candidate video position and the video at the j-th candidate video position.

[0232] In some embodiments, the overall optimization objective can incorporate the location decoder loss. and successor predictor loss By optimizing this combined loss using the backpropagation algorithm, the trained video ranking model can learn to reason about the optimal order of short videos, while also enhancing the content coherence between consecutive video pairs in the set.

[0233] The method described above effectively enhances the content coherence between consecutive videos in a collection by optimizing the model parameters of the successor predictor using the coherence scores of video pairs. The successor predictor enables the trained video ranking model to better understand and organize the content of the video collection by capturing the semantic relationships between any two videos, thereby generating more accurate ranking results.

[0234] Please refer to Figure 8, which illustrates a flowchart of a training method for a video ranking model provided in another embodiment of this application. First, a sample video set is constructed, comprising N sample videos. Then, video features are extracted from each of the N sample videos using a video encoder, resulting in N sample video features. A set context encoder extracts the video context features between the N sample videos. A position decoder, based on the video context features of each of the N sample videos, obtains the probabilities of the N sample videos corresponding to the N candidate video positions, and calculates the loss based on these probabilities. A successor predictor, based on the video pairs of the N sample videos, obtains the coherence score for each video pair, and calculates the loss based on the coherence score of the video pairs. Finally, the overall loss from the above two losses is calculated, and the network is optimized using backpropagation.

[0235] The technical solution provided in this application can extract video features from each sample video through a video encoder, and extract video context features between sample videos based on the video features through a set context encoder. Furthermore, it can obtain the probability of N candidate video positions corresponding to N videos based on the video context features through a position decoder. Since the video context features fully consider the video content correlation between multiple videos, the determined probabilities are more accurate, thereby making the sorting effect of the trained video ranking model more accurate.

[0236] The methods used on the video ranking model and the training side in the instruction manual are corresponding to each other. For details not explained on one side, please refer to the other side.

[0237] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0238] Please refer to Figure 9, which shows a block diagram of a video sorting device based on a video sorting model according to an embodiment of this application. This device has the function of implementing the aforementioned video sorting method based on a video sorting model. This function can be implemented in hardware or by hardware executing corresponding software. The device can be a computer device or can be installed within a computer device. The device 900 may include: an acquisition module 910, a first generation module 920, a second generation module 930, a determination module 940, and a obtaining module 950.

[0239] The acquisition module 910 is used to acquire N videos, where N is an integer greater than 1.

[0240] The first generation module 920 is used to generate video features for each of the N videos through the video encoder, thereby obtaining N video features.

[0241] The second generation module 930 is used to generate video context features between the N videos based on the N video features through the collection context encoder.

[0242] The determination module 940 is used to determine the probability that the N videos correspond to the N candidate video positions based on the video context features using the position decoder.

[0243] The module 950 is used to assign the N videos to the N candidate video positions according to the probability, thereby obtaining a video sequence.

[0244] In some embodiments, the determining module 940 is configured to generate video features with added position embeddings for each of the N videos based on the video context features of each of the N videos and the position embeddings of each of the N candidate video positions, wherein the position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video positions; and the position decoder determines the probability that each of the N videos corresponds to one of the N candidate video positions based on the video features with added position embeddings for each of the N videos.

[0245] In some embodiments, the determining module 940 is configured to determine the probability that the N videos correspond to the first candidate video position based on the video features of the added position embeddings of the N videos and the initial position embeddings, using the position decoder; and to determine the probability that the N videos correspond to the i-th candidate video position based on the video features of the added position embeddings of the N videos, the initial position embeddings, and the position embeddings of the first to (i-1)-th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

[0246] In some embodiments, the determining module 940 is used to take the video features of the added position embedding of each of the N videos as key vectors and value vectors, take the initial position embedding as query vectors, and determine the probability that the N videos correspond to the first candidate video position by the position decoder based on the key vectors, the value vectors and the query vectors.

[0247] In some embodiments, the determining module 940 is used to take the video features of the added position embedding of each of the N videos as key vectors and value vectors, take the initial position embedding and the position embedding of each of the 1st to i-1th candidate video positions as query vectors, and use the position decoder to determine the probability that the N videos correspond to the i-th candidate video position based on the key vector, the value vector and the query vector.

[0248] In some embodiments, the obtaining module 950 is used to assign the video with the highest probability at each of the N candidate video locations to the candidate video location to obtain the video sequence.

[0249] In some embodiments, the obtaining module 950 is used to obtain multiple candidate bundles and the cumulative score of each candidate bundle according to the probability, each candidate bundle corresponding to a ranking result; take the ranking result corresponding to the candidate bundle with the highest cumulative score as the ranking result of the N videos; and assign the N videos to the N candidate video positions according to the ranking result of the N videos to obtain the video sequence.

[0250] In some embodiments, the obtaining module 950 is configured to: when i equals 1, for the first candidate video position, determine the probability score of each of the N videos based on the probability of the N videos corresponding to the first candidate video position; select k candidate videos from the N videos based on the probability scores of the N videos, and generate a candidate bundle for the first candidate video position, where k is a positive integer less than or equal to N; when i is an integer from 2 to N, for the i-th candidate video position, determine the probability of the N videos corresponding to the i-th candidate video position; for each candidate bundle in the candidate bundles of the first i-1 candidate video positions... The candidate set is determined by: 1) determining the probability score of the video not included in the candidate set based on the probability of the N videos corresponding to the i-th candidate video position; 2) selecting k candidate videos based on the probability scores of the videos not included in the candidate set, and adding the k candidate videos to the candidate set respectively, updating the candidate sets for the first i-1 candidate video positions, and obtaining the candidate sets for the first i candidate video positions; 3) for each candidate set in the candidate sets for the first i candidate video positions, obtaining the cumulative score of the candidate set based on the probability scores of the i videos included in the candidate set.

[0251] In some embodiments, the video ranking model further includes a successor predictor; the obtaining module 950 is used to generate, through the successor predictor, a coherence score for the video pair corresponding to the i-th candidate video position in each of the N videos based on the video context features, wherein for each of the N videos, the video pair consists of the video and the video corresponding to the (i-1)-th candidate video position, where i is an integer greater than or equal to 2 and less than or equal to N; based on the probability and the coherence score for the video pair corresponding to the i-th candidate video position in each of the N videos, multiple candidate bundles and the cumulative score of each candidate bundle are obtained.

[0252] In some embodiments, the obtaining module 950 is configured to: when i equals 1, for the first candidate video position, determine the probability score of each of the N videos based on the probability of the N videos corresponding to the first candidate video position; select k candidate videos from the N videos based on the probability scores of the N videos, and generate a candidate bundle for the first candidate video position, where k is a positive integer less than or equal to N; for each candidate bundle in the candidate bundle for the first candidate video position, determine the probability score of one video included in the candidate bundle as the combined score of the first video; when i is an integer from 2 to N, for the first i-1 candidate video positions... For each candidate bundle in the candidate bundle, for each video among the N videos that is not included in the candidate bundle, the combined score of the video is determined based on the probability score of the video and the coherence score of the video pair. Based on the combined score, k candidate videos are selected from the N videos and added to the candidate bundle respectively. The candidate bundles at the first i-1 candidate video positions are updated to obtain the candidate bundles at the first i candidate video positions. For each candidate bundle in the candidate bundles at the first i candidate video positions, the cumulative score of the candidate bundle is obtained based on the combined scores of the i videos included in the candidate bundle.

[0253] In some embodiments, the first generation module 920 is used to obtain the video frame set and text content of each of the N videos, wherein the text content is obtained based on at least one of the audio and text included in the video; for each of the N videos, the video encoder generates video features of the video based on the video frame set and text content of the video.

[0254] In some embodiments, the video encoder includes a visual encoder, a text encoder, and a multimodal feature fusion network; the first generation module 920 is configured to generate visual features of the video based on the video frame set of the video using the visual encoder; generate text features of the video based on the text content of the video using the text encoder; and generate video features of the video based on the visual features and the text features using the multimodal feature fusion network.

[0255] In some embodiments, the first generation module 920 is used to take the visual features as key vectors and value vectors, and the initial video features and the text features as query vectors, and generate video features of the video through the multimodal feature fusion network based on the key vectors, the value vectors and the query vectors. The initial video features are learnable parameters in the training process of the video encoder, and the initial video features are used to learn the features of the video frames included in the video frame set and the features of the text content.

[0256] The method provided in this application embodiment, on the one hand, can extract video features of each video through a video encoder, and extract video context features of each video based on the video features through a set context encoder. The video context features fully consider the video content association between multiple videos, making the sorting result of the obtained video sequence more accurate. On the other hand, the position decoder can generate the probability of N videos corresponding to each candidate video position based on the video context features, that is, based on the learned video feature representation. This probability is used to guide the allocation of each video. This method of comprehensively considering the probability of N videos corresponding to N candidate video positions to determine the video sequence effectively improves the accuracy of the sorting result of the video sequence.

[0257] Please refer to Figure 10, which shows a block diagram of a training apparatus for a video ranking model according to an embodiment of this application. This apparatus has the function of implementing the training method for the video ranking model described above. This function can be implemented in hardware or by hardware executing corresponding software. The apparatus 1000 can be a computer device or can be installed within a computer device. The apparatus 1000 may include: an acquisition module 1010, a first generation module 1020, a second generation module 1030, a determination module 1040, and a first adjustment module 1050.

[0258] The acquisition module 1010 is used to acquire N sample videos, where N is an integer greater than 1.

[0259] The first generation module 1020 is used to generate video features for each of the N sample videos through the video encoder, thereby obtaining N sample video features.

[0260] The second generation module 1030 is used to generate video context features among the N sample videos based on the features of the N sample videos through the collection context encoder.

[0261] The determination module 1040 is used to determine the probability that the N sample videos correspond to the N candidate video positions based on the video context features using the position decoder.

[0262] The first adjustment module 1050 is used to adjust the parameters of the video ranking model with the goal of minimizing the error of the probability of the N sample videos, so as to obtain the trained video ranking model.

[0263] In some embodiments, the determining module 1040 is configured to generate video features with added position embeddings for each of the N sample videos based on the video context features and the position embeddings of each of the N candidate video locations, wherein the position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video locations; and the position decoder determines the probability that each of the N sample videos corresponds to one of the N candidate video locations based on the video features with added position embeddings of each of the N sample videos.

[0264] In some embodiments, the determining module 1040 is configured to determine the probability that the N sample videos correspond to the first candidate video position by means of the position decoder based on the video features of the added position embeddings of the N sample videos and the initial position embeddings; and to determine the probability that the N sample videos correspond to the i-th candidate video position by means of the position decoder based on the video features of the added position embeddings of the N sample videos, the initial position embeddings, and the position embeddings of the first to (i-1)-th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

[0265] In some embodiments, the video ranking model further includes a successor predictor, and the device 1000 further includes a second adjustment module (not shown in FIG10).

[0266] The second adjustment module is used to obtain video pairs for each of the N sample videos. For each of the N sample videos, the video pair consists of the sample video and any one of the N sample videos. The successor predictor generates a coherence score for the video pair based on the video context features of the video pair and the two sample videos included in the video pair. The parameters of the video ranking model are adjusted with the goal of minimizing the error of the coherence score of the positive video pairs in the video pair, to obtain the trained video ranking model. The positive video pair refers to a video pair consisting of two adjacent sample videos from the N sample videos.

[0267] In some embodiments, the first generation module 1020 is used to obtain the video frame set and text content of each of the N sample videos, wherein the text content is obtained based on at least one of the speech and text included in the sample video; for each of the N sample videos, the video encoder generates video features of the sample video based on the video frame set and text content of the sample video.

[0268] In some embodiments, the video encoder includes a visual encoder, a text encoder, and a multimodal feature fusion network; the first generation module 1020 is configured to generate visual features of the sample video based on the set of video frames of the sample video using the visual encoder; generate text features of the sample video based on the text content of the sample video using the text encoder; and generate video features of the sample video based on the visual features and the text features using the multimodal feature fusion network.

[0269] In some embodiments, the first generation module 1020 is used to take the visual features as key vectors and value vectors, and the initial video features and the text features as query vectors, and generate video features of the sample video through the multimodal feature fusion network based on the key vectors, the value vectors and the query vectors. The initial video features are learnable parameters in the training process of the video encoder, and the initial video features are used to learn the features of the video frames included in the video frame set and the features of the text content.

[0270] The technical solution provided in this application can extract video features from each sample video through a video encoder, and extract video context features from each sample video based on the video features through a set context encoder. Furthermore, it can obtain the probability of N sample videos corresponding to each candidate video position based on the video context features through a position decoder. Because the video context features fully consider the video content correlation between multiple sample videos, the generated probabilities are more accurate, thereby making the sorting effect of the trained video ranking model more accurate.

[0271] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0272] Please refer to Figure 11, which shows a structural block diagram of a computer device 1100 provided in one embodiment of this application.

[0273] Typically, computer device 1100 includes a processor 1110 and a memory 1120.

[0274] Processor 1110 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1110 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1110 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1110 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1110 may also include an AI processor for handling computational operations related to machine learning.

[0275] The memory 1120 may include one or more computer-readable storage media, which may be non-transitory. The memory 1120 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1120 is used to store a computer program configured to be executed by one or more processors to implement the video ranking method based on the video ranking model described above, or the training method for the video ranking model described above.

[0276] Those skilled in the art will understand that the structure shown in FIG11 does not constitute a limitation on the computer device 1100, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0277] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the video ranking method based on the video ranking model described above, or the training method of the video ranking model described above.

[0278] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0279] In some embodiments, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium, the processor reading from the computer-readable storage medium and executing the computer program to implement the above-described video ranking method based on the video ranking model, or the above-described video ranking model training method.

[0280] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0281] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video ranking method based on a video ranking model, characterized in that, The video ranking model includes a video encoder, a collection context encoder, and a position decoder; the method includes: Get N videos, where N is an integer greater than 1; The video encoder generates video features for each of the N videos, resulting in N video features; The collection context encoder generates video context features among the N videos based on the N video features. The location decoder determines the probability that each of the N videos corresponds to one of the N candidate video locations based on the video context features. Based on the probability, the N videos are assigned to the N candidate video positions to obtain a video sequence.

2. The method according to claim 1, characterized in that, The step of determining the probability that each of the N videos corresponds to one of the N candidate video positions based on the video context features using the location decoder includes: Based on the video context features and the position embeddings of each of the N candidate video locations, video features with added position embeddings are generated for each of the N videos. The position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video locations. The location decoder determines the probability that each of the N videos corresponds to one of the N candidate video locations based on the video features embedded at the added locations of the N videos.

3. The method according to claim 2, characterized in that, The step of determining the probability that each of the N videos corresponds to one of the N candidate video positions using the location decoder based on the video features embedded at the respective added positions of the N videos includes: The location decoder determines the probability that each of the N videos corresponds to the position of the first candidate video by using the video features of the added location embedding for each of the N videos, as well as the initial location embedding. The location decoder determines the probability that the N videos correspond to the i-th candidate video position based on the video features of the added location embeddings of the N videos, the initial location embedding, and the location embeddings of the 1st to i-1th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

4. The method according to claim 3, characterized in that, The step of determining the probability that the N videos correspond to the position of the first candidate video by means of the position decoder based on the video features of the N videos with added position embeddings and the initial position embeddings includes: The video features embedded at the added positions of the N videos are used as key vectors and value vectors, and the initial position embedding is used as a query vector. The position decoder determines the probability that the N videos correspond to the position of the first candidate video based on the key vector, the value vector, and the query vector.

5. The method according to claim 3, characterized in that, The step of determining the probability that the N videos correspond to the i-th candidate video position using the position decoder based on the video features of the N videos with added position embeddings, the initial position embedding, and the position embeddings of the 1st to i-1th candidate video positions includes: The video features embedded at the added positions of the N videos are used as key vectors and value vectors. The initial position embedding and the position embeddings of the 1st to (i-1th)th candidate video positions are used as query vectors. The position decoder determines the probability that the N videos correspond to the i-th candidate video position based on the key vector, the value vector, and the query vector.

6. The method according to any one of claims 1-5, characterized in that, The step of assigning the N videos to the N candidate video positions according to the probability to obtain a video sequence includes: For each of the N candidate video positions, the video with the highest probability at that candidate video position is assigned to that candidate video position to obtain the video sequence.

7. The method according to any one of claims 1-5, characterized in that, The step of assigning the N videos to the N candidate video positions according to the probability to obtain a video sequence includes: Based on the probability, multiple candidate bundles and the cumulative score of each candidate bundle are obtained, with each candidate bundle corresponding to a ranking result; The ranking result corresponding to the candidate bundle with the highest cumulative score is used as the ranking result of the N videos; Based on the sorting results of the N videos, the N videos are assigned to the N candidate video positions to obtain the video sequence.

8. The method according to claim 7, characterized in that, The step of obtaining multiple candidate bundles and the cumulative score of each candidate bundle based on the probability includes: When i equals 1, for the first candidate video position, the probability score of each of the N videos is determined based on the probability of the N videos corresponding to the first candidate video position; Based on the probability scores of the N videos, k candidate videos are selected from the N videos, and a candidate bundle is generated for the position of the first candidate video, where k is a positive integer less than or equal to N. When i is an integer from 2 to N, for the i-th candidate video position, determine the probability that the N videos correspond to the i-th candidate video position; For each candidate bundle in the candidate bundles of the first i-1 candidate video positions, determine the probability score of the video that is not included in the candidate bundle among the N videos based on the probability of the N videos corresponding to the i-th candidate video position. Based on the probability scores of the videos that are not included in the candidate bundle among the N videos, select k candidate videos and add the k candidate videos to the candidate bundle respectively. Update the candidate bundles at the positions of the first i-1 candidate videos to obtain the candidate bundles at the positions of the first i candidate videos. For each candidate bundle in the candidate bundles of the first i candidate video positions, the cumulative score of the candidate bundle is obtained based on the probability scores of the i videos included in the candidate bundle.

9. The method according to claim 7, characterized in that, The video ranking model also includes a successor predictor; The step of obtaining multiple candidate bundles and the cumulative score of each candidate bundle based on the probability includes: The successor predictor generates a coherence score for the video pair corresponding to the i-th candidate video position in each of the N videos based on the video context features. For each of the N videos, the video pair consists of the video and the video corresponding to the (i-1)-th candidate video position, where i is an integer greater than or equal to 2 and less than or equal to N. Based on the probability and the coherence score of the video pairs corresponding to the i-th candidate video position in each of the N videos, multiple candidate bundles and the cumulative score of each candidate bundle are obtained.

10. The method according to claim 9, characterized in that, The process of obtaining multiple candidate bundles and the cumulative score of each candidate bundle based on the probability and the coherence score of the video pairs corresponding to the i-th candidate video position in the N videos includes: When i equals 1, for the first candidate video position, the probability score of each of the N videos is determined based on the probability of the N videos corresponding to the first candidate video position; Based on the probability scores of the N videos, k candidate videos are selected from the N videos, and a candidate bundle is generated for the position of the first candidate video, where k is a positive integer less than or equal to N. For each candidate bundle in the candidate bundle of the first candidate video location, the probability score of one video included in the candidate bundle is determined as the combined score of the first video. When i is an integer from 2 to N, for each candidate bundle in the candidate bundles of the first i-1 candidate video positions, for each video in the N videos that is not included in the candidate bundle, the combined score of the video is determined based on the probability score of the video and the coherence score of the video pair. Based on the combined score, k candidate videos are selected from the N videos, and the k candidate videos are added to the candidate bundle respectively. The candidate bundles at the positions of the first i-1 candidate videos are updated to obtain the candidate bundles at the positions of the first i candidate videos. For each candidate bundle in the candidate bundles of the first i candidate video positions, the cumulative score of the candidate bundle is obtained based on the combined scores of the i videos included in the candidate bundle.

11. The method according to any one of claims 1-10, characterized in that, The process of generating video features for each of the N videos using the video encoder to obtain N video features includes: Obtain the video frame set and text content of each of the N videos, wherein the text content is obtained based on at least one of the audio and text included in the video; For each of the N videos, the video encoder generates video features for the video based on the video frame set and text content of the video.

12. The method according to claim 11, characterized in that, The video encoder includes a visual encoder, a text encoder, and a multimodal feature fusion network; The step of generating video features of the video based on the video frame set and text content using the video encoder includes: The visual encoder generates visual features of the video based on the set of video frames. The text encoder generates text features of the video based on the text content of the video; The video features of the video are generated by the multimodal feature fusion network based on the visual features and the text features.

13. The method according to claim 12, characterized in that, The step of generating video features for the video based on the visual features and the text features through the multimodal feature fusion network includes: The visual features are used as key vectors and value vectors, and the initial video features and the text features are used as query vectors. The multimodal feature fusion network generates video features of the video based on the key vectors, the value vectors, and the query vectors. The initial video features are learnable parameters in the training process of the video encoder. The initial video features are used to learn the features of the video frames included in the video frame set and the features of the text content.

14. A training method for a video ranking model, characterized in that, The video ranking model includes a video encoder, a collection context encoder, and a position decoder; the method includes: Obtain N sample videos, where N is an integer greater than 1; The video encoder generates video features for each of the N sample videos, resulting in N sample video features; The collection context encoder generates video context features among the N sample videos based on the features of the N sample videos. The location decoder determines the probability that each of the N sample videos corresponds to one of the N candidate video locations based on the video context features. With the goal of minimizing the probability error of the N sample videos, the parameters of the video ranking model are adjusted to obtain the trained video ranking model.

15. The method according to claim 14, characterized in that, The step of determining the probability that the N sample videos correspond to the N candidate video positions respectively by the location decoder based on the video context features includes: Based on the video context features and the position embeddings of the N candidate video locations, video features with added position embeddings are generated for each of the N sample videos. The position embeddings are learnable parameters during the training process of the position decoder and are used to characterize the candidate video locations. The location decoder determines the probability that each of the N sample videos corresponds to one of the N candidate video locations based on the video features embedded at the added locations of the N sample videos.

16. The method according to claim 15, characterized in that, The step of determining the probability that each of the N sample videos corresponds to one of the N candidate video positions using the location decoder based on the video features embedded at the respective added positions of the N sample videos includes: The location decoder determines the probability that each of the N sample videos corresponds to the first candidate video location based on the video features of the added location embeddings of each of the N sample videos and the initial location embeddings. The location decoder determines the probability that the N sample videos correspond to the i-th candidate video position based on the video features of the added location embeddings of the N sample videos, the initial location embedding, and the location embeddings of the 1st to i-1th candidate video positions, where i is an integer greater than or equal to 2 and less than or equal to N.

17. The method according to any one of claims 14-16, characterized in that, The video ranking model further includes a successor predictor, and the method further includes: Obtain video pairs for each of the N sample videos. For each of the N sample videos, the video pair consists of the sample video and any one of the N sample videos. The successor predictor generates a coherence score for the video pair based on the video context features of the video pair and the two sample videos included in the video pair. With the goal of minimizing the error in the coherence score of positive video pairs in the video pair, the parameters of the video ranking model are adjusted to obtain the trained video ranking model. The positive video pair refers to a video pair consisting of two adjacent sample videos in the N sample videos.

18. A video sorting device based on a video sorting model, characterized in that, The video ranking model includes a video encoder, a set context encoder, and a position decoder; the device includes: The acquisition module is used to acquire N videos, where N is an integer greater than 1; The first generation module is used to generate video features for each of the N videos through the video encoder, thereby obtaining N video features; The second generation module is used to generate video context features between the N videos based on the N video features using the collection context encoder; The determination module is used to determine the probability that the N videos correspond to the N candidate video positions based on the video context features using the position decoder; The module is used to assign the N videos to the N candidate video positions according to the probability, thereby obtaining a video sequence.

19. A training device for a video ranking model, characterized in that, The video ranking model includes a video encoder, a set context encoder, and a position decoder; the device includes: The acquisition module is used to acquire N sample videos, where N is an integer greater than 1; The first generation module is used to generate video features for each of the N sample videos through the video encoder, thereby obtaining N sample video features; The second generation module is used to generate video context features between the N sample videos based on the features of the N sample videos through the collection context encoder; The determination module is used to determine the probability that the N sample videos correspond to the N candidate video positions based on the video context features using the position decoder; The first adjustment module is used to adjust the parameters of the video ranking model with the goal of minimizing the error of the probability of the N sample videos, so as to obtain the trained video ranking model.

20. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 13, or the method as claimed in any one of claims 14 to 17.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method as claimed in any one of claims 1 to 13, or the method as claimed in any one of claims 14 to 17.

22. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as claimed in any one of claims 1 to 13, or the method as claimed in any one of claims 14 to 17.

Images