Video processing model training method and video processing method

Abstract

This application relates to a video processing model training method and a video processing method. The video processing model training method includes: acquiring training samples comprising sample videos, sample descriptive text, and time segment labels; the sample descriptive text describing a portion of the video content marked by time segment labels; using the model to be trained, predicting the content saliency scores of multiple sample segments based on their respective segment features obtained from the sample videos using the sample descriptive text; determining salient segments among the multiple sample segments based on the content saliency scores; performing video content retrieval based on the salient segments and the segment features of the multiple sample segments to obtain sample predicted time segments for a portion of the video content; and updating the model to be trained based on the difference between the sample predicted time segments and the time segment labels, as well as the content saliency scores of the multiple sample segments. This method can improve the accuracy of video content retrieval.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video processing model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product, as well as a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, video content retrieval technology has emerged. Video content retrieval refers to accurately locating a specific time segment within a video using natural language queries provided by the user.

[0003] In traditional technology, video content retrieval involves obtaining the search text for retrieving video content, splitting the video into multiple video segments, extracting the segment features of each video segment, extracting text features from the search text, comparing the similarity between the segment features and text features of the multiple video segments, and retrieving the video segments that match the search text.

[0004] However, traditional methods, which rely solely on similarity for retrieval, suffer from low accuracy in video content retrieval. Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical problems by providing a video processing model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of video content retrieval, as well as a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of video content retrieval.

[0006] Firstly, this application provides a method for training a video processing model, including:

[0007] Obtain training samples, which include sample videos, sample description text, and time period labels. The sample description text describes a portion of the video content marked by the time period labels in the sample video.

[0008] Using the model to be trained, the segment features of each of the multiple sample segments are determined based on the sample description text and the video features of multiple sample segments divided from the sample video.

[0009] Using the model to be trained, based on the segment features of each of the multiple sample segments, the content saliency score of each of the multiple sample segments is predicted, and based on the content saliency score, the salient segments among the multiple sample segments are determined.

[0010] Using the model to be trained, video content retrieval is performed based on the salient segments of the samples and the segment features of each of the multiple sample segments to obtain the sample prediction time period of the partial video content;

[0011] The video retrieval loss value is determined based on the difference between the predicted time period of the sample and the time period label, and the significance loss value is determined based on the content significance score of each of the multiple sample segments.

[0012] The training model is updated based on the video retrieval loss value and the saliency loss value until the training model is completed.

[0013] Secondly, this application also provides a video processing model training device, comprising:

[0014] The sample acquisition module is used to acquire training samples, which include sample videos, sample description text, and time period labels. The sample description text describes a portion of the video content marked by the time period labels in the sample video.

[0015] The sample feature extraction module is used to determine the segment features of each of the multiple sample segments based on the sample description text and the video features of multiple sample segments divided from the sample video using the model to be trained.

[0016] The saliency prediction module is used to predict the content saliency score of each of the multiple sample segments based on the segment features of each of the multiple sample segments using the model to be trained, and to determine the salient sample segments among the multiple sample segments based on the content saliency score.

[0017] The content retrieval and prediction module is used to retrieve video content based on the salient segments of the samples and the segment features of each of the multiple sample segments using the model to be trained, and to obtain the sample prediction time period of the part of the video content;

[0018] The loss calculation module is used to determine the video retrieval loss value based on the difference between the predicted time period of the sample and the time period label, and to determine the saliency loss value based on the content saliency scores of each of the multiple sample segments.

[0019] The model update module is used to update the model to be trained based on the video retrieval loss value and the saliency loss value until the model to be trained is fully trained.

[0020] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described video processing method.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described video processing method.

[0022] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described video processing method.

[0023] The aforementioned video processing model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product, based on the acquisition of training samples, determine the segment features of multiple sample segments according to the sample description text and video features of multiple sample segments divided from the sample video using the model to be trained. This allows for the acquisition of segment features closely related to the sample description text. By predicting the content saliency scores of multiple sample segments based on their respective segment features, the saliency of multiple sample segments can be determined. Thus, the salient segments among the multiple sample segments can be identified using the content saliency scores. Furthermore, guided by prior preferences, video content retrieval can be performed based on the salient segments and the segment features of the multiple sample segments, obtaining the predicted time periods of some video content marked by time period tags. Finally, the model to be trained can be updated using the video retrieval loss value determined by the difference between the predicted time periods and the time period tags, and the saliency loss value determined by the content saliency scores of the multiple sample segments, until the model is fully trained, resulting in a video processing model that can be used to accurately retrieve video content. The entire process, guided by salient fragments obtained from content saliency detection as prior preferences, helps to accurately locate key events in sample videos and enhances the semantic understanding of events. At the same time, it fully leverages the synergistic advantages between content saliency detection and video content retrieval, effectively improving the accuracy of video content location. This results in a video processing model that can be used to accurately retrieve video content, thereby improving the accuracy of video content retrieval.

[0024] Sixthly, this application provides a video processing method, including:

[0025] Obtain the video to be processed and the search text used to retrieve video content from the video to be processed;

[0026] Based on the retrieved text and the video features of multiple video segments divided from the video to be processed, the segment features of each of the multiple video segments are determined;

[0027] Based on the segment features of each of the multiple video segments, predict the target salience score of each of the multiple video segments, and determine the salient video segments among the multiple video segments based on the target salience scores;

[0028] Based on the significant video segments and the segment features of each of the multiple video segments, video content retrieval is performed to obtain the predicted video time period of the video content.

[0029] Seventhly, this application also provides a video processing apparatus, comprising:

[0030] The data acquisition module is used to acquire the video to be processed and the search text used to retrieve video content from the video to be processed;

[0031] The feature extraction module is used to determine the segment features of each of the multiple video segments based on the retrieved text and the video features of multiple video segments divided from the video to be processed;

[0032] The saliency detection module is used to predict the target saliency score of each of the multiple video segments based on their respective segment features, and to determine the salient video segments among the multiple video segments based on the target saliency scores.

[0033] The video content retrieval module is used to retrieve video content based on the significant video segments and the segment features of each of the multiple video segments, and to obtain the predicted video time period of the video content.

[0034] Eighthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described video processing method.

[0035] Ninthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described video processing method.

[0036] In a tenth aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described video processing method.

[0037] The aforementioned video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, based on acquiring the video to be processed and the search text, determine the segment features of each of the multiple video segments according to the search text and the video features of multiple video segments divided from the video to be processed. This allows for the acquisition of segment features closely related to the search text. By predicting the target saliency scores of each of the multiple video segments based on their respective segment features, the saliency of the multiple video segments can be determined. Therefore, the salient video segments among the multiple video segments can be identified using the target saliency scores. Furthermore, using the salient video segments as prior preference guidance, video content retrieval can be performed based on the salient video segments and the segment features of the multiple video segments to obtain the predicted video segment of the video content. Throughout this process, using the salient video segments obtained from content saliency detection as prior preference guidance, it helps to accurately locate key events in the video to be processed and enhances the semantic understanding of the events. Simultaneously, it fully utilizes the synergistic advantages between content saliency detection and video content retrieval, effectively improving the accuracy of video content location and thus enhancing the accuracy of video content retrieval. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the application of a video processing model training method in one embodiment.

[0040] Figure 2 This is a flowchart illustrating a video processing model training method in one embodiment;

[0041] Figure 3 This is a schematic diagram of a time period label in one embodiment;

[0042] Figure 4 This is a schematic diagram illustrating the learning of the first saliency marker feature in one embodiment;

[0043] Figure 5 This is a schematic diagram of content saliency detection in one embodiment;

[0044] Figure 6 This is a flowchart illustrating the video processing model training method in another embodiment;

[0045] Figure 7This is a schematic diagram of the structure of the model to be trained in one embodiment;

[0046] Figure 8 This is an application flowchart of a video processing method in one embodiment;

[0047] Figure 9 This is an application flowchart of the video processing method in another embodiment;

[0048] Figure 10 This is a schematic diagram illustrating the process of obtaining a predicted video segment using a video processing model in one embodiment;

[0049] Figure 11 This is a structural block diagram of a video processing model training device in one embodiment;

[0050] Figure 12 This is a structural block diagram of a video processing device in one embodiment;

[0051] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] The video processing model training method provided in this application embodiment, and its application flowchart on a terminal, can be shown as follows: Figure 1 As shown in the diagram, terminal 102 acquires training samples, which include sample videos, sample description text, and time segment tags. The sample description text describes a portion of the video content marked by the time segment tags in the sample video. Using the model to be trained, based on the sample description text and the video features of multiple sample segments divided from the sample video, the segment features of each of the multiple sample segments are determined. Using the model to be trained, based on the segment features of each of the multiple sample segments, the content saliency score of each of the multiple sample segments is predicted. Based on the content saliency score, significant segments among the multiple sample segments are determined. Using the model to be trained, video content retrieval is performed based on the significant segments and the segment features of each of the multiple sample segments to obtain the predicted time segments of some video content. The video retrieval loss value is determined based on the difference between the predicted time segments and the time segment tags, and the saliency loss value is determined based on the content saliency score of each of the multiple sample segments. The model to be trained is updated based on the video retrieval loss value and the saliency loss value until the model to be trained is completed.

[0054] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0055] In one exemplary embodiment, such as Figure 2 As shown, a video processing model training method is provided. This method can be applied to terminals, servers, or systems including both terminals and servers, and is implemented through interaction between the terminal and the server. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. This method is applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps 202 to 212. Wherein:

[0056] Step 202: Obtain training samples. The training samples include sample videos, sample description text, and time period labels. The sample description text describes the portion of the video content marked by the time period labels in the sample videos.

[0057] Training samples refer to the samples used to train the model, including sample videos, sample descriptive text, and time-segment labels. Sample videos are videos used during model training. For example, sample videos can specifically refer to film / TV clips, news clips, documentaries, game videos, and animated videos used in model training. Sample descriptive text is natural language text used to describe the portion of video content marked with time-segment labels within the sample videos. For example, sample descriptive text can specifically refer to Chinese text, English text, etc., used to describe the portion of video content marked with time-segment labels within the sample videos.

[0058] Time segment labels, in this context, describe the time period within a sample video segment. They are primarily used to mark specific segments of video content within the sample video. For example, a time segment label might be in the form of [t1, t2], where t1 is the start time of the time segment within the sample video segment, and t2 is the end time of that time segment. Another example is a time segment label in the form of [c, m], where c is the midpoint of the time segment within the sample video segment, and m is the length of that time segment. For instance... Figure 3 As shown, if a portion of the video content is in the sample video for a period of 4 to 8 seconds and the time period label is in the form of [t1, t2], then the time period label can be [4 8]. If the time period label is in the form of [c, m], then the time period label can be [6 4].

[0059] For example, when the model to be trained needs to be trained, the terminal will obtain training samples, which include sample videos, sample description text, and time period labels. The sample description text describes the portion of the video content marked by the time period labels in the sample video.

[0060] Step 204: Using the model to be trained, determine the segment features of each of the multiple sample segments based on the sample description text and the video features of the multiple sample segments divided from the sample video.

[0061] A sample segment refers to a short portion of a sample video. For example, a sample segment can be a short portion of a sample video defined by a preset segment length. The preset segment length can be configured according to the actual application scenario. For example, the preset segment length could be 2 seconds. The video features of a sample segment refer to the features obtained through video feature extraction. The segment features of a sample segment are video features closely related to the sample description text. They are closely related to the sample description text and can represent certain aspects of the sample segment, aiding in the analysis and recognition of video content.

[0062] For example, the terminal first divides the sample video into multiple sample segments, then extracts video features from each of the multiple sample segments to obtain the video features of each of the multiple sample segments, and extracts text features from the sample description text. Then, through the model to be trained, based on the text features extracted from the sample description text and the video features of each of the multiple sample segments, the segment features of each of the multiple sample segments are obtained.

[0063] Step 206: Using the model to be trained, predict the content saliency score of each of the multiple sample segments based on their respective segment features, and determine the salient segments among the multiple sample segments based on the content saliency score.

[0064] The content saliency score, in video content analysis, is a numerical value used to quantify the saliency of a sample segment. This score reflects the importance of the sample segment within the overall visual scene (specifically, the sample video in this embodiment), and can be obtained through content saliency detection. For example, the content saliency score can be a value between 0 and 1. The closer the content saliency score is to 1, the more important the sample segment is; the closer the content saliency score is to 0, the less important the sample segment is.

[0065] In this embodiment, content saliency detection is mainly used to calculate the relevance score between sample segments and sample description text by utilizing the segment features of sample segments and the first saliency marker features learned based on multiple sample segments and sample description text. This allows for easy access to important content in the sample video, namely the most attractive, exciting, or important segments. A salient segment refers to an important segment in the sample video, i.e., the most attractive, exciting, or important segment.

[0066] Among these, the relevance score is the content salience score. The most attractive, exciting, or important segments can also be called highlights, key segments, or key content. Therefore, content salience detection can also be called highlight content detection or key content detection.

[0067] For example, the terminal will use the model to be trained to perform content saliency detection on multiple sample segments based on their respective segment features, predict the content saliency scores of each of the multiple sample segments, rank the multiple sample segments based on their respective content saliency scores, and determine the salient segments among the multiple sample segments based on the ranking results.

[0068] In specific applications, when performing content saliency detection, the terminal can first determine the saliency features of each of the multiple sample segments based on the segment features of each of the multiple sample segments through the model to be trained. Then, it can use the saliency features of each of the multiple sample segments, as well as the first saliency marker features learned based on the multiple sample segments and sample description text, to predict the content saliency score of each of the multiple sample segments.

[0069] In a specific application, a terminal can use a learnable linear layer in the model to determine the salient features of multiple sample segments based on their individual segment characteristics. These determined salient features are then represented as a vector of a specific dimension output by the learnable linear layer. This specific dimension can be configured according to the actual application scenario when configuring the model to be trained. For example, the specific dimension can be 1.

[0070] Step 208: Using the model to be trained, video content retrieval is performed based on the salient segments of the samples and the segment features of each of the multiple sample segments to obtain the sample prediction time periods of some video content.

[0071] Video content retrieval, also known as cross-modal segment retrieval, refers to accurately locating a specific time segment within a video using a natural language query provided by the user. In this embodiment, the time segment within a portion of video content in the sample video is accurately located using the sample description text in the training samples; the located time segment is the sample prediction time segment.

[0072] For example, after determining the salient segments of the samples, the terminal will extract the positional features of the salient segments of the samples through the model to be trained, and then fuse the segment features of each of the multiple sample segments, the segment features of the salient segments of the samples, and the positional features to obtain the target features of each of the multiple sample segments. Based on the target features of each of the multiple sample segments, the terminal will perform time period prediction to obtain the sample prediction time period of part of the video content.

[0073] Step 210: Determine the video retrieval loss value based on the difference between the sample prediction time period and the time period label, and determine the significance loss value based on the content significance scores of each of the multiple sample segments.

[0074] The video retrieval loss value refers to the loss value generated in the video content retrieval task. It is used to measure the difference between the prediction of the model under training and the true label, and guides the training of the model to optimize the performance of video content retrieval. The saliency loss value refers to the loss value generated in the content saliency detection task. It is used to measure the difference between the prediction of the model under training and the true saliency, and guides the training of the model to optimize the performance of content saliency detection.

[0075] For example, the terminal determines the video retrieval time period loss value based on the difference between the sample prediction time period and the time period label, then determines the video retrieval loss value based on the video retrieval time period loss value, and determines the saliency category loss value based on the content saliency score and saliency label of each of the multiple sample segments, and then determines the saliency loss value based on the saliency category loss value.

[0076] In practical applications, the terminal can use a pre-set time-period prediction loss function to calculate the difference between the predicted time period and the time period label, thereby determining the time period loss value for video retrieval. The time-period prediction loss function can be configured according to the actual application scenario. For example, the time-period prediction loss function can be at least one of the following: a smoothed L1 loss function or a GIOU (Generalized Intersection over Union) loss function.

[0077] The smoothed L1 loss function combines the characteristics of both L1 and L2 loss functions. When the absolute value of the difference between the predicted and true values ​​is less than 1, the loss function behaves as an L2 loss function; when the absolute value of the difference between the predicted and true values ​​is greater than or equal to 1, the loss function behaves as an L1 loss function. In this embodiment, the predicted value is the sample prediction time period, and the true value is the time period label.

[0078] The GIOU (Generalized Intersection over Union) loss function is a loss function used for bounding box regression in object detection tasks. It extends the traditional IOU (Intersection over Union) loss function to address the problem that the IOU loss function cannot provide an effective gradient when two bounding boxes do not intersect, and can better reflect the relative positional relationship between the two bounding boxes. In this embodiment, the two bounding boxes are the sample prediction time period and the time period label.

[0079] In practical applications, after determining the video retrieval time loss value, the terminal will also obtain the sample segment categories of multiple sample segments predicted by video content retrieval, so as to determine the video retrieval classification loss value based on the sample segment categories of multiple sample segments. The video retrieval time period loss value and the video retrieval classification loss value are combined to determine the video retrieval loss value.

[0080] Step 212: Update the training model based on the video retrieval loss value and the saliency loss value until the training model is completed.

[0081] For example, after determining the video retrieval loss value and the saliency loss value, the terminal will update the model parameters related to the video content retrieval task in the training model according to the video retrieval loss value, and update the model parameters related to the content saliency detection task in the training model according to the saliency loss value, until the training model is completed.

[0082] In practical applications, after the model to be trained is updated, the terminal will continue to acquire training samples, train the model, and determine whether the model to be trained meets the stop training condition during the training process. When the model to be trained meets the stop training condition, the trained model is obtained, which can be used as a video processing model to accurately retrieve video content.

[0083] The training stop condition can be configured according to the actual application scenario. For example, the training stop condition can be that both the video retrieval loss value and the saliency loss value determined during the training process are less than the loss value threshold, or that both the video retrieval loss value and the saliency loss value have converged.

[0084] The aforementioned video processing model training method, based on the acquired training samples, uses the model to be trained to determine the segment features of multiple sample segments according to the sample description text and video features of multiple sample segments divided from the sample video. This allows for the acquisition of segment features closely related to the sample description text. By predicting the content saliency scores of multiple sample segments based on their respective segment features, the saliency of multiple sample segments can be determined. Therefore, the salient segments among the multiple sample segments can be identified using the content saliency scores. Furthermore, guided by prior preferences, video content retrieval can be performed based on the salient segments and the segment features of the multiple sample segments, obtaining the predicted time periods of some video content marked by time period tags. The model to be trained can then be updated using the video retrieval loss value determined by the difference between the predicted time periods and the time period tags, and the saliency loss value determined by the content saliency scores of the multiple sample segments, until the model is fully trained, resulting in a video processing model that can be used to accurately retrieve video content. The entire process, guided by salient fragments obtained from content saliency detection as prior preferences, helps to accurately locate key events in sample videos and enhances the semantic understanding of events. At the same time, it fully leverages the synergistic advantages between content saliency detection and video content retrieval, effectively improving the accuracy of video content location. This results in a video processing model that can be used to accurately retrieve video content, thereby improving the accuracy of video content retrieval.

[0085] In an exemplary embodiment, the video content is retrieved using the model to be trained, based on salient segments of the samples and the segment features of each of the multiple sample segments, to obtain the sample prediction time periods of a portion of the video content, including:

[0086] Using the model to be trained, extract the location features of significant segments in the sample;

[0087] Based on the cross-attention interaction mechanism, feature interaction is performed on the segment features of multiple sample segments, the segment features of salient sample segments, and the positional features to obtain the target features of multiple sample segments.

[0088] Based on the target features of multiple sample segments, time periods are predicted to obtain sample predicted time periods for a portion of the video content.

[0089] Location features refer to features used to describe the position of salient segments within the sample video; they indicate the location of the salient segments within the sample video. It should be noted that, in this embodiment, the position of a salient segment within the sample video refers to the time period in which the salient segment is located within the sample video, and can be represented in the same form as the time period label.

[0090] The cross-attention interaction mechanism is primarily used for information exchange between different inputs, enabling the model to effectively align and focus on contexts from different sources, thereby helping the model better capture the correlation between two inputs. In this embodiment, the cross-attention interaction mechanism is used mainly to obtain target features of sample segments that incorporate the features of salient sample segments, guided by prior preferences based on the segment features and positional features of salient sample segments.

[0091] For example, the terminal uses the model to be trained to determine the segment location information of the salient segments in the sample video, extracts the position features of the salient segments based on the segment location information, and then uses a cross-attention interaction mechanism to perform feature interaction on the segment features of each of the multiple sample segments, the segment features of the salient segments, and the position features, so as to fuse the segment features and position features of the salient segments with the segment features of each of the multiple sample segments to obtain the target features of each of the multiple sample segments. Then, based on the target features of each of the multiple sample segments, time period prediction is performed to obtain the sample prediction time period of part of the video content.

[0092] In practical applications, during feature interaction, the terminal performs linear transformations on the segment features of multiple sample segments based on the cross-attention interaction mechanism to obtain the first key vector and the first value vector. It then performs linear transformations on the segment features and positional features of the salient segments of the samples to obtain the first query vector. Attention weights are calculated based on the first query vector and the first key vector to obtain the attention weight matrix. The first value vector is then weighted based on the attention weight matrix to obtain the first interaction feature. The first interaction feature is then processed through residual connections and layer normalization to obtain the target features of multiple sample segments.

[0093] In a specific application, the number of salient segments in a sample is determined based on the number k of segment features required for the video content retrieval task to be used as a query. Since k is a positive integer, when performing a linear transformation on the segment features and positional features of the salient segments in the sample to obtain the first query vector, it is necessary to perform a linear transformation on the segment features and positional features of k salient segments in the sample to obtain the first query vector.

[0094] In a specific application, in order to fully integrate the fragment features and positional features of the salient fragments of a sample with the fragment features of each of the multiple sample fragments, the terminal can perform multiple rounds of feature interaction based on the cross-attention interaction mechanism to obtain the target features of each of the multiple sample fragments.

[0095] In each round of feature interaction, the first query vector is obtained by linearly transforming the fragment features and positional features of the salient segments of the samples. The first key vector and first value vector of the first round of feature interaction are obtained by linearly transforming the fragment features of each of the multiple sample segments. Starting from the second round of feature interaction, the first key vector and first value vector are obtained by linearly transforming the features of each of the multiple sample segments output from the previous round of feature interaction. The output of the final round of feature interaction represents the target features of each of the multiple sample segments.

[0096] In practical applications, the training model primarily uses two fully connected layers to predict time periods. One fully connected layer outputs the range of multiple sample segment intervals based on their respective target features. The other fully connected layer predicts the confidence level of each of these intervals, which can be understood as the degree of confidence that the interval belongs to a portion of the video content. After determining the confidence levels of each interval, the terminal selects the interval with the highest confidence level as the predicted portion of the video content for video content retrieval. The range of the selected interval is used as the predicted time period for this portion of the video content. The sample segments within the selected interval are designated as foreground sample segments, and the remaining sample segments in the video are designated as background sample segments.

[0097] In a specific application, a sample segment interval refers to an interval that includes at least one sample segment. The number of sample segments it contains may be one or more. The interval range has the same form as the time period label, specifically [c, m], where c is the midpoint of the time period within the sample video, and m is the length of the time period within the sample video. The number of sample segment intervals output by the fully connected layer is the same as the number of segment features k required for the video content retrieval task, where k is a positive integer. In this embodiment, feature interaction based on a cross-attention interaction mechanism can use the segment features and positional features of salient sample segments as prior preference guidance to help accurately locate key events in the sample video and enhance the semantic understanding of the events. This yields target features of sample segments that integrate the features of salient sample segments. Furthermore, by predicting the time period based on the target features of multiple sample segments, accurate prediction of the time period labels for a portion of the video content can be achieved, thus obtaining the predicted time period of a portion of the video content.

[0098] In an exemplary embodiment, the location features of salient segments in a sample are extracted using the model to be trained, including:

[0099] Using the model to be trained, the segment location information of the salient segments in the sample video is determined based on the segment features of the salient segments in the sample.

[0100] Feature encoding is performed based on fragment location information to obtain the location features of significant fragments in the sample.

[0101] Segment location information refers to information describing the position of a salient segment within the sample video. The position of a salient segment within the sample video refers to the time period in which the salient segment is located within the sample video. For example, segment location information can be in the same format as time period labels.

[0102] For example, the terminal will use the model to be trained to convert the segment features of the salient segments of the sample into the segment location information of the salient segments of the sample in the sample video, and perform feature encoding based on the segment location information to obtain the location features of the salient segments of the sample.

[0103] In practical applications, the terminal can use the fully connected layer in the model under test, which is used for time period prediction, to convert the segment features of salient segments in the sample into the segment location information of the salient segments in the sample video. The structure of this fully connected layer for time period prediction can be the same as that used for time period prediction in video content retrieval tasks. During feature encoding, the terminal can use a pre-set location feature calculation formula to perform feature encoding based on the segment location information to obtain the location features of the salient segments in the sample.

[0104] In a specific application, the number of significant fragments in the sample is k, where k is a positive integer, and the fragment position information R of the k-th significant fragment is... k It can be calculated using the following formula (1):

[0105] (1)

[0106] in, This represents the segment features of the salient segment of the k-th sample. This represents a fully connected layer used to convert the segment features of the salient segment of the k-th sample into the segment location information of the salient segment of the k-th sample in the sample video. The subscript span indicates that its structure can be the same as the fully connected layer used for time segment prediction in video content retrieval tasks.

[0107] In a specific application, it can be used Let represent the segment features of k salient segments from the samples. belong , The calculation can be performed using the following formula (2):

[0108] (2)

[0109] Where k is a positive integer, representing the number of segment features required for the video content retrieval task to be used as a query element. Let L represent the segment features of multiple sample segments, where L is the number of sample segments. This represents the segment feature of the k-th sample segment. The content significance score of the k-th sample segment. This represents the segment features used to select k significant segments from the segment features of multiple sample segments.

[0110] In a specific application, the pre-set location feature calculation formula can be configured according to the feature dimensions of the location features required in actual use. For example, the feature dimension of the location feature of the k-th salient fragment can be d, where d is a positive integer, thus providing location features with d dimensions. It can be calculated using the following formula (3):

[0111] (3)

[0112] Among them, R k This provides the fragment location information for the salient fragment of the k-th sample.

[0113] In this embodiment, by first determining the segment location information of the significant segments in the sample video, the positional features of the significant segments can be extracted by using the segment location information for feature encoding.

[0114] In an exemplary embodiment, the model to be trained predicts the content saliency score of each of the multiple sample segments based on their respective segment features, and determines the salient segments among the multiple sample segments based on their content saliency scores, including:

[0115] Using the model to be trained, feature mapping is performed on the segment features of multiple sample segments to determine the salient features of each sample segment, and the first salient label feature learned based on multiple sample segments and sample description text is obtained.

[0116] Based on the first saliency marker feature and the saliency features of each of the multiple sample segments, the content saliency scores of each of the multiple sample segments are determined;

[0117] Based on the content saliency score, multiple sample segments are sorted, and the salient segments among the multiple sample segments are determined based on the sorting results.

[0118] Among them, saliency features refer to features used to describe the content saliency of sample segments. For example, saliency features can specifically refer to feature vectors used to describe the content saliency of sample segments. The first saliency label feature refers to the feature representation learned in the content saliency detection task based on the interaction of video features of multiple sample segments and text features extracted from sample description text. It can be regarded as a feature representing the coefficients related to saliency, reflecting the model's ability to understand and capture saliency features. It is optimized during the model training process to improve the performance of the content saliency detection task.

[0119] For example, the terminal uses the model to be trained to perform feature mapping on the segment features of multiple sample segments divided from the sample video, determines the saliency features of each sample segment, and obtains the first saliency marker features learned based on the multiple sample segments and sample description text. The first saliency marker features are mapped to a first coefficient of saliency correlation. Based on the first coefficient and the saliency features of each sample segment, the content saliency score of each sample segment is determined. The sample segments are then ranked according to their content saliency scores. Based on the ranking result and the number of segment features required for the video content retrieval task, salient segments are determined from the multiple sample segments. The number of salient segments is the same as the number of segment features required for the query.

[0120] In practical applications, the terminal can use a learnable linear layer in the model to perform feature mapping on the segment features of multiple sample segments divided from the sample video, thereby determining the salient features of each of the multiple sample segments. The determined salient features of each of the multiple sample segments are then represented as vectors of a specific dimension output by the learnable linear layer. This specific dimension can be configured according to the actual application scenario when configuring the model to be trained. For example, the specific dimension can be 1.

[0121] In practical applications, the process of learning the first saliency marker feature can be as follows: Figure 4 As shown, the terminal, through the encoder in the model to be trained, can perform feature interaction between the initial saliency marker and the video features of multiple sample segments, as well as the text features extracted from the sample description text, thereby learning the first saliency marker features. Furthermore, through the feature interaction performed by the encoder, the segment features of each of the multiple sample segments can also be obtained. The video features of each of the multiple sample segments are obtained by extracting features from each of the multiple sample segments separately.

[0122] In a specific application, such as Figure 5As shown, after the encoder outputs the segment features and first saliency marker features of multiple sample segments, the terminal performs feature mapping on the segment features of the multiple sample segments divided from the sample video through the learnable linear layer in the model to be trained, determines the saliency features of each of the multiple sample segments, and obtains the first saliency marker features. The first saliency marker features are mapped to the first coefficient of saliency correlation through a fully connected layer used for feature mapping. The first coefficient is multiplied by the saliency features of each of the multiple sample segments, and then passed through an activation function to determine the content saliency score of each of the multiple sample segments. According to the content saliency score, the multiple sample segments are sorted. According to the sorting result and the number of segment features k required for the video content retrieval task to be used as the query, k sample segments are selected as salient sample segments, and the segment features of the k salient sample segments (i.e., Figure 5 The selected top-K values ​​are shown below, where k is a positive integer.

[0123] It should be noted that the first coefficient of significance actually reflects the training model's ability to understand and capture significant features, such as... Figure 5 As shown, a smaller first coefficient indicates a poorer understanding and capture of salient features by the model under training, resulting in lower predicted content saliency scores and a weaker ability to distinguish between salient and non-salient content. Conversely, a larger first coefficient indicates a stronger understanding and capture of salient features by the model under training, leading to more significant differences in predicted content saliency scores and a stronger ability to distinguish between salient and non-salient content. It is understandable that through continuous training of the model under training, the learned first saliency marker features will be optimized to improve the performance of the content saliency detection task.

[0124] The activation function can be configured according to the actual application scenario. For example, the activation function can be the sigmoid function, which maps the input value to the interval (0, 1), introducing non-linearity into the model and enabling it to learn more complex functions. For instance, when the activation function is the sigmoid function, the content significance score S(X) of sample fragment i... i It can be predicted using the following formula (4):

[0125] (4)

[0126] Among them, X i Let i be the fragment feature of sample fragment i. MLP() is a learnable linear layer. The output of the learnable linear layer is multiplied by the first coefficient, which is not shown in the formula.

[0127] In this embodiment, based on the segment features of each of the multiple sample segments divided from the sample video, the saliency features of each of the multiple sample segments can be determined through feature mapping. Then, based on the first saliency marker features learned from the multiple sample segments and sample description text, the content saliency scores of each of the multiple sample segments can be determined using the first saliency marker features and the saliency features of each of the multiple sample segments, thus completing the content saliency detection of the multiple sample segments. Therefore, the multiple sample segments can be sorted according to the content saliency scores, and the salient segments among the multiple sample segments can be determined according to the sorting results, thereby achieving the screening of salient segments.

[0128] In one exemplary embodiment, the video processing model training method further includes:

[0129] Using the model to be trained, based on the segment features of multiple sample segments divided from the sample video, the time period to which some video content belongs is predicted.

[0130] The significance loss value is determined based on the content significance scores of multiple sample segments, including:

[0131] Obtain the saliency labels of multiple sample fragments;

[0132] Based on the saliency labels and content saliency scores of multiple sample segments, the saliency category loss value is determined, and based on the time period to which the video content belongs and the differences in time period labels, the time period prediction loss value is determined;

[0133] The significance loss value is determined based on the significance category loss value and the time period prediction loss value.

[0134] The time period to which the video content belongs refers to the time period in which a portion of the video content appears within the sample video. The saliency label is a label describing the saliency category of a sample segment; that is, the saliency label can be used to distinguish sample segments and determine their saliency category. The saliency category can be configured according to the actual application scenario. For example, the saliency category can specifically be foreground sample segments or background sample segments.

[0135] In this context, foreground sample clips refer to the portion of the sample video that is closest to the viewer's object or scene. These are typically the intersection of frames or the main point of interest, and can be used to emphasize specific actions, characters, or objects. Background sample clips, on the other hand, are the portion of the sample video located behind the foreground, providing environment and context for the foreground. Understandably, background sample clips are often used to set the atmosphere of a scene, provide a sense of depth, and can include elements such as sky, architecture, and natural landscapes.

[0136] For example, the terminal will also use the fully connected layer in the model to be trained for time period prediction to predict the time period to which the video content belongs based on the segment features of each of the multiple sample segments divided from the sample video. The structure of the fully connected layer for time period prediction can be the same as the structure of the fully connected layer for time period prediction in the video content retrieval task.

[0137] For example, when determining the saliency loss value, the terminal first obtains the saliency labels of multiple sample segments. Based on the saliency labels and content saliency scores of the multiple sample segments, it determines the saliency category loss value. Then, based on the time period to which the video content belongs and the differences in time period labels, it determines the time period prediction loss value. Finally, it weights the saliency category loss value and the time period prediction loss value to obtain the task collaboration constraint loss. Based on the task collaboration constraint loss, it determines the saliency loss value. The weighting coefficients used when weighting the saliency category loss value and the time period prediction loss value are hyperparameters for balancing the loss and can be configured according to the actual application scenario; specifically, they can be values ​​less than 1.

[0138] In practical applications, each saliency label corresponds to a different category weight. The terminal determines the category weight of each sample segment based on its respective saliency label, and then determines the saliency category loss value based on the category weights of each sample segment and the content saliency score. The category weight corresponding to each saliency label can be configured according to the actual application scenario. For example, the category weight of the saliency label describing the foreground sample segment can be 1, while the category weight of the saliency label describing the background sample segment can be 0.

[0139] In a specific application, based on determining the class weights and content saliency scores of multiple sample segments, the saliency class loss value L... ce It can be calculated using formula (5):

[0140] (5)

[0141] in, This represents the segment feature of the i-th sample segment. Let represent the significance score of the i-th sample segment, and log() represent taking the logarithm. This represents the class weight of the i-th sample segment. When determining the saliency class loss value, we can first calculate the product of the class weights of multiple sample segments and the logarithm of the content saliency score, then add the product of the class weights of multiple sample segments and the logarithm of the content saliency score, and finally take the negative value of the added value to obtain the saliency class loss value.

[0142] In practical applications, the terminal can use a pre-set time-period prediction loss function to calculate the difference between the time period to which the video content belongs and the time period label, and determine the time-period prediction loss value. The time-period prediction loss function can be configured according to the actual application scenario. For example, the time-period prediction loss function can be at least one of the following: a smoothed L1 loss function or a GIOU (Generalized Intersection over Union) loss function.

[0143] The smoothed L1 loss function combines the characteristics of both L1 and L2 loss functions. When the absolute value of the difference between the predicted and true values ​​is less than 1, the loss function behaves as an L2 loss function; when the absolute value of the difference between the predicted and true values ​​is greater than or equal to 1, the loss function behaves as an L1 loss function. In this embodiment, the predicted value is the time period to which the video content belongs, and the true value is the time period label.

[0144] The GIOU loss function is a loss function used for bounding box regression in object detection tasks. It extends the traditional IOU (Intersection over Union) loss function to address the problem that the IOU loss function cannot provide an effective gradient when two bounding boxes do not intersect, and can better reflect the relative positional relationship between the two bounding boxes. In this embodiment, the two bounding boxes are the time period to which the video content belongs and the time period label.

[0145] In a specific application, the time-period prediction loss function may include a smoothed L1 loss function and a GIOU loss function. The weighted saliency class loss value and the time-period prediction loss value then yield the task collaboration constraint loss. It can be calculated using formula (6):

[0146] (6)

[0147] in, , and This is the hyperparameter of the balancing loss, where m is the time period label. It is the predicted time period of the video content. It is the smoothed L1 loss value. It is the GIOU loss value. It is the saliency category loss value, which can be calculated using formula (5).

[0148] In this embodiment, the selection of relevant moments is further refined by classifying the content saliency score using saliency labels, and the saliency category loss value is determined. The classification loss of the video content retrieval task can be used to constrain the content saliency detection loss. By determining the time period prediction loss value based on the time period to which the video content belongs and the difference in time period labels, the time period prediction loss of the video content retrieval task can be used to constrain the content saliency detection loss. Thus, based on the saliency category loss value and the time period prediction loss value, the saliency loss value constrained by the loss of the video content retrieval task can be obtained, thereby improving the performance of the content saliency detection task by utilizing the loss constraint of the video content retrieval task.

[0149] In one exemplary embodiment, determining a saliency loss value based on the saliency category loss value and the time-period prediction loss value includes:

[0150] The significance score loss value is determined based on the content significance score and time period label of each of the multiple sample segments.

[0151] The significance loss value is determined based on the significance category loss value, the time period prediction loss value, and the significance score loss value.

[0152] For example, the terminal can filter out a group of sample segments for calculating the saliency scoring loss value based on the content saliency scores and time period labels of multiple sample segments. Then, it can determine the saliency scoring loss value by utilizing the difference in content saliency scores of at least two sample segments within the sample segment group. The saliency loss value is determined by weighting the saliency category loss value, the time period prediction loss value, and the saliency scoring loss value. The weighting coefficients used when weighting the saliency category loss value, the time period prediction loss value, and the saliency scoring loss value are hyperparameters for balancing the loss and can be configured according to the actual application scenario; specifically, they can be values ​​less than 1.

[0153] In practical applications, at least two sample segments in a sample segment group can be understood as positive and negative segments. The concept of positive and negative segments is analogous to positive and negative samples in object detection. A positive segment refers to a sample video containing important content or themes, while a negative segment refers to a sample video containing background or other non-critical parts. For example, a positive segment could be the one with the highest content saliency score, while a negative segment could be the one with the lowest content saliency score. Another example is that a positive segment could be a sample segment within a time period label, while a negative segment could be a sample segment outside of a time period label.

[0154] In this embodiment, by determining the saliency scoring loss value based on the content saliency scores and time period labels of multiple sample segments, the saliency scoring loss can be used to constrain the content saliency detection task. This combines the loss constraints of the video content retrieval task and the constraints of the saliency scoring loss to improve the performance of the content saliency detection task.

[0155] In an exemplary embodiment, a saliency scoring loss value is determined based on the content saliency scores and time period labels of multiple sample segments, including:

[0156] From the content significance scores of multiple sample segments, determine the maximum and minimum content significance scores;

[0157] Determine the first significance score difference between the maximum and minimum content significance scores, and based on the time period labels, determine the second significance score difference between sample segments within and outside the time period labels;

[0158] The significance score loss value is determined based on the difference between the first and second significance scores.

[0159] For example, the terminal determines the maximum and minimum content saliency scores from the content saliency scores of multiple sample segments. It then calculates a first saliency score difference between the maximum and minimum content saliency scores. Based on time period labels, it identifies sample segments within and outside the time period labels, calculates a second saliency score difference between them, and sums the first saliency score difference with the loss interval value. It then compares the score difference threshold with the sum of the first saliency score difference and the loss interval value, determining the larger of the two. The terminal then calculates the second saliency score difference with the loss interval value, compares the score difference threshold with the sum of the second saliency score difference and the loss interval value, determining the larger of the two. Finally, it sums the larger value determined by the two comparisons to obtain the saliency score loss value. The score difference threshold can be configured according to the actual application scenario. For example, the score difference threshold can specifically be 0. The loss interval can be viewed as a safety factor used to control the gap between correct and incorrect prediction scores, and can be configured according to the actual application scenario.

[0160] In practical applications, when determining the maximum and minimum content saliency scores, the terminal can first filter out multiple sample segments within a time period label from multiple sample segments, and then determine the maximum and minimum content saliency scores from the content saliency scores of each of the multiple sample segments within the time period label. In this way, the differences in saliency scores can be evaluated using the individual content saliency scores of multiple sample segments within the time period label, thereby enabling the performance evaluation of the content saliency detection task.

[0161] In practical applications, when calculating the difference in the second significance score, if there are multiple sample segments within the time period label, the terminal needs to select one sample segment from these segments to calculate the difference in the second significance score. Similarly, if there are multiple sample segments outside the time period label, the terminal also needs to select one sample segment from these segments to calculate the difference in the second significance score.

[0162] In a specific application, when selecting sample segments, the terminal can sort multiple sample segments within a time period label according to their content saliency scores, and select the sample segment with the highest content saliency score to participate in the calculation of the second saliency score difference. Simultaneously, the terminal can also sort multiple sample segments outside the time period label according to their content saliency scores, and select the sample segment with the highest content saliency score to participate in the calculation of the second saliency score difference.

[0163] In a specific application, the significance scoring loss value The calculation can be performed using the following formula (7):

[0164] (7)

[0165] Where n is the score difference threshold, This is the loss interval value. The minimum content saliency score, To achieve the highest content saliency score, The difference in scores is the first significant difference. For sample segments outside the time period label, the content significance score is given. The content significance score of the sample segment within the time period label. This represents the difference in the second significance score.

[0166] In this embodiment, by determining the first saliency score difference between the maximum and minimum content saliency scores, and by determining the second saliency score difference between sample segments within and outside the time period labels based on the time period labels, the performance of the content saliency detection task can be evaluated using the first and second saliency score differences, and the saliency score loss value used for constraints can be determined.

[0167] In an exemplary embodiment, determining the video retrieval loss value based on the difference between the sample prediction time period and the time period label includes:

[0168] Based on the difference between the predicted time period and the time period label, the video retrieval time period loss value is determined, and the sample segment category of each of the multiple sample segments predicted by video content retrieval is obtained.

[0169] The video retrieval classification loss value is determined based on the respective sample segment categories of multiple sample segments;

[0170] The video retrieval loss value is determined based on the video retrieval time period loss value and the video retrieval classification loss value.

[0171] The sample segment category refers to the category of the sample segment determined through video content retrieval. For example, the sample segment category can specifically be foreground sample segment, background sample segment, etc.

[0172] For example, the terminal determines the video retrieval time period loss value based on the difference between the sample prediction time period and the time period label, and obtains the sample segment categories of multiple sample segments predicted by video content retrieval. Based on the sample segment categories of the multiple sample segments, it determines the video retrieval classification loss value. The video retrieval time period loss value and the video retrieval classification loss value are weighted to determine the video retrieval loss value. Among them, the weighting coefficients when weighting the video retrieval time period loss value and the video retrieval classification loss value are hyperparameters for balancing the loss, which can be configured according to the actual application scenario, and can be values ​​less than 1.

[0173] In practical applications, the terminal can use a pre-set time-period prediction loss function to calculate the difference between the predicted time period and the time period label, thereby determining the time period loss value for video retrieval. The time-period prediction loss function can be configured according to the actual application scenario. For example, the time-period prediction loss function can be at least one of the following: a smoothed L1 loss function or a GIOU (Generalized Intersection over Union) loss function.

[0174] In practical applications, during video content retrieval, the model to be trained can output the range and confidence level of multiple sample segment intervals based on the target features of each of the multiple sample segments. After determining the confidence level of each of the multiple sample segment intervals, the terminal selects the sample segment interval with the highest confidence level as the predicted part of the video content for video content retrieval. The range of the selected sample segment interval is used as the predicted time period of the partial video content, and the sample segments in the selected sample segment interval are used as foreground sample segments, while the remaining sample segments in the sample video are used as background sample segments.

[0175] In practical applications, each sample segment category corresponds to a different category weight. The terminal determines the category weight of each sample segment based on its respective category, and then determines the video retrieval classification loss value based on the category weight and confidence level of each sample segment. The category weight corresponding to each sample segment category can be configured according to the actual application scenario. For example, the category weight of a foreground sample segment can be 1, and the category weight of a background sample segment can be 0.

[0176] In a specific application, based on determining the class weights and confidence levels of multiple sample segments, the video retrieval classification loss value L... ce It can be calculated using formula (8):

[0177] (8)

[0178] in, This represents the feature of the i-th sample segment after it has been decoded. This represents the confidence level of the i-th sample segment identified through video content retrieval, and log() represents taking the logarithm. Let L represent the class weight of the i-th sample segment. When determining the video retrieval loss value, we can first calculate the product of the class weights and logarithms of the confidence scores of multiple sample segments, then sum these products, and finally take the negative of the summed value to obtain the video retrieval classification loss value L. ce .

[0179] In a specific application, taking time-period prediction loss functions, including the smoothed L1 loss function and the GIOU loss function, as an example, the video retrieval loss value... The calculation can be performed using the following formula (9):

[0180] (9)

[0181] in, , and This is the hyperparameter of the balancing loss, where m is the time period label. It is the predicted sample prediction period. It is the smoothed L1 loss value. It is the GIOU loss value. It is the video retrieval classification loss value, which can be calculated using formula (8).

[0182] In this embodiment, the video retrieval time period loss value is determined based on the difference between the sample prediction time period and the time period label, and the sample segment category of each of the multiple sample segments predicted by the video content retrieval is obtained. Based on the sample segment category of each of the multiple sample segments, the video retrieval classification loss value is determined. By combining time period prediction and classification, the accurate calculation of the video retrieval loss value for video content retrieval can be achieved.

[0183] In an exemplary embodiment, the model to be trained determines the segment features of each of the multiple sample segments based on the sample description text and the video features of multiple sample segments divided from the sample video, including:

[0184] Using the model to be trained, based on the cross-attention interaction mechanism, the text features extracted from the sample description text and the video features of multiple sample segments divided from the sample video are interacted to determine the segment features of each of the multiple sample segments.

[0185] The cross-attention interaction mechanism is primarily used for information interaction between different inputs, enabling the model to effectively align and focus on contexts from different sources, thereby helping the model better capture the correlation between two inputs. In this embodiment, the cross-attention interaction mechanism is used mainly to fuse the video features of multiple sample segments with the text features extracted from the sample description text, in order to obtain the segment features of multiple sample segments that are closely related to the sample description text.

[0186] For example, the terminal can divide the sample video into multiple sample segments according to a preset segment length, and extract video features from each of the multiple sample segments to obtain the video features of each of the multiple sample segments. It can also extract text features from the sample description text. Through the model to be trained, based on the cross-attention interaction mechanism, the video features of each of the multiple sample segments and the text features extracted from the sample description text are interacted to fuse the video features of each of the multiple sample segments and the text features extracted from the sample description text to obtain the segment features of each of the multiple sample segments that are closely related to the sample description text.

[0187] In practical applications, for each sample segment, the terminal performs dual-stream spatiotemporal feature extraction to obtain the first feature of the sample segment, and then performs video content feature extraction to obtain the second feature. The first and second features are then fused to obtain the video features of the sample segment. Dual-stream spatiotemporal feature extraction refers to using dual-stream convolutional input to extract the spatial and temporal information of the sample segment separately, achieving a comprehensive understanding of the sample segment's content. This can be implemented using the SlowFast model (a dual-channel model for video recognition tasks). The dual-stream convolutional input includes a slow channel and a fast channel. The slow channel is mainly used to capture image features, with fewer frames and a larger number of channels, while the fast channel is mainly used to capture motion features; due to rapid motion changes, it has more frames and a smaller number of channels. Video content feature extraction can be achieved through the image encoding part of the CLIP (Contrastive Language-Image Pretraining) model. Fusion can be achieved by dimensional concatenation.

[0188] In practical applications, the terminal can use a pre-trained video feature extractor to extract video features from multiple sample segments, obtaining the video features of each segment individually. The video feature extractor can be configured according to the actual application scenario; this embodiment does not impose specific limitations. For example, the video feature extractor can specifically be an extractor including at least one of the Slowfast model or the CLIP model.

[0189] In practical applications, when extracting text features from sample description text, the terminal first splits the sample description text into multiple text segments, and then extracts text features from each of the multiple text segments to obtain text features, which are the text features extracted from the sample description text, and are segment-level text features.

[0190] In practical applications, the terminal can extract text features from sample descriptive text using a pre-trained text feature extractor. The text feature extractor can be configured according to the actual application scenario; this embodiment does not impose specific limitations. For example, the text feature extractor could be the text encoder in the CLIP model.

[0191] In practical applications, based on the cross-attention interaction mechanism, the terminal performs linear transformation on the video features of multiple sample segments to obtain the second query vector, and performs linear transformation on the text features extracted from the sample description text to obtain the second key vector and the second value vector. Attention weights are calculated based on the second key vector and the second query vector to obtain the attention weight matrix. Based on the attention weight matrix, the second value vector is weighted to obtain the second interaction feature. The second interaction feature is then processed through residual connections and layer normalization to obtain the segment features of multiple sample segments.

[0192] In a specific application, the feature interaction processing in this embodiment can be represented by formula (10):

[0193] (10)

[0194] in, This is the second query vector. This is the second key vector. It is the transpose of the second key vector. For the second value vector, These are the feature dimensions of video features and text features. It is a function used to normalize the attention score, obtained after feature interaction. By using residual connections and layer normalization, the segment features of each of the multiple sample segments can be obtained.

[0195] In a specific application, in order to fully integrate the text features extracted from the sample description text and the video features of each of the multiple sample segments, the terminal can perform multiple rounds of feature interaction based on the cross-attention interaction mechanism to obtain the segment features of each of the multiple sample segments.

[0196] In each round of feature interaction, the second key vector and second value vector are obtained by linearly transforming the text features extracted from the sample description text. The second query vector in the first round of feature interaction is obtained by linearly transforming the video features of each of the multiple sample segments. Starting from the second round of feature interaction, the second query vector is obtained by linearly transforming the features of each of the multiple sample segments output from the previous round of feature interaction. The output of the final round of feature interaction is the segment features of each of the multiple sample segments.

[0197] In this embodiment, based on the cross-attention interaction mechanism, the video features of multiple sample segments and the text features extracted from the sample description text are interacted, which can achieve full fusion of the video features of multiple sample segments and the text features extracted from the sample description text, thereby obtaining the segment features of multiple sample segments that are closely related to the sample description text.

[0198] Based on the detailed description of the foregoing embodiments, the complete process of the video processing model training method in the embodiments of this application will be described below. In an exemplary embodiment, such as... Figure 6 As shown, a video processing model training method is provided. The method is illustrated using a terminal as an example. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method may include the following steps:

[0199] Step 602: Obtain training samples. The training samples include sample videos, sample description text, and time period labels. The sample description text describes the portion of the video content marked by the time period labels in the sample videos.

[0200] Step 604: Using the model to be trained, based on the cross-attention interaction mechanism, the text features extracted from the sample description text and the video features of multiple sample segments divided from the sample video are interacted to determine the segment features of each of the multiple sample segments.

[0201] Step 606: Using the model to be trained, feature mapping is performed on the segment features of each of the multiple sample segments divided from the sample video to determine the salient features of each of the multiple sample segments, and the first salient label feature learned based on the multiple sample segments and sample description text is obtained.

[0202] Step 608: Based on the first saliency marker feature and the saliency features of each of the multiple sample segments, determine the content saliency score of each of the multiple sample segments, and predict the time period to which the video content of some video content belongs based on the segment features of each of the multiple sample segments divided from the sample video.

[0203] Step 610: Sort multiple sample segments according to their content saliency scores, and determine the salient segments among the multiple sample segments based on the sorting results.

[0204] Step 612: Using the model to be trained, determine the segment location information of the salient segments in the sample video based on the segment features of the salient segments.

[0205] Step 614: Perform feature encoding based on fragment location information to obtain the location features of significant fragments in the sample.

[0206] Step 616: Based on the cross-attention interaction mechanism, feature interaction is performed on the segment features of each of the multiple sample segments, the segment features of the salient segments of the samples, and the positional features to obtain the target features of each of the multiple sample segments.

[0207] Step 618: Based on the target features of each of the multiple sample segments, perform time period prediction to obtain the sample predicted time periods of part of the video content.

[0208] Step 620: Based on the difference between the predicted time period and the time period label, determine the video retrieval time period loss value, and obtain the sample segment category of each of the multiple sample segments predicted by video content retrieval.

[0209] Step 622: Determine the video retrieval classification loss value based on the respective sample segment categories of multiple sample segments.

[0210] Step 624: Determine the video retrieval loss value based on the video retrieval time period loss value and the video retrieval classification loss value.

[0211] Step 626: Obtain the saliency labels of each of the multiple sample fragments.

[0212] Step 628: Determine the saliency category loss value based on the saliency labels and content saliency scores of each of the multiple sample segments, and determine the time period prediction loss value based on the time period to which the video content belongs and the differences in time period labels.

[0213] Step 630: Determine the maximum and minimum content saliency scores from the content saliency scores of each of the multiple sample segments.

[0214] Step 632: Determine the first significance score difference between the maximum and minimum content significance scores, and determine the second significance score difference between sample segments within and outside the time period labels based on the time period labels.

[0215] Step 634: Determine the significance score loss value based on the difference between the first significance score and the difference between the second significance score.

[0216] Step 636: Determine the significance loss value based on the significance category loss value, the time period prediction loss value, and the significance score loss value.

[0217] Step 638: Update the model to be trained based on the video retrieval loss value and the saliency loss value until the model to be trained is completed.

[0218] In an exemplary embodiment, the structure of the model to be trained is as follows: Figure 7 The video processing model training method of this application will be explained using the example shown.

[0219] like Figure 7 As shown, the architecture of the model to be trained mainly consists of two parts: an encoder and a decoder, both employing an N-layer Transformer network architecture, where N is a positive integer, for example, N can be 2. For the encoder, each layer mainly consists of a multi-head self-attention mechanism (MHSA) and a forward fully connected layer (FFN). In the decoder, in addition to MHSA and FFN, each layer is also equipped with an additional cross-attention layer (MHCA).

[0220] For the encoder, the input data consists of training samples. The sample videos and descriptive text within these training samples are first fed into pre-trained feature extractors to extract necessary video and text features, which will be used in subsequent training processes. In the sample video processing section, the sample video is divided into multiple sample segments. Features from each segment are further extracted using a pre-trained feature extractor (which may include Slowfast or CLIP models), and these segments are then concatenated dimensionally to obtain the video features. Where L is the number of sample segments, Let be the video features of the i-th sample segment. For the sample description text, input the sample description text into the text encoder to obtain the text features. Where M is the number of text fragments obtained by splitting the sample description text. Let be the text feature corresponding to the i-th text segment.

[0221] To effectively integrate text feature context into the feature representation of each video clip (i.e., the video features of the sample segments), this application constructs a cross-attention interaction mechanism between video and text modalities. This mechanism aligns and interactively processes video and text features through a cross-attention layer, thereby generating video feature representations closely related to the text, i.e., the segment features of multiple sample segments.

[0222] Specifically, such as Figure 7 As shown, the video features of multiple sample segments output by the feature extractor in the video part are used to generate query vectors, while the text features output by the feature extractor in the text part are used to generate key vectors and value vectors. The cross-attention layer calculates the similarity score between the query vector and the key vector, and uses these scores to weight the corresponding value vectors. The weighted sum of each value vector is calculated so that the text query can be directly associated with relevant information in the video features, as shown in formula (11):

[0223] (11)

[0224] in, For query vector, For key vectors, It is the transpose of the key vector. For value vectors, These are the feature dimensions of video features and text features. It is a function used to normalize the attention score, obtained after feature interaction. By using residual connections and layer normalization, we can obtain video features closely related to the text (i.e., the segment features of each of the multiple sample segments). Where L is the number of sample segments, Let be the segment features of the i-th sample segment.

[0225] For the decoder, this application designs a joint task cooperative decoder, introducing interaction between two tasks in the joint framework to achieve effective gains. Dynamic position queries for motion and semantic content queries are based on semantic saliency predicted along the time dimension by content saliency detection, obtained through event-specific priors. Simultaneously, the binary saliency prediction task in content saliency detection uses an auxiliary loss from the video content retrieval task to predict the accurate moment boundary in the time dimension.

[0226] Regarding the gain of video content retrieval on content saliency detection, since the selected query is preliminary and lacks further refined encoder features, such ambiguous queries may mislead the decoder, thus limiting the effectiveness of video content retrieval. Therefore, this application utilizes classification and regression constraint losses (i.e., using time-segment prediction loss values ​​as constraint losses) from the video content retrieval task in content saliency detection learning. It consists of smoothing L1 loss and GIOU loss to improve the accuracy of detected salient segment locations, and cross-entropy loss (i.e., saliency category loss value). Used to classify the predicted content salience score into foreground and background The selection of relevant time points is refined, and its calculation formula is formula (12):

[0227] (12)

[0228] in, , and This is the hyperparameter of the balancing loss, where m is the time period label. It is the predicted time period of the video content. It is the smoothed L1 loss value. It is the GIOU loss value. It is the saliency category loss value, which can be calculated using formula (13).

[0229] (13)

[0230] in, This represents the segment feature of the i-th sample segment. Let represent the significance score of the i-th sample segment, and log() represent taking the logarithm. This represents the class weight of the i-th sample segment.

[0231] It should be noted that, as Figure 7 As shown in this application, a fully connected layer for time period prediction is added to the content saliency detection task. Based on the segment features of each of the multiple sample segments divided from the sample video, the time period to which the video content belongs is predicted. The structure of this fully connected layer for time period prediction can be the same as the structure of the fully connected layer for time period prediction in the video content retrieval task.

[0232] In the video content retrieval task, time period prediction is mainly achieved through two fully connected layers. One fully connected layer is used for time period prediction. Based on the target features of multiple sample segments, it predicts the segment time period of each of the multiple sample segments. The other fully connected layer (which can be a fully connected layer) predicts the confidence of each of the multiple sample segments based on the target features of each of the multiple sample segments. Here, the confidence can be understood as the confidence that the sample segment belongs to part of the video content. After determining the confidence of each of the multiple sample segments, at least one sample segment can be selected as part of the video content predicted by the video content retrieval based on the confidence of each of the multiple sample segments. The selected at least one sample segment is used as the foreground sample segment, and the other sample segments are used as the background sample segments. The segment time of the foreground sample segment is used as the sample prediction time period of part of the video content. In addition, this application uses a saliency scoring loss to impose constraints. This loss is obtained by calculating the hinge loss between two pairs of positive and negative segments. Specifically, the first pair includes segments with higher and lower scores in the ground real time. The second pair involves a segment within the ground real time and another segment outside the ground real time. The loss calculation formula is as shown in formula (14):

[0233] (14)

[0234] in, This is the loss interval value. The content salience score is given to the segment with the lower score. The higher-scoring segments are given content saliency scores. The content significance score is given for sample segments outside of the actual ground time. The content significance score of the sample segment within the actual ground time.

[0235] Among them, the content significance score S(X) of sample fragment i i The result can be obtained using the following formula (15):

[0236] S(X) i =Sigmoid(MLP(X) i (15)

[0237] Among them, X i For the fragment features of sample fragment i, MLP() is a learnable linear layer, specifically a BinaryMLP. For example... Figure 7 As shown, in the model to be trained, a linear layer can be used to perform feature mapping on the segment features of multiple sample segments to determine the saliency features of each sample segment. The first saliency label feature is mapped to the first coefficient of saliency correlation through a fully connected layer used for feature mapping. The first coefficient is multiplied by the saliency features of each sample segment, and then passed through an activation function to determine the content saliency score of each sample segment.

[0238] in, Figure 7 The example provided illustrates content saliency scores when the first coefficient is small and large. It can be seen that a small first coefficient reflects a poor understanding and capture of salient features by the model being trained, resulting in lower predicted content saliency scores and a weaker ability to distinguish between salient and non-salient content. Conversely, a large first coefficient reflects a stronger understanding and capture of salient features by the model being trained, leading to more pronounced differences in predicted content saliency scores and a stronger ability to distinguish between salient and non-salient content.

[0239] Regarding the gain of content saliency detection for video content retrieval, existing methods employ static decoder queries, where location queries are learned from the object distribution of the training dataset, and content queries are set to a zero vector by default. However, these static queries lack specificity and adaptability to the input video and may fail to capture subtle differences. This application introduces the saliency prediction results of content saliency detection as prior knowledge for the query. Based on the number of decoder queries, k sample segments are selected as salient segments from multiple sample segments. Using a cross-attention interaction mechanism, the segment features and location features of the k salient segments are used to interact with the segment features of the multiple sample segments to achieve decoding, obtaining the target features of each of the multiple sample segments. Finally, the target features of the multiple sample segments are used for time period prediction.

[0240] Specifically, this application uses the highest Top-K content saliency score to select from video features closely related to the text (i.e., the segment features of each of the multiple sample segments). Select content search (i.e., the segment features of significant segments of the sample), the specific formula is as shown in formula (16):

[0241] (16)

[0242] Where k is the number of decoder queries, i.e., the number of segment features required for the video content retrieval task to be used as queries. This represents the segment feature of the k-th sample segment. The content significance score of the k-th sample segment. This represents the segment features used to select k significant segments from the segment features of multiple sample segments.

[0243] Furthermore, this application specifies the query at the k-th position. Corresponding to The location is determined by using the same auxiliary time-segment prediction fully connected layer as in video content retrieval to convert the fragment features of salient segments in the sample into reference locations R. k That is, the fragment location information of the significant fragment of the k-th sample, which can be converted by formula (17):

[0244] (17)

[0245] in, This represents the segment features of the salient segment of the k-th sample. This represents a fully connected layer used to convert the segment features of the salient segment of the k-th sample into the segment location information of the salient segment of the k-th sample in the sample video. The subscript span indicates that its structure can be the same as the fully connected layer used for time segment prediction in video content retrieval tasks.

[0246] Furthermore, by using the location feature calculation formula to encode the location information of the fragments, the location features of significant fragments in the sample can be obtained. Suppose that the feature dimension of the location feature of the salient segment of the k-th sample can be d, where d is a positive integer, then the location feature with d dimensions... It can be calculated using the following formula (18):

[0247] (18)

[0248] Among them, R k This provides the fragment location information for the salient fragment of the k-th sample.

[0249] For video content retrieval, this application uses the same loss function as formula (12) for task training, including smoothing L1 loss function, GIOU loss function and video retrieval classification loss, the specific formula of which can be formula (19):

[0250] (19)

[0251] in, , and This is the hyperparameter of the balancing loss, where m is the time period label. It is the predicted sample prediction period. It is the smoothed L1 loss value. It is the GIOU loss value. It is the video retrieval classification loss value, which can be calculated using formula (20).

[0252] (20)

[0253] in, This represents the feature of the i-th sample segment after it has been decoded. This represents the confidence level of the i-th sample segment identified through video content retrieval, and log() represents taking the logarithm. This represents the class weight of the i-th sample segment, which is determined by the class of the i-th sample segment, such as... Figure 7 As shown, the sample fragment categories can be foreground sample fragments or background sample fragments.

[0254] It is understandable that the loss function value for the entire training process is the sum of the above loss function values, which can be expressed by formula (21):

[0255] ;(twenty one)

[0256] in, The hyperparameters are used to balance the loss.

[0257] During the training phase, a fixed number of training samples are processed in each iteration. First, the sample videos and sample description texts in the training samples are processed by a pre-trained feature extractor to obtain their respective features. Then, these features are fed into the network, and the corresponding loss is calculated according to formula (21). Backpropagation is then performed to optimize the network parameters of the model. After a set number of training cycles, the training process stops. It should be noted that, as Figure 7 As shown, in the content saliency detection task section, besides and In addition, other loss values ​​can be calculated to further optimize the performance of the content saliency detection task, such as saliency marker loss value, loss interval loss value, etc.

[0258] It is understandable that the video processing model training method in this application fully leverages the synergistic advantages between the joint tasks of video content retrieval and content saliency detection, effectively improving the accuracy of video content localization. On one hand, based on the given query context, this method uses the semantic saliency predicted along temporal motion in content saliency detection as a priori preference guide for video content retrieval, helping to accurately locate key events in the video and enhance the semantic understanding of the events. On the other hand, the precise temporal boundary predicted by video content retrieval and most relevant to the query semantics provides a potentially strong distinguishing factor for binary prediction in content saliency detection. This method uses the auxiliary loss of the video content retrieval task to predict the accurate moment boundary in the temporal dimension, helping to improve the binary saliency prediction task in content saliency detection. This method emphasizes the interrelationship between the joint tasks of video content retrieval and content saliency detection and effectively utilizes the synergistic advantages between the two to strengthen the complementarity between the tasks. It also improves the generalization ability of the model through co-training.

[0259] In one exemplary embodiment, the video processing model trained using this application can be deployed on a cloud platform in a video cross-modal segment retrieval and highlight content detection service that requires uploading query content and video to the cloud, thereby improving the user's video browsing experience.

[0260] In an exemplary embodiment, taking a general cross-modal segment retrieval and highlight content detection dataset as an example, as shown in Table 1, the model trained using the video processing model training method of this application significantly improves Rank-1 accuracy, mAP (mean Average Precision), and HIT@1 metrics on the general dataset compared to the baseline model QD-DETR (Query-Dependent DETR, a detection transformer model customized for video moment retrieval and highlight detection tasks) with the same network structure and trained using the same video data. Experimental results show that this method can effectively improve the performance of video cross-modal segment retrieval and highlight detection. Rank-1 accuracy refers to the proportion of times the system ranks the most relevant or interesting item first among all users or queries. In other words, Rank-1 accuracy measures the system's ability to correctly predict user preferences or query intent on the first attempt. mAP is a metric used to evaluate the performance of object detection models, which comprehensively considers the model's accuracy and recall across different categories. The HIT@1 metric is a measure of the performance of a recommender system or search engine. It indicates the percentage of users whose top-1 recommendation (the first-ranked item) is actually of interest to them across all user queries. In other words, HIT@1 measures the success rate of the system's first recommendation to a user.

[0261] Table 1

[0262]

[0263] In an exemplary embodiment, the video processing method provided in this application, when applied to a terminal, can be illustrated as follows: Figure 8 As shown in the diagram, terminal 802 acquires the video to be processed and the retrieval text used to retrieve video content from the video to be processed. Based on the retrieval text and the video features of multiple video segments divided from the video to be processed, it determines the segment features of each of the multiple video segments. Based on the segment features of each of the multiple video segments, it predicts the target salience score of each of the multiple video segments. Based on the target salience score, it determines the salient video segments among the multiple video segments. Based on the salient video segments and the segment features of each of the multiple video segments, it performs video content retrieval to obtain the predicted video time period of the video content.

[0264] The terminal 802 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0265] In one exemplary embodiment, such as Figure 9 As shown, a video processing method is provided. This method can be applied to a terminal, a server, or a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. This method is applied to... Figure 9 Taking the terminal in the example, the explanation includes the following steps 902 to 908. Wherein:

[0266] Step 902: Obtain the video to be processed and the search text used to retrieve video content from the video to be processed.

[0267] For example, when performing video processing, the terminal obtains the video to be processed and the search text used to retrieve video content from the video to be processed. The search text describes the video content in the video to be processed, and the terminal needs to use the search text to retrieve the video content described therein from the video to be processed.

[0268] Step 904: Determine the segment features of each of the multiple video segments based on the retrieved text and the video features of the multiple video segments divided from the video to be processed.

[0269] For example, the terminal divides the video to be processed into multiple video segments according to a preset segment length, extracts video features from each video segment to obtain the video features of each segment, and extracts text features from the search text. Based on a cross-attention interaction mechanism, the video features of each video segment and the text features extracted from the search text are interacted to determine the segment features of each video segment. It can be understood that the segment features of each video segment at this time are features closely related to the search text.

[0270] In practical applications, based on the cross-attention interaction mechanism, the terminal performs linear transformations on the video features of multiple video segments to obtain a third query vector, and performs linear transformations on the text features extracted from the retrieved text to obtain a third key vector and a third value vector. Attention weights are calculated based on the third key vector and the third query vector to obtain an attention weight matrix. Based on the attention weight matrix, the third value vector is weighted to obtain the third interaction feature. The third interaction feature is then processed through residual connections and layer normalization to obtain the segment features of multiple sample segments.

[0271] In a specific application, in order to fully integrate the text features extracted from the retrieved text with the video features of each of the multiple video segments, the terminal can perform multiple rounds of feature interaction based on the cross-attention interaction mechanism to obtain the segment features of each of the multiple video segments.

[0272] Step 906: Based on the segment features of each of the multiple video segments, predict the target salience score of each of the multiple video segments, and determine the salient video segments among the multiple video segments based on the target salience scores.

[0273] For example, the terminal performs content saliency detection on multiple video segments based on their respective segment features, predicts the target saliency score of each video segment, sorts the multiple video segments based on their respective target saliency scores, and determines the salient video segments among the multiple video segments based on the sorting results.

[0274] In practical applications, when performing content saliency detection, the terminal performs feature mapping on the segment features of multiple video segments separately. This allows for the initial determination of the saliency features of each video segment. Then, using these individual saliency features, along with second saliency marker features learned from the video segments and the retrieved text, the target saliency score for each video segment can be predicted. The second saliency marker features refer to the feature representations learned in the content saliency detection task based on the interaction between the video features of the video segments and the text features extracted from the retrieved text. These features can be viewed as characteristics representing coefficients related to saliency.

[0275] In a specific application, a terminal can use a learnable linear layer to perform feature mapping on the segment features of multiple video clips, determining the salient features of each video clip. These determined salient features are then represented as vectors of a specific dimension output by the learnable linear layer. This specific dimension can be configured according to the actual application scenario when configuring the model to be trained. For example, the specific dimension can be 1.

[0276] In a specific application, the terminal maps the second saliency marker features to a second saliency-related coefficient. Based on the second coefficient and the saliency features of each of the multiple video segments, it determines the target content saliency score for each video segment. According to the target content saliency score, the multiple video segments are ranked. Based on the ranking result and the number of segment features required for the video content retrieval task, the salient video segments are determined. The number of salient video segments is the same as the number of segment features required for the query.

[0277] Step 908: Perform video content retrieval based on significant video segments and the segment features of each of the multiple video segments to obtain the predicted video time period of the video content.

[0278] For example, after identifying significant video segments, the terminal extracts the target location features of the significant video segments, and then fuses the segment features of each of the multiple video segments, the segment features of the significant video segments, and the target location features to obtain the target features of each of the multiple video segments. Based on the target features of each of the multiple video segments, the terminal performs time period prediction to obtain the predicted video time period of the video content.

[0279] The aforementioned video processing method, based on the acquisition of the video to be processed and the search text, determines the segment features of each video segment according to the search text and the video features of multiple video segments divided from the video to be processed. This allows for the acquisition of segment features closely related to the search text. By predicting the target saliency scores of each video segment based on these segment features, the saliency of multiple video segments can be determined. Therefore, the salient video segments can be identified using the target saliency scores. Furthermore, using these salient video segments as prior preference guidance, video content retrieval can be performed based on these salient video segments and their respective segment features to obtain the predicted video segment of the video content. This entire process, using the salient video segments obtained from content saliency detection as prior preference guidance, helps to accurately locate key events in the video to be processed and enhances the semantic understanding of these events. Simultaneously, it fully leverages the synergistic advantages between content saliency detection and video content retrieval, effectively improving the accuracy of video content location and thus enhancing the accuracy of video content retrieval.

[0280] In one exemplary embodiment, video content retrieval is performed based on significant video segments and the segment features of each of the multiple video segments to obtain a predicted video segment of the video content, including:

[0281] Extract target location features from significant video segments;

[0282] Based on the cross-attention interaction mechanism, feature interaction is performed on the segment features of multiple video segments, the segment features of salient video segments, and the target location features to obtain the target features of multiple video segments.

[0283] Based on the target features of multiple video segments, time periods are predicted to obtain the predicted video time periods of the video content.

[0284] For example, the terminal determines the target segment location information of the salient video segment in the video to be processed based on the segment features of the salient video segment, and then performs feature encoding based on the target segment location information to obtain the target location features of the salient video segment. Then, based on the cross-attention interaction mechanism, it performs feature interaction on the segment features of each of the multiple video segments, the segment features of the salient video segment, and the target location features to fuse the segment location and target location features of the salient video segment with the segment features of each of the multiple video segments to obtain the target features of each of the multiple video segments. Finally, it performs time period prediction based on the target features of each of the multiple video segments to obtain the predicted video time period of the video content.

[0285] In practical applications, during feature interaction, the terminal performs linear transformations on the segment features of multiple video segments based on a cross-attention interaction mechanism to obtain a fourth key vector and a fourth value vector. It then performs linear transformations on the segment features and target location features of salient video segments to obtain a fourth query vector. Attention weights are calculated based on the fourth query vector and the fourth key vector to obtain an attention weight matrix. The fourth value vector is then weighted based on the attention weight matrix to obtain the fourth interaction feature. By processing the fourth interaction feature through residual connections and layer normalization, the target features of multiple video segments can be obtained.

[0286] In a specific application, in order to fully integrate the segment features and target location features of salient video segments with the segment features of multiple video segments, the terminal can perform multiple rounds of feature interaction based on the cross-attention interaction mechanism to obtain the target features of multiple video segments.

[0287] In practical applications, when predicting the predicted video time period based on the target features of multiple video segments, the terminal outputs the range of multiple video segment intervals and predicts the confidence level of each interval. This confidence level can be understood as the degree of confidence that the video segment interval belongs to the video content retrieved by the search text. After determining the confidence levels of each video segment interval, the terminal selects the interval with the highest confidence level as the predicted video content for the video content retrieval. The range of the selected interval is then used as the predicted video time period, and the selected interval is designated as the foreground video segment, while the remaining video segments in the video to be processed are designated as background video segments.

[0288] In a specific application, a video segment interval refers to an interval that includes at least one video segment. The number of video segments it contains may be one or more. The interval range has the same form as the time period label, specifically [c, m], where c is the midpoint of the time period within the video to be processed, and m is the length of the time period within the video to be processed. In this embodiment, the number of video segment intervals output by the video content retrieval task is the same as the number of segment features k required for the query in the video content retrieval task, where k is a positive integer. In this embodiment, feature interaction based on a cross-attention interaction mechanism can use the segment features of salient video segments and target location features as prior preference guidance to help accurately locate key events in the video to be processed and enhance the semantic understanding of the events. This yields target features of video segments that integrate the features of salient video segments. Furthermore, by performing time period prediction based on the target features of multiple video segments, accurate time period prediction of the video content can be achieved, obtaining the predicted video time period of the video content.

[0289] In one exemplary embodiment, the video processing method further includes:

[0290] The predicted video segments are obtained through a video processing model, which is trained using the aforementioned video processing model training method.

[0291] For example, the predicted video segment of the video content can be obtained through a video processing model. The video processing model is obtained through the above-mentioned video processing model training method. Then, after the terminal obtains the text to be processed and the searched text, it can directly use the video processing model to process the text to be processed and the searched text to obtain the predicted video segment of the video content.

[0292] In practical applications, the terminal first needs to extract features from the video to be processed and the retrieved text separately. After obtaining the video features of each video segment divided from the video to be processed and the text features extracted from the retrieved text, the terminal then inputs the video features of each video segment divided from the video to be processed and the text features extracted from the retrieved text into the video processing model to obtain the predicted video segment of the video content.

[0293] In this embodiment, by using a video processing model that can be used to accurately retrieve video content, accurate prediction of video time periods can be obtained, thereby improving the accuracy of video content retrieval.

[0294] In an exemplary embodiment, taking the prediction of video segments of video content obtained through a video processing model as an example, the video processing method of this application can be as follows: Figure 10 As shown, the terminal first extracts features from the video to be processed and the retrieved text separately, obtaining video features from multiple video segments divided from the video to be processed and text features from the retrieved text. Then, these features are input into the video processing model. Through the encoder in the video processing model, based on a cross-attention interaction mechanism, feature interaction is performed between the video features of each video segment and the text features extracted from the retrieved text to determine the segment features of each video segment. Based on the segment features of each video segment and the learned second saliency marker features, the data is processed through fully connected layers, linear layers, and activation functions. The algorithm performs content saliency detection on multiple video segments, predicts the target saliency score for each segment, and ranks the video segments based on their target saliency scores. It then identifies the salient video segments based on the ranking and extracts their target location features. A decoder, using a cross-attention interaction mechanism, performs feature interaction between the segment features of each video segment, the segment features of the salient video segments, and the target location features to obtain the target features for each video segment. Finally, through two fully connected layers at the decoder output, it predicts the predicted video segment based on the target features of each video segment, thus obtaining the predicted video segment of the video content.

[0295] In this system, one fully connected layer at the decoder output is used to output the range of multiple video segment intervals based on the target features of each video segment. Another fully connected layer is used to predict the confidence level of each video segment interval based on the target features of each video segment. Here, confidence level can be understood as the degree of confidence that the video segment interval belongs to the video content retrieved by the search text. After determining the confidence levels of each video segment interval, the terminal selects the video segment interval with the highest confidence level as the predicted video content for video content retrieval. The range of the selected video segment interval is used as the predicted video segment of the video content, and the selected video segment interval is used as the foreground video segment. The remaining video segments in the video to be processed are used as the background video segments. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0296] Based on the same inventive concept, embodiments of this application also provide a video processing model training apparatus for implementing the video processing model training method described above, and a video processing apparatus for implementing the video processing method described above. The solutions provided by these video processing model training apparatus and video processing apparatus are similar to the solutions described in the above methods. Therefore, the specific limitations of one or more embodiments of the video processing model training apparatus and video processing apparatus provided below can be found in the limitations of the video processing model training method and video processing method described above, and will not be repeated here.

[0297] In one exemplary embodiment, such as Figure 11 As shown, a video processing model training device is provided, including: a sample acquisition module 1102, a sample feature extraction module 1104, a saliency prediction module 1106, a content retrieval prediction module 1108, a loss calculation module 1110, and a model update module 1112, wherein:

[0298] The sample acquisition module 1102 is used to acquire training samples. The training samples include sample videos, sample description text, and time period labels. The sample description text describes the part of the video content marked by the time period labels in the sample video.

[0299] The sample feature extraction module 1104 is used to determine the segment features of each of the multiple sample segments based on the sample description text and the video features of multiple sample segments divided from the sample video using the model to be trained.

[0300] The saliency prediction module 1106 is used to predict the content saliency score of each of the multiple sample segments based on the segment features of each of the multiple sample segments through the model to be trained, and to determine the salient segments among the multiple sample segments based on the content saliency score.

[0301] The content retrieval and prediction module 1108 is used to retrieve video content based on the salient segments of the samples and the segment features of each of the multiple sample segments using the model to be trained, and to obtain the sample prediction time period of some video content.

[0302] The loss calculation module 1110 is used to determine the video retrieval loss value based on the difference between the sample prediction time period and the time period label, and to determine the significance loss value based on the content significance scores of multiple sample segments.

[0303] The model update module 1112 is used to update the model to be trained based on the video retrieval loss value and the saliency loss value until the model to be trained is completed.

[0304] The aforementioned video processing model training device, based on the acquired training samples, uses the model to be trained to determine the segment features of each of the multiple sample segments according to the sample description text and the video features of multiple sample segments divided from the sample video. It can obtain segment features closely related to the sample description text. By predicting the content saliency scores of each of the multiple sample segments based on their respective segment features, it can determine the saliency of multiple sample segments using their individual segment features. Thus, it can use the content saliency scores to identify the salient segments among the multiple sample segments. Furthermore, it can use the salient segments as prior preference guidance to perform video content retrieval based on the salient segments and the segment features of each of the multiple sample segments, obtaining the sample predicted time periods of some video content marked by time period tags. Then, it can use the video retrieval loss value determined by the difference between the sample predicted time periods and the time period tags, and the saliency loss value determined by the content saliency scores of each of the multiple sample segments, to update the model to be trained until the model to be trained is completed, resulting in a video processing model that can be used to accurately retrieve video content. The entire process, guided by salient fragments obtained from content saliency detection as prior preferences, helps to accurately locate key events in sample videos and enhances the semantic understanding of events. At the same time, it fully leverages the synergistic advantages between content saliency detection and video content retrieval, effectively improving the accuracy of video content location. This results in a video processing model that can be used to accurately retrieve video content, thereby improving the accuracy of video content retrieval.

[0305] In an exemplary embodiment, the content retrieval prediction module is further configured to extract the positional features of salient segments of the samples through the model to be trained, and perform feature interaction on the segment features of each of the multiple sample segments, the segment features of the salient segments of the samples, and the positional features based on the cross-attention interaction mechanism to obtain the target features of each of the multiple sample segments, and perform time period prediction based on the target features of each of the multiple sample segments to obtain the sample prediction time period of part of the video content.

[0306] In an exemplary embodiment, the content retrieval prediction module is further configured to determine the segment location information of the salient segments in the sample video based on the segment features of the salient segments of the sample using the model to be trained, and to perform feature encoding based on the segment location information to obtain the location features of the salient segments of the sample.

[0307] In an exemplary embodiment, the saliency prediction module is further configured to perform feature mapping on the segment features of multiple sample segments divided from the sample video using the model to be trained, determine the saliency features of each of the multiple sample segments, obtain the first saliency marker features learned based on the multiple sample segments and sample description text, determine the content saliency score of each of the multiple sample segments based on the first saliency marker features and the saliency features of each of the multiple sample segments, sort the multiple sample segments according to the content saliency scores, and determine the salient sample segments among the multiple sample segments according to the sorting results.

[0308] In an exemplary embodiment, the video processing model training apparatus further includes a time period prediction module. The time period prediction module is used to predict the time period to which the video content belongs based on the segment features of multiple sample segments divided from the sample video using the model to be trained. The loss calculation module is also used to obtain the saliency labels of the multiple sample segments, determine the saliency category loss value based on the saliency labels and content saliency scores of the multiple sample segments, determine the time period prediction loss value based on the time period to which the video content belongs and the difference in time period labels, and determine the saliency loss value based on the saliency category loss value and the time period prediction loss value.

[0309] In an exemplary embodiment, the loss calculation module is further configured to determine a saliency score loss value based on the content saliency scores and time period labels of each of the multiple sample segments, and to determine a saliency loss value based on the saliency category loss value, the time period prediction loss value, and the saliency score loss value.

[0310] In an exemplary embodiment, the loss calculation module is further configured to determine the maximum and minimum content saliency scores from the respective content saliency scores of multiple sample segments, determine a first saliency score difference between the maximum and minimum content saliency scores, and determine a second saliency score difference between sample segments within and outside the time period labels based on the time period labels, and determine a saliency score loss value based on the first and second saliency score differences.

[0311] In an exemplary embodiment, the loss calculation module is further configured to determine the video retrieval time period loss value based on the difference between the sample prediction time period and the time period label, obtain the sample segment category of each of the multiple sample segments predicted by video content retrieval, determine the video retrieval classification loss value based on the sample segment category of each of the multiple sample segments, and determine the video retrieval loss value based on the video retrieval time period loss value and the video retrieval classification loss value.

[0312] In an exemplary embodiment, the sample feature extraction module is further configured to use the model to be trained, based on a cross-attention interaction mechanism, to perform feature interaction between text features extracted from sample description text and video features of multiple sample segments divided from sample videos, and determine the segment features of each of the multiple sample segments.

[0313] In one exemplary embodiment, such as Figure 12 As shown, a video processing apparatus is provided, including: a data acquisition module 1202, a feature extraction module 1204, a saliency detection module 1206, and a video content retrieval module 1208, wherein:

[0314] The data acquisition module 1202 is used to acquire the video to be processed and the search text used to retrieve video content from the video to be processed;

[0315] The feature extraction module 1204 is used to determine the segment features of each of the multiple video segments based on the retrieved text and the video features of the multiple video segments divided from the video to be processed.

[0316] The saliency detection module 1206 is used to predict the target saliency score of each of the multiple video segments based on their respective segment features, and to determine the salient video segments among the multiple video segments based on the target saliency scores.

[0317] The video content retrieval module 1208 is used to retrieve video content based on significant video segments and the segment features of multiple video segments, and to obtain the predicted video time period of the video content.

[0318] The aforementioned video processing device, based on acquiring the video to be processed and the search text, determines the segment features of each video segment according to the search text and the video features of multiple video segments divided from the video to be processed. It can obtain segment features closely related to the search text. By predicting the target saliency score of each video segment based on these segment features, it can determine the saliency of multiple video segments. Therefore, using the target saliency score, it can identify salient video segments among the multiple video segments. Furthermore, using these salient video segments as prior preference guidance, it can perform video content retrieval based on these salient video segments and the segment features of each video segment, obtaining the predicted video segment of the video content. The entire process, using salient video segments obtained from content saliency detection as prior preference guidance, helps to accurately locate key events in the video to be processed and enhances the semantic understanding of the events. Simultaneously, it fully utilizes the synergistic advantages between content saliency detection and video content retrieval, effectively improving the accuracy of video content location and thus enhancing the accuracy of video content retrieval.

[0319] In an exemplary embodiment, the video content retrieval module is further configured to extract target location features of salient video segments, and based on a cross-attention interaction mechanism, perform feature interaction on the segment features of each of the multiple video segments, the segment features of the salient video segments, and the target location features to obtain target features of each of the multiple video segments, and perform time period prediction based on the target features of each of the multiple video segments to obtain the predicted video time period of the video content.

[0320] In an exemplary embodiment, the predicted video segment of the video content is obtained through a video processing model, which is trained using the video processing model training method described above.

[0321] The various modules in the aforementioned video processing model training device and video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0322] In one exemplary embodiment, a computer device is provided. This computer device can be a terminal or a server. Taking the computer device as a terminal as an example, its internal structure diagram can be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a video processing model training method and a video processing method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0323] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0324] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0325] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0326] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0327] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0328] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0329] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0330] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video processing model training method, characterized in that, The method includes: Obtain training samples, which include sample videos, sample description text, and time period labels. The sample description text describes a portion of the video content marked by the time period labels in the sample video. Using the model to be trained, the segment features of each of the multiple sample segments are determined based on the sample description text and the video features of multiple sample segments divided from the sample video. Using the model to be trained, based on the segment features of each of the multiple sample segments, the content saliency score of each of the multiple sample segments is predicted, and based on the content saliency score, the salient segments among the multiple sample segments are determined. Using the model to be trained, video content retrieval is performed based on the salient segments of the samples and the segment features of each of the multiple sample segments to obtain the sample prediction time period of the partial video content; The video retrieval loss value is determined based on the difference between the predicted time period of the sample and the time period label, and the significance loss value is determined based on the content significance score of each of the multiple sample segments. The training model is updated based on the video retrieval loss value and the saliency loss value until the training model is completed.

2. The method according to claim 1, characterized in that, The step of retrieving video content based on the salient segments of the samples and the segment features of each of the multiple sample segments using the model to be trained, and obtaining the sample prediction time period of the partial video content, includes: The location features of significant segments in the sample are extracted using the model to be trained. Based on the cross-attention interaction mechanism, feature interaction is performed on the segment features of each of the multiple sample segments, the segment features and positional features of the salient segments of the samples, to obtain the target features of each of the multiple sample segments. Based on the target features of each of the multiple sample segments, time period prediction is performed to obtain the sample predicted time period of the video content.

3. The method according to claim 2, characterized in that, The step of extracting the location features of significant fragments in the sample using the model to be trained includes: Using the model to be trained, the segment location information of the significant segments in the sample video is determined based on the segment features of the significant segments in the sample. Feature encoding is performed based on the fragment location information to obtain the location features of the significant fragments in the sample.

4. The method according to claim 1, characterized in that, The step of predicting the content saliency score of each of the multiple sample segments based on their respective segment features using the model to be trained, and determining the salient segments among the multiple sample segments based on the content saliency scores, includes: Using the model to be trained, feature mapping is performed on the segment features of each of the multiple sample segments to determine the salient features of each of the multiple sample segments, and the first salient label feature learned based on the multiple sample segments and the sample description text is obtained. Based on the first saliency marker feature and the saliency features of each of the plurality of sample segments, the content saliency score of each of the plurality of sample segments is determined; Based on the content significance score, the multiple sample segments are sorted, and the significant sample segments among the multiple sample segments are determined based on the sorting results.

5. The method according to claim 1, characterized in that, The method further includes: Using the model to be trained, the time period to which the video content belongs is predicted based on the segment features of each of the multiple sample segments divided from the sample video; The step of determining the significance loss value based on the content significance scores of the multiple sample segments includes: Obtain the saliency labels of each of the multiple sample fragments; Based on the saliency labels and content saliency scores of the multiple sample segments, the saliency category loss value is determined, and based on the time period to which the video content belongs and the difference in the time period labels, the time period prediction loss value is determined; The significance loss value is determined based on the significance category loss value and the time period prediction loss value.

6. The method according to claim 5, characterized in that, The determination of the saliency loss value based on the saliency category loss value and the time period prediction loss value includes: Based on the content saliency scores of each of the multiple sample segments and the time period labels, a saliency score loss value is determined; The significance loss value is determined based on the significance category loss value, the time period prediction loss value, and the significance score loss value.

7. The method according to claim 6, characterized in that, The step of determining the saliency score loss value based on the content saliency scores of each of the multiple sample segments and the time period label includes: From the content saliency scores of each of the multiple sample segments, determine the maximum and minimum content saliency scores; Determine a first significance score difference between the maximum content significance score and the minimum content significance score, and determine a second significance score difference between sample segments within the time period label and sample segments outside the time period label based on the time period label; Based on the first significance score difference and the second significance score difference, a significance score loss value is determined.

8. The method according to any one of claims 1 to 7, characterized in that, The step of determining the video retrieval loss value based on the difference between the predicted time period of the sample and the time period label includes: Based on the difference between the predicted time period and the time period label, determine the video retrieval time period loss value, and obtain the sample segment category of each of the multiple sample segments predicted by video content retrieval; Based on the respective sample segment categories of the multiple sample segments, determine the video retrieval classification loss value; The video retrieval loss value is determined based on the video retrieval time period loss value and the video retrieval classification loss value.

9. The method according to any one of claims 1 to 7, characterized in that, The step of determining the segment features of each of the multiple sample segments using the model to be trained, based on the sample description text and video features of multiple sample segments divided from the sample video, includes: Using the model to be trained, based on the cross-attention interaction mechanism, the text features extracted from the sample description text and the video features of multiple sample segments divided from the sample video are interacted to determine the segment features of each of the multiple sample segments.

10. A video processing method, characterized in that, The method includes: Obtain the video to be processed and the search text used to retrieve video content from the video to be processed; Based on the retrieved text and the video features of multiple video segments divided from the video to be processed, the segment features of each of the multiple video segments are determined; Based on the segment features of each of the multiple video segments, predict the target salience score of each of the multiple video segments, and determine the salient video segments among the multiple video segments based on the target salience scores; Based on the significant video segments and the segment features of each of the multiple video segments, video content retrieval is performed to obtain the predicted video time period of the video content.

11. The method according to claim 10, characterized in that, The step of retrieving video content based on the salient video segments and the segment features of each of the plurality of video segments to obtain the predicted video segment of the video content includes: Extract the target location features of the significant video segments; Based on the cross-attention interaction mechanism, feature interaction is performed on the segment features of each of the multiple video segments, the segment features of the salient video segments, and the target location features to obtain the target features of each of the multiple video segments. Based on the target features of each of the multiple video segments, time period prediction is performed to obtain the predicted video time period of the video content.

12. The method according to claim 10, characterized in that, The method further includes: The predicted video segment of the video content is obtained through a video processing model, which is trained using the video processing model training method described in any one of claims 1 to 11.

13. A video processing model training device, characterized in that, The device includes: The sample acquisition module is used to acquire training samples, which include sample videos, sample description text, and time period labels. The sample description text describes a portion of the video content marked by the time period labels in the sample video. The sample feature extraction module is used to determine the segment features of each of the multiple sample segments based on the sample description text and the video features of multiple sample segments divided from the sample video using the model to be trained. The saliency prediction module is used to predict the content saliency score of each of the multiple sample segments based on the segment features of each of the multiple sample segments using the model to be trained, and to determine the salient sample segments among the multiple sample segments based on the content saliency score. The content retrieval and prediction module is used to retrieve video content based on the salient segments of the samples and the segment features of each of the multiple sample segments using the model to be trained, and to obtain the sample prediction time period of the part of the video content; The loss calculation module is used to determine the video retrieval loss value based on the difference between the predicted time period of the sample and the time period label, and to determine the saliency loss value based on the content saliency scores of each of the multiple sample segments. The model update module is used to update the model to be trained based on the video retrieval loss value and the saliency loss value until the model to be trained is fully trained.

14. A video processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire the video to be processed and the search text used to retrieve video content from the video to be processed; The video feature extraction module is used to determine the segment features of each of the multiple video segments based on the retrieved text and the video features of multiple video segments divided from the video to be processed; The saliency detection module is used to predict the target saliency score of each of the multiple video segments based on their respective segment features, and to determine the salient video segments among the multiple video segments based on the target saliency scores. The video content retrieval module is used to retrieve video content based on the significant video segments and the segment features of each of the multiple video segments, and to obtain the predicted video time period of the video content.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images