Video feature model training method and apparatus, video feature extraction method and apparatus

Abstract

This specification provides a video feature model training method and apparatus, and a video feature extraction method and apparatus. In the model training method, video data is acquired, and then the video feature model is trained in the following manner until the training termination condition is met: video data is provided to the video feature model to obtain multimodal features; positive example matching scores are obtained between video frame features and text features in each positive example video pair group; positive example weights are assigned to the positive example video pair group based on the positive example matching scores; a first loss is calculated using the current video feature model in a preset proxy task based on the negative example multimodal features, the positive example multimodal features, and the corresponding positive example weights; and the video feature model is adjusted based on the first loss.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, specifically to video feature model training methods and apparatus, and video feature extraction methods and apparatus. Background Technology

[0002] Video recommendation and video search are the most widely used services on video websites. With the rise of artificial intelligence, machine learning models are replacing traditional recommendation and search methods, providing users with more accurate recommendations and search services. When machine learning models perform video recommendation or search, they need to extract features from the videos and then make recommendations and searches based on the extracted video features. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a video feature model training method and apparatus, and a video feature extraction method and apparatus. Through the technical solutions of these embodiments, the importance of positive example video pairs in video feature model training is dynamically adjusted by using positive example weights. When the positive example matching score is sufficiently high and the number of training rounds increases, the importance of positive example video pairs in video feature model training is reduced by decreasing the positive example weights, thereby increasing the importance of negative example video pairs in video feature model training. This achieves the goal of progressively learning negative examples and improves the robustness of the video feature model.

[0004] According to one aspect of an embodiment of this specification, a method for training a video feature model is provided, comprising: acquiring video data, wherein each video data includes video frame data and text data; training the video feature model in the following manner until a training termination condition is met: providing the video data to the video feature model to obtain multimodal features, wherein each multimodal feature is obtained by multimodal fusion of video frame features and text features, the multimodal features including positive multimodal features for positive video pair groups and negative multimodal features for negative video pair groups, wherein each positive video pair included in each positive video pair group is derived from the same... The video feature model is composed of video frame features and text features. Each negative example video pair group consists of video frame features and text features from different videos. A positive example matching score is obtained between the video frame features and text features in each positive example video pair group. A positive example weight is assigned to the positive example video pair group based on the positive example matching score, wherein the positive example weight decreases with increasing training rounds. Based on the negative example multimodal features, the positive example multimodal features, and the corresponding positive example weights, a first loss is calculated using the current video feature model in a preset proxy task. The video feature model is then adjusted based on the first loss.

[0005] According to another aspect of the embodiments of this specification, a method for extracting video features is also provided, wherein the video feature model used is obtained according to the training method described above, the method comprising: providing video data to be extracted to the video feature model; and obtaining video features output by the video feature model, wherein the video features fuse video frame features and text features in the video data.

[0006] According to another aspect of the embodiments of this specification, an apparatus for training a video feature model is also provided, comprising: a data acquisition unit for acquiring video data, wherein each video data includes video frame data and text data; and a data providing unit for providing the video data to the video feature model to obtain multimodal features, wherein each multimodal feature is obtained by multimodal fusion of video frame features and text features, the multimodal features including positive multimodal features for positive video pair groups and negative multimodal features for negative video pair groups, each positive video pair included in each positive video pair group consists of video frame features and text features from the same video, and each negative video pair included in each negative video pair group consists of video frame features and text features from different sources. The system comprises video frame features and text features of the same video; a scoring unit that obtains the positive example matching score between the video frame features and text features in each positive example video pair group; a weighting unit that obtains the positive example weights assigned to the positive example video pair group based on the positive example matching score, wherein the positive example weights decrease as the training rounds increase; a loss calculation unit that calculates a first loss in a preset proxy task using the current video feature model based on the negative example multimodal features, the positive example multimodal features, and the corresponding positive example weights; and a model adjustment unit that adjusts the video feature model based on the first loss, wherein the data providing unit is triggered when the training termination condition is not met, until the training termination condition is met.

[0007] According to another aspect of the embodiments of this specification, an apparatus for extracting video features is also provided, wherein the video feature model used is obtained according to any of the training methods described above, the apparatus comprising: a data providing unit for providing video data to be extracted to the video feature model; and a feature obtaining unit for obtaining video features output by the video feature model, wherein the video features are a fusion of video frame features and text features in the video data.

[0008] According to another aspect of the embodiments of this specification, an electronic device is also provided, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the video feature model training method or the video feature extraction method as described above. Attached Figure Description

[0009] A further understanding of the nature and advantages of the embodiments described in this specification can be achieved by referring to the following accompanying drawings. In the drawings, similar components or features may have the same reference numerals.

[0010] Figure 1 A flowchart illustrating an example of a method for training a video feature model according to an embodiment of this specification is shown.

[0011] Figure 2 A flowchart illustrating another example of a method for training a video feature model according to an embodiment of this specification is shown.

[0012] Figure 3 A schematic diagram illustrating an example of a matrix formed by positive example matching scores and negative example matching scores according to an embodiment of this specification is shown.

[0013] Figure 4 A flowchart illustrating an example of a method for extracting video features according to an embodiment of this specification is shown.

[0014] Figure 5 A block diagram illustrating an example of an apparatus for training a video feature model according to an embodiment of this specification is shown.

[0015] Figure 6 A block diagram of an example video feature extraction apparatus according to an embodiment of this specification is shown.

[0016] Figure 7 A block diagram of an electronic device for implementing a model training method according to an embodiment of this specification is shown.

[0017] Figure 8 A block diagram of an electronic device for implementing a model training method according to an embodiment of this specification is shown. Detailed Implementation

[0018] The subject matter described herein will be discussed below with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of the embodiments described herein. Various processes or components may be omitted, substituted, or added as needed in the various examples. Furthermore, features described in some examples may be combined in other examples.

[0019] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0020] Video recommendation and video search are the most widely used services on video websites. With the rise of artificial intelligence, machine learning models are replacing traditional recommendation and search methods, providing users with more accurate recommendations and search services. When machine learning models perform video recommendation or search, it is necessary to extract features from the videos, and then make recommendations and searches based on the extracted video features. Given the importance of video feature extraction, a training method is needed to improve the accuracy of video feature extraction models.

[0021] In view of the above, embodiments of this specification provide a video feature model training method and apparatus, and a video feature extraction method and apparatus. In the video feature model training method, video data is acquired, and the video feature model is trained in the following manner until the training termination condition is met: providing video data to the video feature model to obtain multimodal features; obtaining positive example matching scores between video frame features and text features in each positive example video pair group; obtaining positive example weights assigned to the positive example video pair group based on the positive example matching scores; calculating a first loss in a preset proxy task based on negative example multimodal features, positive example multimodal features, and corresponding positive example weights; and adjusting the video feature model based on the first loss. The technical solution of the embodiments in this specification dynamically adjusts the importance of positive video pairs in the training of video feature models by using positive example weights. When the positive example matching score is high enough and the number of training rounds increases, the importance of positive video pairs in the training of video feature models is reduced by decreasing the positive example weights, thereby increasing the importance of negative video pairs in the training of video feature models. This achieves the purpose of progressively learning negative examples and improves the robustness of video feature models.

[0022] The video feature model training method and apparatus, and the video feature extraction method and apparatus provided in the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 A flowchart of an example 100 of a method for training a video feature model according to an embodiment of this specification is shown.

[0024] like Figure 1As shown, video data can be obtained at 110.

[0025] In the embodiments of this specification, each acquired video data may include video frame data and text data. Video frame data includes the individual video frames that make up the video, and text data may include different types of text corresponding to the video, such as subtitles, text corresponding to audio played in the video, lyrics to a song played in the video, or dialogue between characters in the video. In one example, the text data in each video data set can be divided according to video frames, with each piece of text data corresponding to one video frame. The correspondence between text data and video frame data can mean that the text data is presented together with the video frame data during video playback.

[0026] In one example, the acquired video data can be categorized into three types based on the degree of consistency between video frame data and text data: good data, hard data, and noisy data. Good data refers to video data where the consistency between video frame data and text data is high. Hard data refers to video data where the consistency between video frame data and text data is low; for example, video frame data in hard data may only partially match the text data. Noisy data refers to video data where the video frame data and text data are completely inconsistent. In this specification, consistency between video frame data and text data can refer to the consistency between the meaning expressed by the text data and the content presented by the video frame data. Good data and hard data can be used to train video feature models. Noisy data, because it lacks consistency between video frame data and text data, will reduce the performance and robustness of the video feature model when learning from it.

[0027] Based on the above three types of video data, after acquiring the video data, the acquired video data can be cleaned to remove noisy data and retain high-quality data and difficult examples, which can be used as training samples to train the video feature model.

[0028] In one cleaning method, video data for each video can be cleaned separately. For each video's data, a matching score can be calculated between the video frame features and the text features. The matching score characterizes the degree of consistency between the video frame features and the text features; a higher matching score indicates a higher degree of consistency, and a lower matching score indicates a lower degree of consistency.

[0029] In one computational approach, an image-text matching model can be pre-trained. This model predicts the degree of consistency between an image and text, and outputs a matching score to characterize the consistency between the input image and text. When calculating the matching score, video frame data and text data included in the video data to be calculated can be provided to the image-text matching model to obtain the matching score output by the model.

[0030] In one example, before providing the video data to the image-text matching model, the video frame data can be sampled to obtain sampled video frame data. Sampling methods can include random sampling and sampling according to specified rules, such as interval sampling at specified intervals of a certain number of video frames. The sampled video frame data is a subset of the video frame data, and the number of sampled video frame data is less than the number of complete video frame data included in the original video data.

[0031] By sampling, the amount of video frame data provided to the image-text matching model is reduced, thereby reducing the amount of data processing required by the model and improving its processing and data cleaning efficiency.

[0032] In one training method for image-text matching models, a large-scale image-text corpus can be used to train the model. The corpus includes positive image-text pairs with high consistency (which can be used as positive examples) and negative image-text pairs with low consistency (which can be used as negative examples). The image-text matching model can include a visual feature encoding module, a text feature encoding module, a text vectorization model, and a prediction network layer. The prediction network layer can include an MLP (Multilayer Perceptron) network. Images from the image-text corpus can be provided to the visual feature encoding module to obtain corresponding image features. Text from the corpus is provided to the text vectorization model to obtain corresponding text vectors. These text vectors are then provided to the text feature encoding module to obtain corresponding text features. Next, in the MLP network, the image and text features are used to calculate the loss for the image-text matching model in the image-text matching agent task. The model is then adjusted based on the loss until training is complete.

[0033] After calculating the matching score between video frame features and text features in the video data, the matching score for each video can be obtained, and the video data can be cleaned based on the matching score of each video.

[0034] In one example, video data with low matching scores can be identified as noise data, and thus discarded as noise, retaining only high-quality data and difficult examples with high matching scores. In another example, a matching score threshold can be set to identify video data with matching scores below the threshold as noise data.

[0035] In one example, for each video, after performing sampling operations on the video frame data and calculating the corresponding video frame features and text features, this sampling and calculation operation can be repeated multiple times. Each time, the sampled video frame data will be different, resulting in different matching scores each time. For example, the number of matching score calculations for each video can be set to 3. After obtaining multiple matching scores for each video through these multiple operations, these scores can be averaged, and the average value can be used as the matching score for that video.

[0036] In step 120, the acquired video data can be provided to the video feature model to be trained to obtain multimodal features.

[0037] In the embodiments of this specification, when the acquired video data is cleaned, the data provided to the video feature model is the cleaned video data. When the acquired video data is not cleaned, all acquired video data can be provided to the video feature model.

[0038] Video feature models can include visual feature encoding modules, text feature encoding modules, text vectorization models, multimodal feature encoding modules, and prediction network layers, such as MLP networks. In one example, before providing video data to the video feature model, the video frame data for each video can be sampled. Sampling methods can include random sampling and sampling according to specified rules, such as interval sampling at specified intervals. The number of sampled video frames is less than the total number of video frames included in the corresponding video data. Different videos can use the same or different sampling methods. Sampling reduces the number of video frames provided to the video feature model, thereby reducing the data processing load and improving the processing and training efficiency of the video feature model. In another example, sampling can be omitted from the video data for each video, providing the video feature model with all the video data for each video.

[0039] The video data provided to the video feature model can include positive example video pair groups as positive examples and negative example video pair groups as negative examples. Each positive example video pair group can include multiple positive example video pairs belonging to the same video. Each positive example video pair group can include positive example video pairs corresponding to all video frame data in the corresponding video, or it can include only positive example video pairs corresponding to a portion of the video frame data in the corresponding video. The portion of the video frame data can be obtained by sampling the video frame data in the video. Each positive example video pair consists of video frame features and text features from the same video. The video frame features in each positive example video pair originate from one video frame; therefore, there is a one-to-one correspondence between positive example video pairs and video frames. In one example, the text data corresponding to the text features in each positive example video pair is presented in the video corresponding to the video frame features. The text features in different positive example video pairs belonging to the same positive example video pair group can be different. In another example, the text features in each positive example video pair are the features of all text data in the corresponding video data. Therefore, the text features in different positive example video pairs belonging to the same positive example video pair group are the same. Positive video pairs correspond one-to-one with videos. For example, a set of positive video pairs for a video may contain multiple positive video pairs, each corresponding to one video frame in that video. The video frame features and text features in each positive video pair have a high degree of consistency, making the positive video pair sets high-quality data for training video feature models.

[0040] Each negative example video pair set can include multiple negative example video pairs. The video frame features in these multiple negative example video pairs all come from the same video. However, the video frame features and text features in each negative example video pair come from different videos. In other words, each negative example video pair set corresponds to two different videos: one providing video frame features and one providing text features. The consistency between the video frame features and text features in each negative example video pair is relatively low. Negative example video pair sets can serve as hard example data for training video feature models.

[0041] Video frame data from the video data provided to the video feature model can be fed into the visual feature encoding module to obtain corresponding image features. Text data from the video data is fed into the text vectorization model to obtain corresponding text vectors. These text vectors can then be fed into the text feature encoding module to obtain corresponding text features. The dimensions of both image and text features can be specified.

[0042] Then, the image features and text features can be concatenated according to their respective video pairs (including positive and negative video pairs). Each concatenated feature corresponds to one video pair and represents the image and text features in that video pair. Next, each concatenated feature can be provided to the multimodal feature encoding module to obtain the corresponding multimodal features. The dimensionality of the multimodal features can be the same as the dimensionality of the image and text features.

[0043] In the embodiments of this specification, each multimodal feature is obtained by multimodal fusion of video frame features and text features. The obtained multimodal features may include positive multimodal features for positive video pair groups and negative multimodal features for negative video pair groups. The multimodal features obtained for positive video pair groups are positive multimodal features, and the corresponding negative multimodal features are obtained for negative video pair groups.

[0044] At 130, the positive match score between video frame features and text features in each positive video pair group can be obtained.

[0045] In one example, for each positive video pair, a pre-trained image-text matching model can be used to calculate the positive match score between the video frame features and text features in that positive video pair. Each positive video pair corresponds one-to-one with the video data provided to the video feature model, and each positive video pair can represent the corresponding video data. In one example, the positive video pair used for calculating the positive match score is the complete video data, meaning that the positive video pair includes all video frame features from the corresponding video data. In another example, the positive video pair used for calculating the positive match score is partial video data, meaning that the positive video pair includes features corresponding to only a portion of the video frames in the video data.

[0046] In one computational approach, each positive video pair group represents a video, and this group can include positive video pairs for different video frames within the video. For each positive video pair, it can be provided to a text-image matching model to obtain a positive match score, which represents the degree of consistency between the video frame features and text features in the positive video pair.

[0047] After obtaining the positive example matching score for each positive example video pair, the positive example matching score for the corresponding positive example video pair group can be obtained based on the positive example matching score of each positive example video pair. This positive example matching score can represent the degree of consistency of the videos corresponding to the positive example video pair group. In one example, the positive example matching scores of each positive example video pair can be averaged, and the mean value obtained is the positive example matching score of the positive example video pair group.

[0048] In one example, if the positive match score of each positive video pair group has been calculated in the previous training rounds, then in the subsequent training rounds, the positive match score of each positive video pair group calculated in the previous rounds can be directly obtained.

[0049] At 140, the positive weights assigned to the positive video pair groups can be obtained based on the positive match score.

[0050] In the embodiments of this specification, based on the correspondence between positive example video pairs and videos, the positive example weights assigned to positive example video pairs are also the weights of the corresponding videos in that positive example video pair. In one example, the positive example weights can be positive numbers greater than 0 and less than 1. In different training rounds, the positive example weights assigned to each video can change dynamically; for the same video, the assigned positive example weights can differ in different rounds. In one example, for each video, the assigned positive example weights can gradually decrease as the number of training rounds increases. For example, a larger positive example weight is assigned to a video in the early stages of training, and the smaller positive example weight is assigned to that video in the later stages of training. The time step for adjusting the positive example weights can be customized. For example, the positive example weights for the same video can be adjusted once every specified number of training rounds. Alternatively, the positive example weights can be adjusted once per round.

[0051] During training, the weight of positive examples assigned to each video is gradually reduced as the training rounds increase. This gradually reduces the importance of each video as a positive example in the training of the video feature model, while correspondingly increasing the importance of negative examples in the training of the video feature model. This achieves the goal of progressively learning negative examples and improves the robustness of the video feature model.

[0052] In the embodiments of this specification, the assigned positive example weight for each positive example video pair is inversely proportional to the matching score of that positive example video pair. That is, the higher the matching score, the smaller the assigned positive example weight; the lower the matching score, the larger the assigned positive example weight. The lower the matching score corresponding to a positive example video pair, the more inconsistencies there are between the image and text in that positive example video pair. By increasing the corresponding positive example weight, the importance of the inconsistencies between the image and text in the positive example video pair can be increased in the training of the video feature model, which is beneficial for the video feature model to learn from negative examples, thereby improving the robustness of the video feature model.

[0053] In one example, after determining the positive match score for each positive video pair group, the median score and minimum score can be determined from the positive match scores of each positive video pair group. The median score can be the median value among the positive match scores. Then, for each positive video pair group, the positive weight of that positive video pair group is obtained based on its positive match score, median score, and minimum score.

[0054] In one example, for each positive video pair group, the positive weight can be calculated using the following formula:

[0055]

[0056] in, Z represents the positive weight corresponding to the i-th positive example video pair group. diag Z represents the set of positive match scores for each positive video pair group. Min() represents the minimum value, and Min() represents the median value. i,i Let e0 represent the positive match score corresponding to the i-th positive example video pair group, and let e0 represent a small positive value greater than 0, which can avoid the denominator being 0 in the above formula.

[0057] After calculating the positive example weights for each positive example video pair, these weights can be adjusted so that they decrease as the training epochs increase. In one example, for a positive example video pair, if the calculated positive example weights are the same as those used in previous training epochs, the calculated positive example weights for this epoch can be reduced, and the reduced positive example weights can be used as the positive example weights assigned to that positive example video pair.

[0058] In one example, the positive weights assigned to the positive video pair group can be obtained based on the positive match score and the dynamic change coefficient.

[0059] In this example, the positive example matching score can be used to calculate the positive example weight. For instance, the positive example matching score can be used to calculate the positive example weight using the formula described above. The dynamic change coefficient can be used to adjust the calculated positive example weight, and the adjusted positive example weight can be used as the positive example weight assigned to the positive example video pair group.

[0060] The dynamic variation coefficient can increase within a specified range as the training epochs increase. The specified range can be from 0 to 1, including a minimum of 0 and a maximum of 1. The positive example weights can be adjusted using the dynamic variation coefficient according to the following formula:

[0061]

[0062] Where β represents the dynamic variation coefficient. This represents the weight of positive examples after adjustment by the dynamic change coefficient.

[0063] In the above adjustment method, as the dynamic change coefficient gradually increases, the weights of the adjusted positive examples gradually decrease. The dynamic change coefficient can be gradually increased according to a specified time step, which can be represented by the number of training epochs. For example, the dynamic change coefficient can be increased once every specified number of training epochs. In one example, the dynamic change coefficient can be increased by a fixed value each time an increase operation is performed.

[0064] It should be noted that the execution order between 120 and 130 and 140 is not limited. That is, the operation of 120 can be executed first, followed by the operations of 130 and 140; or the operations of 130 and 140 can be executed first, followed by the operation of 120.

[0065] In 150, the first loss can be calculated in the preset agent task using the current video feature model based on the negative example multimodal features and positive example multimodal features and the corresponding positive example weights.

[0066] In the embodiments of this specification, negative multimodal features and positive multimodal features, along with their corresponding positive weights, can be provided to a prediction network layer (e.g., MLP) in a video feature model. The prediction network layer processes the input and outputs the degree of consistency of the multimodal features of the input. A first loss is calculated based on this degree of consistency.

[0067] In one example, the preset proxy task may include a video-text matching task (VTM), in which the loss can be calculated based on the degree of consistency between the video frames and the text in the video.

[0068] In one calculation method, the first loss is calculated according to the following formula:

[0069]

[0070] Where L1 represents the first loss, |B| represents the number of input features, θ represents the prediction network layer, and exp represents the exponential function with the natural constant e as the base. This represents positive multimodal features. This represents the multimodal features of negative examples.

[0071] At 160, the video feature model can be adjusted based on the first loss.

[0072] In one example, when the preset proxy task includes a video-text matching proxy task, for each video pair (including positive and negative video pairs), the video frame features and text features in that video pair can be multiplied to obtain a product matching score for that video pair. The product matching score represents the degree of consistency between the video frame features and text features in the corresponding video pair.

[0073] In one calculation method, each video pair group can include multiple video pairs, and each video pair includes video frame features and corresponding text features. The video frame features and text features in each video pair can be multiplied accordingly to obtain the product matching score of that video pair. Then, the product matching scores of all video pairs included in the targeted video pair group can be averaged, and the resulting mean can be used as the product matching score of the video pair group.

[0074] In one example, the second loss can be calculated using the following formula:

[0075]

[0076] Where <,> denotes matrix multiplication operations on vector features, for example,<a,b> =a×b T T denotes the transpose of the matrix. V Dimensions representing video frame features Represents video frame features, Represents text features. This represents the matrix product between the video frame features and the text features in a positive video pair.

[0077] After obtaining the first and second losses in the video-text matching proxy task, the video feature model can be adjusted based on these two losses. In one example, the sum of the first and second losses can be used as the total loss, and then the video feature model can be adjusted based on this total loss.

[0078] It should be noted that the order in which the operation of calculating the second loss is performed is not limited to the operation of calculating the first loss.

[0079] In the embodiments of this specification, during the loss calculation process, the loss can be calculated based on only one preset proxy task, or it can be calculated based on multiple proxy tasks. Each proxy task can include multiple different loss calculation methods, so that different losses can be obtained in each proxy task according to different loss calculation methods. For example, in the video text matching task mentioned above, the first loss and the second loss can be obtained based on two different loss calculation methods, respectively.

[0080] In one example, in addition to the preset proxy task, a missing word modeling (MLM) proxy task can be added. In the missing word modeling proxy task, text features and / or positive example multimodal features can be processed. A portion of the text content in the input text features and / or positive example multimodal features can be masked, and the prediction network layer can predict the masked content. The masked portion can be a specified proportion, for example, 15% of the text content can be masked.

[0081] In loss calculation methods targeting text features, a portion of the text features can be masked. This masked text feature is then input into a prediction network layer, which predicts the masked content. A third loss can then be calculated based on the predicted value and the label value. In one example, the third loss can be calculated using the following formula:

[0082]

[0083] Where L3 represents the third loss, σ represents the set of masked words, q represents one of the masked words, and L... CE () represents a loss function, such as the Cross-Entropy loss function. q Represents the label value, t q This represents the predicted value for text features.

[0084] In loss calculation methods targeting positive multimodal features, a portion of the text included in the positive multimodal features can be masked. This masked positive multimodal feature is then input into a prediction network layer, which predicts the masked text content. A fourth loss can then be calculated based on the predicted value and the label value. In one example, the fourth loss can be calculated using the following formula:

[0085]

[0086] Where L4 represents the fourth loss, m q This represents the predicted value for text features.

[0087] In one example, a missing word modeling proxy task can be performed simultaneously on text features and positive multimodal features, in which case the third and fourth losses can be obtained.

[0088] After obtaining the third and / or fourth losses through the missing word filling modeling proxy task, the video feature model can be adjusted based on the first loss and the third and / or fourth losses. In one example, the sum of the first, third, and / or fourth losses can be used as the total loss, and then the video feature model can be adjusted based on the total loss.

[0089] In one example, the video feature model can also be adjusted based on a first loss, a second loss, a third loss, and a fourth loss. In another example, the sum of the first, second, third, and fourth losses can be used as the total loss, and then the video feature model can be adjusted based on the total loss.

[0090] At step 170, it can be determined whether the training termination condition is met. If not, return to step 120 above; if yes, the training of the video feature model ends.

[0091] In the embodiments of this specification, the training termination condition may include at least one of the following conditions: the number of training rounds reaches a specified number of rounds threshold, the training duration reaches a specified duration threshold, and the loss is less than a specified loss threshold.

[0092] Figure 2 A flowchart of another example 200 of a method for training a video feature model according to an embodiment of this specification is shown.

[0093] like Figure 1 As shown, video data can be obtained at 210.

[0094] In step 220, the acquired video data can be provided to the video feature model to be trained to obtain multimodal features.

[0095] At 230, the positive match score between video frame features and text features in each positive video pair group can be obtained.

[0096] At 240, the positive weights assigned to the positive video pair groups can be obtained based on the positive match score.

[0097] In this example, the operations from 210 to 240 are the same as... Figure 1 The operations for 110 to 140 in the example are the same, and you can refer to the operation instructions for 110 to 140.

[0098] At 250, the negative example matching score between video frame features and text features in each negative example video pair group can be obtained.

[0099] In the embodiments of this specification, each negative example video pair group may include video frame features from a first video and text features from a second video, wherein the first video and the second video are different. The video frame features from the first video may include features corresponding to each video frame included in the first video, and each video frame may correspond to one video frame feature. Thus, the video frame features in each negative example video pair group can constitute a video frame feature group, which includes video frame features for each video frame.

[0100] In one example, for each negative video pair, a pre-trained image-text matching model can be used to calculate the positive example matching score between the video frame features and text features in that negative video pair. Each negative video pair can correspond to one negative example matching score.

[0101] In one example, if the negative example matching score of each negative example video pair group has been calculated in the previous training rounds, then in the subsequent training rounds, the negative example matching score of each negative example video pair group calculated in the previous rounds can be directly obtained.

[0102] Figure 3 A schematic diagram illustrating an example of a matrix formed by positive and negative example matching scores according to embodiments of this specification is shown. Figure 3 As shown, the positive and negative example matching scores form a matrix, where each row corresponds to a video frame feature and each column corresponds to a text feature. The z-column on the diagonal... 1,1 z 2,2 ..., z i,i These are the positive example matching scores, and the others are the negative example matching scores. Each negative example video pair and each positive example video pair in the same row includes video frame features from the same video. For example, the matching score z in the i-th row... i,1 z i,2 ..., z i,i Each negative example video pair and each positive example video pair includes video frame features from the same video.

[0103] In 260, each negative example video pair can be sampled based on the negative example matching score.

[0104] For each video frame feature group, there can be one positive video pair group, which consists of the video frame data and text data of that video. Furthermore, each video frame feature group can have multiple negative video pair groups. These multiple negative video pair groups all include the same video frame feature group from that video. Additionally, these multiple negative video pair groups also include text feature groups from different videos; different text feature groups from different videos generate different negative video pair groups. Thus, the positive video pair groups including the same video frame feature group and the multiple negative video pair groups constitute a video pair set for that video frame feature group.

[0105] In a set of video pairs targeting a set of video frame features, the lower the negative example matching score of a negative example video pair, the lower the consistency between the video and text data within that negative example video pair. Conversely, the higher the negative example matching score, the higher the consistency between the video and text data within that negative example video pair, making it closer to the positive example video pair and thus more easily confused with it. Therefore, video feature models face greater difficulty in processing negative example video pairs that are closer to the positive example video pairs. In other words, the higher the negative example matching score of a negative example video pair, the more difficult it is to process. Therefore, it is advisable to sample negative example video pairs with high negative example matching scores. These negative example video pairs with high negative example matching scores can be used as counterexamples to train the video feature model, thereby improving the training performance of the video feature model.

[0106] In one example, for each video frame feature group, the negative example video pair group to be sampled can be sampled according to the negative example matching degree score of the negative example video pair group to be sampled that includes the video frame feature group, so as to obtain a specified number of negative example video pair groups that include the video frame feature group.

[0107] In this example, the number of negative example video pairs corresponding to each video frame feature group is the same. Each negative example video pair corresponding to a video frame feature group refers to a negative example video pair that includes that video frame feature group. Each negative example video pair corresponding to a video frame feature group can be used as a sampled negative example video pair for that video frame feature group.

[0108] The specified number is less than the number of negative video pairs corresponding to each video frame feature group, thereby reducing the data processing of the video feature model for negative video pairs. Independent sampling operations can be performed for each video frame feature group.

[0109] In one example, for each video frame feature group, a specified number of negative video pairs with the highest negative match scores can be sampled from the group of negative video pairs to be sampled that includes that video frame feature group.

[0110] In one example, sampling for each negative video pair can be performed in every training round, or it can be performed only in a portion of the training rounds.

[0111] Before performing sampling operations on each negative video pair, a random probability can be used to determine whether to perform the sampling operation. This random probability can increase with the number of training epochs; that is, the random probability is greater in later training epochs, thus increasing the probability of performing sampling operations on each negative video pair in later training epochs.

[0112] In one example, when introducing a dynamic coefficient that increases with each training epoch, a random number can be generated during each training epoch and compared to the dynamic coefficient. If the random number is less than the dynamic coefficient, sampling for each negative video pair is performed in that epoch; if the random number is not less than the dynamic coefficient, sampling for each negative video pair is not performed in that epoch. When the dynamic coefficient is limited to increasing within a specified range, random numbers can be generated within that specified range.

[0113] In this example, since the dynamic change coefficient increases with the number of training rounds, the probability of generating random numbers smaller than the dynamic change coefficient increases with the number of training rounds, thereby increasing the probability of performing sampling operations for each negative video pair.

[0114] In 270, the first loss can be calculated in the preset agent task based on the negative multimodal features and positive multimodal features corresponding to the sampled negative video pair groups, as well as the corresponding positive weights.

[0115] The method for calculating the first loss in this example can be referenced above. Figure 1 The method for calculating the first loss is described above. In this example, by sampling, the number of negative video pairs processed by the video feature model is reduced, thereby improving the processing performance of the video feature model. Furthermore, by sampling the negative video pairs with the highest negative match scores, a high-difficulty negative video pair set for training is obtained, which is beneficial for training a more accurate video feature model and improving the robustness of the video feature model.

[0116] At 280, the video feature model can be adjusted based on the first loss.

[0117] At step 290, it can be determined whether the training termination condition is met. If not, return to step 220 above; if yes, the training of the video feature model ends.

[0118] In this example, the operations of 280 and 290 are the same as... Figure 1 The operations for 160 and 170 in the example are the same; please refer to the operation instructions for 160 and 170.

[0119] The technical solution of the embodiments in this specification dynamically adjusts the importance of positive video pairs in the training of video feature models by using positive example weights. When the positive example matching score is high enough and the number of training rounds increases, the importance of positive video pairs in the training of video feature models is reduced by decreasing the positive example weights, thereby increasing the importance of negative video pairs in the training of video feature models. This achieves the purpose of progressively learning negative examples and improves the robustness of video feature models.

[0120] Figure 4 A flowchart of an example 400 of a method for extracting video features according to an embodiment of this specification is shown.

[0121] Figure 4 The video feature model used in the example shown can be a video feature model trained according to the training method provided in the embodiments of this specification.

[0122] like Figure 4 As shown in Figure 410, the video data to be extracted can be provided to the video feature model.

[0123] At 420, the video features output by the video feature model are obtained. The output video features fuse video frame features and text features from the video data. The output video features can be used in at least one application scenario, such as video allocation, video retrieval, and video recommendation.

[0124] Figure 5 A block diagram of an example of an apparatus (model training apparatus 500) for training a video feature model according to an embodiment of this specification is shown.

[0125] like Figure 5 As shown, the model training device 500 includes: a data acquisition unit 510, a data provision unit 520, a score acquisition unit 530, a weight acquisition unit 540, a loss calculation unit 550, and a model adjustment unit 560.

[0126] The data acquisition unit 510 can be configured to acquire video data, wherein each piece of video data includes video frame data and text data.

[0127] The data providing unit 520 can be configured to provide video data to the video feature model to obtain multimodal features, wherein each multimodal feature is obtained by multimodal fusion of video frame features and text features. The multimodal features include positive multimodal features for positive video pair groups and negative multimodal features for negative video pair groups. Each positive video pair included in each positive video pair group consists of video frame features and text features from the same video, and each negative video pair included in each negative video pair group consists of video frame features and text features from different videos.

[0128] The scoring unit 530 can be configured to obtain the positive match score between video frame features and text features in each positive video pair group.

[0129] The weight acquisition unit 540 can be configured to obtain the positive weights assigned to the positive video pair group based on the positive match score, wherein the positive weights decrease as the training rounds increase.

[0130] The loss calculation unit 550 can be configured to calculate the first loss in a preset agent task based on the negative example multimodal features and positive example multimodal features and the corresponding positive example weights, using the current video feature model.

[0131] The model adjustment unit 560 can be configured to adjust the video feature model based on the first loss, wherein the data providing unit 520 is triggered when the training termination condition is not met, until the training termination condition is met.

[0132] In one example, the model training device 500 may further include a sampling unit. The scoring unit may also be configured to obtain a negative example matching score between video frame features and text features in each negative example video pair group. The sampling unit may be configured to sample each negative example video pair group based on the negative example matching score. The loss calculation unit may also be configured to calculate a first loss in a preset proxy task based on the negative example multimodal features and positive example multimodal features corresponding to the sampled negative example video pair groups, as well as the corresponding positive example weights.

[0133] In one example, the sampling unit can also be configured to: for each video frame feature group, sample the negative example video pair group to be sampled according to the negative example matching degree score of the negative example video pair group to be sampled that includes the video frame feature group, so as to obtain a specified number of negative example video pair groups that include the video frame feature group.

[0134] In one example, the model training device 500 may further include an execution judgment unit, which can be configured to: determine, based on a random probability, whether to perform a sampling operation for each negative example video pair; if the judgment result is yes, then trigger the score acquisition unit 530 to perform the operation of obtaining the negative example matching score. The random probability increases with the increase of training epochs.

[0135] In one example, the weight acquisition unit 540 can also be configured to: obtain the positive weights assigned to the positive video pair group based on the positive match score and a dynamic change coefficient that increases within a specified numerical range as the training rounds increase.

[0136] In one example, the weight acquisition unit 540 can also be configured to: determine the median score and the minimum score from the positive match scores of each positive video pair group; and for each positive video pair group, obtain the positive weight of the positive video pair group based on the positive match score, the median score, and the minimum score of the positive video pair group.

[0137] In one example, the preset proxy task includes a video-text matching proxy task. The loss calculation unit 550 can also be configured to calculate a second loss in the video-text matching task based on the product matching score of each video pair group. Here, the video pair includes positive and negative video pair groups, and the product matching score of each video pair group is obtained by multiplying the video frame features in that video pair group with the corresponding text features. The model adjustment unit 560 can also be configured to adjust the video feature model based on the first and second losses.

[0138] In one example, the loss calculation unit 550 can also be configured to calculate the corresponding third and / or fourth losses in the missing word modeling proxy task based on text features and / or positive multimodal features. The model adjustment unit 560 can also be configured to adjust the video feature model based on the first loss and the third and / or fourth losses.

[0139] Figure 6 A block diagram of an example of a video feature extraction apparatus 600 according to an embodiment of this specification is shown.

[0140] The video feature model used by the video feature extraction device 600 can be obtained according to the training method provided in the embodiments of this specification. For example... Figure 6 As shown, the video feature extraction device 600 includes a data providing unit 610 and a feature acquisition unit 620.

[0141] The data providing unit 610 can be configured to provide the video data to be extracted to the video feature model.

[0142] The feature acquisition unit 620 can be configured to obtain video features output by the video feature model, wherein the video features are a fusion of video frame features and text features in the video data.

[0143] Reference above Figures 1 to 6 This specification describes embodiments of the video feature model training method and apparatus, and the video feature extraction method and apparatus according to the embodiments of this specification.

[0144] The video feature model training device and video feature extraction device in the embodiments of this specification can be implemented in hardware, software, or a combination of hardware and software. Taking software implementation as an example, as a logical device, it is formed by the processor of the device in which it resides reading the corresponding computer program instructions from the memory into memory and executing them. In the embodiments of this specification, the video feature model training device and video feature extraction device can be implemented, for example, using electronic devices.

[0145] Figure 7 A block diagram of an electronic device 700 for implementing a model training method according to an embodiment of this specification is shown.

[0146] like Figure 7 As shown, the electronic device 700 may include at least one processor 710, a memory (e.g., non-volatile memory) 720, a RAM 730, and a communication interface 740, and the at least one processor 710, memory 720, RAM 730, and communication interface 740 are connected together via a bus 750. The at least one processor 710 executes at least one computer-readable instruction (i.e., the elements implemented in software above) stored or encoded in the memory.

[0147] In one embodiment, computer-executable instructions are stored in memory, which, when executed, cause at least one processor 710 to: acquire video data, wherein each video data includes video frame data and text data; train a video feature model until a training termination condition is met by: providing video data to the video feature model to obtain multimodal features; obtaining positive example matching scores between video frame features and text features in each positive example video pair group; obtaining positive example weights assigned to the positive example video pair group based on the positive example matching scores, wherein the positive example weights decrease with increasing training epochs; calculating a first loss in a preset proxy task using the current video feature model based on the negative example multimodal features and positive example multimodal features and the corresponding positive example weights; and adjusting the video feature model based on the first loss.

[0148] Figure 8 A block diagram of an electronic device 800 for implementing a model training method according to an embodiment of this specification is shown.

[0149] like Figure 8 As shown, the electronic device 800 may include at least one processor 810, a memory (e.g., non-volatile memory) 820, a RAM 830, and a communication interface 840, and the at least one processor 810, memory 820, RAM 830, and communication interface 840 are connected together via a bus 850. The at least one processor 810 executes at least one computer-readable instruction (i.e., the elements implemented in software above) stored or encoded in the memory.

[0150] In one embodiment, computer-executable instructions are stored in memory that, when executed, cause at least one processor 810 to: provide video data to be extracted to a video feature model; and obtain video features output by the video feature model, wherein the video features incorporate video frame features and text features from the video data.

[0151] It should be understood that the computer-executable instructions stored in the memory, when executed, cause at least one processor 710 and processor 810 to perform the above combinations as described in the various embodiments of this specification. Figure 1-6 The description includes various operations and functions.

[0152] According to one embodiment, a program product, such as a machine-readable medium, is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figure 1-6 The description includes various operations and functions.

[0153] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0154] In this case, the program code itself, which can be read from a readable medium, can perform the functions of any of the above embodiments. Therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0155] The computer program code required for the operation of each part of this manual can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB, .NET, and Python; conventional procedural programming languages ​​such as C, Visual Basic 2003, Perl, COBOL 2002, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service, such as Software as a Service (SaaS).

[0156] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0157] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0158] Not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted as needed. The execution order of each step is not fixed and can be determined as required. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0159] The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" over other embodiments. Detailed descriptions are included for the purpose of providing an understanding of the described techniques. However, these techniques may be practiced without these detailed descriptions. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0160] The optional embodiments of the present specification have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present specification are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present specification, various simple modifications can be made to the technical solutions of the embodiments of the present specification, and these simple modifications all fall within the protection scope of the embodiments of the present specification.

[0161] The foregoing description of this specification is provided to enable any person skilled in the art to implement or use the content of this specification. Various modifications to the content of this specification will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of protection of this specification. Therefore, this specification is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for training a video feature model, comprising: Acquire video data, where each video data includes video frame data and text data; Train the video feature model as follows until the training termination condition is met: The video data is provided to the video feature model to obtain multimodal features, wherein each multimodal feature is obtained by multimodal fusion of video frame features and text features. The multimodal features include positive multimodal features for positive video pair groups and negative multimodal features for negative video pair groups. Each positive video pair group consists of video frame features and text features from the same video, and each negative video pair group consists of video frame features and text features from different videos. The positive example matching score between video frame features and text features in each positive example video pair group is obtained; The positive example weights assigned to the positive example video pair group are obtained based on the positive example matching score, wherein the positive example weights decrease as the training rounds increase; Based on the negative example multimodal features and the positive example multimodal features, along with their corresponding positive example weights, a first loss is calculated using the current video feature model in a preset agent task; and The video feature model is adjusted based on the first loss.

2. The method of claim 1, further comprising: The negative example matching score between video frame features and text features in each negative example video pair group is obtained; as well as The negative example video pairs are sampled based on the negative example matching score. Based on the negative example multimodal features and the positive example multimodal features, along with their corresponding positive example weights, the calculation of the first loss in the preset proxy task includes: Based on the negative multimodal features corresponding to the negative video pair groups obtained after sampling, the positive multimodal features, and the corresponding positive weights, the first loss is calculated in the preset agent task.

3. The method as described in claim 2, wherein, Sampling of each negative example video pair group based on the negative example matching score includes: For each video frame feature group, the negative example video pair group to be sampled is sampled according to the negative example matching degree score of the negative example video pair group to be sampled that includes the video frame feature group, so as to obtain a specified number of negative example video pair groups that include the video frame feature group.

4. The method of claim 3, wherein, For each video frame feature group, the negative example video pair group to be sampled is sampled according to the negative example matching score of the negative example video pair group to be sampled, which includes the video frame feature group, to obtain a specified number of negative example video pair groups including the video frame feature group, including: For each video frame feature group, sample a specified number of negative video pairs with the highest negative example matching score from the negative example video pair group that includes that video frame feature group.

5. The method of claim 2, further comprising: Whether to perform sampling operations for each negative example video pair is determined based on a random probability, wherein the random probability increases with the increase of training rounds.

6. The method of claim 5, wherein, Determining whether to perform a resampling operation for each negative video pair based on random probability includes: With the introduction of a dynamic variation coefficient, random numbers belonging to a specified numerical range are generated in each training round, and the dynamic variation coefficient increases within the specified numerical range as the number of training rounds increases. When the random number is less than the dynamic change coefficient, a sampling operation is performed for each negative example video pair in this round of training; and When the random number is not less than the dynamic change coefficient, sampling operations for each negative video pair are not performed in this round of training.

7. The method of claim 1, wherein, The positive weights assigned to the positive video pair group based on the positive match score include: The positive example weights assigned to the positive example video pair groups are obtained based on the positive example matching score and the dynamic change coefficient that increases within a specified numerical range as the training rounds increase.

8. The method of claim 1, wherein, The positive weights assigned to the positive video pair group based on the positive match score include: Determine the median score and minimum score from the positive match scores of each positive video pair group; and For each positive example video pair, the positive example weight of the positive example video pair is obtained based on the positive example matching score, the median score, and the minimum score of the positive example video pair.

9. The method as described in claim 1, wherein the preset proxy task includes a video text matching proxy task. The method further includes: Based on the product matching score of each video pair group, a second loss is calculated in the video-text matching task, wherein the video pair includes positive example video pair groups and negative example video pair groups, and the product matching score of each video pair group is obtained by multiplying the video frame features in that video pair group by the corresponding text features; and Adjusting the video feature model based on the first loss includes: The video feature model is adjusted based on the first loss and the second loss.

10. The method of claim 1, further comprising: Based on the text features and / or the positive example multimodal features, calculate the corresponding third loss and / or fourth loss in the missing character fill modeling proxy task; as well as Adjusting the video feature model based on the first loss includes: The video feature model is adjusted based on the first loss, the third loss, and / or the fourth loss.

11. The method of claim 1, wherein, After acquiring the video data, the method further includes: For each video, calculate the matching score between the video frame features and text features in the video data; and The video data is cleaned based on the matching score corresponding to each video data point; The video data is provided to the video feature model to obtain multimodal features, including: The cleaned video data is provided to the video feature model to obtain multimodal features.

12. A method for extracting video features, wherein, The video feature model used is obtained through the training method described in any one of claims 1 to 11. The method includes: The video data to be extracted is provided to the video feature model; and The video features output by the video feature model are obtained, wherein the video features are a fusion of video frame features and text features from the video data.

13. An apparatus for training a video feature model, comprising: The data acquisition unit acquires video data, wherein each video data includes video frame data and text data; The data providing unit provides the video data to the video feature model to obtain multimodal features. Each multimodal feature is obtained by multimodal fusion of video frame features and text features. The multimodal features include positive multimodal features for positive video pair groups and negative multimodal features for negative video pair groups. Each positive video pair group consists of video frame features and text features from the same video, and each negative video pair group consists of video frame features and text features from different videos. The scoring unit obtains the positive match score between video frame features and text features in each positive video pair group; The weighting unit obtains the positive weights assigned to the positive video pair group based on the positive match score, wherein the positive weights decrease as the training rounds increase; The loss calculation unit, based on the negative example multimodal features and the positive example multimodal features and their corresponding positive example weights, calculates the first loss in a preset agent task using the current video feature model; and The model adjustment unit adjusts the video feature model based on the first loss. If the training termination condition is not met, the data providing unit is triggered until the training termination condition is met.

14. The apparatus of claim 13, further comprising a sampling unit. The scoring unit is further configured to obtain a negative example matching score between video frame features and text features in each negative example video pair group. The sampling unit samples each negative example video pair group based on the negative example matching score; The loss calculation unit is further configured to calculate a first loss in a preset agent task based on the negative multimodal features corresponding to the sampled negative video pair group, the positive multimodal features, and the corresponding positive weights.

15. The apparatus of claim 14, further comprising: The execution judgment unit determines whether to perform sampling operations for each negative example video pair based on random probability. If the judgment result is yes, the scoring unit is triggered to perform the operation of obtaining the negative example matching score. The random probability increases with the increase of training rounds.

16. An apparatus for extracting video features, wherein, The video feature model used is obtained through the training method described in any one of claims 1 to 11. The device includes: The data providing unit provides the video data to be extracted to the video feature model; and The feature acquisition unit obtains the video features output by the video feature model, wherein the video features are a fusion of video frame features and text features from the video data.

17. An electronic device comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method as described in any one of claims 1-11.

18. An electronic device comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method as claimed in claim 12.

Images