Model training method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a model training method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: acquiring video data and text data used for describing the video data; performing feature extraction on the text data to obtain text features, and performing feature extraction on each image frame in the video data to obtain image features; searching co-occurrence features in a feature space formed by the text features and the image features; determining a target probability that the co-occurrence feature appears in the feature space under the condition that the text feature appears in the feature space; determining context features whose similarity with the co-occurrence features in the feature space is greater than a threshold value, and determining aggregation features according to the target probability and the context features; and training a neural network model by using the aggregation features. According to the method and the device, the technical problem that the overall understanding of a model on a scene appearing in the video is limited due to the fact that fine-grained information in the video cannot be effectively captured in a related text video retrieval technology is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a model training method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] Text-to-video retrieval is a technology that queries video databases for relevant video clips based on text descriptions. This technology plays an important role in many practical applications. For example, in relevant learning models, learners can quickly find relevant teaching video clips based on text descriptions according to their learning needs. This can not only improve learning efficiency, but also help students review and consolidate knowledge more specifically.

[0003] However, the current field of text-video retrieval faces many challenges, which are mainly reflected in the accuracy and robustness when dealing with complex scenes and fine-grained information. Related methods have difficulty in effectively capturing the nuances and fine-grained information in videos, and their ability to understand the co-occurrence context is also limited, which limits the model's overall understanding of the scene and the effective use of the association between co-occurrence features.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a model training method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problem of limiting the model's overall understanding of the scene appearing in the video due to the inability of related text video retrieval technology to effectively capture fine-grained information in the video.

[0006] According to one aspect of an embodiment of the present application, a model training method is provided, including: acquiring video data and text data for describing the video data; performing feature extraction processing on the text data to obtain text features, and performing feature extraction processing on each image frame in the video data to obtain image features; searching for co-occurrence features in a feature space composed of text features and image features, wherein a matching relationship exists between the text features and the image features in the co-occurrence features; determining a target probability of the co-occurrence features appearing in the feature space when the text features appear in the feature space; determining context features in the feature space whose similarity with the co-occurrence features is greater than a preset threshold, and determining aggregate features based on the target probability and the context features; and training a neural network model using the aggregate features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

[0007] Optionally, determining a target probability of a co-occurrence feature appearing in a feature space when a text feature appears in the feature space includes: determining a first similarity between the text feature and the kth co-occurrence feature in the feature space, wherein k is a positive integer not greater than K, K is the number of co-occurrence features in the feature space, and K is a positive integer greater than 2; determining a second similarity between the text feature and the kth co-occurrence feature at the i-th position to obtain n second similarities, wherein i is a positive integer not greater than n, n is a channel dimension of the image feature, and n is a positive integer greater than 2; summing the n second similarities to obtain a first numerical value; determining a ratio between the first similarity and the first numerical value as a first probability corresponding to the jth co-occurrence feature to obtain K first probabilities; obtaining a mixing weight corresponding to each co-occurrence feature, and performing a weighted summation on the K first probabilities according to the mixing weight corresponding to each co-occurrence feature to obtain a target probability.

[0008] Optionally, the target probability of the co-occurrence feature appearing in the feature space when the text feature appears in the feature space is determined by the following formula: Among them, x c is the co-occurrence feature, x t is the text feature, π k is the mixed weight corresponding to the kth co-occurrence feature, s k (x c , x t ) is the first similarity between the text feature and the kth co-occurring feature in the feature space, s k (x i , x t ) is the second similarity between the text feature and the kth co-occurrence feature at the ith position.

[0009] Optionally, the neural network model is trained using aggregated features, including: convolution processing on text features, upsampling processing on image features, connecting the processed features to the aggregated features to obtain target features, and using the target features to train the neural network model.

[0010] Optionally, the target features are used to train the neural network model, including: obtaining the output result of the neural network model classifying the target features, and in the output result, taking the text features and the image features that successfully match the text features as positive samples, and taking the image features that fail to match the text features as negative samples; and training the neural network model using the positive samples, negative samples and a preset cross entropy loss function until the function value of the preset cross entropy loss function is less than a preset threshold.

[0011] Optionally, after determining the aggregate features, the method further includes: performing random inactivation processing on the aggregate features, decomposing the processed aggregate features into multiple sub-vectors, calculating the attention weight between each sub-vector and other sub-vectors; and performing weighted summation processing on the sub-vectors using the attention weight between each sub-vector and other sub-vectors.

[0012] Optionally, determining the aggregate feature according to the target probability and the context feature includes: multiplying the target probability and the context feature to obtain the aggregate feature.

[0013] According to another aspect of the embodiment of the present application, a model training device is also provided, including: an acquisition module, used to acquire video data and text data for describing the video data; a processing module, used to perform feature extraction processing on the text data to obtain text features, and perform feature extraction processing on each image frame in the video data to obtain image features; a search module, used to search for co-occurrence features in a feature space composed of text features and image features, wherein there is a matching relationship between the text features and the image features in the co-occurrence features; a first determination module, used to determine the target probability of the co-occurrence features appearing in the feature space when the text features appear in the feature space; a second determination module, used to determine the context features in the feature space whose similarity with the co-occurrence features is greater than a preset threshold, and determine the aggregate features based on the target probability and the context features; a training module, used to train the neural network model using the aggregate features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, and the storage medium includes a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the above model training method.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above model training method is executed when the program is running.

[0016] According to another aspect of the embodiments of the present application, a computer program is also provided, wherein the above model training method is implemented when the computer program is executed by a processor.

[0017] According to another aspect of the embodiments of the present application, a computer program product is also provided, which includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above model training method is implemented.

[0018] In an embodiment of the present application, video data and text data used to describe the video data are obtained; feature extraction processing is performed on the text data to obtain text features, and feature extraction processing is performed on each image frame in the video data to obtain image features; co-occurrence features are searched in a feature space composed of text features and image features, wherein there is a matching relationship between the text features and the image features in the co-occurrence features; the target probability of the co-occurrence features appearing in the feature space when the text features appear in the feature space is determined; context features whose similarity with the co-occurrence features in the feature space is greater than a preset threshold are determined, and aggregate features are determined based on the target probability and the context features; and a neural network model is trained using the aggregate features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data, thereby achieving the purpose of effectively capturing fine-grained information in the video, thereby achieving the technical effect of improving the model's overall understanding of the scene appearing in the video, and further solving the technical problem of limiting the model's overall understanding of the scene appearing in the video due to the inability to effectively capture fine-grained information in the video in the related text video retrieval technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flow chart of a model training method according to an embodiment of the present application;

[0021] Figure 2 is a structural diagram of a model training device according to an embodiment of the present application;

[0022] Figure 3 It is a hardware structure block diagram of a computer terminal according to a model training method of an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present application, a method embodiment of a model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] Figure 1 is a flow chart of a model training method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0027] Step S101, obtaining video data and text data for describing the video data.

[0028] For example, video data can be collected from a video database, which can be teaching videos, meeting records, advertising clips, etc. At the same time, text data related to the video data can be collected, such as video titles, descriptions, subtitles, or user comments. Ensure that the text data can accurately describe the key content or scenes of the video.

[0029] Step S102, performing feature extraction processing on the text data to obtain text features, and performing feature extraction processing on each image frame in the video data to obtain image features.

[0030] The text encoder of the CLIP model is used to process the collected text data to generate text embedding vectors. Text embedding not only contains the semantic information of each word, but also captures the correlation between words through context encoding, providing a high-dimensional vector representation for text features.

[0031] The video clip is split into a series of consecutive frames, and each frame is processed using CLIP's image encoder to extract frame-level feature embeddings. This will produce a sequence of frame embeddings, where each embedding represents the feature representation of an image frame.

[0032] Specifically, the CLIP model is used as the basic model to process text and video data. First, a natural language query sentence t is input, and the text encoder ψ of CLIP is used to process the sentence to generate a text embedding c t ∈R D , the embedding not only contains the semantic information of each word, but also captures the relationship between words. Next, input the video clip v and split it into a series of consecutive frames v f , and use CLIP's image encoder φ to process each frame and extract frame-level feature embedding By extracting features from each frame in the video, a frame embedding sequence is obtained Where F represents the number of video frames and D represents the dimension of feature embedding.

[0033] Step S103, searching for co-occurrence features in the feature space formed by the text features and the image features, wherein in the co-occurrence features, there is a matching relationship between the text features and the image features.

[0034] The extracted text features and image features are mapped into a common feature space for cross-modal comparison and matching. In the feature space, the similarity between image features and text features is calculated, for example based on dot product similarity or more complex similarity functions, to find those features that appear in both the video frame and the text description.

[0035] Step S104, determining the target probability of the co-occurrence feature appearing in the feature space when the text feature appears in the feature space.

[0036] The Softmax function is used to convert the similarity between the target feature and the co-occurrence feature into a probability distribution, thereby predicting the probability of the co-occurrence feature appearing given the text feature.

[0037] Step S105 , determining context features in the feature space whose similarity with the co-occurrence features is greater than a preset threshold, and determining aggregate features based on the target probability and the context features.

[0038] Identify contextual features that have high similarity to co-occurrence features, which are closely related to co-occurrence features in video frames and can provide additional semantic context information. And build an aggregated co-occurrence feature module, which aggregates co-occurrence context features in the entire scene using co-occurrence probabilities and contextual feature information in a self-attention or non-local manner. This will generate aggregated features that contain key co-occurrence and contextual information in the scene.

[0039] Step S106, using the aggregated features to train a neural network model, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

[0040] The pre-trained ResNet model is used as the basic network, and the aggregation co-occurrence feature module and the global pooling feature branch are added to it to build a co-occurrence feature network. The Res-4 and Res-5 layers of ResNet apply the dilated convolution strategy to obtain a suitable step size while reducing the amount of calculation.

[0041] The collected video data and text data are organized into batches to ensure that each batch contains diverse video and text pairs. Matching text-video pairs are used as positive samples, while other text-video combinations are used as negative samples for model training.

[0042] The model is trained using the cross entropy loss function to minimize the symmetric loss of text to video and video to text. During the training process, the random dropout technique is used to enhance the robustness of the model and avoid overfitting; at the same time, the multi-head attention mechanism is used to enhance the model's ability to capture different features and improve the generalization performance of the model.

[0043] During the training process, the model parameters are adjusted, including the jointly learned scaling parameter α, as well as the weights in the aggregated co-occurrence feature module and the ResNet model. Through continuous iterative optimization, the model reaches the best state in matching text and video features, improving the accuracy of text-video retrieval.

[0044] In step 106, the neural network model may be loaded into the memory, for example, the raw data of the neural network model may be loaded from the non-volatile memory into the volatile memory, so that the processor runs the neural network model. The raw data of the neural network model refers to unprocessed data, and generally includes parameters and structural data of the neural network model. The structural data may be a calculation relationship based on the parameters, such as a forward propagation calculation relationship between intermediate layers and neurons. Specifically, the structural data may include code related to the structure of the neural network model, such as code for performing related calculations between intermediate layers and neurons.

[0045] In one embodiment, an area for loading a neural network model can be divided in the memory, which may include a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related codes, and the parameters referenced by it can point to the address of a specific parameter in the parameter storage area through a pointer. During the training process of the neural network model, it may be necessary to frequently update the parameters, and the parameter values ​​in the parameter storage area can be updated.

[0046] Through the above steps, this application can effectively improve the accuracy and robustness of text-video retrieval, especially when dealing with complex scenes and fine-grained information. This method is not only suitable for learning scenarios of operator learning platforms, but can also be widely used in other occasions that require text-based query of video content, such as online education, media retrieval and entertainment recommendation systems.

[0047] The following Figure 1 The steps shown are exemplary and explanatory.

[0048] According to some optional embodiments of the present application, determining the target probability of a co-occurrence feature appearing in a feature space when a text feature appears in the feature space can be achieved by the following method:

[0049] Determine a first similarity between a text feature and a k-th co-occurrence feature in a feature space, where k is a positive integer not greater than K, K is the number of co-occurrence features in the feature space, and K is a positive integer greater than 2; determine a second similarity between the text feature and the k-th co-occurrence feature at the i-th position, and obtain n second similarities, where i is a positive integer not greater than n, n is the channel dimension of the image feature, and n is a positive integer greater than 2; sum the n second similarities to obtain a first value; determine a ratio between the first similarity and the first value as a first probability corresponding to the j-th co-occurrence feature, and obtain K first probabilities; obtain a mixing weight corresponding to each co-occurrence feature, and perform a weighted summation on the K first probabilities according to the mixing weight corresponding to each co-occurrence feature to obtain a target probability.

[0050] Specifically, the second similarity between each co-occurrence feature and all positions of the image feature in the feature space can be determined by the following method: Obtain a co-occurrence feature set in the feature space ({C 1 , C 2 , ..., C K}), where K represents the number of co-occurring features, and (K>2). For each co-occurring feature C K , calculate the text features and C K The first similarity between them, where k∈[1, K]. Determine the second similarity between the text feature and the kth co-occurrence feature at the ith position, and then obtain the second similarity between each co-occurrence feature and the image feature at all positions in the feature space.

[0051] In deep learning, the position in the feature space can be concretized as the position of the pixel point or feature vector on the feature map, which carries the information of the local area in the input image or video frame. For example, in the feature extraction process of a video frame, the CLIP model or other similar models will output a three-dimensional feature tensor with a shape of ([C,H,W]), where (C) represents the number of feature channels, which can be understood as the dimension of the feature space. (H) and (W) represent the height and width of the feature map, respectively, representing the position in the feature space. The position (i) can be a specific ((h,w)) coordinate, representing a point in the feature space.

[0052] For each co-occurrence feature (C K ), at each position (i) (i.e., each pixel or each feature vector position), calculate the text feature (T) and the image feature (F i ). The concept of position in feature space is crucial for understanding image and video content because it allows the model to capture local details that are necessary for recognizing and understanding objects, textures, colors, and actions in the scene. By considering features at different positions, the model is able to more fully understand the input data, thereby providing more accurate predictions and matches in the downstream task of text-video retrieval.

[0053] Specifically, the target probability of the co-occurrence feature appearing in the feature space when the text feature appears in the feature space is determined by the following formula:

[0054]

[0055] Among them, x c is the co-occurrence feature, x t is the text feature, π k is the mixed weight corresponding to the kth co-occurrence feature, s k (x c , x t ) is the first similarity between the text feature and the kth co-occurring feature in the feature space, s k (x i , x t ) is the second similarity between the text feature and the kth co-occurrence feature at the ith position.

[0056] and is feature x t and x c The target and co-occurrence vector representation of . The vector representation is given by and Given, where Φc and Φ t is a learnable transformation using a feed-forward network.

[0057] Through the above steps, the input convolutional feature map with a shape of C×H×W is regarded as a C-dimensional feature set X={x 1 ,...,x N , where N = H * W is the total number of features. The input features are obtained by using Φ t and Φ c Transform to vector space. Transform the target Φ t and co-occurrence transformation Φ c We use shared weights and apply 3×3 average pooling with a step size of 2 before the co-occurrence transformation to reduce the amount of computation. We predict the probability of co-occurring features based on the similarity in the vector space. We then aggregate the co-occurrence context based on the co-occurrence probability and transform it by learning a transformation function Φ in the vector space. t and Φ c To learn the co-occurrence feature distribution.

[0058] According to other optional embodiments of the present application, training a neural network model using aggregated features can be achieved by the following method: performing convolution processing on text features and upsampling processing on image features, connecting the processed features to the aggregated features to obtain target features, and using the target features to train the neural network model.

[0059] Convolution operations are performed on text features and image features to enhance the ability to capture local information and understand contextual semantics. Image features are upsampled to restore the spatial resolution of the features to facilitate subsequent fusion and comparison with text features. Upsampling can be achieved through methods such as bilinear interpolation, nearest neighbor interpolation, or transposed convolution (upsampled convolution).

[0060] Connect the text features after convolution processing and the image features after upsampling. For example, this can be achieved by splicing or element-by-element addition to form a feature representation that combines text and image information. Connect the aggregated features calculated in step S105 to the above-mentioned fused features. The connection operation can also be done by splicing or element-by-element addition, depending on the matching of the model architecture and feature dimensions. The connected features are passed through additional convolutional layers or fully connected layers to generate the final target features. Make necessary adjustments or transformations to the generated target features to ensure that they meet the input requirements of the neural network model.

[0061] Furthermore, the neural network model is trained using the target features by the following method: an output result of the neural network model classifying the target features is obtained, and in the output result, the text features and the image features that successfully match the text features are used as positive samples, and the image features that fail to match the text features are used as negative samples; the neural network model is trained using the positive samples, the negative samples and the preset cross entropy loss function until the function value of the preset cross entropy loss function is less than a preset threshold.

[0062] Specifically, the target features are input into the constructed neural network model for classification prediction. The output of the model is a probability distribution or direct category prediction, indicating the probability that each text-image pair belongs to a positive sample (matching success) or a negative sample (matching failure).

[0063] According to the output of the model, the matching relationship between the text feature (T) and the image feature (F) is classified. If the model predicts that the matching degree between the text and the image is higher than a preset threshold, the text-image pair is marked as a positive sample; conversely, if the matching degree is lower than the threshold, it is marked as a negative sample.

[0064] Define the cross entropy loss function. For each text-image pair, use its actual positive and negative sample labels and the model's predicted probability distribution to calculate the cross entropy loss. If a text-image pair is a positive sample (the actual label is a successful match), the model's predicted probability distribution is expected to have a higher value on a successful match class; conversely, if it is a negative sample, the model is expected to predict a higher probability of a failed match class.

[0065] In a training batch, the cross entropy losses of all text-image pairs are summed to get a batch loss. The batch loss is back-propagated to each layer of the model to calculate the gradient of each model parameter. Based on these gradients, the model parameters are updated using an optimizer (such as stochastic gradient descent SGD, Adam, etc.) to minimize the loss function.

[0066] Repeat the above steps until the model reaches the preset training round or the function value of the cross entropy loss function is less than the preset threshold, that is, the model has converged to a good enough state.

[0067] Preferably, the cross entropy loss function is as follows:

[0068]

[0069] L=L t2v +L v2t

[0070] Among them, s(t i ,v j ) is the text ti and video v j The cosine similarity between , B is the batch size, and λ is a learnable scaling parameter. By bootstrapping from the pre-trained CLIP model and combining the co-occurrence feature network, training with this loss function allows the model to learn to match text with the most semantically similar subregions in its corresponding video.

[0071] Through the above steps, the neural network model can learn how to use aggregated features to effectively classify text-image pairs and determine whether they match. The cross entropy loss function in the model training process serves as a guide to help the model adjust parameters so that the predicted probability is as close as possible to the actual positive and negative sample labels. When the model training is completed, it will be used in the actual text-video retrieval task to retrieve the video clip that best matches the input text description. This deep learning-based method can handle complex feature associations and improve the accuracy and robustness of text-video retrieval. In practical applications, the training and evaluation process can be accelerated on the GPU to improve training speed and efficiency.

[0072] Preferably, after determining the aggregate features, the following steps may be performed: performing random inactivation processing on the aggregate features, decomposing the processed aggregate features into multiple sub-vectors, and calculating the attention weight between each sub-vector and other sub-vectors; and performing weighted sum processing on the sub-vectors using the attention weight between each sub-vector and other sub-vectors.

[0073] Specifically, the generated aggregated features (A) are obtained from the aggregate co-occurrence feature module. During the training phase, random dropout techniques are applied to the elements or sub-vectors in the aggregated features (A). Random dropout helps the model learn more robust representations and prevent overfitting by randomly "deactivating" or ignoring some elements in the features and setting their values ​​to 0 with a certain probability (e.g., 0.5). The aggregated features (A) that have been subjected to random dropout processing are decomposed into multiple sub-vectors. An attention mechanism is designed for each sub-vector, such as calculating the query, key, and value vector representations, where the query vector can be a summary of the aggregated features or each sub-vector itself. For each sub-vector, the attention weight between it and other sub-vectors is calculated. Each sub-vector is weighted summed with other sub-vectors using the attention weight to generate a weighted aggregated feature representation. The weighted sum results of all sub-vectors are merged to form the final weighted aggregated feature (A').

[0074] Through the above steps, a multi-head attention mechanism can be introduced into the neural network model to process the aggregated features more finely. The random dropout technology enhances the robustness of the model, allowing it to maintain good performance when facing partially inactivated features. Decomposing the processed aggregated features into multiple sub-vectors and calculating the attention weights between the sub-vectors further enhances the model's understanding and utilization of the complex relationships between features, and helps capture deeper co-occurrence feature associations between text descriptions and video content. Finally, through weighted summation, the model can generate a more comprehensive and semantic feature representation.

[0075] Preferably, the aggregate feature z is determined based on the target probability and context features. t , which can be achieved by the following method: multiplying the target probability with the context feature to obtain the aggregate feature. This can be achieved by the following formula:

[0076]

[0077] Among them, ψ c is the co-occurrence context at position c, and the co-occurrence context is given by ψ c =Ψ(x c ), where ψ is a learnable feed-forward network transformation.

[0078] Figure 2 is a structural diagram of a model training device according to an embodiment of the present application, such as Figure 2 As shown, the device comprises:

[0079] The acquisition module 21 is used to acquire video data and text data used to describe the video data.

[0080] The processing module 22 is used to perform feature extraction processing on the text data to obtain text features, and perform feature extraction processing on each image frame in the video data to obtain image features.

[0081] The search module 22 is used to search for co-occurrence features in the feature space formed by text features and image features, wherein in the co-occurrence features, there is a matching relationship between the text features and the image features.

[0082] The first determination module 24 is used to determine the target probability of the co-occurrence feature appearing in the feature space when the text feature appears in the feature space.

[0083] The second determination module 25 is used to determine the context features whose similarity with the co-occurrence features in the feature space is greater than a preset threshold, and determine the aggregated features according to the target probability and the context features.

[0084] The training module 26 is used to train the neural network model using the aggregated features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

[0085] Optionally, the first determination module 24 is also used to perform the following steps: determine a first similarity between the text feature and the kth co-occurrence feature in the feature space, wherein k is a positive integer not greater than K, K is the number of co-occurrence features in the feature space, and K is a positive integer greater than 2; determine a second similarity between the text feature and the kth co-occurrence feature at the i-th position to obtain n second similarities, wherein i is a positive integer not greater than n, n is the channel dimension of the image feature, and n is a positive integer greater than 2; sum the n second similarities to obtain a first value; determine the ratio between the first similarity and the first value as the first probability corresponding to the j-th co-occurrence feature to obtain K first probabilities; obtain a mixing weight corresponding to each co-occurrence feature, and perform a weighted summation on the K first probabilities according to the mixing weight corresponding to each co-occurrence feature to obtain a target probability.

[0086] Optionally, the first determination module 24 is further configured to determine, when the text feature appears in the feature space, the target probability of the co-occurrence feature appearing in the feature space by using the following formula: Among them, x c is the co-occurrence feature, x t is the text feature, π k is the mixed weight corresponding to the kth co-occurrence feature, s k (x c , x t ) is the first similarity between the text feature and the kth co-occurring feature in the feature space, s k (x i , x t ) is the second similarity between the text feature and the kth co-occurrence feature at the ith position.

[0087] Optionally, the training module 26 is also used to perform the following steps: convolution processing on text features, upsampling processing on image features, connecting the processed features to aggregate features to obtain target features, and using the target features to train the neural network model.

[0088] Optionally, the training module 26 is also used to perform the following steps: obtaining the output result of the neural network model for classifying the target features, in the output result, taking the text features and the image features that successfully match the text features as positive samples, and taking the image features that fail to match the text features as negative samples; using the positive samples, negative samples and a preset cross entropy loss function to train the neural network model until the function value of the preset cross entropy loss function is less than a preset threshold.

[0089] Optionally, the model training device is also used to execute the following steps after determining the aggregate features: perform random inactivation processing on the aggregate features, decompose the processed aggregate features into multiple sub-vectors, and calculate the attention weight between each sub-vector and other sub-vectors; use the attention weight between each sub-vector and other sub-vectors to perform weighted summation processing on the sub-vectors.

[0090] Optionally, the second determination module 25 is further configured to perform the following steps: multiplying the target probability by the context feature to obtain an aggregate feature.

[0091] It should be noted that the above Figure 2 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0092] It should be noted that Figure 2 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0093] Figure 3 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a model training method. Figure 3 As shown, the computer terminal 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission module 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.

[0094] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0095] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, the above-mentioned model training method is realized. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the computer terminal 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0096] The transmission module 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 30. In one example, the transmission module 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0097] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 30 .

[0098] It should be noted that, in some optional embodiments, the above Figure 3 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 3 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0099] It should be noted that Figure 3 The computer terminal shown is used to execute Figure 1 The model training method shown, therefore the relevant explanations in the execution method of the above commands are also applicable to the electronic device and will not be repeated here.

[0100] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above model training method.

[0101] A program for a non-volatile storage medium to perform the following functions: acquiring video data and text data for describing the video data; performing feature extraction processing on the text data to obtain text features, and performing feature extraction processing on each image frame in the video data to obtain image features; searching for co-occurrence features in a feature space composed of text features and image features, wherein a matching relationship exists between the text features and the image features in the co-occurrence features; determining a target probability of the co-occurrence features appearing in the feature space when the text features appear in the feature space; determining context features in the feature space whose similarity with the co-occurrence features is greater than a preset threshold, and determining aggregate features based on the target probability and the context features; and training a neural network model using the aggregate features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

[0102] An embodiment of the present application also provides an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above model training method is executed when the program is running.

[0103] The processor is used to run a program that performs the following functions: obtain video data and text data used to describe the video data; perform feature extraction processing on the text data to obtain text features, and perform feature extraction processing on each image frame in the video data to obtain image features; search for co-occurrence features in a feature space composed of text features and image features, wherein there is a matching relationship between the text features and the image features in the co-occurrence features; determine the target probability of the co-occurrence features appearing in the feature space when the text features appear in the feature space; determine context features in the feature space whose similarity with the co-occurrence features is greater than a preset threshold, and determine aggregate features based on the target probability and the context features; and train a neural network model using the aggregate features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

[0104] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0105] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0106] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0111] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Acquire video data and text data for describing the video data; Performing feature extraction processing on the text data to obtain text features, and performing feature extraction processing on each image frame in the video data to obtain image features; Searching for co-occurrence features in a feature space formed by the text features and the image features, wherein a matching relationship exists between the text features and the image features in the co-occurrence features; Determining a target probability that the co-occurrence feature appears in the feature space when the text feature appears in the feature space; Determine a context feature in the feature space whose similarity with the co-occurrence feature is greater than a preset threshold, and determine an aggregate feature based on the target probability and the context feature; The aggregated features are used to train a neural network model, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

2. The method according to claim 1, characterized in that Determining a target probability that the co-occurrence feature appears in the feature space when the text feature appears in the feature space includes: Determine a first similarity between the text feature and the kth co-occurrence feature in the feature space, wherein k is a positive integer not greater than K, K is the number of the co-occurrence features in the feature space, and K is a positive integer greater than 2; Determine the second similarity between the text feature and the kth co-occurrence feature at the i-th position, and obtain n second similarities, where i is a positive integer not greater than n, and n is the channel dimension of the image feature. n is a positive integer greater than 2; Performing a sum operation on n of the second similarities to obtain a first value; Determine the ratio between the first similarity and the first value as the first probability corresponding to the jth co-occurrence feature, and obtain K first probabilities; A mixing weight corresponding to each of the co-occurrence features is obtained, and according to the mixing weight corresponding to each of the co-occurrence features, a weighted sum is performed on the K first probabilities to obtain the target probability.

3. The method according to claim 1, characterized in that The target probability of the co-occurrence feature appearing in the feature space when the text feature appears in the feature space is determined by the following formula: Among them, x c is the co-occurrence feature, x t is the text feature, π k is the mixed weight corresponding to the kth co-occurrence feature, s k (x c , x t ) is the first similarity between the text feature and the kth co-occurrence feature in the feature space, s k (x i , x t ) is the second similarity between the text feature and the kth co-occurrence feature at the ith position.

4. The method according to claim 1, characterized in that: The neural network model is trained using the aggregated features, including: The text features are convolved, and the image features are upsampled, and the processed features are connected to the aggregated features to obtain target features, and the neural network model is trained using the target features.

5. The method according to claim 4, characterized in that The neural network model is trained using the target features, including: Obtaining an output result of the neural network model classifying the target feature, wherein the text feature and the image feature that successfully matches the text feature are used as positive samples, and the image feature that fails to match the text feature is used as a negative sample; The neural network model is trained using the positive samples, the negative samples and a preset cross entropy loss function until a function value of the preset cross entropy loss function is less than a preset threshold.

6. The method according to claim 1, characterized in that After determining the aggregation feature, the method further includes: Performing random inactivation processing on the aggregated features, decomposing the processed aggregated features into multiple sub-vectors, and calculating the attention weight between each of the sub-vectors and other sub-vectors; The sub-vectors are weighted summed using the attention weights between each of the sub-vectors and other sub-vectors.

7. The method according to claim 1, characterized in that Determining an aggregate feature according to the target probability and the context feature includes: The target probability is multiplied by the context feature to obtain the aggregate feature.

8. A model training device, characterized in that: include: An acquisition module, used for acquiring video data and text data for describing the video data; A processing module, used for performing feature extraction processing on the text data to obtain text features, and performing feature extraction processing on each image frame in the video data to obtain image features; A search module, used for searching for co-occurrence features in a feature space formed by the text features and the image features, wherein a matching relationship exists between the text features and the image features in the co-occurrence features; A first determination module, used to determine a target probability of the co-occurrence feature appearing in the feature space when the text feature appears in the feature space; A second determination module is used to determine a context feature whose similarity with the co-occurrence feature in the feature space is greater than a preset threshold, and determine an aggregate feature according to the target probability and the context feature; A training module is used to train a neural network model using the aggregated features, wherein the neural network model is used to retrieve target video data corresponding to the text data in the video data.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the model training method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the model training method described in any one of claims 1 to 7 when running.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method described in any one of claims 1 to 7 is implemented.

Images