Multi-task learning model training method and device, multi-task learning model application method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and machine learning, and discloses a multi-task learning model training method and device, a multi-task learning model application method and device, equipment and a medium. According to the method, the video sample, the corresponding query text, the video sample segment and the starting and ending moments marked by the video sample segment are utilized to perform multi-task learning to obtain the multi-task learning model, and the processing of video description generation and video segment retrieval can be completed only by training one model, so that the resource consumption of a server is reduced; the later maintenance cost of the model is reduced; when multi-task learning is carried out, a mean square error loss function and a cross entropy loss function are respectively calculated, a loss function of the whole model is calculated according to the mean square error loss function and the cross entropy loss function, and model parameters are optimized according to the loss function of the whole model; the video description generation task and the video clip retrieval task help each other and learn together, so that the feature alignment capability of the model is improved, and the accuracy of the model is further improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and machine learning technology, and in particular to a training and application method, device, computer equipment and storage medium for a multi-task learning model. Background Art

[0002] In insurance marketing projects, agents do not need to browse videos. They can quickly obtain video information during the marketing process to users by reading video description text. At the same time, agents can retrieve video clips to quickly and accurately locate video clips in a long video, and use the retrieved video clips for user stickiness marketing, which greatly improves the work efficiency of insurance agents.

[0003] In the prior art, video description generation and video clip retrieval are two independent tasks, and each task is treated as a separate task for machine learning, that is, the video description generation task and the video clip retrieval task are both single-task learning. Among them, the video description generation task uses the alignment of text and video to encode the video features, and then generates description text through a text encoder, so as to automatically describe the content of the video in text; the video clip retrieval task, by first semantically segmenting the video, cutting it into multiple small video clips, and then calculating the similarity of the text to the video clip features, calculates which video clip the current text is most similar to, and retrieves the corresponding clip.

[0004] After research, the inventors found that the semantic feature alignment method between video and text was used in the processing of the video description generation task and the video clip retrieval task, and both tasks are cross-modal tasks of video and text. In the prior art, the video description generation task and the video clip retrieval task are separately learned as single tasks, which often results in low feature alignment ability of the model, affecting the accuracy of the model; and when deploying the tasks, two different models need to be deployed on the server to handle the two tasks of video description generation and video clip retrieval, which consumes more server resources and has high later maintenance costs for the model. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a training and application method, device, computer equipment and storage medium for a multi-task learning model to solve the technical problems that the video description generation task and the video clip retrieval task are respectively learned as single tasks, the feature alignment ability of the model is low, which in turn affects the accuracy of the model, and the model consumes more server resources and the subsequent maintenance cost of the model is high.

[0006] First, a training method for a multi-task learning model is provided, including:

[0007] Obtaining a video sample, a query text, a video sample segment group, and k frame vectors for model training, wherein the video sample segment group includes a plurality of video sample segments obtained by slicing the video sample, and the dimension of each frame vector is 64;

[0008] Encode the query text using a text encoder to obtain a feature vector of the query text;

[0009] Encode the video sample using 3D-CNN to obtain the feature vector of the video sample;

[0010] Perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a cross-attention weighted vector;

[0011] After concatenating the k box vectors, concatenate them with the weighted vector of the cross attention, perform self-attention calculation on the concatenated vector, and obtain the weighted vector of the self-attention;

[0012] The k*64 dimensions of the weighted vector of self-attention are segmented to obtain k new box vectors, and the k new box vectors are respectively input into the first group of MLP layers with different parameters, and the mean square error loss function is calculated based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group;

[0013] The k new box vectors are respectively input into the second set of MLP layers with shared parameters. The output results of the second set of MLP layers are passed through the text decoder to generate video text description, and the cross entropy loss function is calculated based on the results of video text description generation and the query text.

[0014] The loss function of the entire model is calculated according to the mean square error loss function and the cross entropy loss function, the loss value gradient is calculated, and then backpropagation is performed to optimize the model parameters until convergence to obtain a trained multi-task learning model.

[0015] In a second aspect, a method for applying a multi-task learning model is provided, wherein the multi-task learning model is trained according to the training method of the multi-task learning model, and the method for applying the multi-task learning model includes:

[0016] Video description generation: input the video material into the trained multi-task learning model to obtain the video description text corresponding to the video material;

[0017] Video clip retrieval: input the video material and the search text into the trained multi-task learning model to obtain the start time and end time of the video sample segment in the video material that matches the search text.

[0018] In a third aspect, a training device for a multi-task learning model is provided, comprising:

[0019] A training data acquisition module, used to acquire video samples, query text, video sample segment groups and k frame vectors for model training, wherein the video sample segment group includes multiple video sample segments obtained by slicing the video sample, and the dimension of each frame vector is 64;

[0020] A text encoding module, used for encoding the query text using a text encoder to obtain a feature vector of the query text;

[0021] A video encoding module is used to encode video samples using 3D-CNN to obtain feature vectors of video samples;

[0022] A first attention calculation module is used to perform cross attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a cross attention weighted vector;

[0023] The second attention calculation module is used to concatenate the k box vectors and then concatenate them with the weighted vector of the cross attention, and perform self-attention calculation on the concatenated vector to obtain the weighted vector of the self-attention;

[0024] A first loss calculation module is used to segment the rear k*64 dimensions of the self-attention weighted vector to obtain k new box vectors, input the k new box vectors into a first group of MLP layers with different parameters, and calculate a mean square error loss function based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group;

[0025] A second loss calculation module is used to input the k new box vectors into the second group of MLP layers with shared parameters respectively, and the output results of the second group of MLP layers are passed through a text decoder to generate a video text description, and a cross entropy loss function is calculated based on the result of the video text description generation and the query text;

[0026] The model optimization module is used to calculate the loss function of the entire model based on the mean square error loss function and the cross entropy loss function, calculate the loss value gradient, and then perform backpropagation to optimize the model parameters until convergence to obtain a trained multi-task learning model.

[0027] In a fourth aspect, a device for applying a multi-task learning model is provided, wherein the multi-task learning model is trained according to the training method of the multi-task learning model, and the device for applying the multi-task learning model comprises:

[0028] The video description generation module is used to input the video material into the trained multi-task learning model to obtain the video description text corresponding to the video material;

[0029] The video clip retrieval module is used to input the video material and the retrieval text into the trained multi-task learning model to obtain the starting time and the ending time of the video sample clip in the video material that matches the retrieval text.

[0030] In a fifth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the training method of the multi-task learning model when executing the computer program.

[0031] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the training method of the multi-task learning model are implemented.

[0032] The training and application methods, devices, computer equipment and storage media of the multi-task learning model above use video samples, corresponding query texts, video sample segments obtained by slicing the video samples, and the start and end times annotated by the video sample segments to perform multi-task learning to obtain a multi-task learning model. Only one model needs to be trained to complete the processing of the two tasks of video description generation and video segment retrieval, which reduces server resource consumption and reduces the later maintenance cost of the model. Moreover, when performing multi-task learning, the mean square error loss function is calculated during the learning process of the video segment retrieval task, and the cross entropy loss function is calculated during the learning process of the video description generation task. The loss function of the entire model is calculated based on the mean square error loss function and the cross entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, which improves the feature alignment capability of the model and thereby improves the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0034] Figure 1 is an exemplary system architecture diagram in which the present invention can be applied;

[0035] Figure 2 is a flow chart of a training method for a multi-task learning model in one embodiment of the present invention;

[0036] Figure 3 is a flow chart of an application method of a multi-task learning model in one embodiment of the present invention;

[0037] Figure 4 is a structural schematic diagram of a training device for a multi-task learning model in one embodiment of the present invention;

[0038] Figure 5 is a structural schematic diagram of an application device of a multi-task learning model in one embodiment of the present invention;

[0039] Figure 6 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the system architecture includes a client, a network, and a server. The network is used to provide a medium for a communication link between the client and the server, that is, the client can communicate with the server through the network. The network includes various connection types, such as wired, wireless communication links or fiber optic cables, etc. Users can use the client to interact with the server through the network to receive or send messages, etc. Various communication client applications can be installed on the client, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc. The server can be a server that provides various services, such as a background server that supports the pages displayed on the client.

[0042] The embodiments of the present application provide a method, device, computer equipment and medium for training and applying a multi-task learning model. Among them, the training method and application method of the multi-task learning model can be applied to a server. The client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0043] Figure 2 A flowchart of a training method for a multi-task learning model provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the training method of the multi-task learning model includes steps S10 to S80:

[0044] S10: Obtain video samples, query text, a video sample segment group and k frame vectors for model training, wherein the video sample segment group includes a plurality of video sample segments obtained by slicing the video sample.

[0045] This step is mainly used to obtain a training data set for model training. The training data set includes several video samples, several query texts corresponding to the video samples, several groups of video sample clips, and k box vectors.

[0046] For a video sample, we manually cut and slice the video sample based on the integrity of semantic expression, and divide it into several video sample segments. We mark the start and end time of each video sample segment, and manually annotate the video sample segment with a text summary to describe the content of the current video sample segment:<V,S> =<v1,s1,ts1,te1> ,<v2,s2,ts2,te2> ,<v3,s3,ts3,te3> ,..., <v n ,s n ,ts n ,te n >, V represents the entire video sample, v i represents the i-th video sample segment, s i represents the text description of the i-th video sample segment, S is the text description of all s i The text splicing as a whole, ts i Indicates the starting time of the i-th video sample segment, te i represents the end time of the i-th video sample segment, i=1,2,3,...,n.

[0047] A plurality of video sample segments obtained by slicing a video sample constitute a video sample segment group, and a plurality of video sample segments can be sliced ​​to obtain a plurality of video sample segment groups. It should be noted that, in other embodiments, a video sample may be sliced ​​using other different slicing standards, for example, a video sample may be sliced ​​using an average duration slicing standard, that is, a video sample may be evenly divided into a plurality of video sample segments of equal duration.

[0048] The box vector can be set manually, the initial value of the box vector is random, and the number of box vectors (k) can be set accordingly as needed. In order to facilitate matrix calculation, it can be set to a power of 2. For example, in this embodiment, 16 box vectors are manually set, each with a dimension of 64, for a total of 1024 dimensions. These 16 box vectors will be used as input to the model. After training, the model will output 16 vectors equivalently, that is, 16 significant box feature vectors can be obtained. These 16 box feature vectors determine how a video sample is sliced ​​into 16 video sample segments.

[0049] The video samples, query text, video sample segment groups and frame vectors can be stored in a server or client. Before performing multi-task learning model training, the video samples, query text, video sample segment groups and frame vectors need to be obtained from the server or client. Then, a video sample, a query text corresponding to the video sample, a video sample segment group corresponding to the video sample and k frame vectors are used as inputs of the model to perform model training.

[0050] S20: Encode the query text using a text encoder to obtain a feature vector of the query text.

[0051] The step S20 comprises the following steps:

[0052] S21: Segment the query text into m words, and use a text encoder to encode each word into a 1024-dimensional vector.

[0053] S22: Use the multi-head self-attention layer of the text encoder to perform self-attention operation on the m words to obtain multiple weighted vectors.

[0054] S23: concatenate multiple weighted vectors in the order of word segmentation and restore them into a 1024-dimensional vector.

[0055] S24: Normalize the restored vector to obtain a feature vector of the query text.

[0056] For steps S21-S24, after the query text is segmented to obtain m words, a text encoder is used to perform text encoding on the m words. The text encoder is a self-attention mechanism layer. In this embodiment, the self-attention layer of the text encoder is provided with 8 heads, so each attention head processes 128 dimensions of the 1024 dimensions of the m word vectors, that is, each attention head performs self-attention operations on the m words. Specifically, the 128-dimensional vectors of the m words are copied into 3 copies, multiplied by three 128*128 matrices Q, K, and V respectively, and the output result is the output of the Q matrix multiplied by the output of the V matrix, and then a 128-dimensional attention weight is obtained through a normalized exponential function (or softmax function), and then the corresponding dimension weight value of this attention weight is multiplied by the output result of the V matrix to obtain a 128-dimensional weighted vector. Since the self-attention layer has 8 heads, performing self-attention operations on m words will obtain 8 128-dimensional weighted vectors, which are concatenated according to the word segmentation order of step S21 to restore them to a 1024-dimensional vector. The restored vector is normalized, that is, the mean of the vector is subtracted and divided by the variance, and the resulting 1024-dimensional vector is the feature vector of the query text.

[0057] S30: Encode the video sample using 3D-CNN to obtain a feature vector of the video sample.

[0058] Three-dimensional convolutional neural network (3D-CNN) is a mature model structure that can encode images in time series. Using 3D-CNN to encode video samples, a 1024-dimensional video feature vector can be obtained, and then the 1024-dimensional video feature vector is normalized, that is, the mean of the vector is subtracted and divided by the variance, and finally the feature vector of the preprocessed video sample is obtained.

[0059] S40: Perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a cross-attention weighted vector.

[0060] This step requires cross-attention calculation of the feature vector of the query text and the feature vector of the video sample. Specifically, the feature vector of the video sample is copied twice, used as the K vector and V vector of the QKV matrix respectively, and the feature vector of the query text is used as the Q vector of the QKV matrix; the transposition of the Q vector * K vector is calculated by the softmax function to obtain a set of weight vectors, which are then multiplied with the V vector to obtain the cross-attention weighted vector, denoted as V_W. This cross-attention weighted vector is based on the feature vector of the video sample and weights the query text and the video sample.

[0061] S50: concatenate the k box vectors and then concatenate them with the weighted vector of the cross attention, perform self-attention calculation on the concatenated vectors, and obtain the weighted vector of the self-attention.

[0062] The step S50 comprises the following steps:

[0063] S51: concatenate the k box vectors to obtain a vector of k*64 dimensions, and multiply the vector of k*64 dimensions by a fully connected layer to obtain a new vector;

[0064] S52: normalize the new vector and concatenate it with the weighted vector of the cross attention;

[0065] S53: Copy the vector obtained by concatenating it with the weighted vector of the cross attention three times, and use them as the Q vector, K vector and V vector of the QKV matrix respectively. Multiply the transpose of the Q vector and the K vector and then calculate it through the softmax function. The obtained weight vector is then multiplied with the V vector to obtain the weighted vector of self-attention.

[0066] The number of box vectors (k) can be set accordingly before model training. In this embodiment, k=16 is used as an example to illustrate the process of steps S51-S53: 16 box vectors are concatenated to obtain a 1024-dimensional vector, and this 1024-dimensional vector is multiplied by a 1024*1024 fully connected layer to obtain a new 1024-dimensional vector. This new 1024-dimensional vector is normalized, that is, the mean of the vector is subtracted and divided by the variance, and then concatenated with V_W after normalization to form a 2048-dimensional vector. This 2048-dimensional vector is self-attention calculated, copied three times, multiplied by a new QKV matrix, and the transpose of the Q vector and the K vector is multiplied and calculated by the softmax function to obtain a weight vector, which is then multiplied by the V vector to obtain a weighted vector of self-attention, recorded as S_W. This self-attention weighted vector combines the vector features of 16 boxes, and the corresponding dimensions represent the corresponding features. For example, the first 1024 dimensions still represent the features of video fusion text, and each 64 dimensions in the last 1024 dimensions represent the features of a box.

[0067] S60: Segment the rear k*64 dimensions of the self-attention weighted vector to obtain k new box vectors, input the k new box vectors into the first group of MLP layers with different parameters respectively, and calculate the mean square error loss function based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group.

[0068] The number of frame vectors (k) can be set accordingly as needed before model training. In this embodiment, k=16 is used as an example to illustrate the process of step S60: the 1024 dimensions after S_W are divided into 64-dimensional units to obtain 16 new frame vectors; these 16 new frame vectors are respectively input into the first group of 16 MLP layers with different parameters, and each MLP layer outputs a 2-dimensional vector, the first value of the 2-dimensional vector represents the start time corresponding to the current frame, and the second value of the 2-dimensional vector represents the end time corresponding to the current frame. Based on the output results of the first group of 16 MLP layers with different parameters and the start and end times of the corresponding video sample segments in the video sample segment group, the mean square error (MSE) loss function is calculated, which is recorded as L_MSE. It should be noted here that if the number of video sample fragments obtained by slicing a video sample is less than 16, the extra frames will not participate in the calculation of the mean square error loss function; if the number of video sample fragments obtained by slicing a video sample is greater than 16, then only the first 16 video sample fragments are taken in chronological order for the calculation of the mean square error loss function.

[0069] S70: k new box vectors are respectively input into the second group of MLP layers with shared parameters. The output results of the second group of MLP layers are passed through a text decoder to generate a video text description, and a cross entropy loss function is calculated based on the result of the video text description generation and the query text.

[0070] The number of box vectors (k) can be set accordingly as needed before model training. In this embodiment, k=16 is taken as an example to illustrate the process of step S70: the 16 new box vectors obtained by 1024-dimensional segmentation after S_W are respectively input into the second group of 16 shared parameter MLP layers, and the number of layers of the 16 shared parameter MLP layers is 2, wherein the first layer parameter matrix is ​​64*256, and the second layer parameter matrix is ​​256*1024, and the 16 shared parameter MLP layers correspond to output 16 1024-dimensional vectors, which are then passed through a transformer structured text decoder for video text description generation, and the cross entropy loss function is calculated based on the result of the video text description generation and the query text, denoted as L_CE.

[0071] S80: Calculate the loss function of the entire model based on the mean square error loss function and the cross entropy loss function, calculate the loss value gradient, and then perform back propagation to optimize the model parameters until convergence to obtain a trained multi-task learning model.

[0072] The training method of the multi-task learning model provided in the embodiment of the present invention uses video samples, corresponding query texts, video sample segments obtained by slicing the video samples, and the start and end times annotated by the video sample segments to perform multi-task learning to obtain a multi-task learning model. Only one model needs to be trained to complete the processing of the two tasks of video description generation and video segment retrieval, which reduces server resource consumption and reduces the later maintenance cost of the model. Moreover, when performing multi-task learning, the mean square error loss function is calculated in the video segment retrieval task learning process, and the cross entropy loss function is calculated in the video description generation task learning process. The loss function of the entire model is calculated based on the mean square error loss function and the cross entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, which improves the feature alignment capability of the model and thereby improves the accuracy of the model.

[0073] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0074] Figure 3 A flow chart of an application method of a multi-task learning model provided in an embodiment of the present invention, wherein the multi-task learning model is based on Figure 2 The training method of the multi-task learning model in the embodiment shown is obtained by training. Figure 3 As shown, the application method of the multi-task learning model includes steps S101 to S102:

[0075] S101: Video description generation: input the video material into the trained multi-task learning model to obtain the video description text corresponding to the video material.

[0076] S102: Video segment retrieval: input the video material and the retrieval text into the trained multi-task learning model to obtain the start time and end time of the video sample segment in the video material that matches the retrieval text.

[0077] The application method of the multi-task learning model provided by the present invention can be applied to various application scenarios. For example, in the field of insurance applications, if the user only inputs video material to the multi-task learning model, the model will process the video material and output the corresponding video description text. The user can quickly obtain video information without browsing the video, which greatly improves the work efficiency of insurance agents; if the user inputs video material and search text to the multi-task learning model, the model will output the start and end time of the video sample segment in the video material that matches the search text, so as to quickly and accurately locate the video segment in a long video.

[0078] The application method of the multi-task learning model provided in the embodiment of the present invention is that the multi-task learning model is obtained by multi-task learning using video samples, corresponding query texts, video sample segments obtained by slicing the video samples, and the start and end times annotated by the video sample segments. Only one model is needed to complete the processing of the two tasks of video description generation and video segment retrieval, thereby reducing server resource consumption and reducing the later maintenance cost of the model. Moreover, when training the multi-task learning model, the mean square error loss function is calculated during the learning process of the video segment retrieval task, and the cross entropy loss function is calculated during the learning process of the video description generation task. The loss function of the entire model is calculated based on the mean square error loss function and the cross entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, thereby improving the feature alignment capability of the model and thus improving the accuracy of the model.

[0079] In one embodiment, a training device for a multi-task learning model is provided, and the training device for the multi-task learning model corresponds one-to-one to the training method for the multi-task learning model in the above embodiment. Figure 4 As shown, the training device of the multi-task learning model includes a training data acquisition module 10, a text encoding module 20, a video encoding module 30, a first attention calculation module 40, a second attention calculation module 50, a first loss calculation module 60, a second loss calculation module 70 and a model optimization module 80. The functional modules are described in detail as follows:

[0080] The training data acquisition module 10 is used to execute step S10 in the training method of the multi-task learning model in the above embodiment, that is, to obtain video samples, query text, video sample segment groups and k frame vectors for model training, wherein the video sample segment group includes multiple video sample segments obtained by slicing the video sample, and the dimension of each frame vector is 64.

[0081] The text encoding module 20 is used to execute step S20 in the training method of the multi-task learning model in the above embodiment, that is, to use a text encoder to perform text encoding on the query text to obtain a feature vector of the query text.

[0082] The video encoding module 30 is used to execute step S30 in the training method of the multi-task learning model in the above embodiment, that is, to encode the video sample by using 3D-CNN to obtain the feature vector of the video sample.

[0083] The first attention calculation module 40 is used to execute step S40 in the training method of the multi-task learning model in the above embodiment, that is, to perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a cross-attention weighted vector.

[0084] The second attention calculation module 50 is used to execute step S50 in the training method of the multi-task learning model in the above embodiment, that is, to splice the k box vectors and then splice them with the weighted vector of cross attention, perform self-attention calculation on the vector obtained after splicing, and obtain the weighted vector of self-attention.

[0085] The first loss calculation module 60 is used to execute step S60 in the training method of the multi-task learning model in the above embodiment, that is, to segment the rear k*64 dimensions of the self-attention weighted vector to obtain k new box vectors, and input the k new box vectors into the first group of MLP layers with different parameters respectively, and calculate the mean square error loss function based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group.

[0086] A second loss calculation module 70 is used to execute step S70 in the training method of the multi-task learning model in the above embodiment, that is, to input the k new box vectors into the MLP layer of the second group of shared parameters respectively, and the output results of the second group of MLP layers are passed through the text decoder to generate a video text description, and a cross entropy loss function is calculated based on the result of the video text description generation and the query text;

[0087] The model optimization module 80 is used to execute step S80 in the training method of the multi-task learning model in the above embodiment, that is, to calculate the loss function of the entire model according to the mean square error loss function and the cross entropy loss function, calculate the loss value gradient, and then perform back propagation to optimize the model parameters until convergence to obtain a trained multi-task learning model.

[0088] In one embodiment, the text encoding module 20 is specifically used for:

[0089] Segment the query text into m words, each of which is a 1024-dimensional vector.

[0090] Use the multi-head self-attention layer of the text encoder to perform self-attention operations on m words to obtain multiple weighted vectors;

[0091] Multiple weighted vectors are concatenated in the order of word segmentation to restore them into a 1024-dimensional vector;

[0092] The restored vector is normalized to obtain the feature vector of the query text.

[0093] In one embodiment, the video encoding module 30 is specifically configured to:

[0094] Use 3D-CNN to encode video samples in time sequence to obtain video feature vectors;

[0095] The obtained video feature vector is normalized to obtain the feature vector of the video sample.

[0096] In one embodiment, the first attention calculation module 40 is specifically used to:

[0097] The feature vector of the video sample is copied twice, and used as the K vector and V vector of the QKV matrix respectively. The feature vector of the query text is used as the Q vector of the QKV matrix.

[0098] The transposition of the Q vector * K vector is then calculated through the softmax function to obtain a set of weight vectors, which are then multiplied with the V vector to obtain the cross-attention weighted vector.

[0099] In one embodiment, the second attention calculation module 50 is specifically used to:

[0100] Concatenate the k box vectors to obtain a k*64 dimensional vector, and multiply the k*64 dimensional vector by a fully connected layer to obtain a new vector;

[0101] Normalize the new vector and concatenate it with the weighted vector of the cross attention.

[0102] The vector obtained by concatenating it with the weighted vector of the cross attention is copied three times as the Q vector, K vector and V vector of the QKV matrix respectively. The transpose of the Q vector and the K vector are multiplied and then calculated by the softmax function. The obtained weight vector is then multiplied with the V vector to obtain the weighted vector of self-attention.

[0103] The training device of the multi-task learning model provided in the embodiment of the present invention uses video samples, corresponding query texts, video sample segments obtained by slicing the video samples, and the start and end times annotated by the video sample segments to perform multi-task learning to obtain a multi-task learning model. Only one model needs to be trained to complete the processing of the two tasks of video description generation and video segment retrieval, which reduces server resource consumption and reduces the later maintenance cost of the model. Moreover, when performing multi-task learning, the mean square error loss function is calculated in the video segment retrieval task learning process, and the cross entropy loss function is calculated in the video description generation task learning process. The loss function of the entire model is calculated based on the mean square error loss function and the cross entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, which improves the feature alignment capability of the model and thereby improves the accuracy of the model.

[0104] For the specific definition of the training device of the multi-task learning model, please refer to the definition of the training method of the multi-task learning model above, which will not be repeated here. Each module in the above-mentioned training device of the multi-task learning model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] Figure 5 A flow chart of an application device of a multi-task learning model provided in an embodiment of the present invention, wherein the application device of the multi-task learning model corresponds one-to-one to the application method of the multi-task learning model in the above embodiment, wherein the multi-task learning model is based on Figure 2 The training method of the multi-task learning model in the embodiment shown is obtained by training. Figure 5 As shown, the application device of the multi-task learning model includes a video description generation module 101 and a video clip retrieval module 102.

[0106] The video description generation module 101 is used to input the video material into the trained multi-task learning model to obtain the video description text corresponding to the video material.

[0107] The video segment retrieval module 102 is used to input the video material and the retrieval text into the trained multi-task learning model to obtain the start time and end time of the video sample segment in the video material that matches the retrieval text.

[0108] The application device of the multi-task learning model provided in the embodiment of the present invention is that the multi-task learning model is obtained by multi-task learning using video samples, corresponding query texts, video sample segments obtained by slicing the video samples, and the start and end times annotated by the video sample segments. Only one model is needed to complete the processing of the two tasks of video description generation and video segment retrieval, thereby reducing server resource consumption and reducing the later maintenance cost of the model. Moreover, when training the multi-task learning model, the mean square error loss function is calculated during the learning process of the video segment retrieval task, and the cross entropy loss function is calculated during the learning process of the video description generation task. The loss function of the entire model is calculated based on the mean square error loss function and the cross entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, thereby improving the feature alignment capability of the model and thus improving the accuracy of the model.

[0109] For the specific definition of the application device of the multi-task learning model, please refer to the definition of the application method of the multi-task learning model above, which will not be repeated here. Each module in the above-mentioned application device of the multi-task learning model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0110] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other external electronic devices through a network connection. When the computer program is executed by the processor, it is implemented Figure 2 Functions or steps of the training method of the multi-task learning model in the illustrated embodiment.

[0111] In one embodiment, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, Figure 2 Functions or steps of the training method of the multi-task learning model in the illustrated embodiment.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. The non-company software tools or components that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A training method for a multi-task learning model, characterized in that: include: Obtaining a video sample, a query text, a video sample segment group, and k frame vectors for model training, wherein the video sample segment group includes a plurality of video sample segments obtained by slicing the video sample, and the dimension of each frame vector is 64; Encode the query text using a text encoder to obtain a feature vector of the query text; Encode the video sample using 3D-CNN to obtain the feature vector of the video sample; Perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a cross-attention weighted vector; After concatenating the k box vectors, concatenate them with the weighted vector of the cross attention, perform self-attention calculation on the concatenated vector, and obtain the weighted vector of the self-attention; The k*64 dimensions of the weighted vector of self-attention are segmented to obtain k new box vectors, and the k new box vectors are respectively input into the first group of MLP layers with different parameters, and the mean square error loss function is calculated based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group; The k new box vectors are respectively input into the second set of MLP layers with shared parameters. The output results of the second set of MLP layers are passed through the text decoder to generate video text description, and the cross entropy loss function is calculated based on the results of video text description generation and the query text. The loss function of the entire model is calculated according to the mean square error loss function and the cross entropy loss function, the loss value gradient is calculated, and then backpropagation is performed to optimize the model parameters until convergence to obtain a trained multi-task learning model.

2. The training method of the multi-task learning model according to claim 1, characterized in that: The step of encoding the query text by using a text encoder to obtain a feature vector of the query text includes: Segment the query text into m words, each of which is a 1024-dimensional vector. Use the multi-head self-attention layer of the text encoder to perform self-attention operations on m words to obtain multiple weighted vectors; Multiple weighted vectors are concatenated in the order of word segmentation to restore them into a 1024-dimensional vector; The restored vector is normalized to obtain the feature vector of the query text.

3. The training method of the multi-task learning model according to claim 1, characterized in that: The method of encoding the video sample by 3D-CNN to obtain the feature vector of the video sample includes: Use 3D-CNN to encode video samples in time sequence to obtain video feature vectors; The obtained video feature vector is normalized to obtain the feature vector of the video sample.

4. The training method of the multi-task learning model according to claim 1, characterized in that: The cross-attention calculation is performed on the feature vector of the query text and the feature vector of the video sample to obtain a cross-attention weighted vector, including: The feature vector of the video sample is copied twice, and used as the K vector and V vector of the QKV matrix respectively. The feature vector of the query text is used as the Q vector of the QKV matrix. The transposition of the Q vector * K vector is then calculated through the softmax function to obtain a set of weight vectors, which are then multiplied with the V vector to obtain the cross-attention weighted vector.

5. The training method of the multi-task learning model according to claim 1, characterized in that: The k frame vectors are concatenated and then concatenated with the weighted vector of the cross attention, and the self-attention calculation is performed on the vector obtained after the concatenation to obtain the weighted vector of the self-attention, including: Concatenate the k box vectors to obtain a k*64 dimensional vector, and multiply the k*64 dimensional vector by a fully connected layer to obtain a new vector; Normalize the new vector and concatenate it with the weighted vector of the cross attention. The vector obtained by concatenating it with the weighted vector of the cross attention is copied three times as the Q vector, K vector and V vector of the QKV matrix respectively. The transpose of the Q vector and the K vector are multiplied and then calculated by the softmax function. The obtained weight vector is then multiplied with the V vector to obtain the weighted vector of self-attention.

6. A method for applying a multi-task learning model, characterized in that: include: Video description generation: input the video material into the trained multi-task learning model to obtain the video description text corresponding to the video material; Video clip retrieval: input the video material and the retrieval text into the trained multi-task learning model to obtain the start time and end time of the video sample clip in the video material that matches the retrieval text; The multi-task learning model is trained according to the training method of the multi-task learning model according to any one of claims 1 to 5.

7. A training device for a multi-task learning model, characterized in that: include: A training data acquisition module, used to acquire video samples, query text, video sample segment groups and k frame vectors for model training, wherein the video sample segment group includes multiple video sample segments obtained by slicing the video sample, and the dimension of each frame vector is 64; A text encoding module, used for encoding the query text using a text encoder to obtain a feature vector of the query text; A video encoding module is used to encode video samples using 3D-CNN to obtain feature vectors of video samples; A first attention calculation module is used to perform cross attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a cross attention weighted vector; The second attention calculation module is used to concatenate the k box vectors and then concatenate them with the weighted vector of the cross attention, and perform self-attention calculation on the concatenated vector to obtain the weighted vector of the self-attention; A first loss calculation module is used to segment the rear k*64 dimensions of the self-attention weighted vector to obtain k new box vectors, input the k new box vectors into a first group of MLP layers with different parameters, and calculate a mean square error loss function based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group; A second loss calculation module is used to input the k new box vectors into the second group of MLP layers with shared parameters respectively, and the output results of the second group of MLP layers are passed through a text decoder to generate a video text description, and a cross entropy loss function is calculated based on the result of the video text description generation and the query text; The model optimization module is used to calculate the loss function of the entire model based on the mean square error loss function and the cross entropy loss function, calculate the loss value gradient, and then perform backpropagation to optimize the model parameters until convergence to obtain a trained multi-task learning model.

8. An application device of a multi-task learning model, characterized in that: include: The video description generation module is used to input the video material into the trained multi-task learning model to obtain the video description text corresponding to the video material; A video clip retrieval module is used to input the video material and the retrieval text into the trained multi-task learning model to obtain the start time and end time of the video sample clip in the video material that matches the retrieval text; The multi-task learning model is trained according to the training method of the multi-task learning model according to any one of claims 1 to 5.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the training method of the multi-task learning model as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed by a processor, the steps of the training method of the multi-task learning model as described in any one of claims 1 to 5 are implemented.

Images