Training, application method, device and equipment of multi-task learning model and medium

Abstract

This invention relates to the fields of artificial intelligence and machine learning, and discloses a training and application method, apparatus, device, and medium for a multi-task learning model. This invention utilizes video samples, corresponding query text, video sample segments, and the start and end times of the labeled video sample segments to perform multi-task learning to obtain a multi-task learning model. Only one model needs to be trained to complete both video description generation and video segment retrieval, reducing server resource consumption and lowering the later maintenance costs of the model. Furthermore, during multi-task learning, the mean squared error loss function and cross-entropy loss function are calculated separately. Based on these functions, the loss function of the entire model is calculated, and the model parameters are optimized based on the overall model loss function. The video description generation task and the video segment retrieval task help each other and learn together, improving the model's feature alignment ability and thus enhancing the model's accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning, and in particular to a method, apparatus, computer device, and storage medium for training and applying a multi-task learning model. Background Technology

[0002] In insurance marketing projects, agents can quickly obtain video information during the marketing process by reading video description text without having to browse the videos. At the same time, agents can quickly and accurately locate video segments in a long video through video clip retrieval and use the retrieved video segments for marketing to enhance user engagement, which greatly improves the work efficiency of insurance agents.

[0003] In existing technologies, video description generation and video segment retrieval are two independent tasks, each treated as a separate machine learning task; that is, both video description generation and video segment retrieval are single-task learning processes. The video description generation task uses text-video alignment to encode video features, then uses a text encoder to generate descriptive text, thereby automating the text description of the video content. The video segment retrieval task first semantically segments the video into multiple smaller video segments, then calculates the similarity between the text and the video segment features to determine which video segment is most similar to the current text, and then retrieves the corresponding segment.

[0004] The inventors discovered through research that both video description generation and video clip retrieval tasks utilize semantic feature alignment methods between video and text, and both tasks are cross-modal tasks involving video and text. Existing technologies often employ separate single-task learning for video description generation and video clip retrieval, resulting in low model feature alignment capabilities and impacting model accuracy. Furthermore, task deployment requires deploying two different models on a server to handle these two tasks, consuming significant server resources and incurring high maintenance costs. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a training and application method, apparatus, computer equipment and storage medium for a multi-task learning model, so as to solve the technical problems that when video description generation task and video segment retrieval task are learned separately in single task, the model has low feature alignment ability, which affects the accuracy of the model, as well as the model requires a lot of server resources and has high maintenance costs.

[0006] Firstly, a training method for a multi-task learning model is provided, including:

[0007] Obtain video samples, query text, video sample segment groups, and k bounding box vectors for model training. The video sample segment groups include multiple video sample segments obtained by slicing the video samples, and each bounding box vector has a dimension of 64.

[0008] The query text is encoded using a text encoder to obtain the feature vector of the query text;

[0009] The video samples are encoded using 3D-CNN to obtain the feature vectors of the video samples;

[0010] Cross-attention is calculated between the feature vector of the query text and the feature vector of the video sample to obtain a weighted vector of cross-attention.

[0011] The k box vectors are concatenated and then concatenated with the weighted vector of cross attention. Self-attention is calculated on the concatenated vector to obtain the weighted vector of self-attention.

[0012] The weighted vector of self-attention is split into k*64 dimensions to obtain k new bounding box vectors. The k new bounding box vectors are then input into the first group of MLP layers with different parameters. Based on the output of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group, the mean squared error loss function is calculated.

[0013] The k new bounding box vectors are input into the second set of shared parameters MLP layers respectively. The output of the second set of MLP layers is processed by a text decoder to generate video text descriptions. The cross-entropy loss function is calculated based on the video text descriptions and the query text.

[0014] The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function. The gradient of the loss value is calculated, and then backpropagation is performed to optimize the model parameters until convergence, resulting in a trained multi-task learning model.

[0015] Secondly, a method for applying a multi-task learning model is provided. The multi-task learning model is trained using the aforementioned training method, and the method for applying the multi-task learning model includes:

[0016] Video description generation involves inputting video footage into a trained multi-task learning model to obtain the corresponding video description text.

[0017] Video clip retrieval involves inputting video footage and search text into a trained multi-task learning model to obtain the start and end times of video sample clips in the video footage that match the search text.

[0018] Thirdly, a training device for a multi-task learning model is provided, comprising:

[0019] The training data acquisition module is used to acquire video samples, query text, video sample segment groups, and k bounding box vectors for model training. The video sample segment group includes multiple video sample segments obtained by slicing the video samples, and each bounding box vector has a dimension of 64.

[0020] The text encoding module is used to encode the query text using a text encoder to obtain the feature vector of the query text.

[0021] The video encoding module is used to encode video samples using 3D-CNN to obtain the feature vectors of the video samples;

[0022] The first attention calculation module is used to perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain the weighted vector of cross-attention.

[0023] The second attention calculation module is used to concatenate k box vectors and then concatenate them with the weighted vector of cross attention. The concatenated vector is then used to calculate self-attention and obtain the weighted vector of self-attention.

[0024] The first loss calculation module is used to segment the last k*64 dimensions of the self-attention weighted vector to obtain k new bounding box vectors. The k new bounding box vectors are input into the first group of MLP layers with different parameters respectively. Based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group, the mean squared error loss function is calculated.

[0025] The second loss calculation module is used to input k new bounding box vectors into the second set of shared parameters MLP layers respectively. The output of the second set of MLP layers is processed by a text decoder to generate video text descriptions, and the cross-entropy loss function is calculated based on the video text descriptions and the query text.

[0026] The model optimization module is used to calculate the loss function of the entire model based on the mean squared error loss function and the cross-entropy loss function, calculate the gradient of the loss value, and then perform backpropagation to optimize the model parameters until convergence, thus obtaining a trained multi-task learning model.

[0027] Fourthly, an application device for a multi-task learning model is provided, wherein the multi-task learning model is trained according to the training method described above, and the application device for the multi-task learning model includes:

[0028] The video description generation module is used to input video footage into a trained multi-task learning model and obtain video description text corresponding to the video footage.

[0029] The video clip retrieval module is used to input video footage and search text into a trained multi-task learning model to obtain the start and end times of video sample clips in the video footage that match the search text.

[0030] Fifthly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method for the multi-task learning model described above.

[0031] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the steps of the training method for the above-described multi-task learning model are implemented.

[0032] The training, application methods, apparatus, computer equipment, and storage media of the aforementioned multi-task learning model utilize video samples, corresponding query text, video sample segments obtained by slicing the video samples, and the start and end times of the video sample segments to perform multi-task learning and obtain a multi-task learning model. Only one model needs to be trained to complete both video description generation and video segment retrieval tasks, reducing server resource consumption and lowering the later maintenance costs of the model. Furthermore, during multi-task learning, the mean squared error loss function is calculated during the video segment retrieval task, and the cross-entropy loss function is calculated during the video description generation task. The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function, and the model parameters are optimized based on the overall model loss function. The video description generation task and the video segment retrieval task help each other and learn together, improving the model's feature alignment ability and thus improving the model's accuracy. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is an exemplary system architecture diagram in which the present invention can be applied;

[0035] Figure 2 This is a flowchart illustrating a training method for a multi-task learning model according to an embodiment of the present invention.

[0036] Figure 3 This is a flowchart illustrating an application method of a multi-task learning model according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a training device for a multi-task learning model according to an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the structure of an application device for a multi-task learning model according to an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the system architecture includes a client, a network, and a server. The network serves as the medium providing a communication link between the client and the server; that is, the client can communicate with the server through the network. Networks include various connection types, such as wired, wireless communication links, or fiber optic cables. Users can use the client to interact with the server through the network to receive or send messages, etc. Various communication client applications can be installed on the client, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platforms, etc. The server can be a server that provides various services, such as a backend server that supports the pages displayed on the client.

[0042] This application provides a method, apparatus, computer device, and medium for training and applying a multi-task learning model. The training and application methods for the multi-task learning model can be applied to a server. Clients can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments; unless otherwise specified, the following embodiments and features can be combined with each other.

[0043] Figure 2 A flowchart illustrating the training method of the multi-task learning model provided in this embodiment of the invention is shown below. Figure 2 As shown, the training method of the multi-task learning model includes steps S10 to S80:

[0044] S10: Obtain video samples, query text, video sample segment groups, and k bounding box vectors for model training, wherein the video sample segment groups include multiple video sample segments obtained by slicing the video samples.

[0045] This step is mainly used to obtain the training dataset for model training. The training dataset includes several video samples, several query texts corresponding to the video samples, several groups of video sample segments, and k bounding box vectors.

[0046] For a given video sample, based on the completeness of semantic expression, the video sample is manually edited and segmented into several video sample segments. The start and end times of each video sample segment are marked, and the video sample segments are manually annotated with text summaries describing the content of the current video sample segment.<V,S> =<v1,s1,ts1,te1> ,<v2,s2,ts2,te2> ,<v3,s3,ts3,te3> ,..., <v n ,s n ,ts n ,te n >, V represents the entire video sample, v i Let s represent the i-th video sample segment. i Let S represent the text description of the i-th video sample segment, and let S be the sum of all s. i The text is concatenated as a whole, ts i Let te represent the start time of the i-th video sample segment. i Let i represent the termination time of the i-th video sample segment, where i = 1, 2, 3, ..., n.

[0047] A video sample is divided into several video sample segments by slicing a single video sample. These segments are then grouped into several video sample segment groups. It should be noted that in other embodiments, different slicing standards can be used to slice a video sample. For example, an average duration slicing standard can be used to slice a video sample, which means dividing a video sample into multiple video sample segments of equal duration.

[0048] The bounding box vectors can be manually set, with initial values ​​randomized. The number of bounding box vectors (k) can be set as needed; for ease of matrix calculation, it can be a power of 2. For example, in this embodiment, 16 bounding box vectors are manually set, each with a dimension of 64, for a total of 1024 dimensions. These 16 bounding box vectors serve as input to the model. After training, the model will output 16 vectors, resulting in 16 salient bounding box feature vectors. These 16 feature vectors determine how a video sample is sliced ​​into 16 video sample segments.

[0049] The video samples, query text, video sample segment groups, and bounding box vectors can be stored on the server or client. Before training the multi-task learning model, the video samples, query text, video sample segment groups, and bounding box vectors need to be obtained from the server or client. Then, a video sample, a query text corresponding to the video sample, a video sample segment group corresponding to the video sample, and k bounding box vectors are used as the input to the model for model training.

[0050] S20: Use a text encoder to encode the query text to obtain the feature vector of the query text.

[0051] Step S20 includes the following steps:

[0052] S21: Segment the query text to obtain m words, and use a text encoder to encode each word into a 1024-dimensional vector.

[0053] S22: Use the multi-head self-attention layer of the text encoder to perform self-attention operations on m words to obtain multiple weighted vectors.

[0054] S23: Concatenate multiple weighted vectors according to the word segmentation order to restore a 1024-dimensional vector.

[0055] S24: Normalize the restored vector to obtain the feature vector of the query text.

[0056] For steps S21-S24, after segmenting the query text into m words, a text encoder is used to encode these m words. The text encoder is a self-attention mechanism layer. In this embodiment, the self-attention layer of the text encoder has 8 heads. Therefore, each attention head processes 128 dimensions out of the 1024 dimensions of the m word vectors, that is, each attention head performs self-attention operation on the m words. Specifically, the 128-dimensional vectors of the m words are copied 3 times and multiplied by three 128*128 matrices Q, K, and V respectively. The output result is obtained by multiplying the output of the Q matrix and the output of the V matrix, and then passing it through a normalized exponential function (or softmax function) to obtain a 128-dimensional attention weight. Then, the corresponding dimension weight value of this attention weight is multiplied by the output result of the V matrix to obtain a 128-dimensional weighted vector. Since the self-attention layer has 8 heads, performing self-attention operations on m words will result in 8 weighted vectors of 128 dimensions. These 8 weighted vectors are then concatenated according to the word segmentation order in step S21 to reconstruct a single 1024-dimensional vector. The reconstructed vector is then normalized by subtracting its mean and dividing by its variance. The resulting 1024-dimensional vector is the feature vector of the query text.

[0057] S30: Encode the video samples using 3D-CNN to obtain the feature vectors of the video samples.

[0058] Three-dimensional convolutional neural networks (3D-CNN) are mature model structures that can encode images temporally. Encoding video samples using 3D-CNN yields a 1024-dimensional video feature vector. This 1024-dimensional video feature vector is then normalized by subtracting the mean and dividing by the variance, ultimately resulting in the preprocessed feature vector of the video sample.

[0059] S40: Perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain the weighted vector of cross-attention.

[0060] This step requires calculating cross-attention between the feature vectors of the query text and the feature vectors of the video samples. Specifically, the feature vectors of the video samples are copied twice, becoming the K and V vectors of the QKV matrix, respectively, and the feature vectors of the query text are used as the Q vector of the QKV matrix. The transpose of the Q vector multiplied by the K vector is then processed using the softmax function to obtain a set of weight vectors. These weight vectors are then multiplied by the V vector to obtain the cross-attention weighted vector, denoted as V_W. This cross-attention weighted vector is based on the feature vectors of the video samples and weights the query text and video samples accordingly.

[0061] S50: Concatenate the k bounding box vectors and then concatenate them with the weighted vector of cross attention. Perform self-attention calculation on the concatenated vector to obtain the weighted vector of self-attention.

[0062] Step S50 includes the following steps:

[0063] S51: Concatenate the k box vectors to obtain a k*64 dimension vector, and multiply the k*64 dimension vector by a fully connected layer to obtain a new vector;

[0064] S52: After normalizing the new vector, concatenate it with the weighted vector of cross-attention;

[0065] S53: After concatenating the weighted vector with the cross-attention vector, make three copies of the resulting vector and use them as the Q vector, K vector, and V vector of the QKV matrix, respectively. Multiply the transposes of the Q vector and K vector and then calculate the weight vector using the softmax function. Multiply the resulting weight vector with the V vector to obtain the weighted vector of self-attention.

[0066] The number of box vectors (k) can be set as needed before model training. In this embodiment, k=16 is used as an example to illustrate the process of steps S51-S53: The 16 box vectors are concatenated to obtain a 1024-dimensional vector. This 1024-dimensional vector is multiplied by a 1024*1024 fully connected layer to obtain a new 1024-dimensional vector. This new 1024-dimensional vector is normalized by subtracting the mean and dividing by the variance. After normalization, it is concatenated with V_W to form a 2048-dimensional vector. Self-attention is calculated on this 2048-dimensional vector. It is copied three times and multiplied by a new QKV matrix. The transposes of the Q and K vectors are multiplied and then processed by the softmax function to obtain the weight vector. This weight vector is then multiplied by the V vector to obtain the self-attention weighted vector, denoted as S_W. This self-attention weighted vector fuses the vector features of 16 boxes, and the corresponding dimensions represent the corresponding features. For example, the first 1024 dimensions still represent the features of video-fused text, and the last 1024 dimensions each represent the features of a box.

[0067] S60: Segment the last k*64 dimensions of the self-attention weighted vector to obtain k new bounding box vectors. Input the k new bounding box vectors into the first group of MLP layers with different parameters respectively. Calculate the mean squared error loss function based on the output of the first group of MLP layers and the start and end times of the corresponding video sample segments within the video sample segment group.

[0068] The number of bounding box vectors (k) can be set as needed before model training. In this embodiment, step S60 is illustrated using k=16 as an example: the 1024 dimensions after S_W are divided into 64-dimensional units, resulting in 16 new bounding box vectors. These 16 new bounding box vectors are then input into the first group of 16 MLP layers with different parameters. Each MLP layer outputs a 2D vector, where the first value represents the start time of the current bounding box and the second value represents the end time. Based on the output of the first group of 16 MLP layers with different parameters and the start and end times of the corresponding video sample segments within the video sample segment group, the mean squared error (MSE) loss function, denoted as L_MSE, is calculated. It is important to note that if the number of video sample segments obtained by slicing a video sample is less than 16, the extra bounding boxes will not be included in the calculation of the mean squared error loss function; if the number of video sample segments obtained by slicing a video sample is greater than 16, then only the first 16 video sample segments in chronological order will be used for the calculation of the mean squared error loss function.

[0069] S70: Input the k new bounding box vectors into the second set of shared parameters MLP layers respectively. The output of the second set of MLP layers is processed by a text decoder to generate video text descriptions. Based on the video text descriptions and the query text, the cross-entropy loss function is calculated.

[0070] The number of bounding box vectors (k) can be set as needed before model training. In this embodiment, k=16 is used as an example to illustrate the process of step S70: The 16 new bounding box vectors obtained by the 1024-dimensional segmentation after S_W are input into the second group of 16 shared parameter MLP layers. The 16 shared parameter MLP layers have 2 layers, where the first layer parameter matrix is ​​64*256 and the second layer parameter matrix is ​​256*1024. The 16 shared parameter MLP layers output 16 vectors of 1024 dimensions. Then, a transformer-structured text decoder is used to generate video text descriptions. The cross-entropy loss function, denoted as L_CE, is calculated based on the result of the generated video text description and the query text.

[0071] S80: Calculate the loss function of the entire model based on the mean squared error loss function and the cross-entropy loss function, calculate the gradient of the loss value, and then perform backpropagation to optimize the model parameters until convergence, thus obtaining the trained multi-task learning model.

[0072] The multi-task learning model training method provided in this embodiment of the invention utilizes video samples, corresponding query text, video sample segments obtained by slicing the video samples, and the start and end times of the video sample segments to perform multi-task learning to obtain a multi-task learning model. Only one model needs to be trained to complete the processing of both video description generation and video segment retrieval tasks, reducing server resource consumption and lowering the later maintenance cost of the model. Moreover, during multi-task learning, the mean squared error loss function is calculated during the video segment retrieval task learning process, and the cross-entropy loss function is calculated during the video description generation task learning process. The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, improving the feature alignment ability of the model, thereby improving the accuracy of the model.

[0073] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0074] Figure 3 This is a flowchart illustrating an application method of a multi-task learning model provided in an embodiment of the present invention, wherein the multi-task learning model is based on... Figure 2 The multi-task learning model shown in the embodiment is trained using the training method described above. Figure 3 As shown, the application method of the multi-task learning model includes steps S101 to S102:

[0075] S101: Video description generation. Input the video footage into the trained multi-task learning model to obtain the video description text corresponding to the video footage.

[0076] S102: Video segment retrieval, inputting video footage and search text into a trained multi-task learning model to obtain the start and end times of video sample segments in the video footage that match the search text.

[0077] The application method of the multi-task learning model provided by this invention can be applied to various application scenarios. For example, in the field of insurance applications, if a user only inputs video material into the multi-task learning model, the model will process the video material and output corresponding video description text. The user can quickly obtain video information without browsing the video, which greatly improves the work efficiency of insurance agents. If a user inputs video material and search text into the multi-task learning model, the model will output the start and end times of the video sample segments in the video material that match the search text, thereby quickly and accurately locating video segments in a long video.

[0078] The multi-task learning model provided in this embodiment of the invention utilizes video samples, corresponding query text, video sample segments obtained by slicing the video samples, and the start and end times of the video sample segments to perform multi-task learning. Only one model is needed to complete both video description generation and video segment retrieval tasks, reducing server resource consumption and lowering the later maintenance costs of the model. Furthermore, during the training of the multi-task learning model, the mean squared error loss function is calculated during the video segment retrieval task, and the cross-entropy loss function is calculated during the video description generation task. The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function, and the model parameters are optimized based on the overall model loss function. The video description generation task and the video segment retrieval task help each other and learn together, improving the model's feature alignment ability and thus improving the model's accuracy.

[0079] In one embodiment, a training apparatus for a multi-task learning model is provided, which corresponds one-to-one with the training method for the multi-task learning model in the above embodiments. For example... Figure 4 As shown, the training device for this multi-task learning model includes a training data acquisition module 10, a text encoding module 20, a video encoding module 30, a first attention calculation module 40, a second attention calculation module 50, a first loss calculation module 60, a second loss calculation module 70, and a model optimization module 80. Detailed descriptions of each functional module are as follows:

[0080] The training data acquisition module 10 is used to execute step S10 in the training method of the multi-task learning model in the above embodiment, that is, to acquire video samples, query text, video sample segment groups and k bounding box vectors for model training, wherein the video sample segment group includes multiple video sample segments obtained by slicing the video samples, and each bounding box vector has a dimension of 64.

[0081] The text encoding module 20 is used to execute step S20 in the training method of the multi-task learning model in the above embodiment, that is, to use the text encoder to encode the query text and obtain the feature vector of the query text.

[0082] The video encoding module 30 is used to execute step S30 in the training method of the multi-task learning model in the above embodiment, that is, to encode the video samples using 3D-CNN to obtain the feature vector of the video samples.

[0083] The first attention calculation module 40 is used to execute step S40 in the training method of the multi-task learning model in the above embodiment, that is, to perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain the weighted vector of cross-attention.

[0084] The second attention calculation module 50 is used to execute step S50 in the training method of the multi-task learning model in the above embodiment, that is, to concatenate k box vectors and then concatenate them with the weighted vector of cross attention, and to perform self-attention calculation on the concatenated vector to obtain the weighted vector of self-attention.

[0085] The first loss calculation module 60 is used to execute step S60 in the training method of the multi-task learning model in the above embodiment, that is, to divide the last k*64 dimensions of the self-attention weighted vector to obtain k new bounding box vectors, input the k new bounding box vectors into the first group of MLP layers with different parameters respectively, and calculate the mean squared error loss function based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group.

[0086] The second loss calculation module 70 is used to execute step S70 in the training method of the multi-task learning model in the above embodiment, that is, to input k new box vectors into the second set of shared parameters MLP layers respectively, the output results of the second set of MLP layers are processed by the text decoder to generate video text description, and the cross-entropy loss function is calculated based on the result of the video text description generation and the query text.

[0087] The model optimization module 80 is used to execute step S80 in the training method of the multi-task learning model in the above embodiment, that is, to calculate the loss function of the entire model based on the mean squared error loss function and the cross-entropy loss function, calculate the gradient of the loss value, and then perform backpropagation to optimize the model parameters until convergence, so as to obtain the trained multi-task learning model.

[0088] In one embodiment, the text encoding module 20 is specifically used for:

[0089] The query text is segmented into m words, each of which is a 1024-dimensional vector;

[0090] The text encoder uses a multi-head self-attention layer to perform self-attention operations on m words, resulting in multiple weighted vectors;

[0091] Multiple weighted vectors are concatenated according to the word segmentation order to restore a 1024-dimensional vector;

[0092] The restored vector is normalized to obtain the feature vector of the query text.

[0093] In one embodiment, the video encoding module 30 is specifically used for:

[0094] 3D-CNN is used to encode video samples in a temporal manner to obtain video feature vectors;

[0095] The obtained video feature vectors are normalized to obtain the feature vectors of the video samples.

[0096] In one embodiment, the first attention calculation module 40 is specifically used for:

[0097] The feature vectors of the video samples are copied twice and used as the K vector and V vector of the QKV matrix, respectively. The feature vectors of the query text are used as the Q vector of the QKV matrix.

[0098] The transpose of Q vector * K vector is then processed by the softmax function to obtain a set of weight vectors. These weight vectors are then multiplied by V vector to obtain the weighted vector for cross-attention.

[0099] In one embodiment, the second attention calculation module 50 is specifically used for:

[0100] Concatenate k bounding box vectors to obtain a k*64 dimensional vector. Multiply the k*64 dimensional vector by a fully connected layer to obtain a new vector.

[0101] The new vector is normalized and then concatenated with the weighted vector of the cross-attention.

[0102] The vector obtained by concatenating the weighted vector of cross attention is copied three times and used as the Q vector, K vector and V vector of the QKV matrix respectively. The transpose of the Q vector and K vector are multiplied and then processed by the softmax function. The resulting weight vector is then multiplied by the V vector to obtain the weighted vector of self attention.

[0103] The training device for the multi-task learning model provided in this embodiment of the invention utilizes video samples, corresponding query text, video sample segments obtained by slicing the video samples, and the start and end times of the video sample segments to perform multi-task learning to obtain a multi-task learning model. Only one model needs to be trained to complete the processing of both video description generation and video segment retrieval tasks, reducing server resource consumption and lowering the later maintenance cost of the model. Moreover, during multi-task learning, the mean squared error loss function is calculated during the video segment retrieval task learning process, and the cross-entropy loss function is calculated during the video description generation task learning process. The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, improving the feature alignment ability of the model, thereby improving the accuracy of the model.

[0104] Specific limitations regarding the training apparatus for the multi-task learning model can be found in the limitations on the training method for the multi-task learning model described above, and will not be repeated here. Each module in the training apparatus for the aforementioned multi-task learning model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0105] Figure 5 This is a flowchart illustrating an application device for a multi-task learning model provided in an embodiment of the present invention. This application device corresponds one-to-one with the application method of the multi-task learning model in the above embodiments. The multi-task learning model is based on... Figure 2 The multi-task learning model shown in the embodiment is trained using the training method described above. Figure 5 As shown, the application device of the multi-task learning model includes a video description generation module 101 and a video segment retrieval module 102.

[0106] The video description generation module 101 is used to input video footage into a trained multi-task learning model to obtain video description text corresponding to the video footage.

[0107] The video clip retrieval module 102 is used to input video footage and retrieval text into a trained multi-task learning model to obtain the start and end times of video sample clips in the video footage that match the retrieval text.

[0108] The multi-task learning model application device provided in this embodiment of the invention utilizes video samples, corresponding query text, video sample segments obtained by slicing the video samples, and the start and end times of the video sample segments to perform multi-task learning. Only one model is needed to complete the processing of both video description generation and video segment retrieval tasks, reducing server resource consumption and lowering the later maintenance cost of the model. Moreover, during the training of the multi-task learning model, the mean squared error loss function is calculated during the learning process of the video segment retrieval task, and the cross-entropy loss function is calculated during the learning process of the video description generation task. The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function, and the model parameters are optimized based on the loss function of the entire model. The video description generation task and the video segment retrieval task help each other and learn together, improving the feature alignment ability of the model, thereby improving the accuracy of the model.

[0109] Specific limitations regarding the application device for the multi-task learning model can be found in the above description of the application method for the multi-task learning model, and will not be repeated here. Each module in the application device of the aforementioned multi-task learning model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0110] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with other external electronic devices via a network connection. When the computer program is executed by the processor, it implements… Figure 2 The functions or steps of the training method for the multi-task learning model in the illustrated embodiment.

[0111] In one embodiment, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program being implemented when executed by a processor. Figure 2 The functions or steps of the training method for the multi-task learning model in the illustrated embodiment.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The above-described embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

Claims

1. A training method for a multi-task learning model, characterized in that, include: Obtain video samples, query text, video sample segment groups, and k bounding box vectors for model training. The video sample segment groups include multiple video sample segments obtained by slicing the video samples, and each bounding box vector has a dimension of 64. The query text is encoded using a text encoder to obtain the feature vector of the query text; The video samples are encoded using 3D-CNN to obtain the feature vectors of the video samples; Cross-attention is calculated between the feature vector of the query text and the feature vector of the video sample to obtain a weighted vector of cross-attention. The k box vectors are concatenated and then concatenated with the weighted vector of cross attention. Self-attention is calculated on the concatenated vector to obtain the weighted vector of self-attention. The weighted vector of self-attention is split into k*64 dimensions to obtain k new bounding box vectors. The k new bounding box vectors are then input into the first group of MLP layers with different parameters. Based on the output of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group, the mean squared error loss function is calculated. The k new bounding box vectors are input into the second set of shared parameters MLP layers respectively. The output of the second set of MLP layers is processed by a text decoder to generate video text descriptions. The cross-entropy loss function is calculated based on the video text descriptions and the query text. The loss function of the entire model is calculated based on the mean squared error loss function and the cross-entropy loss function. The gradient of the loss value is calculated, and then backpropagation is performed to optimize the model parameters until convergence, resulting in a trained multi-task learning model.

2. The training method for the multi-task learning model as described in claim 1, characterized in that, The step of using a text encoder to encode the query text to obtain the feature vector of the query text includes: The query text is segmented into m words, each of which is a 1024-dimensional vector; The text encoder uses a multi-head self-attention layer to perform self-attention operations on m words, resulting in multiple weighted vectors; Multiple weighted vectors are concatenated according to the word segmentation order to restore a 1024-dimensional vector; The restored vector is normalized to obtain the feature vector of the query text.

3. The training method for the multi-task learning model as described in claim 1, characterized in that, The process of encoding video samples using 3D-CNN to obtain feature vectors for the video samples includes: 3D-CNN is used to encode video samples in a temporal manner to obtain video feature vectors; The obtained video feature vectors are normalized to obtain the feature vectors of the video samples.

4. The training method for the multi-task learning model as described in claim 1, characterized in that, The step of performing cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain a weighted vector for cross-attention includes: The feature vectors of the video samples are copied twice and used as the K vector and V vector of the QKV matrix, respectively. The feature vectors of the query text are used as the Q vector of the QKV matrix. The transpose of Q vector * K vector is then processed by the softmax function to obtain a set of weight vectors. These weight vectors are then multiplied by V vector to obtain the weighted vector for cross-attention.

5. The training method for the multi-task learning model as described in claim 1, characterized in that, The process of concatenating k bounding box vectors and then concatenating them with the weighted vector of cross-attention, followed by performing self-attention calculation on the concatenated vector to obtain the self-attention weighted vector, includes: Concatenate k bounding box vectors to obtain a k*64 dimensional vector. Multiply the k*64 dimensional vector by a fully connected layer to obtain a new vector. The new vector is normalized and then concatenated with the weighted vector of the cross-attention. The vector obtained by concatenating the weighted vector of cross attention is copied three times and used as the Q vector, K vector and V vector of the QKV matrix respectively. The transpose of the Q vector and K vector are multiplied and then processed by the softmax function. The resulting weight vector is then multiplied by the V vector to obtain the weighted vector of self attention.

6. A method for applying a multi-task learning model, characterized in that, include: Video description generation involves inputting video footage into a trained multi-task learning model to obtain the corresponding video description text. Video clip retrieval involves inputting video footage and search text into a trained multi-task learning model to obtain the start and end times of video sample clips in the video footage that match the search text. The multi-task learning model is trained using the training method for the multi-task learning model according to any one of claims 1 to 5.

7. A training device for a multi-task learning model, characterized in that, include: The training data acquisition module is used to acquire video samples, query text, video sample segment groups, and k bounding box vectors for model training. The video sample segment group includes multiple video sample segments obtained by slicing the video samples, and each bounding box vector has a dimension of 64. The text encoding module is used to encode the query text using a text encoder to obtain the feature vector of the query text. The video encoding module is used to encode video samples using 3D-CNN to obtain the feature vectors of the video samples; The first attention calculation module is used to perform cross-attention calculation on the feature vector of the query text and the feature vector of the video sample to obtain the weighted vector of cross-attention. The second attention calculation module is used to concatenate k box vectors and then concatenate them with the weighted vector of cross attention. The concatenated vector is then used to calculate self-attention and obtain the weighted vector of self-attention. The first loss calculation module is used to segment the last k*64 dimensions of the self-attention weighted vector to obtain k new bounding box vectors. The k new bounding box vectors are input into the first group of MLP layers with different parameters respectively. Based on the output results of the first group of MLP layers and the start and end times of the corresponding video sample segments in the video sample segment group, the mean squared error loss function is calculated. The second loss calculation module is used to input k new bounding box vectors into the second set of shared parameters MLP layers respectively. The output of the second set of MLP layers is processed by a text decoder to generate video text descriptions, and the cross-entropy loss function is calculated based on the video text descriptions and the query text. The model optimization module is used to calculate the loss function of the entire model based on the mean squared error loss function and the cross-entropy loss function, calculate the gradient of the loss value, and then perform backpropagation to optimize the model parameters until convergence, thus obtaining a trained multi-task learning model.

8. An application device for a multi-task learning model, characterized in that, include: The video description generation module is used to input video footage into a trained multi-task learning model and obtain video description text corresponding to the video footage. The video clip retrieval module is used to input video footage and search text into a trained multi-task learning model to obtain the start and end times of video sample clips in the video footage that match the search text. The multi-task learning model is trained using the training method for the multi-task learning model according to any one of claims 1 to 5.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the training method for the multi-task learning model as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the multi-task learning model as described in any one of claims 1 to 5.

Images