A cross-category video temporal localization method, system and storage medium based on adversarial multimodal domain adaptation

Abstract

The present invention discloses a cross-category video temporal localization method, system and storage medium based on adversarial multimodal domain adaptation, belonging to the field of computer vision. Videos of different categories and corresponding query texts are obtained, and visual features and text features are extracted; the visual features and text features of the target category video are calibrated with semantic information by a cross-modal feature calibrator; the visual features of the target category video are randomly masked and reconstructed by a video feature reconstructor; the video features and text features are fused by a cross-modal feature fusion device; the video features and text features are learned by a domain discriminator to express unimodal domain invariant features for the video features and text features, and the initial fused features are learned to express cross-modal domain invariant features; the final fused features of the source category video are predicted by a dual affine predictor. The present invention realizes temporal localization for cross-category videos and improves the generalization ability of the model to deal with unknown target videos.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a cross-category video temporal positioning method, system and storage medium based on adversarial multimodal domain adaptation. Background Art

[0002] The task of video temporal localization aims to give the temporal boundaries of video clips corresponding to the query text from unprocessed videos. Currently, traditional methods include fully supervised learning and weakly supervised learning. Fully supervised learning takes a long time in the training phase and requires a large amount of manually labeled data, which is time-consuming and laborious; although weakly supervised learning does not require a large amount of manually labeled data, due to the lack of sufficient labeled data, the model performance is far behind that of the fully supervised learning model. In addition, both of the above methods are based on the premise that the training data and the test data are distributed in the same way, without considering the domain offset between different categories of scenes in the real world. The models trained based on the above two methods have poor generalization ability for unknown category data and cannot meet the needs of real-world scenes well. Summary of the invention

[0003] In view of the above problems, the present invention provides a cross-category video temporal localization method based on adversarial multimodal domain adaptation to improve the generalization ability of the model to deal with unknown target data.

[0004] To this end, the technical solution adopted in the present invention is as follows:

[0005] In a first aspect, the present invention provides a cross-category video temporal localization method based on adversarial multimodal domain adaptation, comprising the following steps:

[0006] S1: Obtain the source category video, the target category video and the query text of each corresponding video, extract the initial visual features of the video and the initial text features of the query text, and encode them as the final visual features and text features;

[0007] S2: Using a cross-modal feature calibrator, semantic information is calibrated on the visual features and text features of the target category video obtained in step S1;

[0008] S3: randomly masking the visual features of the target category video obtained in step S1 and reconstructing the visual features through a video feature reconstructor to obtain a reconstructed visual feature;

[0009] S4: fusing the video features and text features obtained in step S1 through a cross-modal feature fuser to obtain initial fused features and final fused features of the source category video, as well as initial fused features and final fused features of the target category video;

[0010] S5: Performing unimodal domain invariant feature expression learning on the video features and text features obtained in step S1 through a domain discriminator, and performing cross-modal domain invariant feature expression learning on the initial fusion features obtained in step S4;

[0011] S6: predicting the final fusion features of the source category video obtained in step S4 through a dual affine predictor to obtain the prediction probabilities of all possible results corresponding to each query text, and the one with the largest prediction probability is the final prediction result.

[0012] In a second aspect, the present invention provides a cross-category video time positioning system based on adversarial multimodal domain adaptation, which is used to implement the above-mentioned cross-category video time positioning method based on adversarial multimodal domain adaptation.

[0013] In a third aspect, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned cross-category video temporal positioning method based on adversarial multimodal domain adaptation.

[0014] Compared with the prior art, the advantages of the present invention are:

[0015] The present invention proposes a basic training module, which is mainly used for supervised learning of labeled source category data; proposes an adversarial domain identification module, which is used to learn domain invariant features; proposes a cross-module feature calibration module, which is used to reduce the semantic gap between different modal features in target category data; and proposes a video reconstruction module, which is used to learn temporal semantic relations and identifiable feature expressions. The present invention proposes and implements the video time localization task for cross-category video data for the first time, and the experimental results show that the model proposed by the present invention has good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the overall architecture of a cross-category video temporal localization method based on adversarial multimodal domain adaptation according to an exemplary embodiment;

[0017] Figure 2 is a schematic diagram of an adversarial domain identification module according to an exemplary embodiment;

[0018] Figure 3 is a schematic diagram of a cross-modal feature calibration module according to an exemplary embodiment;

[0019] Figure 4 is a schematic diagram of a video feature reconstruction module according to an exemplary embodiment; Figure 5 The figure is a schematic diagram of a device terminal with data processing capability according to an exemplary embodiment. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The accompanying drawings are only schematic diagrams of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.

[0021] like Figure 1 As shown, the present invention proposes a cross-category video time positioning method based on adversarial multimodal domain adaptation, comprising the following steps:

[0022] S1: First, the semantic information of the source category video and the target category video is extracted through a 3D convolutional neural network to obtain visual features And the feature encoding is performed through the visual encoder to finally obtain the encoded visual features Then, the Glove model is used to extract the semantic information of the source category query text and the target category query text to obtain the text features. And the feature encoding is performed through the text encoder to finally obtain the encoded text features

[0023] This paper implements the temporal localization task for cross-category videos for the first time, where the input data includes labeled source category data and unlabeled target category data, so as to improve the generalization ability of the model for videos of unknown categories. The input data is defined as follows:

[0024]

[0025]

[0026] Among them, V s is the source category video, Q s is the query text set of source category videos, T s is the label set of source category data, Represents the i-th source video in the source category video and its corresponding query text, represents the real time boundary of the i-th query text matching the i-th source video in the source category video; V t is the target category video, Q t is the query text set of target category videos, T t is the label set of the target category data, which is empty in this embodiment; represents the i-th target video in the target category video and its corresponding query text; B represents the batch size, that is, the number of source videos or target videos and their corresponding query texts input into the model each time.

[0027] In this step, the same visual feature extractor and visual encoder are used to obtain and encode visual features of videos of different categories, and the same text feature extractor and text encoder are used to obtain and encode text features of query texts corresponding to videos of different categories.

[0028] Specifically, the implementation process of step S1 is:

[0029] S1-1: Extract the input source category video V through a 3D convolutional neural network s and target category video V t Semantic information of Extract the query text Q of the source category video through the Glove model s and the query text Q of the target category video t Semantic information of S1-2: Initial visual features obtained in step S1-1 and initial text features The images are projected to the same hidden dimension through a convolution layer and a linear projection layer respectively, and encoded through a multi-head attention layer to obtain the encoded visual features and text features as the final features for subsequent calculations; the encoded visual features and text features are expressed as:

[0030]

[0031]

[0032]

[0033]

[0034] Where n is the number of frames in each video, m is the query text length, and d is the hidden dimension. is the visual feature of the source category video, is the visual feature of the i-th source category video, is the visual feature of the jth frame of the i-th source category video, is the visual feature of the target category video, is the visual feature of the i-th target category video, is the visual feature of the jth frame of the i-th target category video, is the text feature of the query text of the source category video, is the text feature of the query text of the i-th source category video, is the text feature of the jth word in the query text of the i-th source category video, is the text features of the query text of the target category video, The text features of the query text of the i-th target category video, The text features of the jth word in the query text of the i-th target category video;

[0035] In this embodiment, the initial visual features are encoded A convolutional layer and a linear projection layer are recorded as visual encoders, which will be used to encode the initial text features. The convolutional layer and the linear projection layer of are denoted as text encoder.

[0036] S2: Visual features obtained in step S1 are calibrated by a cross-modal feature calibrator and text features Perform semantic information calibration, such as Figure 3 shown.

[0037] In this step, the cross-modal feature calibrator applies the loss function This makes the visual features and text features in the positive samples closer in the semantic space, and further reduces the similarity between the visual features and text features in the negative samples in the semantic space.

[0038] Loss Function The definition is as follows:

[0039]

[0040] Among them, l(.,.) is used to calculate the cosine similarity between vectors; is the visual feature obtained in step S1 and text features Positive samples obtained by averaging along the time axis;

[0041] and It is for and The corresponding negative samples selected, Z V - is for The negative sample set, Z Q - is for is a set of negative samples; Δ is the boundary.

[0042] The visual features and text features of the target category after encoding in step S1 are defined as follows:

[0043]

[0044] calculate and The average value along the time axis gives the positive sample:

[0045]

[0046] definition Negative samples Negative samples

[0047] The loss function of the cross-modal feature calibrator during training The calculation process is as follows:

[0048]

[0049] S3: The initial visual features of the target category video obtained in step S1 are reconstructed by the video feature reconstructor Randomly mask and reconstruct visual features to obtain the reconstructed visual features V recon , and further learn the temporal semantic relationship and discriminative features between visual features and text features, such as Figure 4 As shown;

[0050] S3-1: Randomly compare the initial visual features of the target category video obtained in step S1 Perform mask operation with β probability and encode it through visual encoder to obtain the encoded mask visual features

[0051] S3-2: The mask visual features obtained in step S3-1 are processed by the cross-modal feature fuser And the text features of the query text of the target category video obtained in step S1 Fusion is performed to obtain the initial fusion feature F of the mask video m ;

[0052] S3-3: Mask visual features obtained according to step S3-1 and the initial fusion feature F obtained in step S3-2 m Reconstruct the video features and obtain the reconstructed visual features V recon , the calculation process is as follows:

[0053]

[0054] Among them, Conv1D is a 1-dimensional convolutional layer, ReLU is the activation function, Represents an element-wise add operation.

[0055] In this embodiment, the loss calculation process of the video feature reconstructor in step S3 during the training process is as follows:

[0056]

[0057] Among them, MSELoss is the mean square error loss.

[0058] S4: The video features and text features obtained in step S1 are respectively fused through the cross-modal feature fusion and Perform feature fusion to obtain fusion feature F s , F t ,

[0059] S4-1: Calculate the visual features of the source category video obtained in step S1 and text features The cross-modal similarity matrix S∈R n×m ; In this embodiment, cosine similarity is used to calculate the cross-modal similarity matrix;

[0060] S4-2: Use the SoftMax function to normalize the cross-modal similarity matrix S obtained in step S4-1 along the rows and columns to obtain the similarity density matrix S r and S c ;

[0061] S4-3: Similarity density matrix S obtained in step S4-2 r and S c , calculate the video-query text attention matrix A v ∈R n×d and query text-video attention matrix A q ∈R m×d , the calculation process is as follows:

[0062]

[0063]

[0064] S4-4: Video-query text attention matrix A calculated according to step S4-3 v and query text-video attention matrix A q Calculate the initial fusion feature F of the source category video s , the calculation process is as follows:

[0065]

[0066] Among them, W f ∈R4d×d , b f ∈R d , both of which are learnable parameters; represents the initial fusion features of the i-th source category video, represents the initial fused features of the jth frame of the i-th source category video.

[0067] S4-5: Initial fusion feature F obtained according to step S4-4 s The final fusion features of the source category video are obtained through the multi-head attention mechanism in, represents the final fusion feature of the i-th source category video, represents the final fusion feature of the jth frame of the i-th source category video;

[0068] Similarly, according to the method of steps S4-1 to S4-5, Replace with Will Replace with Get the initial fusion feature F of the target category video t And the final fusion feature

[0069] S5: The video features obtained in step S1 are respectively processed by the domain identifier and text features Perform unimodal domain invariant feature expression learning and the initial fusion feature F obtained in step S4 s 、F t Conduct cross-modal domain invariant feature expression learning to reduce the domain gap between different modal features;

[0070] In this embodiment, Figure 2 As shown, the domain discriminator contains the visual feature discriminator D v , Text feature discriminator D q And the fusion feature discriminator D f .

[0071] For video features and text features Video feature discriminator D v and text feature discriminator D q The calculation process is as follows:

[0072]

[0073]

[0074] Among them, MLP vand MLP p It is a multi-layer perceptron for visual features and text features, GRL is the gradient reversal layer, and σ is the sigmoid function.

[0075] For the initial fusion feature F s / F t , fusion feature discriminator D f The calculation process is as follows:

[0076] D f (F k )=σ(MLP f (GRL(F k ))),k∈{s,t}

[0077] Among them, MLP f It is a multi-layer perceptron.

[0078] Video feature discriminator D v , Text feature discriminator D q And the fusion feature discriminator D f The output is a scalar between [0,1] indicating the probability that the input feature is from the target category.

[0079] During the training process, we first define the domain labels That is, the feature label from the source class is 0, and the feature label from the target class is 1.

[0080] The adversarial loss calculation process of the domain discriminator during training is as follows:

[0081]

[0082] Among them, BCELoss is the binary cross entropy loss.

[0083] S6: The final fusion feature of the source category video obtained in step S4 through the dual affine predictor The prediction is performed to obtain the predicted probabilities of all possible results corresponding to each query text. The one with the largest predicted probability among all possible results corresponding to the query text is the final prediction result.

[0084] S6-1: Define the candidate segments P = (fs, fe) of the query text matching for each video in the source category video, where fs is the start frame number, fe is the end frame number, fs, fe∈[1,n]. The final fusion features of each video obtained from step S4 are Extract the fusion features corresponding to the start frame and the end frame of the candidate segment Generate the hidden features r of the start frame and the end frame through the linear layer fs 、r fe ∈Rd , the calculation process is as follows:

[0085]

[0086] Where W1 / W2∈R 4d×d , b1 / b2∈R d are learnable parameters of the linear layer.

[0087] Since each video in the source category video consists of n video frames, all candidate segments can be represented as P = (fs, fe), fs∈[1,n], fe∈[1,n], fs≤fe.

[0088] S6-2: Introduce the hidden feature r obtained in step S6-1 fs 、r fe , through the biaffine mechanism, we get the matrix containing the predicted probabilities of all candidate video segments P The frames corresponding to the maximum value of matrix M are combined into the final prediction result. P The calculation process is as follows:

[0089]

[0090] Among them, U m ∈R d×d ,W m ∈R d ,b m ∈R is a learnable parameter, M P Represents the probability that the video segment P is the video segment corresponding to the query text.

[0091] In this embodiment, the dual affine predictor in step S6 uses a scaled intersection over union (IoU) function as a supervisory signal during the training process to complete the training process. The specific contents are as follows:

[0092] The IoU score of the video clip P P The calculation process is as follows:

[0093]

[0094] Where τ=(τ fs ,τ fe ) is the real time boundary of the query text matching. Max(Score) represents the IoU score matrix The maximum value in .

[0095] The loss calculation process of the biaffine predictor during training is as follows:

[0096]

[0097] Preferably, the final loss function during the training phase is It is expressed as:

[0098]

[0099] Among them, λ1, λ2, and λ3 are hyperparameters.

[0100] In the present embodiment, a cross-category video time positioning system based on adversarial multimodal domain adaptation is also provided, and the system is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible.

[0101] The system comprises:

[0102] A data acquisition and feature extraction module is used to acquire the source category video, the target category video and the query text of each corresponding video, extract the initial visual features of the video and the initial text features of the query text, and encode them as the final visual features and text features;

[0103] A cross-modal feature calibration module, which is used to perform semantic information calibration on the obtained visual features and text features of the target category video through a cross-modal feature calibrator;

[0104] A video feature reconstruction module, which is used to randomly mask the visual features of the target category video through a video feature reconstructor and reconstruct the visual features to obtain reconstructed visual features;

[0105] A cross-modal feature fusion module, which is used to fuse the obtained video features and text features through a cross-modal feature fuser to obtain initial fusion features and final fusion features of the source category video, as well as initial fusion features and final fusion features of the target category video;

[0106] A domain identification module, which is used to perform unimodal domain invariant feature expression learning on the obtained video features and text features respectively through a domain identifier, and to perform cross-modal domain invariant feature expression learning on the obtained initial fusion features;

[0107] The dual affine prediction module is used to predict the final fusion features of the source category video obtained through the dual affine predictor, and obtain the prediction probabilities of all possible results corresponding to each query text. The one with the largest prediction probability is the final prediction result.

[0108] The above-mentioned data acquisition and feature extraction module, cross-modal feature fusion module, and dual affine prediction module constitute the basic training module.

[0109] The implementation process of the functions and effects of each module in the above system is specifically described in the implementation process of the corresponding steps in the above method. For example, a specific process of the above system may be:

[0110] (1) Obtaining the source category video, the target category video, and the query text of each corresponding video, extracting the initial visual features of the video and the initial text features of the query text, and encoding them as the final visual features and text features;

[0111] (2) Using a cross-modal feature calibrator, the visual features and text features of the target category video obtained in step 1 are semantically calibrated;

[0112] (3) using a video feature reconstructor to randomly mask the visual features of the target category video obtained in step 1 and reconstruct the visual features to obtain reconstructed visual features;

[0113] (4) The video features and text features obtained in step 1 are fused by a cross-modal feature fuser to obtain initial fused features and final fused features of the source category video, as well as initial fused features and final fused features of the target category video;

[0114] (5) Using the domain discriminator, the video features and text features obtained in step 1 are respectively subjected to single-modal domain-invariant feature expression learning, and the initial fusion features obtained in step 4 are subjected to cross-modal domain-invariant feature expression learning;

[0115] (6) The final fusion features of the source category video obtained in step 4 are predicted through a dual affine predictor to obtain the prediction probabilities of all possible results corresponding to each query text. The one with the largest prediction probability is the final prediction result.

[0116] For the system embodiment, since it basically corresponds to the method embodiment, the specific implementation of each step can refer to the description of the method part, which will not be repeated here. The system embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0117] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of any device with data processing capabilities reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 5 As shown, it is a hardware structure diagram provided by this embodiment, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the system in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0118] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the above-mentioned cross-category video time positioning method based on adversarial multimodal domain adaptation is implemented.

[0119] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0120] The technical effect of the present invention is verified by experiments below.

[0121] The present invention uses the ActivityNet Captions dataset and the Charades-STA dataset for model training and verification, wherein the ActivityNet Captions dataset contains five categories: sports, social, diet, housework, and personal care; the Charades-STA dataset contains five categories: kitchen, bedroom, living room, bathroom, and corridor.

[0122] The experimental parameters of the present invention are set as follows: the AdamW optimizer with a decay rate of 1e-6 is used to optimize the training process; the learning rate is 1e-3; the batch size is 64; for the ActivityNet Captions dataset, the droupout is 0.4; for the Charades-STA dataset, the droupout is 0.3; λ1, λ2, and λ3 are 0.5, 0.2, and 0.5 respectively; and the random mask probability β is 0.2.

[0123] First, the present invention verifies the performance of the basic training module (AMDA-base), and the statistical results are shown in Tables 1 and 2. Although AMDA-base uses a simpler architecture, its performance is almost the same as that of other state-of-the-art models in the video temporal localization task, which also provides excellent basic performance for the model (AMDA) proposed in the present invention in the cross-category video temporal localization task.

[0124] Table 1 Performance of AMDA-base on the ActivityNet Captions dataset

[0125]

[0126] Table 2 Performance of AMDA-base on the Charades-STA dataset

[0127]

[0128] Next, the present invention verifies the domain shift between different categories of data based on AMDA-base, and the statistical results are shown in Tables 3 and 4. The ActivityNet Captions dataset uses sports as the source category data, and the other four categories are target category data. The Charades-STA dataset uses kitchen as the source category data, and the other four categories are target category data. Source category only means that the input data only has labeled source categories, target category only means that the input data only has labeled target categories, and joint training input data is labeled source categories and target categories. The experimental results show that there are obvious domain gaps between different categories of data, and joint training has the best results because it carries labels for source and target category data.

[0129] Table 3 Performance of different data categories on the ActivityNet Captions dataset

[0130]

[0131] Table 4 Performance of different data categories on the Charades-STA dataset

[0132]

[0133] Finally, the present invention verifies the performance of the AMDA model in a modular form, where the input data is labeled source category data and unlabeled target data. The ActivityNet Captions dataset uses sports as the source category data, and the remaining 4 categories are target category data. The Charades-STA dataset uses kitchen as the source category data, and the remaining 4 categories are target category data. The statistical results are shown in Tables 5 and 6. From the results, it can be seen that the phenotypic performance of the AMDA model is the best, indicating that the AMDA model proposed in the present invention has good generalization ability in the cross-category video time positioning task.

[0134] Table 5 Performance of AMDA model on ActivityNet Captions dataset

[0135]

[0136] Table 6 Performance of AMDA model on Charades-STA dataset

[0137]

[0138] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A cross-category video temporal localization method based on adversarial multimodal domain adaptation, characterized in that: The steps include: S1: Get the source category video, the target category video and the query text of each corresponding video, and extract the initial visual features of the video and the initial text features of the query text After encoding, it is used as the final visual feature and text features S2: Visual features of the target category video obtained in step S1 through the cross-modal feature calibrator and text features Perform semantic information calibration; S3: randomly masking the visual features of the target category video obtained in step S1 and reconstructing the visual features through a video feature reconstructor to obtain a reconstructed visual feature; S4: fusing the video features and text features obtained in step S1 through a cross-modal feature fuser to obtain initial fused features and final fused features of the source category video, as well as initial fused features and final fused features of the target category video; The step S4 comprises: S4-1: Calculate the visual features obtained in step S1 and text features The cross-modal similarity matrix S between them, where k∈{s,t}, k=s, Represents the visual features of the source category video, is the text feature of the query text of the source category video, when k=t, is the visual feature of the target category video, The text features of the query text of the target category video; S4-2: Normalize the cross-modal similarity matrix S obtained in step S4-1 along the rows and columns to obtain the similarity density matrix S r and S c ; S4-3: Similarity density matrix S obtained in step S4-2 r and S c , calculate the video-query text attention matrix A v and query text-video attention matrix A q ; S4-4: Video-query text attention matrix A calculated according to step S4-3 v and query text-video attention matrix A q Calculate the initial fusion feature F of the source category video k ; S4-5: Initial fusion feature F obtained according to step S4-4 k The final fusion feature is obtained through the multi-head attention mechanism S5: Performing unimodal domain invariant feature expression learning on the video features and text features obtained in step S1 through a domain discriminator, and performing cross-modal domain invariant feature expression learning on the initial fusion features obtained in step S4; S6: predicting the final fusion features of the source category video obtained in step S4 through a dual affine predictor to obtain the prediction probabilities of all possible results corresponding to each query text, and the one with the largest prediction probability is the final prediction result.

2. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 1 is characterized in that: The step S1 comprises: S1-1: Define input data: Among them, V s is the source category video, Q s is the query text set of source category videos, T s is the label set of source category data, Represents the i-th source video in the source category video and its corresponding query text, represents the real time boundary of the i-th query text matching the i-th source video in the source category video; V t is the target category video, Q t is the query text set of target category videos, T t is the label set of the target category data, which is empty in this embodiment; represents the i-th target video in the target category video and its corresponding query text; B represents the batch size; S1-2: Extract source category video V s and target category video V t Semantic information of Extract the query text Q of the source category video s and the query text Q of the target category video t Semantic information of S1-3: Initial visual features obtained in step S1-2 and initial text features Encode them respectively to obtain the encoded visual features and text features as the final features; the encoded visual features and text features are expressed as: Where n is the number of frames in each video, m is the query text length, and d is the hidden dimension. is the visual feature of the source category video, is the visual feature of the i-th source category video, is the visual feature of the jth frame of the i-th source category video, is the visual feature of the target category video, is the visual feature of the i-th target category video, is the visual feature of the jth frame of the i-th target category video, is the text feature of the query text of the source category video, is the text feature of the query text of the i-th source category video, is the text feature of the jth word in the query text of the i-th source category video, is the text features of the query text of the target category video, The text features of the query text of the i-th target category video, The text features of the jth word in the query text of the i-th target category video.

3. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 2 is characterized in that: In step S3, the initial visual features of the target category video obtained in step S1 are Randomly mask and reconstruct visual features.

4. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 3 is characterized in that: The step S3 comprises: S3-1: Randomly compare the initial visual features of the target category video obtained in step S1 Perform mask operation with β probability and encode it through visual encoder to obtain the encoded mask visual features S3-2: The mask visual features obtained in step S3-1 are processed by the cross-modal feature fuser And the text features of the query text of the target category video obtained in step S1 Fusion is performed to obtain the initial fusion feature F of the mask video m ; S3-3: Mask visual features obtained according to step S3-1 and the initial fusion feature F obtained in step S3-2 m Reconstruct the video features and obtain the reconstructed visual features V recob , the calculation process is as follows: Among them, Conv1D is a 1-dimensional convolutional layer, ReLU is the activation function, Represents an element-wise addition operation; The training loss of the cross-modal feature fuser adopts mean square error loss.

5. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 2, characterized in that: The step S2 comprises: S2-1: Calculate the average of the visual features and text features of the target category video along the time axis: in, Represents the visual features of all frames of the i-th target category video The mean of Represents the visual features of all frames of the i-th target category video The set of means of represents the mean of the text features of all words in the query text of the i-th source category video, represents the set of mean values ​​of text features of all words in the query text of the i-th source category video; S2-2: Exploitation Construct positive and negative samples: Will As a positive sample, As a negative sample, it is expressed as: S2-3: Using positive and negative samples to train a cross-modal feature calibrator, the visual features and text features of the target category video are calibrated for semantic information; the loss function of the cross-modal feature calibrator is: in, represents the cross-modal feature calibrator loss, represents the triple loss, B represents the batch size, Δ is the boundary, l(.,.) is used to calculate the cosine similarity between vectors, Z V - represents a set of negative samples of visual features, Z Q -A set of negative samples representing text features.

6. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 1, characterized in that: In the step S4: Video-query text attention matrix A v and query text-video attention matrix A q The calculation process is as follows: Initial fusion feature F of source category video k The calculation process is as follows: k∈{s,t} Among them, W f , b f is a learnable parameter; When k=s, F s represents the initial fusion features of the source category video, represents the initial fusion features of the i-th source category video, represents the initial fusion feature of the jth frame of the i-th source category video; when k = t, F t represents the initial fusion features of the target category video, represents the initial fusion features of the i-th target category video, Represents the initial fusion features of the jth frame of the i-th target category video The final fusion features are as follows: Among them, when k = s, represents the final fusion feature of the i-th source category video, represents the final fusion feature of the jth frame of the i-th source category video; when k = t, represents the final fusion feature of the i-th target category video, represents the final fused features of the jth frame of the i-th target category video.

7. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 2, characterized in that: The domain identifier includes a visual feature identifier D v , Text feature discriminator D q And the fusion feature discriminator D f ; The step S5 comprises: The visual feature identifier D v The visual features of the source category video obtained in step S1 are used to identify and visual features of target category videos The calculation process is as follows: The text feature identifier D q The text features of the query text used to identify the source category video in step S1 and the text features of the query text of the target category video The calculation process is as follows: The fusion feature discriminator D f The initial fusion feature F used to identify the source category video obtained in step S4 s and the initial fusion feature F of the target category video t , the calculation process is as follows: D f (F k )=σ(MLP f (GRL(F k ))),k∈{s,t} Among them, MLP v 、MLP p 、MLP f They are multi-layer perceptrons for visual features, text features and initial fusion features, GRL is the gradient reversal layer, and σ is the sigmoid function; The video feature identifier D v , Text feature discriminator D q And the fusion feature discriminator D f The output is a scalar between [0,1] indicating the probability that the input feature is from the target category.

8. The cross-category video temporal localization method based on adversarial multimodal domain adaptation according to claim 6, characterized in that: The step S6 comprises: S6-1: define candidate segments P = (fs, fe) matching the query text of each video in the source category video, where fs is the start frame number, fe is the end frame number, fs, fe∈[1,n], and n is the number of frames in the video; The final fused features of each video obtained from step S4 Extract the fusion features corresponding to the start frame and the end frame of the candidate segment Generate the hidden features r of the start frame and the end frame through the linear layer fs 、r fe , the calculation process is as follows: Among them, W1, W2, b1, b2 are learnable parameters of the linear layer; Since each video in the source category video consists of n video frames, all candidate segments can be represented as P = (fs, fe), fs∈[1,n], fe∈[1,n], fs≤fe; S6-2: Introduce the hidden feature r obtained in step S6-1 fs 、r fe , through the biaffine mechanism, we get the matrix containing the predicted probabilities of all candidate video segments P Combine the frames corresponding to the maximum value of the matrix M into the final prediction result; M P The calculation process is as follows: Among them, U m , W m , b m is a learnable parameter, M P represents the probability that the video segment P is the video segment corresponding to the query text, and the superscript T represents transposition. Indicates splicing.

9. A cross-category video temporal positioning system based on adversarial multimodal domain adaptation, used to implement the cross-category video temporal positioning method based on adversarial multimodal domain adaptation as described in any one of claims 1 to 8, the system comprising: A data acquisition and feature extraction module is used to acquire the source category video, the target category video and the query text of each corresponding video, extract the initial visual features of the video and the initial text features of the query text, and encode them as the final visual features and text features; A cross-modal feature calibration module, which is used to perform semantic information calibration on the obtained visual features and text features of the target category video through a cross-modal feature calibrator; A video feature reconstruction module, which is used to randomly mask the visual features of the target category video through a video feature reconstructor and reconstruct the visual features to obtain reconstructed visual features; A cross-modal feature fusion module, which is used to fuse the obtained video features and text features through a cross-modal feature fuser to obtain initial fusion features and final fusion features of the source category video, as well as initial fusion features and final fusion features of the target category video; A domain identification module, which is used to perform unimodal domain invariant feature expression learning on the obtained video features and text features respectively through a domain identifier, and to perform cross-modal domain invariant feature expression learning on the obtained initial fusion features; The dual affine prediction module is used to predict the final fusion features of the source category video obtained through the dual affine predictor, and obtain the prediction probabilities of all possible results corresponding to each query text. The one with the largest prediction probability is the final prediction result.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the cross-category video time positioning method based on adversarial multimodal domain adaptation as described in any one of claims 1 to 8.

Images