A video text retrieval method, device and medium for noise correlation

Abstract

The present invention discloses a method for video text retrieval targeting noise correlation, comprising the following steps: initializing parameters of a neural network model; inputting training data into the neural network model and performing calculations to obtain corresponding video representations and text representations; calculating the similarity between the corresponding video and text using optimal transmission based on the video and text representations; calculating a video matching loss function using a backpropagation algorithm based on the similarity between the video and text, and updating the parameters of the neural network model using an optimizer; repeating the above steps until a converged neural network model is obtained; and inputting the search text into the converged neural network model to complete the retrieval of the video text. The present invention can improve the robustness, matching accuracy, and retrieval performance of video data with noise correlation by realigning the misaligned portions of the video time sequence.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video text retrieval method targeting noise correlation. Background Art

[0002] Video text retrieval is the hottest issue in the field of multimodal analysis. It aims to retrieve corresponding videos from a text database and plays an indispensable role in current short video recommendation systems. Video text retrieval requires exploring the deep correlation between videos and texts to achieve accurate matching of cross-modal data. In recent years, a number of video matching methods based on deep neural networks (DNNs) have been proposed, and have made significant progress in various practical scenarios. In the study of video retrieval, video retrieval is currently a hot new research direction. Its core difficulty is how to encode the temporal relationship in the video. Current video retrieval solutions are mainly divided into two categories: designing a customized video network structure and using dynamic time warping (DTW) technology to encode the video.

[0003] The success of current video matching relies on an implicit assumption: that the visual content of a video is correctly temporally aligned with the corresponding text. However, in practice, there exists the problem of "noise correlation," whereby the video content and text are misaligned. According to statistics, only approximately 30% of video-text pairs in the HowTo100M instruction video are visually aligned, and a mere 15% are perfectly aligned. Due to this significant visual-text mismatch, existing video-text retrieval methods often achieve suboptimal and erroneous results, with low precision and an inability to accurately obtain matching results. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the video text retrieval method for noise correlation provided by the present invention solves the problems of noise correlation and low accuracy existing in the existing video text retrieval methods.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A method for video text retrieval based on noise correlation is provided, which comprises the following steps:

[0007] S1, initialize the parameters of the neural network model;

[0008] S2. Input the training data into the neural network model and perform calculations to obtain corresponding video representations and text representations;

[0009] S3, calculating the similarity between the corresponding video and text through optimal transmission based on the video representation and the text representation;

[0010] S4. Calculate the video matching loss function based on the similarity between the video and the text through the back propagation algorithm, and update the parameters of the neural network model through the optimizer;

[0011] S5. Repeat steps S2 to S4 until a converged neural network model is obtained;

[0012] S6. Input the search text into the converged neural network model, calculate the search text representation and the video representation obtained in step S2, and obtain the video with the highest similarity to the search text as the long video text search result.

[0013] Provided is a video-text retrieval device, comprising: a memory storing executable instructions; and a processor configured to execute the executable instructions in the memory to implement a video-text retrieval method for noise association.

[0014] A readable storage medium is provided, which stores executable instructions. When the executable instructions are executed by a processor, a video text retrieval method for noise correlation is implemented.

[0015] Furthermore, the neural network model includes a video network, a text network and an optimizer; the video network includes an S3D-G video encoding network and a Bert network A; the text network includes a Bert network B; and the training data includes a video set and a text set corresponding to the video.

[0016] Furthermore, the specific steps of step S2 are as follows:

[0017] S2-1. Randomly sample N videos and texts from the training data; segment the corresponding videos according to the timestamps of the texts to obtain n video clips and m sampled text titles corresponding to each sampled video, where n = m.

[0018] S2-2, inputting n video clips corresponding to each sampled video into the S3D-G video coding network respectively, according to the formula:

[0019] v' i =f S3D (v i )∈R t×d

[0020] Get the representation v' corresponding to each frame of the video clip corresponding to each sampled video i ; Among them, v i represents the video segment corresponding to the i-th sampled video, and d represents the video segment v i The vector dimension of t represents the duration of the video segment corresponding to the i-th sampled video, R represents a constant, and f S3D(·) represents the coding function of the S3D-G video coding network;

[0021] S2-3, respectively represent each frame of the video clip corresponding to each sampled video v' i Input to Bert network A, according to the formula:

[0022]

[0023]

[0024] Get the video representation V corresponding to each sampled video i ; Among them, f Bert (·) represents the Bert network A, AvgPool(·) represents the average pooling function, represents the representation of each frame of the video clip, and t represents the number of video frames in the video clip;

[0025] S2-4, input m sample text titles into the Bert network B, according to the formula:

[0026]

[0027]

[0028] Get the text representation T corresponding to each sample text title respectively i ; where t i represents the i-th sample text title, g Bert (·) represents the Bert network B, It represents the representation of all character tokens in the current text title, and l represents the number of tokens in the current text.

[0029] Furthermore, the specific steps of step S3 are:

[0030] S3-1, according to the formula:

[0031] [S] ab =[V a ·T b ]S∈R n×m

[0032] Get the similarity matrix [S] ab ; Among them, V a represents the ath video clip, T b represents the bth sampled text title, n represents the total number of video clips, m represents the total number of sampled text titles, and R represents a constant;

[0033] S3-2, according to the formula:

[0034] Q *=Diag(κ1)exp(S / ε)Diag(κ2)

[0035] μ=[1 n ,m],ν=[1 m ,n] T

[0036]

[0037] Get the initial optimal transmission assignment Q * ; Among them, the initial values ​​of κ1 and κ2 are both 1, ε represents the scaling constant, exp(·) represents the exponential function, Diag(·) represents the diagonal matrix function, μ and ν represent the probability distribution of the video corresponding to the text, 1 n represents a unit vector of dimension n, 1 m represents a unit vector of dimension m, [·] T represents the transposed matrix;

[0038] S3-3, according to the formula:

[0039] G=tr(Q *T S)

[0040] Get the similarity G between the video and the text; where Q *T represents the optimal transmission assignment Q * The transposed matrix, tr(Q *T S) represents the optimal transmission assignment Q * The transposed matrix Q *T and similarity matrix [S] ab The trace of the product.

[0041] Furthermore, the formula of the video matching loss function in step S4 is as follows:

[0042]

[0043]

[0044]

[0045] loss=loss videoA +loss videoB

[0046] Among them, loss represents the video matching loss function, loss videoA Represents the first loss function, loss videoB represents the second loss function, N represents the number of sampled videos, τ represents the temperature coefficient of the softmax normalized exponential function, log(·) represents the logarithmic function, ∑(·) represents the summation function, exp(·) represents the exponential function, Gii represents the similarity value corresponding to the i-th video and the i-th text, G ij represents the similarity value corresponding to the i-th video and the j-th text, G ji represents the similarity value corresponding to the j-th video and the i-th text, max(·) represents the maximum value function, (·) T represents the transposed matrix, Represents the representation of all character tokens in the current text title. Represents the representation of each frame of the video clip, Represents the representation of all character tokens in the current text title and the representation of each frame of the video clip The fineness similarity, l represents the number of current text tokens, t represents the number of video frames in the video clip, Represents the representation of all character tokens in the current text title T i The oth row of Representation of each frame of the video clip The pth row of .

[0047] Furthermore, the method for obtaining the retrieved text representation in step S6 is the same as that in step S2.

[0048] The beneficial effects of the present invention are as follows: the noisy video text retrieval method can improve the robustness of the matching model for real noise-related video data by realigning the time-series misaligned parts in the video data, and significantly improve the matching accuracy and retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a specific flow chart of the present invention. DETAILED DESCRIPTION

[0050] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0051] like Figure 1 As shown, a video text retrieval method for noise association includes the following steps:

[0052] S1, initialize the parameters of the neural network model;

[0053] S2. Input the training data into the neural network model and perform calculations to obtain corresponding video representations and text representations;

[0054] S3, calculating the similarity between the corresponding video and text through optimal transmission based on the video representation and the text representation;

[0055] S4. Calculate the video matching loss function based on the similarity between the video and the text through the back propagation algorithm, and update the parameters of the neural network model through the optimizer;

[0056] S5. Repeat steps S2 to S4 until a converged neural network model is obtained;

[0057] S6. Input the search text into the converged neural network model, calculate the search text representation and the video representation obtained in step S2, and obtain the video with the highest similarity to the search text as the long video text search result.

[0058] A video-text retrieval device includes: a memory storing executable instructions; and a processor configured to execute the executable instructions in the memory to implement a video-text retrieval method for noise correlation.

[0059] A readable storage medium stores executable instructions. When the executable instructions are executed by a processor, a video text retrieval method for noise correlation is implemented.

[0060] The neural network model includes a video network, a text network and an optimizer; the video network includes an S3D-G video encoding network and a Bert network A; the text network includes a Bert network B; and the training data includes a video set and a text set corresponding to the video.

[0061] The specific steps of step S2 are as follows:

[0062] S2-1. Randomly sample N videos and texts from the training data; segment the corresponding videos according to the timestamps of the texts to obtain n video clips and m sampled text titles corresponding to each sampled video, where n = m.

[0063] S2-2, inputting n video clips corresponding to each sampled video into the S3D-G video coding network respectively, according to the formula:

[0064] v' i =f S3D (v i )∈R t×d

[0065] Get the representation v' corresponding to each frame of the video clip corresponding to each sampled video i ; Among them, v i represents the video segment corresponding to the i-th sampled video, and d represents the video segment v iThe vector dimension of t represents the duration of the video segment corresponding to the i-th sampled video, R represents a constant, and f S3D (·) represents the coding function of the S3D-G video coding network;

[0066] S2-3, respectively represent each frame of the video clip corresponding to each sampled video v' i Input to Bert network A, according to the formula:

[0067]

[0068]

[0069] Get the video representation V corresponding to each sampled video i ; Among them, f Bert (·) represents the Bert network A, AvgPool(·) represents the average pooling function, represents the representation of each frame of the video clip, and t represents the number of video frames in the video clip;

[0070] S2-4, input m sample text titles into the Bert network B, according to the formula:

[0071]

[0072]

[0073] Get the text representation T corresponding to each sample text title respectively i ; where t i represents the i-th sample text title, g Bert (·) represents the Bert network B, It represents the representation of all character tokens in the current text title, and l represents the number of tokens in the current text.

[0074] Specific steps of step S3:

[0075] S3-1, according to the formula:

[0076] [S] ab =[V a ·T b ]S∈R n×m

[0077] Get the similarity matrix [S] ab ; Among them, V a represents the ath video clip, T b represents the bth sampled text title, n represents the total number of video clips, m represents the total number of sampled text titles, and R represents a constant;

[0078] S3-2, according to the formula:

[0079] Q * =Diag(κ1)exp(S / ε)Diag(κ2)

[0080] μ=[1 n ,m],ν=[1 m ,n] T

[0081]

[0082] Get the initial optimal transmission assignment Q * ; Among them, the initial values ​​of κ1 and κ2 are both 1, ε represents the scaling constant, exp(·) represents the exponential function, Diag(·) represents the diagonal matrix function, μ and ν represent the probability distribution of the video corresponding to the text, 1 n represents a unit vector of dimension n, 1 m represents a unit vector of dimension m, [·] T represents the transposed matrix;

[0083] S3-3, according to the formula:

[0084] G=tr(Q *T S)

[0085] Get the similarity G between the video and the text; where Q *T represents the optimal transmission assignment Q * The transposed matrix, tr(Q *T S) represents the optimal transmission assignment Q * The transposed matrix Q *T and similarity matrix [S] ab The trace of the product.

[0086] The formula of the video matching loss function in step S4 is as follows:

[0087]

[0088]

[0089]

[0090] loss=loss videoA +loss videoB

[0091] Among them, loss represents the video matching loss function, loss videoA Represents the first loss function, loss videoBrepresents the second loss function, N represents the number of sampled videos, τ represents the temperature coefficient of the softmax normalized exponential function, log(·) represents the logarithmic function, ∑(·) represents the summation function, exp(·) represents the exponential function, G ii represents the similarity value corresponding to the i-th video and the i-th text, G ij represents the similarity value corresponding to the i-th video and the j-th text, G ji represents the similarity value corresponding to the j-th video and the i-th text, max(·) represents the maximum value function, (·) T represents the transposed matrix, Represents the representation of all character tokens in the current text title. Represents the representation of each frame of the video clip, Represents the representation of all character tokens in the current text title and the representation of each frame of the video clip The fineness similarity, l represents the number of current text tokens, t represents the number of video frames in the video clip, Represents the representation of all character tokens in the current text title The oth row of Representation of each frame of the video clip The pth row of .

[0092] The method of obtaining the retrieved text representation in step S6 is the same as that in step S2.

[0093] In one embodiment of the present invention, noise association refers to a phenomenon in which the description of a video text is irrelevant to the current video content. This phenomenon is usually caused by inconsistency between a person's speech and actual actions, for example, a person may describe something irrelevant to the video content.

[0094] This paper uses the HowTo100M dataset as a training set and the YouCookII dataset of real-life cooking videos as a test set for video text retrieval. Recall rates R@1, R@5, and R@10 are used as metrics. Experiments were conducted using the video retrieval method provided by this invention, the VT-TWINS method, the MIL-NCE method, the MCN method, and the TAN method. R@K is defined as the percentage of correct samples returned among the first K samples in the retrieved data; a higher value indicates higher retrieval accuracy and better results.

[0095] The experimental data comparison is shown in Table 1:

[0096] Table 1 Comparison of experimental data of different retrieval methods in YouCookII

[0097]

[0098]

[0099] As can be seen from Table 1, among the aforementioned video text retrieval methods, the present invention achieves the best recall rates R@1, R@5, and R@10. This data demonstrates that even when noise correlation is present in the video, the present invention achieves the highest retrieval accuracy.

[0100] In summary, the noisy video text retrieval method provided by the present invention can improve the robustness of the matching model for real noise-related video data by realigning the time-misaligned parts in the video data, and significantly improve the matching accuracy and retrieval performance.

Claims

1. A video text retrieval method for noise correlation, characterized by: The following steps are involved: S1, initialize the parameters of the neural network model; S2. Input the training data into the neural network model and perform calculations to obtain corresponding video representations and text representations; S3, calculating the similarity between the corresponding video and text through optimal transmission based on the video representation and the text representation; S4. Calculate the video matching loss function based on the similarity between the video and the text through the back propagation algorithm, and update the parameters of the neural network model through the optimizer; S5. Repeat steps S2 to S4 until a converged neural network model is obtained; S6. Input the search text into the converged neural network model, calculate the search text representation and the video representation obtained in step S2, and obtain the video with the highest similarity to the search text as the long video text search result; The formula of the video matching loss function in step S4 is as follows: in, represents the video matching loss function, represents the first loss function, represents the second loss function, Indicates the number of sampled videos, represents the temperature coefficient of the softmax normalized exponential function, represents the logarithmic function, represents the summation function, represents the exponential function, Indicates the Videos and The similarity value corresponding to the text, Indicates the Videos and The similarity value corresponding to the text, Indicates the Videos and The similarity value corresponding to the text, represents the maximum function, represents the transposed matrix, Represents the representation of all character tokens in the current text title. represents the representation of each frame of the video clip, Represents the representation of all character tokens in the current text title and the representation of each frame of the video clip The similarity of fineness, Indicates the number of tokens in the current text. Indicates the number of video frames in the video clip. Represents the representation of all character tokens in the current text title No. OK, Representation of each frame of the video clip No. OK.

2. The method for video text retrieval based on noise association according to claim 1, characterized in that: The neural network model includes a video network, a text network and an optimizer; the video network includes an S3D-G video encoding network and a Bert network A; the text network includes a Bert network B; and the training data includes a video set and a text set corresponding to the video.

3. The method for video text retrieval based on noise association according to claim 2, characterized in that: The specific steps of step S2 are as follows: S2-1, randomly sample the training data to get a batch of The corresponding video is cut according to the timestamp of the text to obtain n video clips and m sample text titles corresponding to each sampled video; ; S2-2, inputting n video clips corresponding to each sampled video into the S3D-G video coding network respectively, according to the formula: Get the representation of each frame of the video clip corresponding to each sampled video ;in, Indicates the The video clips corresponding to the sampled videos, Indicates a video clip The vector dimension of Indicates the number of video frames in the video clip. represents a constant, Represents the encoding function of the S3D-G video coding network; S2-3, respectively represent each frame of the video clip corresponding to each sampled video Input to Bert network A, according to the formula: Get the video representation corresponding to each sampled video ;in, represents the Bert network A, represents the average pooling function, Represents the representation of each frame of the video clip, Indicates the number of video frames in the video clip; S2-4, input m sample text titles into the Bert network B, according to the formula: Get the text representation corresponding to each sample text title ;in, Indicates the Sample text titles, represents the Bert network B, Represents the representation of all character tokens in the current text title. Indicates the number of tokens in the current text.

4. The method for video text retrieval based on noise association according to claim 2, wherein: The specific steps of step S3 are: S3-1, according to the formula: ， Get the similarity matrix ;in, Indicates the Video clips, Indicates the Sample text titles, Indicates the total number of video clips, Indicates the total number of sampled text titles, represents a constant; S3-2, according to the formula: Get the initial optimal transmission assignment ;in, and The initial value of is 1. represents the scaling constant, represents the exponential function, represents a diagonal matrix function, and Represents the probability distribution of video corresponding to text, Indicates the dimension The unit vector of Indicates the dimension The unit vector of represents the transposed matrix; S3-3, according to the formula: Get the similarity between video and text ;in, represents the optimal transmission assignment The transposed matrix of represents the optimal transmission assignment The transposed matrix of and similarity matrix The trace of the product.

5. The method for video text retrieval based on noise association according to claim 2, characterized in that: The method for obtaining the retrieved text representation in step S6 is the same as that in step S2.

6. A video-text retrieval device, characterized by: The device comprises: a memory storing executable instructions; and The processor is configured to execute the executable instructions in the memory to implement the video text retrieval method for noise association according to any one of claims 1 to 5.

7. A readable storage medium having executable instructions stored thereon, characterized in that: When the executable instructions are executed by the processor, the noise-related video text retrieval method according to any one of claims 1 to 5 is implemented.

Images