Cross-modal video sensitive event retrieval method and system based on frequency domain enhancement

Abstract

The invention belongs to the technical field of cross-modal video moment positioning, and relates to a cross-modal video sensitive event retrieval method and system based on frequency domain enhancement. The method comprises the following steps: acquiring data of a text mode and data of a video mode, and respectively encoding the data of the two modes to obtain features of the two modes; projecting the features of the two modals to a frequency domain through fast Fourier transform, and enhancing the features of the video modals in the frequency domain to obtain frequency domain enhanced video features; generating positive and negative candidate video segments for the frequency domain enhanced video features through Gaussian distribution, respectively representing candidate video segments closest to a retrieval request and candidate video segments closest to the retrieval request, and selecting the candidate video segment closest to the retrieval request as a retrieval result so as to realize cross-modal video moment positioning; and multi-scale contrast learning is carried out on the positive and negative candidate video segments, so that the generation quality of the positive and negative candidate video segments is improved, and the accuracy of cross-modal video sensitive event retrieval is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal video moment positioning, and in particular relates to a cross-modal video sensitive event retrieval method and system based on frequency domain enhancement. Background Art

[0002] Cross-modal video moment localization determines the time segment (start time, end time) of the event corresponding to the description in the uncut full video by giving it a natural language description as a retrieval request.

[0003] Unlike single computer vision tasks, cross-modal video time location requires not only video processing, but also the combination of information from other modalities (mainly natural language) to improve the ability to understand the video content, so as to locate the start and end time of the event. To improve the accuracy of the model for the start and end time of an event, research can be carried out in the following two aspects: 1) Make the model more fully utilize cross-modal data; 2) Make the retrieval of time intervals more accurate.

[0004] In the existing cross-modal video sensitive event retrieval algorithms, there are generally two retrieval schemes for natural language retrieval requests: 1) Method based on candidate set generation: This method first cuts out several candidate video segments from the video sample as the candidate set of the retrieval result, and then selects the candidate segment closest to the retrieval request as the retrieval result through multi-classification. 2) Method based on direct regression: This method does not construct a candidate set, but directly regresses the video sample to directly predict the start and end frames of the retrieval result.

[0005] Cross-modal video moment localization requires the model to establish the relationship between language and visual modalities, model multimodal features, and have a deep understanding of natural language and video content. In other words, based on the single-modality sensitive event temporal localization task, only a limited number of predefined labels can be used to classify and locate actions in the video, while cross-modal video moment localization can use natural language commonly used by humans to form queries, which contains richer information including the subject and object of the action and the description of the subject and object.

[0006] However, the introduction of natural language modality as a retrieval request in video moment localization has greatly increased the complexity of the data. The high cost required to obtain a large number of finely annotated text-video pairs has actually become the biggest obstacle encountered in the development of this technology. At the same time, due to the lack of finely annotated data, the existing cross-modal video moment localization model shows low generalization performance in actual application scenarios, resulting in serious performance loss in the actual application scenarios, which leads to a disconnect between the experimental environment and the actual application scenarios. Summary of the invention

[0007] The purpose of this invention is to achieve weakly supervised cross-modal video moment localization, so that the machine can accurately locate the corresponding sensitive events in the uncut video through a natural language request even if no finely annotated text-video pairs are used in training.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A cross-modal video sensitive event retrieval method based on frequency domain enhancement includes the following steps:

[0010] Obtain data in two modalities, text and video, which are text search sentences and uncut video segments, respectively, encode the data in the two modalities respectively, and obtain features of the two modalities;

[0011] The features of the two modes are projected into the frequency domain respectively through fast Fourier transform, and the features of the video mode are enhanced in the frequency domain to obtain the video features enhanced in the frequency domain;

[0012] Positive and negative candidate video segments are generated for frequency-domain enhanced video features through Gaussian distribution, representing the candidate video segments closest to and least close to the retrieval request, respectively. The candidate video segment closest to the retrieval request is selected as the retrieval result, thereby achieving cross-modal video moment positioning.

[0013] Furthermore, the enhancement of the features of the video modality in the frequency domain includes:

[0014] Use fast Fourier transform to project the features of the two input modes into the frequency domain respectively;

[0015] The TopK method is used to screen the amplitude of each frequency in the video features, and the video features with the highest K frequencies are selected for feature interaction with the text features, so that the two features are fused to obtain fused features;

[0016] The fused features obtained by interaction are regarded as background semantics and fed back to the video features as residuals for background elimination. Then, inverse fast Fourier transform is performed to obtain video features with event semantic enhancement.

[0017] Furthermore, the above method also includes the following steps: performing multi-scale comparative learning on the positive and negative candidate video segments to improve the generation quality of the positive and negative candidate video segments and improve the accuracy of cross-modal video sensitive event retrieval.

[0018] Furthermore, the multi-scale contrastive learning includes contrastive learning between positive and negative candidate video segments within a sample and contrastive learning between samples.

[0019] Furthermore, the comparative learning between the positive and negative candidate video segments within the sample includes:

[0020] For each video sample, the positive candidate video segments and negative candidate video segments within the sample are used as positive and negative sample pairs in comparative learning;

[0021] During model training, the difference between positive and negative sample pairs is constructed through contrastive learning loss, so that the positive candidate video segments in the same sample are as similar as possible, and the negative candidate video segments are as different as possible.

[0022] Furthermore, the comparative learning between the samples includes:

[0023] For different video samples, the positive candidate video segments within the same sample and the positive candidate video segments in different samples are used as positive and negative sample pairs in contrastive learning;

[0024] During model training, the difference between positive and negative sample pairs is constructed through contrastive learning loss, so that the positive candidate video segments in the same sample are as similar as possible, and the positive candidate video segments in each sample are as different as possible from the positive candidate video segments in other samples.

[0025] A cross-modal video sensitive event retrieval system based on frequency domain enhancement, comprising:

[0026] The feature extraction module is used to obtain data in two modes, text and video, which are text search sentences and uncut video segments, respectively, and encode the data of the two modes to obtain features of the two modes;

[0027] The frequency domain enhancement module is used to project the features of the two modes into the frequency domain respectively through fast Fourier transform, and enhance the features of the video mode in the frequency domain to obtain the video features enhanced in the frequency domain;

[0028] The Gaussian candidate set generation module is used to generate positive and negative candidate video segments for frequency domain enhanced video features using Gaussian distribution, which respectively represent the candidate video segments closest to the retrieval request and the candidate video segments least close to the retrieval request, and select the candidate video segment closest to the retrieval request as the retrieval result, thereby achieving cross-modal video moment positioning.

[0029] Furthermore, the above system also includes:

[0030] The multi-scale contrastive learning module is used to perform multi-scale contrastive learning of positive and negative candidate video segments within and between samples, improve the generation quality of positive and negative candidate video segments, and improve the accuracy of cross-modal video sensitive event retrieval.

[0031] The beneficial effects of the present invention are as follows:

[0032] The present invention proposes a frequency domain enhanced cross-modal video sensitive event retrieval method using text sentences as retrieval requests in a weakly supervised scenario. Unlike fully supervised, the video moment positioning technology of the present invention in a weakly supervised scenario only uses simple annotated video samples for moment positioning, rather than timestamp-level text-video annotation data. In a weakly supervised scenario, the cross-modal video moment positioning method of the present invention no longer relies on finely annotated data samples, which greatly reduces the data annotation cost of model training while improving the generalization ability of the algorithm in actual application scenarios.

[0033] In view of the problem that the existing video moment positioning algorithm has low accuracy in detecting sensitive events, the present invention uses a frequency domain enhancement method to enhance the event semantics in video samples, thereby better utilizing the multimodal information in the samples and improving the accuracy of the algorithm.

[0034] For samples with low shooting quality, the present invention uses a Gaussian generator to generate positive and negative candidate video segments, and improves the generation quality of the Gaussian generator through comparative learning, so that the method of the present invention has more advantages in difficult samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the main flow chart of the present invention.

[0036] Figure 2 Schematic diagram of the frequency domain enhancement module.

[0037] Figure 3 The difficult sample result display diagram. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0039] The main contents of the present invention include:

[0040] 1. The present invention proposes a weakly supervised cross-modal video moment localization method. Different from the existing cross-modal video moment localization method that relies on finely annotated text-video pairs, the present invention does not require the dataset to provide the specific frame in the video sample where the search request starts and ends, but only provides the corresponding search statement for each video sample, and the supervision information is relatively weak.

[0041] 2. In order to improve the retrieval accuracy of sensitive events in videos, the present invention divides the video corpus into an event semantic part containing sensitive events and a background semantic part that does not contain sensitive events, and distinguishes the event semantics from the background semantics through the frequency domain decomposition method, thereby enhancing the semantics of sensitive events.

[0042] 3. In order to further improve the detection accuracy of difficult samples, the present invention is based on the candidate set generation method, constructs several positive and negative sample candidate segments through Gaussian distribution, and optimizes the candidate segments through comparative learning, which further improves the detection accuracy of the method of the present invention.

[0043] Figure 1 This is the main flow chart of the present invention. The multimodal video sensitive event retrieval method based on frequency domain enhancement proposed by the present invention includes the following two networks:

[0044] 1. Feature Network

[0045] The feature network consists of a feature extraction module and a frequency domain enhancement module.

[0046] (1) The input of the feature extraction module contains two modal data, namely, text retrieval sentences and uncut video segments. This module uses two different encoders to encode the natural language and video data into high-dimensional feature tensors as the output of this module.

[0047] (2) Figure 2 This is the detailed workflow of the frequency domain enhancement module. First, the frequency domain enhancement module uses fast Fourier transform (FFT) to project the features of the two input modes to the frequency domain respectively. Then, the TopK method is used to screen the amplitudes of each frequency in the video features, and the video features with the highest K frequencies are selected for feature interaction with the text features, that is, the video features and text features are shared parameters, so that the two features are fused with each other, and finally the fused features are obtained. The fused features obtained by the interaction will be regarded as background semantics and fed back to the video features as residuals for background elimination, that is, the background semantics in the original video features are weakened, and then the inverse fast Fourier transform is performed, and finally the video features with event semantics enhancement are obtained, thereby realizing the enhancement of video features in the frequency domain.

[0048] 2. Positioning network

[0049] The localization network consists of a Gaussian candidate set generation module and a multi-scale contrastive learning module.

[0050] (1) The Gaussian candidate set generation module generates positive and negative candidate video segments for the frequency-domain enhanced video features through Gaussian distribution, where the positive candidate video segment represents the video segment closest to the retrieval request, and the negative candidate video segment represents the video segment closest to (i.e., farthest from) the retrieval request. The candidate video segment closest to the retrieval request is selected as the retrieval result, thereby achieving cross-modal video moment positioning.

[0051] The generation of positive and negative candidate video segments for the frequency-domain enhanced video features through Gaussian distribution can be achieved by a method known in the art, that is, assigning a new weight to the video features, which comes from the Gaussian distribution.

[0052] (2) The multi-scale contrastive learning module conducts multi-scale contrastive learning on the positive and negative candidate video segments within and between samples, improving the generation quality of Gaussian positive and negative candidate video segments, and ultimately improving the accuracy of cross-modal video sensitive event retrieval.

[0053] Among them, multi-scale contrastive learning refers to the contrastive learning between positive and negative candidate video segments within the sample and the contrastive learning between samples.

[0054] The comparative learning between the positive and negative candidate video segments within the sample specifically includes the following steps:

[0055] 1) For each video sample, the Gaussian candidate set generation module generates positive candidate video segments and negative candidate video segments. The positive and negative candidates within the sample use the positive and negative candidate video segments as positive and negative sample pairs in comparative learning.

[0056] 2) During model training, the difference between positive and negative sample pairs is constructed through contrastive learning loss, so that the positive candidate video segments in the same sample are as similar as possible, and the negative candidate video segments are as different as possible.

[0057] The comparative learning between the samples specifically includes the following steps:

[0058] 1) For different video samples, the positive candidate video segments within the same sample and the positive candidate video segments in different samples are used as positive and negative sample pairs in contrastive learning.

[0059] 2) During model training, the difference between positive and negative sample pairs is constructed through contrastive learning loss, so that the positive candidate video segments in the same sample are as similar as possible, and the positive candidate video segments in each sample are as different as possible from the positive candidate video segments in other samples.

[0060] In summary, the present invention proposes a frequency domain enhanced cross-modal video sensitive event retrieval algorithm that uses text sentences as retrieval requests in a weakly supervised scenario. For samples with low shooting quality, the present invention uses a Gaussian generator to generate positive and negative candidate video segments, and improves the generation quality of the Gaussian generator through contrastive learning, making the method of the present invention more advantageous in difficult samples. Figure 3 As shown, "this method" refers to the method proposed in the present invention, and the reference method refers to the existing baseline method of this task, which also uses Gaussian distribution to generate candidate video segments for retrieval, named CPL.

[0061] Another embodiment of the present invention provides a cross-modal video sensitive event retrieval system based on frequency domain enhancement, which includes:

[0062] The feature extraction module is used to obtain data in two modes, text and video, which are text search sentences and uncut video segments, respectively, and encode the data of the two modes to obtain features of the two modes;

[0063] The frequency domain enhancement module is used to project the features of the two modes into the frequency domain respectively through fast Fourier transform, and enhance the features of the video mode in the frequency domain to obtain the video features enhanced in the frequency domain;

[0064] A Gaussian candidate set generation module is used to generate positive and negative candidate video segments for frequency domain enhanced video features using Gaussian distribution, which represent the candidate video segments closest to the retrieval request and the candidate video segments least close to the retrieval request, respectively, and select the candidate video segment closest to the retrieval request as the retrieval result, thereby achieving cross-modal video moment positioning;

[0065] The multi-scale contrastive learning module is used to perform multi-scale contrastive learning of positive and negative candidate video segments within and between samples, improve the generation quality of positive and negative candidate video segments, and improve the accuracy of cross-modal video sensitive event retrieval.

[0066] The division of the above modules is only an example. In actual applications, the above functions can be assigned to different functional modules as needed to complete all or part of the functions described in the above method. The specific working process of each module can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0067] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0068] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0069] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A cross-modal video sensitive event retrieval method based on frequency domain enhancement, characterized in that: The following steps are involved: Obtain data in two modalities, text and video, which are text search sentences and uncut video segments, respectively, encode the data in the two modalities respectively, and obtain features of the two modalities; The features of the two modes are projected into the frequency domain respectively through fast Fourier transform, and the features of the video mode are enhanced in the frequency domain to obtain the video features enhanced in the frequency domain; Positive and negative candidate video segments are generated for frequency-domain enhanced video features through Gaussian distribution, representing the candidate video segments closest to and least close to the retrieval request, respectively. The candidate video segment closest to the retrieval request is selected as the retrieval result, thereby achieving cross-modal video moment positioning.

2. The method according to claim 1, characterized in that The method of enhancing the features of the video modality in the frequency domain includes: Use fast Fourier transform to project the features of the two input modes into the frequency domain respectively; The TopK method is used to screen the amplitude of each frequency in the video features, and the video features with the highest K frequencies are selected for feature interaction with the text features, so that the two features are fused to obtain fused features; The fused features obtained by interaction are regarded as background semantics and fed back to the video features as residuals for background elimination. Then, inverse fast Fourier transform is performed to obtain video features with event semantic enhancement.

3. The method according to claim 1 or 2, characterized in that: The following steps are also included: Multi-scale comparative learning is performed on positive and negative candidate video segments to improve the generation quality of positive and negative candidate video segments and improve the accuracy of cross-modal video sensitive event retrieval.

4. The method according to claim 3, characterized in that The multi-scale contrastive learning includes contrastive learning between positive and negative candidate video segments within a sample and contrastive learning between samples.

5. The method according to claim 4, characterized in that The comparative learning between the positive and negative candidate video segments within the sample includes: For each video sample, the positive candidate video segments and negative candidate video segments within the sample are used as positive and negative sample pairs in comparative learning; During model training, the difference between positive and negative sample pairs is constructed through contrastive learning loss, so that the positive candidate video segments in the same sample are as similar as possible, and the negative candidate video segments are as different as possible.

6. The method according to claim 4, characterized in that The contrastive learning between the samples includes: For different video samples, the positive candidate video segments within the same sample and the positive candidate video segments in different samples are used as positive and negative sample pairs in contrastive learning; During model training, the difference between positive and negative sample pairs is constructed through contrastive learning loss, so that the positive candidate video segments in the same sample are as similar as possible, and the positive candidate video segments in each sample are as different as possible from the positive candidate video segments in other samples.

7. A cross-modal video sensitive event retrieval system based on frequency domain enhancement, characterized in that: include: The feature extraction module is used to obtain data in two modes, text and video, which are text search sentences and uncut video segments, respectively, and encode the data of the two modes to obtain features of the two modes; The frequency domain enhancement module is used to project the features of the two modes into the frequency domain respectively through fast Fourier transform, and enhance the features of the video mode in the frequency domain to obtain the video features enhanced in the frequency domain; The Gaussian candidate set generation module is used to generate positive and negative candidate video segments for frequency domain enhanced video features using Gaussian distribution, which respectively represent the candidate video segments closest to the retrieval request and the candidate video segments least close to the retrieval request, and select the candidate video segment closest to the retrieval request as the retrieval result, thereby achieving cross-modal video moment positioning.

8. The system according to claim 7, characterized in that Also includes: The multi-scale contrastive learning module is used to perform multi-scale contrastive learning of positive and negative candidate video segments within and between samples, improve the generation quality of positive and negative candidate video segments, and improve the accuracy of cross-modal video sensitive event retrieval.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images