A method, system and product for locating target video segments based on semantic alignment

Abstract

The present invention belongs to the technical field of target video segment positioning, and provides a method, system and product for positioning a target video segment based on semantic alignment. The technical scheme is to obtain a global alignment loss function and a local semantic alignment loss function by calculating the alignment distribution probability based on the semantic features of the two modalities of video and text at different levels; align the semantic features of the two modalities of video and text according to the global alignment loss function and the local semantic alignment loss function; infer the missing modal semantic features, and complete the real semantic features corresponding to the missing modal semantic features to obtain the complete semantic features of the two modalities of video and text; replace the input visual and text features according to the complete semantic features of the two modalities of video and text, and locate the target video segment. The present invention can perform effective cross-modal semantic alignment modeling, thereby overcoming the problems of missing multimodal information and fuzzy semantic information, and realizing accurate retrieval of target video segments.

Description

Technical Field

[0001] The present invention belongs to the technical field of target video segment positioning, and in particular relates to a target video segment positioning method, system and product based on semantic alignment. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the task of locating target video clips, the key task is whether effective semantic alignment can be performed. Semantic alignment aims to ensure that videos and texts that express the same content have similar vector representations.

[0004] In order to achieve effective semantic alignment, most of the early methods used global query sentence level alignment as a semantic interaction method, but due to the lack of effective representation of multimodal semantic details, the positioning accuracy was not ideal. To make up for this deficiency, a method of fine-grained cross-modal feature aggregation through frame-by-frame and word interaction was proposed. Many methods further introduced the attention mechanism and explored different ways of using the attention mechanism for frame-by-frame and word-by-word interaction to capture the relationship between visual clues and text queries.

[0005] In order to better model the semantic consistency between local phrase information and fragment features in query sentences, some methods proposed a method to establish a temporal relationship between phrase-level representation and visual features, which improved the performance of fragment retrieval. In order to further explore the semantic role of local phrases in fragment retrieval, some methods proposed to use a self-attention mechanism to model the relationship between words and extract local phrase features for local-global video text interaction. In this process, the visual cues of the target fragment are highlighted to suppress irrelevant cues, and the local context of the target fragment and the global context of other fragments are jointly modeled to correctly understand the long-distance relationship between entities.

[0006] However, the above target video segment localization models all ignore some issues:

[0007] 1) Information loss during acquisition. Due to the limitations of some acquisition equipment, such as the inherent pixel resolution of the camera, the ability to adapt to low-light conditions, and the shooting angle, the integrity of the image information may be affected, resulting in information loss. Environmental factors can also have an impact, such as obstructions such as cars and buildings, or extreme weather such as sandstorms and heavy rains, which can lead to incomplete information.

[0008] 2) The ambiguity of visual information. The video that needs to be positioned may have problems such as poor image quality, motion blur, and blurred focus, which cannot reflect the information well, thus having a relatively large impact on the positioning effect.

[0009] 3) Diversity of language descriptions. Even for the same video clip, the description text may show significant differences due to different describers. This subjectivity may lead to the omission of key information, which in turn has an adverse impact on the positioning accuracy of the video clip. Summary of the invention

[0010] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a method, system and product for locating a target video segment based on semantic alignment, which reduces the influence of multimodal information missing and semantic information ambiguity on the semantic alignment process, and can actively complete the missing modal information, thereby achieving semantic alignment between the target video segment and the text, and further achieving accurate target video segment positioning.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] A first aspect of the present invention provides a method for locating a target video segment based on semantic alignment, comprising the following steps:

[0013] Obtaining a candidate set of video clips and texts, and preprocessing the obtained video clips and texts;

[0014] The preprocessed video clips and texts are respectively subjected to feature extraction at different semantic levels to obtain semantic features of the two modalities of video and text at different semantic levels;

[0015] Based on the semantic features of the two modalities of video and text at different semantic levels, the global alignment loss function and the local semantic alignment loss function are obtained by calculating the alignment distribution probability;

[0016] According to the global alignment loss function and the local semantic alignment loss function, semantic alignment training is performed to obtain semantically aligned video clips and text semantic features;

[0017] According to the semantically aligned video clips and text semantic features, the missing video clips and text semantic features are inferred, and the real semantic features corresponding to the missing modal semantic features are completed to obtain complete semantic features of both video and text modalities;

[0018] Replace the input video and text features with the complete semantic features of the two modalities;

[0019] Based on the replaced video and text features, similarity calculation is performed to locate the target video segment.

[0020] Furthermore, based on the semantic features of the two modalities of video and text at different semantic levels, the global alignment loss function is obtained by calculating the alignment distribution probability, including:

[0021] Based on all semantic features of video modality and text modality, the probability distribution of video-aligned text and the probability distribution of text-aligned video are calculated respectively;

[0022] The loss function of global semantic alignment is obtained based on the probability distribution of video-aligned text and the probability distribution of text-aligned video.

[0023] Furthermore, based on the semantic features of the two modalities of video and text at different semantic levels, the local semantic alignment loss function is obtained by calculating the alignment distribution probability, including:

[0024] A multi-head cross-modal attention mechanism is used to calculate the confidence of each level of semantic features in the video modality and text modality respectively;

[0025] Compare the confidences of the semantic features at the same level corresponding to the two modalities, and take the minimum confidence value of the semantic features at the same level as the weight;

[0026] Based on the weights, the probability distribution of video-aligned text and the probability distribution of text-aligned video, we obtain the confidence-enhanced feature alignment objective, namely the local semantic alignment loss function.

[0027] Furthermore, the missing modal semantic features are inferred, and the real semantic features corresponding to the missing modal semantic features are completed, including:

[0028] Connect the semantically aligned video modality and text modality semantic feature representations to obtain a global semantic variable;

[0029] Connect the semantic modality representations in the video modality and text modality after semantic alignment at the corresponding level to obtain semantic variables at the hierarchical semantic level;

[0030] Based on global semantic variables and semantic variables at the hierarchical semantic level, a variational multimodal autoencoder is used to generate pseudo-modal global semantic features and pseudo-modal hierarchical semantic features through the hidden layer output between the encoder and the decoder;

[0031] A reconstruction target is defined by pseudo-modal global semantic features and pseudo-modal hierarchical semantic features, and real semantic features corresponding to the pseudo-modal global semantic features and pseudo-modal hierarchical semantic features are generated according to the reconstruction target.

[0032] Furthermore, when inferring the missing modal semantic features and completing the real semantic features corresponding to the missing modal semantic features, Kullback-Leibler divergence regularization is introduced, and the Kullback-Leibler divergence is used to control the parameters in the variational multimodal autoencoder.

[0033] Furthermore, multi-semantic features of both video and text modalities are extracted at the event, behavior and subject levels respectively.

[0034] Furthermore, the step of replacing the input video and text features with the complete semantic features of the two modalities of video and text includes:

[0035] The semantic features obtained by multiplying the semantic features of multiple levels of the video modality with the corresponding weights are connected to obtain the global video comprehensive features;

[0036] The semantic features of multiple levels of text modality are connected with the semantic features obtained by multiplying the corresponding weights to obtain the global text comprehensive features;

[0037] The global video comprehensive features are used to replace the input visual features, and the global text comprehensive features are used to replace the input text features.

[0038] A second aspect of the present invention provides a target video segment positioning system based on semantic alignment, comprising:

[0039] A data preprocessing module, which is used to obtain a candidate set of video clips and texts, and preprocess the obtained video clips and texts;

[0040] A feature extraction module is used to extract features of different semantic levels from the preprocessed video clips and texts, respectively, to obtain semantic features of the two modalities of video and text at different semantic levels;

[0041] The semantic alignment module is used for semantic features of two modalities, video and text, at different semantic levels. The global alignment loss function and the local semantic alignment loss function are obtained by calculating the alignment distribution probability. According to the global alignment loss function and the local semantic alignment loss function, semantic alignment training is performed to obtain semantic features of video clips and text after semantic alignment.

[0042] The missing information completion module is used to infer the missing video clips and text semantic features based on the semantically aligned video clips and text semantic features, and complete the real semantic features corresponding to the missing modal semantic features to obtain complete semantic features of both video and text modalities;

[0043] The video positioning module is used to replace the input video and text features according to the complete semantic features of the two modalities of video and text; based on the replaced video and text features, similarity calculation is performed to locate the target video segment.

[0044] Furthermore, in the missing information completion module, the missing modal semantic features are inferred, and the real semantic features corresponding to the missing modal semantic features are completed, including:

[0045] Connect the semantically aligned video modality and text modality semantic feature representations to obtain a global semantic variable;

[0046] Connect the semantic modality representations in the video modality and text modality after semantic alignment at the corresponding level to obtain semantic variables at the hierarchical semantic level;

[0047] Based on global semantic variables and semantic variables at the hierarchical semantic level, a variational multimodal autoencoder is used to generate pseudo-modal global semantic features and pseudo-modal hierarchical semantic features through the hidden layer output between the encoder and the decoder;

[0048] A reconstruction target is defined by pseudo-modal global semantic features and pseudo-modal hierarchical semantic features, and real semantic features corresponding to the pseudo-modal global semantic features and pseudo-modal hierarchical semantic features are generated according to the reconstruction target.

[0049] A third aspect of the present invention provides a program product.

[0050] A program product, which is a computer program product, comprises a computer program. When the computer program is executed by a processor, the steps in the method for locating a target video segment based on semantic alignment as described in the first aspect are implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. Based on the semantic features of the two modalities of video and text at different levels, the present invention obtains the global alignment loss function and the local semantic alignment loss function by calculating the alignment distribution probability, and helps to achieve accurate video clip positioning through effective cross-modal semantic alignment modeling;

[0053] 2. In response to the widespread problems of missing multimodal information and ambiguous semantic information, the present invention obtains complete semantic features of both video and text modalities by inferring the missing modal semantic features and completing the real semantic features corresponding to the missing modal semantic features. It can alleviate the negative impact of missing multimodal information and ambiguous semantic information, making the model have better effects in practical applications, and at the same time can further complete the missing modal information, which will help to obtain complete semantic content and achieve more accurate positioning.

[0054] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0056] Figure 1 This is a flow chart of a method for locating a target video segment based on semantic alignment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0059] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] The target video moment localization task is to return the corresponding segment of a given video based on the user's description information. The process mainly includes extracting video features, extracting text features, processing features, and locating video segments.

[0061] This task has great application value in real-world scenarios, for example: finding specific clips in surveillance videos to help the police quickly solve cases; helping to quickly review online videos to prevent the adverse effects of uploading illegal videos; helping users quickly find clips of interest in long videos, etc.

[0062] Technical issues mentioned in the background technology:

[0063] First, due to the limitations of the collection equipment, information is missing and complete information cannot be obtained;

[0064] Second, how to accurately locate the corresponding segment when there is ambiguity in the parts of the visual modality;

[0065] Third, how to respond to users’ diverse text inputs and return the video clips they expect.

[0066] The present invention solves the challenges brought by missing multimodal information and ambiguous semantic information by effectively fusing the features of video modality and text modality, and achieves more accurate semantic matching. The specific research contents include: how to reduce the impact of missing multimodal information and ambiguous semantic information on the semantic alignment process, and how to enable the model to actively complete the missing modal information, so as to achieve semantic alignment between the target video segment and the text, and then achieve accurate positioning of the target video segment.

[0067] Embodiment 1

[0068] like Figure 1 As shown, this embodiment provides a method for locating a target video segment based on semantic alignment, comprising the following steps:

[0069] Step 1: Obtain a candidate set of video clips and texts, and pre-process the obtained video clips and texts;

[0070] Step 2: extracting features at different semantic levels from the preprocessed video clips and texts; in this embodiment, the multi-level semantic features of the two modalities of video and text include features at three levels: event, behavior, and subject;

[0071] In order to extract features at the three levels of events, behaviors, and entities in both video and text modes, we first need to obtain the unprocessed basic features. Then, six different fully connected layers are used to extract the basic features corresponding to the three levels of events, behaviors, and entities in the two modes to obtain the corresponding hierarchical features.

[0072] In this embodiment, the video mode is represented as , the text modality is represented as .

[0073] Assume that we need to semantically align a video clip feature with a query text feature, where the multi-semantic feature corresponding to the video modality is represented as , the text modality corresponds to the multi-semantic feature representation as , , They correspond to event features, behavior features and entity features respectively.

[0074] Step 3: Based on the semantic features of the two modalities of video and text at different semantic levels, the global alignment loss function and the local semantic alignment loss function are obtained by calculating the alignment distribution probability; including:

[0075] Step 301: Based on the semantic features of the two modalities of video and text at different semantic levels, a global alignment loss function is obtained by calculating the alignment distribution probability; specifically, the following steps are performed:

[0076] Step 3011: Based on features of different semantic levels of the video modality and the text modality, respectively calculate the probability distribution of video-aligned text and the probability distribution of text-aligned video;

[0077] Taking visual alignment of text as an example, assuming that the current alignment sample pair is subscripted as , the probability distribution calculation formula of visually aligned text is:

[0078] ,

[0079] in, represents semantic features from other text modalities in the same batch, , B represents all samples in the batch, j Indicates other samples in the batch except the current aligned sample.

[0080] pass , actually got the video With each query The relative probability of a match. The higher the degree of alignment, the higher the probability, approaching 1; the lower the degree of alignment, the lower the probability, approaching 0.

[0081] Step 3012: Based on the probability distribution of visually aligned text and the probability distribution of textually aligned video, a loss function of global semantic alignment is obtained:

[0082] ,

[0083] in, is the probability distribution of video-to-text alignment, is the alignment distribution probability of the text-aligned video.

[0084] It reflects the degree of alignment between the video mode and the text mode. It is part of the global loss function. Reducing the value of the global loss function by gradient calculation can reduce value, thereby achieving the purpose of improving the degree of modal alignment.

[0085] Step 302: Perform modality weighting through a multi-head cross-modal attention mechanism and combine the alignment probability distribution to finally obtain a local alignment loss function; specifically, the following steps are included:

[0086] Step 3021: using a multi-head cross-modal attention mechanism to calculate the confidence of each level of semantic features in the video modality and the text modality respectively;

[0087] For video mode or text mode, calculate each semantic feature in its mode separately The instance-level confidence of is used to capture the relative importance between different semantic features. This confidence measures the relative size of the attention score weight between a specific semantic feature and other semantic features in all attention head calculations.

[0088] Step 3022: Compare the minimum confidence value of the features of the same level corresponding to the two modalities as the weight , which is used to reduce the alignment error caused by semantic uncertainty between modalities.

[0089] Step 3033: Based on weight , the probability distribution of video alignment text and the probability distribution of text alignment video obtain the confidence-enhanced feature alignment target, that is, the local semantic alignment loss function ;

[0090] Confidence-enhanced feature alignment goal , to ensure that the weight of each semantic feature can be dynamically adjusted during the training process. The specific formula is as follows:

[0091] ,

[0092] ,

[0093] in, For representation and semantics (Event, behavior, entity) corresponding feature alignment targets are used to dynamically adjust semantics features to achieve semantic feature alignment.

[0094] As you can see, the weight The introduction is Compared with the one obtained in step 2 The main difference between same, is also part of the overall loss function. In subsequent training, by reducing In fact, it is to improve the degree of alignment between each level, so as to achieve the purpose of improving the alignment accuracy at the fragment level.

[0095] Step 4: Perform semantic alignment training based on the global alignment loss function and the local semantic alignment loss function to obtain the multimodal semantic features after semantic alignment;

[0096] The obtained global alignment loss function and local semantic alignment loss function are used in the alignment process of the multimodal semantic features extracted in step 2, respectively, to align the semantic features of the two modalities of video and text.

[0097] Through steps 3 and 4, the influence of modal noise and incompleteness can be effectively reduced.

[0098] Step 5: Based on the multimodal semantic features after semantic alignment, infer the missing modal semantic features, and complete the real semantic features corresponding to the missing modal semantic features to obtain complete semantic features of both video and text modalities;

[0099] In step 5, in this embodiment, a variational multimodal autoencoder framework is proposed to enable the model to infer complete semantic content even when some modal information is missing. Specifically, the following steps are included:

[0100] Step 501: Connect the semantic modality representations in the video modality and the text modality after semantic alignment to obtain a global semantic variable:

[0101] ,

[0102] in, , is the corresponding trainable weight, For modal Corresponding semantics The feature representation of .

[0103] Step 502: Connect the semantic modality representations in the video modality and the text modality after the corresponding hierarchical semantic alignment to obtain semantic variables at the hierarchical semantic level;

[0104] Taking event-level feature semantics as an example, in order to coordinate the representation of video modality and text modality in event-level feature semantics, the same method of global semantic variables is used to connect and , denoted as .

[0105] Step 503: Based on the global semantic variables obtained in step 501 and the semantic variables at the hierarchical semantic level obtained in step 502, a variational multimodal autoencoder is used to generate pseudo-modal hierarchical semantic features and pseudo-modal global semantic features through the hidden layer output between the encoder and the decoder;

[0106] Taking event-level feature semantics as an example, the pseudo-modal event semantic-level features generated by the hidden layer output between the encoder and the decoder are generated using the variational multimodal autoencoder. , and reparameterize it into:

[0107] ,

[0108] ,

[0109] ,

[0110] in, and is the parameter in the variational autoencoder, and Represents encoder and decoder respectively.

[0111] Step 504: Define the reconstruction target by using the pseudo-modal hierarchical semantic features and the pseudo-modal global semantic features to ensure that the generated pseudo-modal features and As close as possible to the real features and :

[0112] ,

[0113] It reflects the gap between the pseudo-modal features generated by the model and the real features. The larger the value, the greater the gap. It can make the generated pseudo-modal features approach the real features, thus ensuring that the pseudo-modal features generated by the model are sufficiently connected to the real features, which can effectively help the fragment positioning.

[0114] In this embodiment, in order to maintain the stability of the spatial distribution and avoid overfitting of the variational multimodal autoencoder framework, so as to have better generalization performance, Kullback-Leibler (KL) divergence regularization is introduced to maintain the stability of the spatial distribution:

[0115] ,

[0116] The introduction of KL divergence can effectively avoid overfitting of the model, so that the model can not only perform well on the training set, but also perform well in practical applications.

[0117] Step 6: Replace the input visual and text features according to the complete semantic features of the two modalities of video and text; specifically including:

[0118] Step 601: Connect the semantic features of multiple levels of the video modality and the semantic features obtained by multiplying the corresponding weights to obtain a global video comprehensive feature;

[0119] Step 602: Connect the semantic features of multiple levels of text modality and the semantic features obtained by multiplying the corresponding weights to obtain a global text comprehensive feature;

[0120] Step 603: Use global video comprehensive features to replace input visual features, and use global text comprehensive features to replace input text features.

[0121] In this embodiment, three levels of features and corresponding weights are used to obtain global multimodal comprehensive features:

[0122] ,

[0123] ,

[0124] Finally, the above features will be used and By replacing the original input visual and text features, cross-modal semantic alignment is achieved, thereby achieving accurate video clip positioning.

[0125] Step 7: Accurately locate the target video segment based on the replaced video and text features;

[0126] In this embodiment, the cosine similarity between the replaced video and text features is calculated to obtain the similarity between the videos in the candidate video segment set and the query statement, and the video segments with the highest similarity are selected according to the similarity, thereby achieving accurate positioning of the target video segment.

[0127] Embodiment 2

[0128] This embodiment provides a target video segment positioning system based on semantic alignment, including:

[0129] A data preprocessing module, which is used to obtain a candidate set of video clips and texts, and preprocess the obtained video clips and texts;

[0130] A feature extraction module is used to extract features of different semantic levels from the preprocessed video clips and texts, respectively, to obtain semantic features of the two modalities of video and text at different semantic levels;

[0131] The semantic alignment module is used for semantic features of two modalities, video and text, at different semantic levels. The global alignment loss function and the local semantic alignment loss function are obtained by calculating the alignment distribution probability. According to the global alignment loss function and the local semantic alignment loss function, semantic alignment training is performed to obtain semantic features of video clips and text after semantic alignment.

[0132] The missing information completion module is used to infer the missing video clips and text semantic features based on the semantically aligned video clips and text semantic features, and complete the real semantic features corresponding to the missing modal semantic features to obtain complete semantic features of both video and text modalities;

[0133] The video positioning module is used to replace the input video and text features according to the complete semantic features of the two modalities of video and text; based on the replaced video and text features, similarity calculation is performed to locate the target video segment.

[0134] It should be noted that the specific implementation process of each module is consistent with the content of Example 1 and will not be repeated here.

[0135] Embodiment 3

[0136] This embodiment provides a program product, which is a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the method for locating a target video segment based on semantic alignment as described in the first embodiment are implemented.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for locating a target video segment based on semantic alignment, characterized in that: The steps include: Obtaining a candidate set of video clips and texts, and preprocessing the obtained video clips and texts; Perform feature extraction at different semantic levels on the preprocessed video clips and texts respectively to obtain semantic features of the two modalities of video and text at different semantic levels; Based on the semantic features of the two modalities of video and text at different semantic levels, the global alignment loss function and the local semantic alignment loss function are obtained by calculating the alignment distribution probability; According to the global alignment loss function and the local semantic alignment loss function, semantic alignment training is performed to obtain semantically aligned video clips and text semantic features; According to the semantically aligned video clips and text semantic features, the missing modal semantic features are inferred, and the real semantic features corresponding to the missing modal semantic features are completed to obtain complete semantic features of both video and text modalities; Replace the input video and text features with the complete semantic features of the two modalities; Based on the replaced video and text features, similarity calculation is performed to locate the target video segment; Based on the semantic features of the two modalities of video and text at different semantic levels, the global alignment loss function is obtained by calculating the alignment distribution probability, including: Based on all semantic features of video modality and text modality, the probability distribution of video-aligned text and the probability distribution of text-aligned video are calculated respectively; The loss function of global semantic alignment is obtained based on the probability distribution of video-to-text alignment and the probability distribution of text-to-video alignment; Based on the semantic features of the two modalities of video and text at different semantic levels, the local semantic alignment loss function is obtained by calculating the alignment distribution probability, including: A multi-head cross-modal attention mechanism is used to calculate the confidence of each level of semantic features in the video modality and text modality respectively; Compare the confidences of the semantic features at the same level corresponding to the two modalities, and take the minimum confidence value of the semantic features at the same level as the weight; Based on the weights, the probability distribution of video-aligned text and the probability distribution of text-aligned video, we obtain the confidence-enhanced feature alignment target, i.e., the local semantic alignment loss function. Infer the missing modal semantic features and complete the real semantic features corresponding to the missing modal semantic features, including: Connect the semantically aligned video modality and text modality semantic feature representations to obtain a global semantic variable; Connect the semantic modality representations in the video modality and text modality after semantic alignment at the corresponding level to obtain semantic variables at the hierarchical semantic level; Based on global semantic variables and semantic variables at the hierarchical semantic level, a variational multimodal autoencoder is used to generate pseudo-modal global semantic features and pseudo-modal hierarchical semantic features through the hidden layer output between the encoder and the decoder; A reconstruction target is defined by pseudo-modal global semantic features and pseudo-modal hierarchical semantic features, and real semantic features corresponding to the pseudo-modal global semantic features and pseudo-modal hierarchical semantic features are generated according to the reconstruction target.

2. The method for locating a target video segment based on semantic alignment according to claim 1, characterized in that: When inferring the missing modal semantic features and completing the real semantic features corresponding to the missing modal semantic features, Kullback-Leibler divergence regularization is introduced, and Kullback-Leibler divergence is used to control the parameters in the variational multimodal autoencoder.

3. The method for locating a target video segment based on semantic alignment according to claim 1, characterized in that: Multi-semantic features of video and text modalities are extracted at three levels: event, behavior and subject.

4. The method for locating a target video segment based on semantic alignment according to claim 1, characterized in that: The step of replacing the input video and text features according to the complete semantic features of the two modalities of video and text includes: The semantic features obtained by multiplying the semantic features of multiple levels of the video modality with the corresponding weights are connected to obtain the global video comprehensive features; The semantic features of multiple levels of text modality are connected with the semantic features obtained by multiplying the corresponding weights to obtain the global text comprehensive features; The global video comprehensive features are used to replace the input visual features, and the global text comprehensive features are used to replace the input text features.

5. A target video segment positioning system based on semantic alignment, using a target video segment positioning method based on semantic alignment as claimed in any one of claims 1 to 4, characterized in that: include: A data preprocessing module, which is used to obtain a candidate set of video clips and texts, and preprocess the obtained video clips and texts; A feature extraction module is used to extract features of different semantic levels from the preprocessed video clips and texts, respectively, to obtain semantic features of the two modalities of video and text at different semantic levels; The semantic alignment module is used for semantic features of two modalities, video and text, at different semantic levels. The global alignment loss function and the local semantic alignment loss function are obtained by calculating the alignment distribution probability. According to the global alignment loss function and the local semantic alignment loss function, semantic alignment training is performed to obtain semantic features of video clips and text after semantic alignment. The missing information completion module is used to infer the missing modal semantic features based on the semantically aligned video clips and text semantic features, and to complete the real semantic features corresponding to the missing modal semantic features to obtain complete semantic features of both video and text modalities; The video positioning module is used to replace the input video and text features according to the complete semantic features of the two modalities of video and text; based on the replaced video and text features, similarity calculation is performed to locate the target video segment.

6. A program product, the program product being a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps in the method for locating a target video segment based on semantic alignment as described in any one of claims 1 to 4 are implemented.

Images