Long video positioning method, system and device and storage medium

Abstract

The invention discloses a long video positioning method, system and device and a storage medium, which are one-to-one corresponding schemes, in the scheme, a step-by-step positioning mode is adopted, firstly, a highlight video frame is identified through video and text guidance, so that a video clip aligned with a query text is obtained, and the problem that in an existing method, a clip is directly extracted from a long video, so that the video clip cannot be directly extracted is avoided. The problem of lack of overall perception of a long video on the text is solved; further positioning on the basis of obtaining the video clip aligned with the query text to obtain an accurate moment; in the step-by-step positioning mode, each step of positioning further strengthens the relation between the video and the query text, so that a more accurate positioning result can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of long video positioning technology, and in particular to a long video positioning method, system, device and storage medium. Background Art

[0002] Long video localization (LVTG) aims to locate the start and end time of a text description in a long video.

[0003] Most existing methods directly select a video clip from a long video by sliding a window and locate the moment in the selected video clip. However, the above methods have the following defects: (1) The sliding window method directly selects a video clip from a long video, lacking the perception of the long video for the text description, so it is difficult to obtain a video clip aligned with the text. (2) The position of the sliding window is fixed. When the real video clip falls in this position, the same video clip will be obtained. When the real video clip falls near the slice position, it will also cause the real video clip to be truncated. Therefore, it is difficult to obtain accurate positioning results.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a long video positioning method, system, device and storage medium, which can obtain video clips aligned with query text through the perception of text description, and then accurately predict the moment corresponding to the query text.

[0006] The objective of the present invention is achieved through the following technical solutions: A long video positioning method, comprising: Step 1: Obtain video data and query text; Step 2: extracting video features from the video data, and extracting text features from the query text; Step 3: using the similarity between the video features and the text features, identifying the highlighted video frames, and thereby locating the video segments aligned with the query text in the video data; Step 4: Use the attention mechanism to fuse the video features and text features corresponding to the video clip, encode them through the encoder, and then predict the moment in the video clip corresponding to the query text through the decoder.

[0007] A long video positioning system, used to implement the above method, comprising: A data acquisition module, used to acquire video data and query text; A feature extraction module, used to extract video features from the video data and text features from the query text; A highlight mining module, used to identify highlight video frames by using the similarity between the video features and the text features, and thereby locate the video segments aligned with the query text in the video data; The segment positioning module is used to fuse the video features and text features corresponding to the video segment by using the attention mechanism, encode them through the encoder, and then predict the moment in the video segment corresponding to the query text through the decoder.

[0008] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0009] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0010] It can be seen from the technical solution provided by the present invention that the present invention adopts a step-by-step positioning method. First, the highlighted video frames are identified through video and text guidance to obtain a video clip aligned with the query text, and then the precise moment is further located in the video clip. Each step of positioning further strengthens the connection between the video and the query text, thereby obtaining a more accurate positioning result. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0012] Figure 1 A flowchart of a long video positioning method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a model architecture of a long video positioning method provided by an embodiment of the present invention; Figure 3 A schematic diagram of a fragment positioning module provided by an embodiment of the present invention; Figure 4 A schematic diagram of a long video positioning system provided by an embodiment of the present invention; Figure 5 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0014] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0015] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0016] The following is a detailed description of a long video positioning method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. The instruments used in the embodiments of the present invention, if the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0017] Embodiment 1 The embodiment of the present invention provides a long video positioning method, which adopts a step-by-step positioning method, firstly obtaining a suitable video segment according to the query text and the long video, and then further locating the precise moment in the video segment. Figure 1 As shown, the method mainly includes the following steps: Step 1: Get video data and query text.

[0018] In the embodiment of the present invention, the video data is recorded as a long video that needs to be located, and the query text is a natural language text used to describe a specific event, scene or content of interest.

[0019] Step 2: extracting video features from the video data, and extracting text features from the query text.

[0020] In the embodiment of the present invention, a conventional video encoder and a text encoder may be used to complete the corresponding video feature and text feature extraction tasks.

[0021] Step 3: using the similarity between the video features and the text features, identify the highlighted video frames, and thereby locate the video clip aligned with the query text in the video data.

[0022] The preferred implementation of this step is as follows: (1) Linearly map the text features.

[0023] (2) The video features include features of each video frame. The video features are downsampled and the similarity with the text features after linear mapping is calculated to obtain the similarity score between each video frame and the query text.

[0024] (3) Select the highlighted video frames based on the similarity scores, and locate the video segments aligned with the query text by combining the highlighted video frames.

[0025] In the embodiment of the present invention, the video frames may be sorted according to the similarity, with the higher the similarity score, the higher the sorting, and finally the first m video frames are selected as the highlight video frames.

[0026] Preferably, the highlighted video frame may be taken as the center and a segment with a fixed window length may be expanded to obtain a video segment aligned with the query text.

[0027] Step 4: Use the attention mechanism to fuse the video features and text features corresponding to the video clip, encode them through the encoder, and then predict the moment in the video clip corresponding to the query text through the decoder.

[0028] In the embodiment of the present invention, the query Q in the attention mechanism is calculated using the video features corresponding to the video clip, and the key K and value V in the attention mechanism are calculated using the text features, and then fused through the cross attention mechanism: ; in, To fusion features, is the video feature corresponding to the video clip, d is the dimension of key K, T is the transposed sign, MLP is a multi-layer perceptron, and softmax is a normalized exponential function.

[0029] In the embodiment of the present invention, step 3 is implemented by a highlight mining module, and step 4 is implemented by a segment positioning module, which together constitute a long video positioning model; the video positioning model is trained using segment loss, and the segment loss is expressed as: ; in, For fragment loss; is the moment retrieval loss, which measures the difference between the real moment and the predicted moment, i.e., the moment in the video clip that the decoder predicts the query text to correspond to; For significant loss, is the contrast loss, and is the weight coefficient corresponding to the loss.

[0030] Moreover, in the training phase, based on the similarity score and the highlight loss function, we mine video frames with high similarity to the text. First, we calculate the cosine similarity between the downsampled video frame and the text, and use the sigmoid function (S-shaped growth curve) to scale the calculated cosine similarity between 0 and 1 to obtain the corresponding similarity score. Next, we use the highlight loss function, expressed as: ; Where M is the number of frame features (equivalent to the number of video frames), represents the cumulative gain of the best ranking, is the indicator function, is the indicator function corresponding to the i-th video frame, which is used to determine whether the i-th video frame is in the real moment. If it is in the real moment, the output is 1, otherwise it is 0. S represents the similarity score, is a differentiable estimate of the i-th video frame, expressed as: ; in, is the temperature coefficient, , Correspondingly, it represents the similarity score between the i-th video frame and the u-th video frame.

[0031] Based on the above highlight loss function, the similarity scores between the video frames in the real moment and the query text can be increased, and the similarity scores between the video frames not in the real moment and the query text can be reduced. Then, the top m video frames with the highest similarity scores are selected as highlight video frames.

[0032] Based on the above solution provided by the embodiment of the present invention, the long video can be positioned at the moment and the start and end times can be output. In practice, it can be installed on a computer in the form of software to provide real-time positioning; it can also be installed on a server to provide large-scale background positioning.

[0033] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0034] 1. Overall introduction of the plan.

[0035] like Figure 2 As shown, the overall model architecture of the above method of the present invention is demonstrated. On the left are two types of inputs, namely long videos and query texts. The content of the query texts is only an example and does not constitute a limitation. The above two types of inputs are respectively subjected to feature extraction through corresponding encoders (both have frozen parameters and do not participate in subsequent training) to obtain video features and text features. Afterwards, the highlight mining module is used to locate the video clip from the long video. Specifically, the highlight mining module identifies the highlight video frames through the guidance of the video and text, and then obtains the video clips aligned with the text; finally, the precise moment is further located in the video clip through the clip positioning module. The represents dot product, and the number of words N is the length of the query text. The following mainly introduces the highlight mining module and the fragment positioning module in detail.

[0036] 1. Highlight mining module.

[0037] The highlight mining module is provided with a downsampling module and a linear layer. Exemplarily, a convolution module can be used as a downsampling module to downsample the video data.

[0038] In order to enhance the understanding of text in long videos, the present invention linearly maps the query text through a linear layer and then calculates the cosine similarity with the downsampled video; then, the sigmoid function (S-shaped growth curve) is used to scale the calculated cosine similarity between 0 and 1 to obtain the similarity score between each video frame and the query text.

[0039] During the training phase, highlight loss is used to discover positive samples in video frames. Video frames contained in real video clips are considered positive samples, and others are negative samples. By learning highlight loss, video frames with highlights aligned with text can be retrieved from long videos. The highlight loss function is expressed as: ; Where M is the number of frame features, represents the cumulative gain of the best ranking, is the indicator function, is the indicator function corresponding to the i-th video frame, which is used to determine whether the i-th video frame is in the real moment. If it is in the real moment, the output is 1, otherwise it is 0. S represents the similarity score, is a differentiable estimate of the i-th video frame, expressed as: ; in, is the temperature coefficient, , Correspondingly, it represents the similarity score between the i-th video frame and the u-th video frame.

[0040] Based on the above highlight loss function, the similarity score between each video frame and the query text can be adjusted. The similarity score between the video frame in the real moment and the query text can be increased, and the similarity score between the video frame not in the real moment and the query text can be reduced. Then, the first m frames that closely correspond to the query text, that is, the first m video frames with the highest similarity scores, are selected as the video frames with highlights aligned with the text. These m frames are used as the center of the video segment, and a segment with a fixed window length is expanded. The extended frames in the segment use the features extracted by the encoder for subsequent operations. In this way, the extracted video segment is more semantically consistent with the query text, ensuring that the extracted video segment captures the real segment as completely as possible while retaining more semantic information.

[0041] In the inference stage, after obtaining the similarity scores, the first m video frames with the highest similarity scores are selected, and then the video clips are obtained in the same way as in the training stage.

[0042] Those skilled in the art can understand that the highlighted video frame is a general term in the art, which refers to the most representative or important video frames in the video. Specifically in the present invention, it refers to the video frames related to the query text.

[0043] 2. Fragment positioning module.

[0044] The present invention uses a segment positioning module to further locate the video segment obtained by the highlight module, aiming to obtain the start and end time of the video corresponding to the given query text. The segment positioning module mainly includes three parts, namely, an attention mechanism (which can be a cross attention mechanism), an encoder and a decoder.

[0045] (1) Attention mechanism.

[0046] In the embodiment of the present invention, cross attention is used to enhance the understanding of visual and textual information. Specifically: using the characteristics of the video clip Calculation query , using text features Calculation Key Sum , the fusion feature is obtained by the following formula : ; in, , and are three weight matrices, which are used to map the input vector to the query space, key space, and value space respectively. Features of the above video clips Refers to the features of all video frames in a video clip obtained by a video encoder.

[0047] (2) Encoder.

[0048] In order to better model the relationship between complex visual representations (i.e. the aforementioned fusion features), Mamba can be used as an encoder. Mamba is a new selective structural state space model that performs well in long sequence modeling tasks. It not only significantly reduces the computational cost, but also enhances the ability to interpret long sequence data. By selectively retaining historical information, it solves the long-range dependency problem while maintaining computational efficiency. Input features First, it passes through the normalization layer. Since it needs to be processed by two branches later, the output of the normalization layer is recorded as and , two linear layers are applied to transform and The dimension from Convert to The information obtained is recorded as and ,in, is the dimension of the input features, is the converted dimension. If the current sublayer , then input feature for , if the current layer , then input feature Output features for the previous layer. Apply a one-dimensional convolutional layer and SiLU activation function, and then calculate the output result through the SSM (state space model) module, denoted as .at last, And after SiLU activation function The point product is passed through the third linear layer to obtain the output features. The SiLU activation function is the Sigmoid gated linear unit, and the formula is as follows: ; in, is the input of the SiLU activation function, is the output of the SiLU activation function.

[0049] like Figure 3 The figure shows an example of the structure of the fragment localization module, which shows the specific structure of the encoder, which includes L sub-layers, each of which includes: a normalization layer, three linear layers, a convolutional layer, a state space model and two activation function layers (For example, using SiLU activation function); in the current sublayer, the normalization layer will normalize the input features. If it is the first sublayer, the input features are the fused features obtained through the attention mechanism. If it is not the first sublayer, the input features are the output features of the previous sublayer, and the output features of the last sublayer are used as the input of the decoder; then it is divided into two branches; in the first branch, the normalized output is processed in sequence by the first linear layer and the first activation function layer to obtain the output of the first branch; in the second branch, the normalized output is processed in sequence by the second linear layer, the convolution layer, the second activation function layer and the state space model to obtain the output of the second branch; the outputs of the two branches are fused and passed through the third linear layer to obtain the output features of the current sublayer.

[0050] (3) Decoder.

[0051] In the embodiment of the present invention, DAB-DETR can be used as a decoder. DAB-DETR is a common target detection method. Its basic framework is composed of a Transformer (transformation neural network) of an encoder-decoder (encoder-decoder) framework. Figure 3 In the figure, it is written as Transformer decoder, which represents the center coordinates and duration of the moment by designing learnable moments. This method pools the features around the center coordinates, calculates the cross attention of the learnable moment and the output of the encoder in the decoder and obtains the cross attention map, and adjusts the cross attention map by the duration of the moment. This part belongs to the internal principle of the decoder, and can be specifically referred to conventional technology, which will not be elaborated in this invention.

[0052] 3. Loss function.

[0053] The above two modules constitute a long video positioning model. The two modules in the model need to be trained, and the training loss is the segment loss , expressed as: ; Among them, the right side of the above formula is three losses, and is the weight coefficient corresponding to the loss.

[0054] is the moment retrieval loss, which measures the difference between the real moment and the predicted moment, where the predicted moment is the moment in the video clip corresponding to the query text predicted by the decoder. The loss is expressed as: ; Among them, the three parts on the right side of the above formula together constitute the moment retrieval loss, , and are the weight coefficients of the three parts, is the L1 norm symbol, and are the real time and the predicted time, both of which correspond to several video frames. is the generalized intersection-over-union loss used as the positioning moment, Is the cross entropy loss used as the classification prediction time belongs to the foreground or background, and Use the predicted time The specific calculation method can refer to conventional technology and will not be described in detail in the present invention.

[0055] The saliency loss is calculated using the output of the highlight mining module combined with the hinge loss max. The hinge loss consists of two pairs (i.e., the two terms on the right side of the equal sign in the following equation). The first pair represents the saliency score of a random frame in a real moment in a real video clip. Greater than the saliency score outside of the real moment in the real video clip The second pair represents the saliency score of a random frame at a real moment in a real video clip. Greater than the maximum saliency score of a random frame in the negative sample video clip , the loss is expressed as: ; in, is the error bound, which is a hyperparameter. For example, it can be set to , Q refers to the query text, S(.) is the significance score function (using a linear layer), is the feature of a random video frame in the real moment in the video clip containing the real moment, is the feature of a random video frame that is not in the real moment in the video clip containing the real moment, It is the feature of a random video frame in the video clip without the real moment. The video clip containing the real moment is obtained through the highlight mining module, and the feature of each video frame is obtained through the video encoder.

[0056] Taking a 200-frame video as an example, assuming that the video clips containing the real moments (i.e., the real video clips) obtained by the highlight mining module are the 50th to 100th frames, and the real moments are the 50th to 60th frames, then is the feature of any frame between the 50th and 60th frames, is the feature of any frame from the 61st to the 100th frame, and the video clip without the real moment is the video clip excluding the 50th to the 100th frame (i.e., the negative sample video clip), whose length is equal to the video clip of the real moment. The feature of any frame in the negative sample video clip is .

[0057] The contrast loss can further strengthen the association between visual features and text features. There are J proposals in total, where J is the batch size. Each proposal is a sample, including a video clip and a moment mark. The proposal containing the real video clip is a positive sample, and the moment mark it contains is the real moment. The rest of the proposals are negative samples. For each sample, the features of the video frame in the moment mark are averaged and pooled, and then the contrast loss is calculated by the following formula, expressed as: ; in, is the mean pooling result of the features of the video frame in the time label of the positive sample pos, is the mean pooling result of the features of the video frame in the time label of the jth candidate corresponding sample, j=1,…,J, Category tag embeddings for query text descriptions.

[0058] 2. Description of the effects.

[0059] Existing methods usually extract clips directly from long videos, lacking the overall perception of the long video to the text. The present invention uses a step-by-step positioning method to first locate a suitable video clip from the long video, and then further locate the precise moment in the video clip. Each step of positioning further strengthens the connection between the video and the text, and brings into play the positioning potential of the model, thereby promoting more accurate positioning results.

[0060] Extensive experiments on two long video temporal localization datasets, Ego4D and MAD, show that the present invention significantly achieves the most advanced performance, with an average index improvement of 0.47% for Ego4D and 1.96% for MAD.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0062] Embodiment 2 The present invention also provides a long video positioning system, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes: A data acquisition module, used to acquire video data and query text; A feature extraction module, used to extract video features from the video data and text features from the query text; A highlight mining module, used to identify highlight video frames by using the similarity between the video features and the text features, and thereby locate the video segments aligned with the query text in the video data; The segment positioning module is used to fuse the video features and text features corresponding to the video segment by using the attention mechanism, encode them through the encoder, and then predict the moment in the video segment corresponding to the query text through the decoder.

[0063] Considering that the main technical details involved in the above system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0064] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0065] Embodiment 3 The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0066] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0067] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0068] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0069] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0070] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A long video positioning method, characterized in that: include: Step 1: Obtain video data and query text; Step 2: extracting video features from the video data, and extracting text features from the query text; Step 3: using the similarity between the video features and the text features, identifying the highlighted video frames, and thereby locating the video segments aligned with the query text in the video data; Step 4: Use the attention mechanism to fuse the video features and text features corresponding to the video clip, encode them through the encoder, and then predict the moment in the video clip corresponding to the query text through the decoder.

2. A long video positioning method according to claim 1, characterized in that: The step of identifying the highlighted video frame by using the similarity between the video feature and the text feature, and thereby locating the video segment aligned with the query text in the video data comprises: Performing linear mapping on the text features; The video features include features of each video frame, the video features are downsampled, and then the similarity with the text features after linear mapping is calculated to obtain the similarity score between each video frame and the query text; The highlighted video frames are selected based on the similarity scores, and the video segments aligned with the query text are located by combining the highlighted video frames.

3. A long video positioning method according to claim 2, characterized in that: The step of combining the highlighted video frames to locate the video segment aligned with the query text comprises: The highlighted video frame is taken as the center and a segment with a fixed window length is expanded to obtain a video segment aligned with the query text.

4. A long video positioning method according to claim 1, characterized in that: The method of fusing the video features and text features corresponding to the video clip using the attention mechanism includes: The query Q in the attention mechanism is calculated using the video features corresponding to the video clip, and the key K and value V in the attention mechanism are calculated using the text features, and then fused through the cross attention mechanism: ； in, To fusion features, is the video feature corresponding to the video clip, d is the dimension of key K, T is the transposed sign, MLP is a multi-layer perceptron, and softmax is a normalized exponential function.

5. A long video positioning method according to claim 1, characterized in that: The encoder comprises L sub-layers, each of which comprises: a normalization layer, three linear layers, a convolutional layer, a state space model and two activation function layers; In the current sublayer, the normalization layer will normalize the input features. If it is the first sublayer, the input features are the fused features obtained through the attention mechanism. If it is not the first sublayer, the input features are the output features of the previous sublayer. Then it is divided into two branches. In the first branch, the normalized output is processed in sequence by the first linear layer and the first activation function layer to obtain the output of the first branch. In the second branch, the normalized output is processed in sequence by the second linear layer, the convolution layer, the second activation function layer and the state space model to obtain the output of the second branch. The outputs of the two branches are fused and passed through the third linear layer to obtain the output features of the current sublayer.

6. A long video positioning method according to claim 1, characterized in that: The step 3 is implemented by the highlight mining module, and the step 4 is implemented by the segment positioning module, which together constitute a long video positioning model; the video positioning model is trained using segment loss, and the segment loss is expressed as: ； in, is the fragment loss; is the moment retrieval loss, which measures the difference between the real moment and the predicted moment, i.e., the moment in the video clip that the decoder predicts the query text to correspond to; For significant loss, is the contrast loss, and is the weight coefficient corresponding to the loss; The significance loss is expressed as: ； in, is the error margin, Q is the query text, S(.) is the significance score function, is the feature of a random video frame in the real moment in the video clip containing the real moment, is the feature of a random video frame that is not in the real moment in the video clip containing the real moment, is a feature of a random video frame in a video clip that does not contain a real moment, the video clip containing the real moment is obtained by a highlight mining module, and the features of each video frame are obtained by a video encoder; When calculating the contrast loss, there are J candidates in total, where J is the batch size. Each candidate is a sample, including a video clip and a time mark. The candidate containing the real video clip is a positive sample, and the time mark it contains is the real time. The rest of the candidates are negative samples. For each sample, the features of the video frame in the time mark are mean pooled, and then the contrast loss is calculated by the following formula, which is expressed as: ； in, is the mean pooling result of the features of the video frame in the time label of the positive sample pos, is the mean pooling result of the features of the video frame in the time label of the jth candidate corresponding sample, j=1,…,J, Category tag embeddings for query text descriptions.

7. A long video positioning method according to claim 6, characterized in that: During the training process, the highlight loss function is calculated through the highlight mining module to adjust the similarity scores between each video frame and the query text, and then the top m video frames with the highest similarity scores are selected as the highlight video frames; the highlight video frame is taken as the center and a segment with a fixed window length is expanded to obtain the corresponding video segment.

8. A long video positioning system, characterized in that: For implementing the method described in any one of claims 1 to 7, the system comprises: A data acquisition module, used to acquire video data and query text; A feature extraction module, used to extract video features from the video data and text features from the query text; A highlight mining module, used to identify highlight video frames by using the similarity between the video features and the text features, and thereby locate the video segments aligned with the query text in the video data; The segment positioning module is used to fuse the video features and text features corresponding to the video segment by using the attention mechanism, encode them through the encoder, and then predict the moment in the video segment corresponding to the query text through the decoder.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images