Joint time-of-flight and highlight detection method and system based on multi-scale difference

Abstract

The application provides a joint time retrieval and highlight detection method and system based on multi-scale difference, and the technical points are as follows: input video and text data, extract visual and text features; perform cross-modal alignment of segment-text feature semantics; use cross-attention mechanism to highlight video segments related to expression and text query; use difference information perception to mine the difference information between adjacent segments, use multi-scale convolution and graph convolution network to mine the multi-scale relationship between segments, and obtain the joint representation of multi-scale difference perception; use the task head of time retrieval and highlight detection respectively to predict the final result, calculate the respective penalty function, perform back propagation to optimize the parameters of the whole network, and output the final highlight segment and retrieval time after the overall network parameters are converged. The application mainly alleviates the problem that the existing method does not consider that the time duration and highlight distribution corresponding to different user queries exist significant differences.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image and video processing technology, and more specifically, to a method and system for joint time retrieval and specular detection based on multi-scale difference. Background Technology

[0002] Compared to still images, videos contain more complex activities and richer semantic information. However, the sheer volume of videos also makes them more difficult to understand and harder to quickly retrieve desired videos and segments. Therefore, tasks related to efficient searching and browsing of video content, such as time-based retrieval and highlight detection, have attracted widespread attention. Time-based retrieval aims to locate moments in a video relevant to a query, while highlight detection aims to find the most engaging segments within the video.

[0003] Recently, Lei et al. first attempted to jointly solve moment retrieval and highlight detection in the paper "Detection of Moments and Highlights in Videos via Natural Language Queries". They constructed the QVHighlights dataset with joint annotations for moment retrieval and highlight detection and proposed a baseline model based on a detection converter, called Moment-DETR. Since then, several papers have designed various methods to address different problems in joint moment retrieval and highlight detection. For example, to reduce network parameters and save computational costs, Li et al. proposed a lightweight cross-modal attention module for modal interaction in the paper "A Lightweight Multimodal Interaction Model for Joint Video Moment Retrieval and Highlight Detection". Xu et al. introduced pooling operations into the interaction module in the paper "Video Moment and Highlight Detection with Cross-Modal Converter" to emphasize visual features relevant to the query. Although recent methods have achieved satisfactory results in joint moment retrieval and highlight detection, they are still plagued by three problems.

[0004] Existing methods typically design different cross-modal attention modules to directly interact with unaligned visual and textual features to generate joint representations, ignoring the natural semantic gaps between different modalities, which weakens modal complementarity.

[0005] These methods typically use multi-head self-attention modules to model the global correlation between video segments. However, due to significant differences in the duration of moments and highlight distributions corresponding to different user queries, global correlation features struggle to adequately capture these differences. Since videos usually record the continuous, gradual progression of events, the transitions between different events are relatively smooth, resulting in similar semantic information between adjacent segments. This makes it difficult for existing methods to accurately distinguish between different event boundaries, leading to blurred moment boundaries and unprominent highlight segments. Summary of the Invention

[0006] This invention addresses the technical problems existing in the prior art by providing a joint time retrieval and highlight detection method and system based on multi-scale difference, which mainly alleviates the problem that existing methods do not take into account the significant differences in the duration of time and highlight related to different user queries.

[0007] According to a first aspect of the present invention, a joint time retrieval and specular detection method based on multi-scale difference is provided, comprising the following steps:

[0008] Input video and text data, and use a pre-trained feature extraction framework to extract visual and text features;

[0009] Use the fragment-text alignment module to perform cross-modal semantic alignment of fragment-text feature semantics;

[0010] Use cross-attention mechanisms to highlight video segments that are relevant to expression and text queries;

[0011] Differential information perception is used to mine the differential information between adjacent segments, and multi-scale convolution and graph convolutional networks are used to mine the multi-scale relationship between segments to obtain a joint representation of multi-scale differential perception.

[0012] The task heads for time retrieval and highlight detection are used to predict the final result, and the penalty functions for time retrieval and highlight detection are calculated separately. Backpropagation is then performed to optimize the parameters of the entire network. After all the parameters of the network converge, the highlight fragments and retrieval times are output.

[0013] Based on the above technical solution, the present invention can also be improved as follows.

[0014] Optionally, the extraction of visual and textual features using a pre-trained feature extraction framework includes:

[0015] The input video is segmented into non-overlapping and spaced segments;

[0016] A visual feature extractor is used to extract video features from each segment, and a text encoder is used to extract word-level text features for the text.

[0017] Optionally, the segment-text alignment module includes a video segment-text feature correlation score penalty function and a related-unrelated segment correlation score penalty function; the video segment-text feature correlation score penalty function is used to perform cross-modal alignment, mapping video segments related to the text to positions close to the text in the joint space, and the related-unrelated segment correlation score penalty function is used to amplify the similarity difference between related segments and unrelated segments, so that the similarity between related segments is greater than the similarity with unrelated segments.

[0018] Optionally, the video segment-text feature correlation score penalty function is calculated as follows:

[0019]

[0020]

[0021] in, These are segment-level video features, where L is the number of video segments. These are global text features obtained using mean pooling. Represents V and Q g Cosine similarity between them; The classification labels are constructed based on the annotations in the dataset. If the i-th segment belongs to a part of the retrieved time, then C... i It is 1 if it is true, otherwise it is 0.

[0022] Optionally, the relevant-irrelevant segment correlation score penalty function is calculated as follows:

[0023]

[0024]

[0025] in, This represents the correlation between the segments. This is used to indicate whether a fragment is truly relevant to the query; if both the i-th and j-th fragments are relevant to the query, then... It is 1 if it is true, otherwise it is 0.

[0026] Optionally, the method of using cross-attention mechanisms to highlight video segments relevant to expressions and text queries is represented as follows:

[0027]

[0028] Where Z is the joint representation of relevant segments of the highlighted text; It is a word-level text feature, where N is the number of words;

[0029]

[0030] e is the base of the natural logarithm, y i This represents the i-th element of vector Y.

[0031] Optionally, the step of using differential information perception to mine differential information between adjacent segments, and employing multi-scale convolutional and graph convolutional networks to mine multi-scale relationships between segments, to obtain a joint representation of multi-scale differential perception includes:

[0032] Multi-scale differential modeling is performed on the video segment-level representation. The multi-scale differential modeling is divided into the following steps: first, extracting the difference information between adjacent segments; then, using multi-scale convolutional and graph convolutional networks to encode the difference-aware video segment features at multiple scales; and finally, using a linear layer to fuse the above encoding results to obtain the multi-scale differential-aware video segment-level representation.

[0033] Optionally, the penalty function retrieved at the specified time is as follows:

[0034]

[0035] in, 'm' represents the start and end timestamps of the predicted time and the start and end timestamps of the actual relevant time; ||·|| represents the L1 matrix norm. Let y represent the result and label of binary classification of L segments, and λ represent the result and label of binary classification of L segments. L1 and λ CE It is an adjustable parameter used to balance the two penalty functions.

[0036] Optionally, the penalty function for specular detection is as follows:

[0037] L margin =max(0,Δ+S(z) low )-S(z high ))

[0038] Where Δ is the margin, S(·) is the specular fraction mapping function, and z low and z high These are a pair of video feature vectors, one from a low-scoring segment within a relevant timeframe and the other from a high-scoring segment within an irrelevant timeframe.

[0039] According to a second aspect of the present invention, a joint time retrieval and specular detection system based on multi-scale difference is provided, comprising:

[0040] The data acquisition module is used to input video and text data and extract visual and text features using a pre-trained feature extraction framework.

[0041] The semantic alignment module is used to perform cross-modal semantic alignment of fragment-text features using the fragment-text alignment module, and uses a cross-attention mechanism to highlight video fragments that are relevant to the expression and text query.

[0042] The joint representation module is used to mine the differential information between adjacent segments using differential information perception. It employs multi-scale convolution and graph convolutional networks to mine the multi-scale relationships between segments and obtains a joint representation of multi-scale differential perception.

[0043] The reverse network optimization module is used to predict the final result using the task heads of time retrieval and highlight detection respectively, and to calculate the penalty functions of time retrieval and highlight detection respectively. It performs backpropagation to optimize the parameters of the entire network. After all the parameters of the network converge, the highlight fragment and the retrieval time are output.

[0044] The technical effects and advantages of this invention are as follows:

[0045] This invention proposes a joint video moment retrieval and highlight detection method based on the inherent multi-scale characteristics of videos. By extracting the difference information between adjacent segments in a video and then encoding it using multi-scale convolutional and graph convolutional networks, a joint representation with multi-scale difference perception is obtained. Compared to existing attention-based methods, the obtained joint representation possesses the multi-scale difference information of the video, which is beneficial for the model to identify semantically similar adjacent segments, while also addressing the issue of varying moment durations corresponding to different queries.

[0046] Simultaneously, it alleviates the problem of inherent semantic gaps between different modalities. Since features of different modalities are extracted by different feature extractors, inherent semantic gaps exist, which can interfere with subsequent modal interactions and fusion. Existing methods directly perform modal interactions, ignoring the semantic gap problem, thus impairing the complementarity of different modalities. Attached Figure Description

[0047] Figure 1 The flowchart illustrates a joint time retrieval and specular detection method based on multi-scale difference, as provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that the purpose of these embodiments is to improve the following aspects:

[0050] 1. Existing joint solutions for time retrieval and specular detection do not take into account the natural semantic differences between different modalities.

[0051] 2. Existing methods generally use multi-head self-attention to model the global relationship between segments, which makes it difficult to capture the variable duration of moments related to the query.

[0052] 3. Because the transitions between unedited videos are relatively smooth, existing methods have difficulty accurately distinguishing different event boundaries, resulting in blurred predicted time boundaries and poor highlight results.

[0053] Understandably, given the deficiencies in the background technology, this invention proposes a joint time retrieval and specular detection method based on multi-scale difference, referring to... Figure 1 As shown, the implementation steps of this embodiment of the invention are as follows:

[0054] Step 1: Input video and text data, and use a pre-trained feature extraction framework to extract visual and text features;

[0055] The aforementioned technical solution specifically includes: First, inputting a video of a living area uploaded to a video platform by a video creator and a text description of an event in the video, segmenting the input video into non-overlapping segments with a 2-second interval. Next, applying a visual feature extractor to extract video features from each segment. For the text, a text encoder is used to extract word-level text features.

[0056] Step 2: Use the fragment-text alignment module to perform cross-modal alignment of fragment-text feature semantics;

[0057] In the above technical solution, the segment-text alignment module includes a video segment-text feature correlation score penalty function and a related-unrelated segment feature similarity score penalty function. The former is used for cross-modal alignment, mapping text-related video segments to positions in the joint space that are close to the text. The latter is used to amplify the similarity difference between related and unrelated segments, making the similarity between related segments greater than the similarity with unrelated segments. Specifically, the video segment-text feature correlation score penalty function is as follows:

[0058]

[0059]

[0060] in, These are segment-level video features, where L is the number of video segments. These are global text features obtained using mean pooling. Represents V and Q g Cosine similarity between them; The classification labels are constructed based on the annotations in the dataset. If the i-th segment belongs to a part of the retrieved time, then C... i It is 1 if it is true, otherwise it is 0.

[0061] The similarity score penalty function for relevant-irrelevant fragment features is as follows:

[0062]

[0063]

[0064] in, This represents the correlation between the segments. This indicates whether a fragment is truly relevant to the query. Specifically, if both the i-th and j-th fragments are relevant to the query, then... It is 1 if it is true, otherwise it is 0.

[0065] Step 3: Use cross-attention mechanisms to highlight video segments that are relevant to the expression and text query;

[0066]

[0067] Where Z is the joint representation of relevant segments of the highlighted text; It is a word-level text feature, where N is the number of words;

[0068]

[0069] e is the base of the natural logarithm, y i This represents the i-th element of vector Y.

[0070] Step 4: Use differential information perception to mine the differential information between adjacent segments, and use multi-scale convolution and graph convolutional networks to mine the multi-scale relationship between segments to obtain the joint representation of multi-scale differential perception.

[0071] The above technical solution specifically includes: multi-scale differential modeling of video segment-level representations. Multi-scale differential modeling consists of two steps: first, extracting the difference information between adjacent segments; then, using multi-scale convolutional and graph convolutional networks respectively to encode the difference-aware video segment features at multiple scales; finally, using linear layers to fuse the above encoding results to obtain a multi-scale differential-aware video segment-level representation. The implementation of adjacent differential sensing is as follows:

[0072] D={d i =z j -z i |1≤i≤L-1,j=i+1}, (7)

[0073] Z D =Linear(Concat(D,Z)) (8)

[0074] Where D represents the set of differential information between adjacent segments, and d in D L Set it to zero because it's the last segment of the video; Z D It is a fragment-level joint feature that aggregates differential information. Linear(·) is a linear projection layer, and Concat(·) represents the concatenation operation.

[0075] The multi-scale coding fusion process is as follows:

[0076] Z l =Linear(MSC(Z) D (9)

[0077] Z g =GCC(Z) D (10)

[0078]

[0079] Among them, multi-scale convolution (MSC) is used to capture multi-scale local features between video segments. And use the Graph Convolutional Component (GCC) to obtain global features.

[0080] Then, Z l and Z g Further fusion via multi-head self-attention (MSA) generates a multi-scale differential perceptual representation.

[0081] Step 5: After obtaining the multi-scale differential perception joint representation of the video segment, the task heads for time retrieval and specular detection are used to predict the final result. Based on the prediction result, the boundary penalty function for time retrieval and the specular score penalty function for specular detection are calculated. Backpropagation is performed to optimize the parameters of the entire network. After all the parameters of the network converge, the specular segment and the retrieved time are output.

[0082] The time-boundary penalty function includes two penalty functions used to measure the accuracy of the time-boundary, as follows:

[0083]

[0084] in, 'm' represents the start and end timestamps of the predicted time and the start and end timestamps of the actual relevant time; ||·|| represents the L1 matrix norm. Let y represent the result and label of binary classification of L segments, and λ represent the result and label of binary classification of L segments. L1 and λ CE It is an adjustable parameter used to balance the two penalty functions.

[0085] The highlight detection highlight score penalty function includes a penalty function for measuring the highlight score distribution, as shown below:

[0086] L margin =max(0,Δ+S(z) low )-S(z high(13)

[0087] Where Δ is the margin, S(·) is the specular fraction mapping function, and z low and z high These are a pair of video feature vectors, one from a low-scoring segment within a relevant timeframe and the other from a high-scoring segment within an irrelevant timeframe.

[0088] This invention proposes a joint time-of-view retrieval and highlight detection solution based on multi-scale difference. First, segment-text alignment is used to ensure meaningful association in subsequent modal interactions. Then, a multi-scale difference perceptual network is used to extract the difference information of adjacent segments and perform multi-scale modeling. Overall, this invention is a joint time-of-view retrieval and highlight detection solution based on multi-scale difference, which can alleviate the inherent problem of semantic inconsistency between different modalities and address the issue of large differences in the duration of video moments corresponding to different queries. Furthermore, to further extract the difference information of adjacent segments to assist the model in identifying semantically similar but fundamentally different video segments, users can obtain accurate relevant times and prominent highlight segments.

[0089] The effects of this invention can be further illustrated by the following experiments.

[0090] Experimental conditions

[0091] This invention utilizes a Gen Intel® Core™ i9-13900KF CPU, a GeForce GTX 3090 Ti GPU, 64GB of RAM, and a Linux operating system, employing the PyTorch deep learning framework for model building and simulation. To comprehensively evaluate its performance, we validated the multi-scale difference network on the Joint Moment Retrieval and Highlight Detection dataset (QVHighlights) and the independent Highlight Detection dataset (TVSum). QVHighlights contains 10,148 short video clips (150 seconds each), each clip having at least one text query annotation describing a relevant moment. Each query has an average of approximately 1.8 irrelevant moments. We selected 8,768 query videos from this dataset for training and validation of the network using the multi-scale difference-based Joint Moment Retrieval and Highlight Detection method, and used an additional 1,542 query videos for test accuracy evaluation. The TVSum dataset includes 10 domains, with 5 videos per domain, and a random training / test ratio of 0.8 / 0.2 was used for training and testing.

[0092] Experimental content

[0093] First, we use the mean precision of time retrieval (IoU is uniformly sampled across 10 thresholds in {0.5, 0.95}) and the Top-5 mean precision of highlight detection as standards to simultaneously evaluate the time retrieval and highlight detection performance of the method of this invention. Then, we conduct experiments and tests based on the QVHighlights dataset according to the steps given in the specific implementation method.

[0094] These methods directly perform modal interactions without considering the variable duration of relevant moments. The comparison results are shown in Table 1 below:

[0095] Table 1 Performance Comparison Results

[0096]

[0097] As clearly shown in Table 1, the joint time-of-view retrieval and specular detection method based on multi-scale difference adopted in this invention performs optimally. This is because the multi-scale difference module fully leverages the multi-granularity characteristics of the video and explicitly extracts the difference information between adjacent segments, which helps the model accurately identify semantically similar adjacent segments, thereby improving the discriminative power of the joint representation. Furthermore, to ensure meaningful associations in cross-modal interactions, this invention also constructs a segment-text alignment module to optimize feature representations between different modalities. Therefore, the method of this invention exhibits superior performance, and comparative experiments on public datasets further validate the advanced effectiveness of this method.

[0098] In summary, the multi-scale difference-based joint time retrieval and specular detection method described in this invention has the following advantages over existing joint time retrieval and specular detection methods:

[0099] 1. The joint representation obtained in the feature encoding stage possesses multi-scale differential information. This invention's embodiment is a method for joint video time-of-view retrieval and specular highlight detection based on the inherent multi-scale characteristics of video. It extracts differential information from adjacent segments in the video and then encodes it using multi-scale convolutional and graph convolutional networks to obtain a multi-scale differential-aware joint representation. Compared to previous attention-based methods, the obtained joint representation possesses multi-scale differential information from the video, which is beneficial for the model to identify semantically similar adjacent segments and also addresses the issue of varying time durations corresponding to different queries.

[0100] 2. It alleviates the inherent semantic gap between different modalities. Since features of different modalities are extracted by different feature extractors, there are inherent semantic gaps that can interfere with subsequent modal interactions and fusion. Previous methods directly performed modal interactions, ignoring the semantic gap problem and thus impairing the complementarity of different modalities.

[0101] Additionally, embodiments of the present invention also provide a joint time retrieval and specular detection system based on multi-scale difference, comprising:

[0102] The data acquisition module is used to input video and text data and extract visual and text features using a pre-trained feature extraction framework.

[0103] The semantic alignment module is used to perform cross-modal semantic alignment of fragment-text features using the fragment-text alignment module; it uses a cross-attention mechanism to highlight video fragments that are relevant to the expression and text query.

[0104] The joint representation module is used to mine the differential information between adjacent segments using differential information perception. It employs multi-scale convolution and graph convolutional networks to mine the multi-scale relationships between segments and obtains a joint representation of multi-scale differential perception.

[0105] The reverse network optimization module is used to predict the final result using the task heads of time retrieval and highlight detection respectively, and to calculate the penalty functions of time retrieval and highlight detection respectively. It performs backpropagation to optimize the parameters of the entire network. After all the parameters of the network converge, the highlight fragment and the retrieval time are output.

[0106] It is understood that the joint time retrieval and highlight detection system based on multi-scale difference provided by the present invention corresponds to the joint time retrieval and highlight detection method based on multi-scale difference provided in the foregoing embodiments. The relevant technical features of the joint time retrieval and highlight detection system based on multi-scale difference can be referred to the relevant technical features of the joint time retrieval and highlight detection method based on multi-scale difference, and will not be repeated here.

[0107] In summary, the joint time-of-view retrieval and specular detection method and system based on multi-scale difference proposed in this invention extracts the difference information of adjacent segments in a video and then encodes it using multi-scale convolutional and graph convolutional networks to obtain a joint representation of multi-scale difference perception. Compared with existing attention-based methods, the obtained joint representation has multi-scale difference information of the video, which is beneficial for the model to identify semantically similar adjacent segments, and also addresses the problem of varying time durations corresponding to different queries.

[0108] Simultaneously, it alleviates the problem of inherent semantic gaps between different modalities. Since features of different modalities are extracted by different feature extractors, inherent semantic gaps exist, which can interfere with subsequent modal interactions and fusion. Existing methods directly perform modal interactions, ignoring the semantic gap problem, thus impairing the complementarity of different modalities.

[0109] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0110] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0111] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A joint time retrieval and specular detection method based on multi-scale difference, characterized in that, Includes the following steps: Input video and text data, and use a pre-trained feature extraction framework to extract visual and text features; A fragment-to-text alignment module is used to perform cross-modal semantic alignment of fragment-to-text features. The fragment-to-text alignment module includes a video fragment-to-text feature correlation score penalty function and a correlation-irrelevant fragment correlation score penalty function. The video fragment-to-text feature correlation score penalty function is used to perform cross-modal alignment, mapping video fragments related to the text to positions in the joint space that are close to the text. The correlation-irrelevant fragment correlation score penalty function is used to amplify the similarity difference between related fragments and irrelevant fragments, making the similarity between related fragments greater than the similarity with irrelevant fragments. Use cross-attention mechanisms to highlight video segments that are relevant to expression and text queries; Differential information perception is used to mine the differential information between adjacent segments. Multi-scale convolution and graph convolutional networks are employed to mine the multi-scale relationships between segments, obtaining a joint representation of multi-scale differential perception. This includes: calculating the differential information set between the joint representations of adjacent video segments. Set the difference information corresponding to the last video segment to zero, and set the difference information set... Joint representation with video clip level The fragments are stitched together and then subjected to a linear projection layer to obtain differential-aware joint features at the fragment level. Multi-scale encoding of the segment-level joint features of the differential sensing is performed using multi-scale convolutional and graph convolutional networks respectively, and the encoding results are fused using linear layers to obtain the multi-scale differential sensing video segment-level joint representation. After obtaining the multi-scale differential sensing joint representation of the video segment, the task heads for time retrieval and highlight detection are used to predict the final result, and the penalty functions for time retrieval and highlight detection are calculated respectively. Backpropagation is then performed to optimize the parameters of the entire network. After all the parameters of the network converge, the highlight segment and the retrieved time are output.

2. The method for joint time retrieval and specular detection based on multi-scale difference according to claim 1, characterized in that, The extraction of visual and textual features using a pre-trained feature extraction framework includes: The input video is segmented into non-overlapping and spaced segments; A visual feature extractor is used to extract video features from each segment, and a text encoder is used to extract word-level text features for the text.

3. The method for joint time retrieval and specular detection based on multi-scale difference according to claim 1, characterized in that, The video segment-text feature correlation score penalty function is calculated as follows: in, It represents and Cosine similarity between them; These are segment-level video features. The number of video clips. The dimension of the feature space representing the video clip features and text features; These are global text features obtained using mean pooling; The classification labels are constructed based on the annotations in the dataset. If the first... If a segment belongs to a part of the retrieved time, then It is 1 if it is true, otherwise it is 0. This represents the i-th element of the similarity vector S, i.e., the feature of the i-th video segment and the global text feature. Cosine similarity between them.

4. The method for joint time retrieval and specular detection based on multi-scale difference according to claim 1, characterized in that, The relevant-irrelevant segment correlation score penalty function is calculated as follows: in, This represents the correlation between the segments. These are segment-level video features. The number of video clips. The dimension of the feature space representing the video clip features and text features; Used to indicate whether a fragment is truly relevant to the query; if the first fragment... The segment and the first If all fragments are related to the query, then It is 1 if it is true, otherwise it is 0. Representing the correlation matrix The Middle line, number The element of the column, i.e., the first Features of video segments With the Features of video segments Cosine similarity between them.

5. The method for joint time retrieval and specular detection based on multi-scale difference according to claim 1, characterized in that, The use of cross-attention mechanisms to highlight video segments relevant to expression and text queries is represented as follows: in, It highlights the joint representation of relevant segments of the text; These are segment-level video features. The number of video clips. Feature dimensions representing video clip features and word-level text features; It is a word-level text feature. The number of words; in, It is the base of the natural logarithm. Representing vectors The One element, Representing vectors The One element, This is the index of the summation term in the denominator of the Softmax normalized denominator.

6. The method for joint time retrieval and specular detection based on multi-scale difference according to claim 1, characterized in that, The penalty function for the time-based retrieval is as follows: in, This represents the penalty function value at the predicted time. and This indicates the start and end timestamps of the predicted time, as well as the start and end timestamps of the actual relevant time. Denotes the L1 matrix norm; and It is an adjustable parameter used to balance the two penalty functions. This is used to penalize the learning of query-related fragment features, encouraging the model to identify fragment features located within the relevant time points. In the formula, and Indicates to The results and labels of binary classification of the segments.

7. The method for joint time retrieval and specular detection based on multi-scale difference according to claim 1, characterized in that, The penalty function for specular detection is as follows: in, This represents the penalty function value at the highlight moment. It's the margin. It is a highlight fraction mapping function. and These are a pair of video feature vectors, one from a low-scoring segment within a relevant timeframe and the other from a high-scoring segment within an irrelevant timeframe.

8. A joint time-based retrieval and specular detection system based on multi-scale difference, characterized in that, include: The data acquisition module is used to input video and text data and extract visual and text features using a pre-trained feature extraction framework. A semantic alignment module is used to perform cross-modal semantic alignment of fragment-text features using a fragment-text alignment module. The fragment-text alignment module includes a video fragment-text feature relevance score penalty function and a relevant-irrelevant fragment relevance score penalty function. The video fragment-text feature relevance score penalty function performs cross-modal alignment, mapping text-related video fragments to positions in the joint space that are close to the text. The relevant-irrelevant fragment relevance score penalty function amplifies the similarity difference between relevant and irrelevant fragments, making the similarity between relevant fragments greater than the similarity with irrelevant fragments. A cross-attention mechanism is used to highlight video fragments that express relevance to the text query. The joint representation module is used to mine differential information between adjacent segments using differential information perception. It employs multi-scale convolutional and graph convolutional networks to mine multi-scale relationships between segments, obtaining a joint representation of multi-scale differential perception. This includes: calculating the set of differential information between the joint representations of adjacent video segments. Set the difference information corresponding to the last video segment to zero; set the difference information set... Joint representation with video clip level The fragments are stitched together and then passed through a linear projection layer to obtain differential-aware joint features at the fragment level. Then, multi-scale convolutional and graph convolutional networks are used to encode the segment-level joint features of the differential sensing at multiple scales, and linear layers are used to fuse the encoding results to obtain the multi-scale differential sensing video segment-level joint representation. The reverse network optimization module is used to predict the final result using the task heads of time retrieval and highlight detection respectively, and to calculate the penalty functions of time retrieval and highlight detection respectively. It performs backpropagation to optimize the parameters of the entire network. After all the parameters of the network converge, the highlight fragment and the retrieval time are output.

Images