Video segment positioning method and device, and electronic device

Abstract

The application discloses a video segment positioning method and device and electronic equipment. The method comprises the following steps: obtaining a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video respectively; each video feature in the first feature sequence and the second feature sequence comprises a time sequence feature; the first feature sequence and the second feature sequence are subjected to feature information interaction to obtain a first enhanced sequence and a second enhanced sequence; each enhanced feature of the first enhanced sequence and the second enhanced sequence comprises a time sequence consistency regularization term; a target similarity matrix is obtained according to the first enhanced sequence and the second enhanced sequence; the target similarity matrix is input into a target detection model; the target detection model outputs a detection result; the detection result comprises a probability value of a video segment in the second video and the first video satisfying a similarity condition, a time point position of the video segment in the first video, and a time point position of the video segment in the second video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a video segment positioning method and device and electronic equipment. BACKGROUND

[0002] At present, the image frame features of a query video and a reference video are extracted through a convolutional neural network, and then dynamic time warping (DTW) algorithm and the like are combined to perform time sequence alignment on the image frames, so as to locate the starting time point and the ending time point of the similar video segment in the reference video and the query video, thereby realizing video segment positioning of the query video in the reference video.

[0003] However, the video segment positioning scheme cannot be used to position the video segment of the query video in the reference video when the query video has a variable speed or is played in reverse. SUMMARY

[0004] In view of the above problems, the present application provides a video segment positioning method, device and electronic equipment to achieve the purpose of … The specific scheme is as follows:

[0005] The first aspect of the present application provides a video segment positioning method, comprising:

[0006] obtaining a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video respectively; each video feature in the first feature sequence and the second feature sequence includes a time sequence feature, and the time sequence feature represents the time sequence of the corresponding video frame in the video;

[0007] interacting feature information of the first feature sequence and the second feature sequence to obtain a first enhanced sequence and a second enhanced sequence; each enhanced feature in the first enhanced sequence and the second enhanced sequence includes a time sequence consistency regularization term; the time sequence consistency regularization term represents the difference between the corresponding video frame and other video frames that satisfy the adjacent condition in time sequence in the video;

[0008] obtaining a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; a matrix element in the target similarity matrix represents the similarity between the corresponding video frame in the first video and the corresponding video frame in the second video;

[0009] inputting the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result includes: a probability value of a video segment in the second video and the first video that satisfies a similarity condition, a time point of the video segment in the first video, and a time point of the video segment in the second video.

[0010] In a possible implementation, obtaining the first feature sequence corresponding to the first video comprises:

[0011] performing video feature extraction on each frame of video frames contained in the first video to obtain a first initial sequence;

[0012] adding a corresponding time sequence feature in a first time coding sequence to each item of video features in the first initial sequence to obtain the first feature sequence; each time sequence feature in the first time coding sequence is different;

[0013] In a possible implementation, obtaining the second feature sequence corresponding to the second video comprises:

[0014] performing video feature extraction on each frame of video frames contained in the second video to obtain a second initial sequence;

[0015] adding a corresponding time sequence feature in a second time coding sequence to each item of video features in the second initial sequence to obtain the second feature sequence.

[0016] In a possible implementation, performing feature information interaction on the first feature sequence and the second feature sequence to obtain a first enhanced sequence comprises:

[0017] inputting the first feature sequence into a self-attention layer to obtain a first initial enhanced sequence;

[0018] inputting the first initial enhanced sequence and the second feature sequence into a cross-attention layer to obtain the first enhanced sequence;

[0019] In a possible implementation, performing feature information interaction on the first feature sequence and the second feature sequence to obtain a second enhanced sequence comprises:

[0020] inputting the second feature sequence into the self-attention layer to obtain a second initial enhanced sequence;

[0021] inputting the second initial enhanced sequence and the first feature sequence into the cross-attention layer to obtain the second enhanced sequence;

[0022] In a possible implementation, the self-attention layer comprises a multi-head attention processing part and a time sequence consistency processing part; the multi-head attention processing part is configured to obtain semantic relationships between video frames in an input video; and the time sequence consistency processing part is configured to obtain a difference degree between video frames that satisfy a proximity condition in time sequence in the input video.

[0023] In a possible implementation, obtaining a target similarity matrix according to the first enhanced sequence and the second enhanced sequence comprises:

[0024] According to a dynamic temperature coefficient, correlation calculation is performed on the first enhancement sequence and the second enhancement sequence to obtain an initial similarity matrix;

[0025] The dynamic temperature coefficient represents a difference degree between the first video and the second video.

[0026] The matrix elements in the initial similarity matrix are subjected to matrix normalization processing to obtain a target similarity matrix.

[0027] In a possible implementation, the matrix elements in the initial similarity matrix are determined based on vector inner product results between each enhancement feature in the first enhancement sequence and each enhancement feature in the second enhancement sequence.

[0028] In a possible implementation, the dynamic temperature coefficient is obtained in the following manner:

[0029] The vector inner product results between each enhancement feature in the first enhancement sequence and each enhancement feature in the second enhancement sequence are obtained.

[0030] According to the vector inner product results, a global average similarity between the first video and the second video is obtained.

[0031] According to the global average similarity, the dynamic temperature coefficient is obtained.

[0032] In a possible implementation, the loss function of the target detection model comprises a first loss part and a second loss part.

[0033] The first loss part is used to optimize the probability value, and the second loss part is used to optimize the time point bit.

[0034] In a possible implementation, the feature sequences corresponding to a first sample video and a second sample video in a training sample video group of the target detection model are added with class label items; the class label items represent video global features of the sample videos to which the class label items belong.

[0035] The loss function of the target detection model at least comprises loss weights corresponding to the training sample video group, the loss weights representing a number of key frames in the first sample video and an information entropy corresponding to the first sample video; the key frames are video frames in which corresponding enhancement features in the training sample video group and corresponding class label items satisfy a similarity condition; and the information entropy is an information entropy of a sequence composed of the enhancement features corresponding to the first sample video.

[0036] The second aspect of the application provides a video segment positioning device, comprising:

[0037] The feature obtaining unit is configured to obtain a first feature sequence corresponding to the first video and a second feature sequence corresponding to the second video respectively; each video feature in the first feature sequence and the second feature sequence is added with a time sequence feature, and the time sequence feature represents a time sequence of the corresponding video frame in the video;

[0038] The feature enhancing unit is configured to perform feature information interaction between the first feature sequence and the second feature sequence to obtain a first enhanced sequence and a second enhanced sequence; each enhanced feature in the first enhanced sequence and the second enhanced sequence comprises a time sequence consistency regular term; the time sequence consistency regular term represents a difference degree between the corresponding video frame and other video frames that satisfy a neighboring condition in time sequence in the video;

[0039] The matrix obtaining unit is configured to obtain a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; a matrix element in the target similarity matrix represents a similarity degree between a corresponding video frame in the first video and a corresponding video frame in the second video;

[0040] The matrix processing unit is configured to input the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result comprises a probability value of a video segment that satisfies a similarity condition in the second video and the first video, a time point of the video segment in the first video, and a time point of the video segment in the second video.

[0041] The third aspect of the present application provides a computer program product, comprising computer readable instructions, when the computer readable instructions are executed on an electronic device, the electronic device implements the video segment positioning method of the first aspect or any implementation manner of the first aspect.

[0042] The fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected with the processor, wherein:

[0043] The memory is configured to store a computer program;

[0044] The processor is configured to execute the computer program, so that the electronic device can implement the video segment positioning method of the first aspect or any implementation manner of the first aspect.

[0045] The fifth aspect of the present application provides a computer storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the video segment positioning method of the first aspect or any implementation manner of the first aspect.

[0046] By the technical solutions, the method, device and electronic equipment for locating a video segment are provided, for a first video and a second video needing to be located, first feature sequences and second feature sequences containing video features with time sequence characteristics are obtained, after the feature information of the two feature sequences is interacted, first enhanced sequences and second enhanced sequences are obtained, each enhanced feature in the enhanced sequences contains a time sequence consistency regular term, which is used to represent the difference between the corresponding video frame and other video frames meeting the adjacent condition in time sequence, and thus, after a target similarity matrix is obtained according to the enhanced features, the video segment meeting the similarity condition in the first video and the second video can be located by processing the target similarity matrix by using a target detection model. It can be seen that, in the present application, the time sequence characteristics are added to the feature sequences, and the time sequence consistency regular term representing the difference between the video frames meeting the adjacent condition in time sequence is added to the enhanced features through the feature information interaction, so that the constraint between the video frames meeting the adjacent condition in time sequence is set, the consistency in time sequence of the two videos is maintained, and even if the first video or the second video is played in reverse or at a variable speed, the video segment can be located according to the same, so that the reliability of the video segment location is improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0048] Figure 1 A flowchart of a method for locating a video segment according to an embodiment of the present application is shown in FIG. 1.

[0049] Figure 2 An example diagram of a video segment in a first video and a second video is shown in FIG. 2.

[0050] Figure 3 A partial flowchart of a method for locating a video segment according to an embodiment of the present application is shown in FIG. 3.

[0051] Figure 4 A structural schematic diagram of a device for locating a video segment according to an embodiment of the present application is shown in FIG. 4.

[0052] Figure 5 A structural schematic diagram of an electronic device according to an embodiment of the present application is shown in FIG. 5.

[0053] Figure 6 A flowchart of a method for locating a video segment according to an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0054] The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0055] The embodiments of the present application will be described below in conjunction with the drawings. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or equipment containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0057] Reference Figure 1 An implementation flowchart of a video segment positioning method provided by the embodiments of the present application, which can be but is not limited to applicable to electronic devices with image processing functions, for example, through the application program with image processing functions deployed in the local computer or the cloud service with image processing functions provided by the cloud side server to implement the technical solutions in the embodiments. The technical solutions in the embodiments are mainly used to realize positioning of video segments in a video group and improve positioning reliability.

[0058] Specifically, the method in the embodiments can include the following steps:

[0059] Step 101: Obtain a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video, respectively.

[0060] Among them, each video feature of the first feature sequence and the second feature sequence includes a time sequence feature, and the time sequence feature represents the time sequence of the corresponding video frame in the video.

[0061] In an implementation manner, the first feature sequence in step 101 can be obtained by the following manner:

[0062] First, video feature extraction is performed on each frame of video frame contained in the first video Q to obtain a first initial sequence, the first initial sequence includes a plurality of video features, and the video feature corresponding to the i-th frame of video frame in the first video in the first initial sequence can be represented as wherein the first initial sequence can be represented as wherein q is the number of video frames in the first video;

[0063] Then, a corresponding time sequence feature in the first time coding sequence is added to each video feature in the first initial sequence to obtain a first feature sequence.

[0064] The time sequence features in the first time coding sequence can also be referred to as time coding information, and each time sequence feature in the first time coding sequence is different. Specifically, the time sequence features (time coding information) in the first time coding sequence are determined based on the time sequence positions of the corresponding video frames in the first video. The first time coding sequence can be represented as The first feature sequence can be represented as

[0065] In an implementation, the second feature sequence in step 101 can be obtained by the following method:

[0066] First, video feature extraction is performed on each video frame included in the second video R to obtain a second initial sequence. The second initial sequence includes a plurality of video features. The video feature corresponding to the i-th video frame in the second video in the second initial sequence can be represented as The second initial sequence can be represented as wherein r is the number of video frames in the second video;

[0067] Then, a corresponding time sequence feature in the second time coding sequence is added to each video feature in the second initial sequence to obtain a second feature sequence.

[0068] The time sequence features in the second time coding sequence can also be referred to as time coding information, and each time sequence feature in the second time coding sequence is different. Specifically, the time sequence features (time coding information) in the second time coding sequence are determined based on the time sequence positions of the corresponding video frames in the second video. The second time coding sequence can be represented as The second feature sequence can be represented as

[0069] Further, the first feature sequence and the second feature sequence can also be added with corresponding class label items in the embodiment.

[0070] Specifically, the first class label item is added to the first feature sequence in the embodiment. The first class label item represents the video global feature of the first video. The first class label item can be represented as The corresponding first feature sequence can be represented as

[0071] ​​​In addition, in the embodiment, a second type of mark item is added in the second feature sequence, the second type of mark item represents a video global feature of the second video, and the second type of mark item can be used to represent , and the corresponding second feature sequence can be represented as .

[0072] Step 102: performing feature information interaction on the first feature sequence and the second feature sequence to obtain a first enhanced sequence and a second enhanced sequence.

[0073] wherein each enhanced feature of the first enhanced sequence and the second enhanced sequence respectively comprises a time sequence consistency regular term; the time sequence consistency regular term represents a difference degree between a corresponding video frame and other video frames adjacent in time sequence in the video to which the corresponding video frame belongs.

[0074] It should be noted that the video frames adjacent in time sequence means that the video frames are adjacent or close in time sequence (with an interval of at least one video frame).

[0075] In a specific implementation, the time sequence consistency regular term can be represented as , and x is the corresponding first feature sequence or the second feature sequence. The time sequence consistency regular term is used to set a time sequence constraint on the enhanced features corresponding to the video frames satisfying the adjacent relationship, so as to conform to the continuity between the corresponding video frames. In principle, the closer the video frames in time sequence in a video, the smaller the difference degree between the video frames, and if the difference degree between the more adjacent video frames is greater, then is set to be greater, so that the time sequence constraint set on the enhanced features corresponding to the video frames is stronger.

[0076] For example, the time sequence consistency regular term in the first enhanced sequence can be obtained through formula (1):

[0077] (1)

[0078] wherein is a vector inner product result (a vector inner product structure can be used to measure the similarity between video features) between an i th video feature in the first feature sequence (i.e., a video feature corresponding to an i th video frame in the first video in the first feature sequence) and a j th video feature in the first feature sequence (i.e., a video feature corresponding to a j th video frame in the first video in the first feature sequence); is a sum of absolute values of vector inner product results between the i th video feature in the first feature sequence and all video features in the first feature sequence (used for normalizing ); is a time sequence varying decay term, so that the more adjacent the video frames in time sequence (i.e., The smaller the corresponding enhanced feature has the stronger timing constraint; The value can be 2, for example, according to the business requirements.

[0079] In an implementation manner, when the first feature sequence and the second feature sequence are interacted for feature information in step 102 to obtain the first enhanced sequence, the following manner can be used:

[0080] First, the first feature sequence is input into a self-attention layer to obtain a first initial enhanced sequence. The self-attention layer can be represented as .

[0081] Then, the first initial enhanced sequence and the second feature sequence are input into a cross-attention layer to obtain the first enhanced sequence. The first enhanced sequence can be represented as , m is the number of enhanced features in the first enhanced sequence. The cross-attention layer can be represented as .

[0082] Wherein, the self-attention layer and the cross-attention layer constitute an attention layer, and the attention layer can be represented as . Correspondingly, the first enhanced sequence can be obtained by formula (2):

[0083] (2)

[0084] Wherein, the enhanced features in the first enhanced sequence represent the dependency relationship between the video frames in the first video and the mutual dependency relationship between the first video and the second video.

[0085] Specifically, the self-attention layer includes a multi-head attention part and a timing consistency processing part. The multi-head attention processing part MHA is used to obtain the semantic relationship between the video frames in the input video. The timing consistency processing part is used to obtain the difference degree between the video frames that meet the adjacent condition in the timing in the input video.

[0086] Wherein, the self-attention layer can be represented by formula (3):

[0087] (3)

[0088] The cross-attention layer can be represented by formula (4):

[0089] (4)

[0090] Wherein, taking the first enhanced sequence as an example, x in formula (3) and formula (4) is ; y in formula (4) is Wherein, MHA represents the processing part of the multi-head attention mechanism, the core of which is to divide the input query (Q), key (K), and value (V) into multiple heads after linear transformation, and then calculate the attention in parallel, and finally splice the results. The calculation process can be represented by formula (5):

[0091] (5)

[0092] Wherein, Q, K, and V are video features in the video feature sequence; and the calculation of each head is realized by formula (6):

[0093] (6)

[0094] Wherein, the definition of the attention function Attention is formula (7):

[0095] (7)

[0096] Wherein, , and are the linear transformation matrices of the query Q, key K, and value V of the i-th head, is the linear transformation matrix of the spliced output, is the dimension of the query Q and the key K, used to scale the dot product result to stabilize the gradient. Taking the first enhanced sequence as an example, combining formula (3), formula (5), formula (6), and formula (7) in the , the input of MHA is the video feature in the first feature sequence, i.e. Q=K=V; combining formula (4), formula (5), formula (6), and formula (7) in the , the input of MHA comes from the features of different videos, Q comes from the first feature sequence, and K=V comes from the second feature sequence, in order to capture the dependency within a single first feature sequence and between two feature sequences (the first feature sequence and the second feature sequence).

[0097] It is particularly noted that the ordinary self-attention layer is represented as MHA(x, x, x), which can capture long-range dependencies, but does not explicitly constrain the temporal consistency (such as the order of video frames should not be reversed), resulting in insufficient robustness of the self-attention layer to transformations such as reversal and speed change. Therefore, in this embodiment, a temporal consistency processing part, i.e. a temporal consistency regular term Reg(x), is added to the self-attention layer, so that the attention weight between the video features corresponding to the video frames with closer distance is more strongly constrained by the temporal consistency, in order to meet the temporal continuity between the video frames.

[0098] In an implementation manner, the feature information interaction between the first feature sequence and the second feature sequence in step 102 to obtain the second enhanced sequence can be implemented in the following manner:

[0099] First, the second feature sequence is input into a self-attention layer to obtain a second initial enhanced sequence.

[0100] Then, the second initial enhanced sequence and the first feature sequence are input into a cross-attention layer to obtain the second enhanced sequence. The second enhanced sequence can be represented as , where n is the number of enhanced features in the second enhanced sequence.

[0101] The second enhanced sequence can be obtained by formula (8):

[0102] (8)

[0103] The enhanced features in the second enhanced sequence represent the dependency relationship between the video frames in the second video and the mutual dependency relationship between the first video and the second video.

[0104] Taking the second enhanced sequence as an example, x in formula (3) and formula (4) is ; y in formula (4) is . In combination with represented by formula (3), formula (5), formula (6) and formula (7), the input of MHA is the video features in the second feature sequence, i.e. Q=K=V; in combination with represented by formula (4), formula (5), formula (6) and formula (7), the input of MHA comes from the features of different videos, Q comes from the second feature sequence, and K=V comes from the first feature sequence, so as to capture the dependency relationship within a single second feature sequence and between two feature sequences (the first feature sequence and the second feature sequence).

[0105] Step 103: obtaining a target similarity matrix according to the first enhanced sequence and the second enhanced sequence.

[0106] The matrix elements in the target similarity matrix represent the similarity between the corresponding video frames in the first video and the corresponding video frames in the second video.

[0107] For example, the matrix element in the i-th row and the j-th column in the target similarity matrix represents the similarity between the i-th video frame in the first video and the j-th video frame in the second video.

[0108] Specifically, in this embodiment, the similarity between the enhanced features in the first enhanced sequence and the enhanced sequence in the second enhanced sequence is calculated to obtain the target similarity matrix.

[0109] Step 104: inputting the target similarity matrix into the target detection model; the target detection model outputs a detection result.

[0110] The detection result includes: a probability value of the second video and the first video containing a video segment meeting the similarity condition, a time point position of the video segment in the first video, and a time point position of the video segment in the second video.

[0111] For example, as shown in Figure 2 , the time point position of the video segment in the first video can be represented by , and the time point position of the video segment in the second video can be represented by .

[0112] Specifically, the target detection model can be an artificial intelligence model trained by a large number of training sample video groups, such as a machine learning model or a neural network model.

[0113] Each training sample video group includes a first sample video and a second sample video. Based on this, the first sample video and the second sample video can be processed according to the processing flow in steps 101 to 103 in this embodiment to obtain corresponding first sample feature sequences such as and second sample feature sequences such as , then the first sample feature sequences and the second sample feature sequences are interacted to obtain first sample enhanced sequences such as and second sample enhanced sequences such as , then according to the first sample enhanced sequences and the second sample enhanced sequences, a sample similarity matrix is obtained, the matrix elements in the sample similarity matrix represent the similarity between the corresponding video frames in the first sample video and the corresponding video frames in the second sample video; finally, the sample similarity matrix is input into the target detection model, the target detection model outputs a detection sample result, the detection sample result includes: a probability value of a video sample segment meeting the similarity condition in the second sample video and the first sample video, a time point position of the video sample segment in the first sample video, and a time point position of the video sample segment in the second sample video, a loss function is used to process the detection sample result and the sample label data (video label segments meeting the similarity condition and probability label values 1 or 0 marked in the first sample video and the second sample video) corresponding to the training sample video group to obtain a loss value, and the network parameters in the target detection model are adjusted according to the loss value.

[0114] In a specific implementation, the loss function of the target detection model includes a first loss part and a second loss part, the first loss part is used to optimize the probability value, and the first loss part can be realized by ​The BCE loss is used to determine whether a video sample segment is a video segment copied from the second sample video to the first sample video, and to predict a probability value of each candidate segment in the prediction sample video being a copied video segment Here, i represents the ith training sample video group, and the probability label value is represented by The second loss part is used to optimize the time point position, and the second loss part can be realized by The IOU loss is used to regress the time boundary of the video segment, and the coordinates of the bounding box are predicted The video label segment in the first sample video and the second sample video can be represented by

[0115] Based on this, the loss function Loss can be represented by formula (9) in the embodiment:

[0116] (9)

[0117] Wherein, The value can be set according to business requirements, such as 5. In formula (9), i represents the ith training sample video group, and the Loss obtained by formula (9) represents the loss value generated by training the target detection model using the ith training sample video group.

[0118] Further, the respective feature sequences of the first sample video and the second sample video in the training sample video group of the target detection model are added with class label items, and the class label items represent the video global features of the corresponding sample video.

[0119] Wherein, the loss function of the target detection model at least includes the loss weight corresponding to the training sample video group, and the loss weight can be represented by The loss weight represents the number of key frames in the first sample video and the information entropy corresponding to the first sample video. The key frame is a video frame in which the corresponding enhanced feature and the corresponding class label item in the training sample video group satisfy the similarity condition, and the information entropy is the information entropy of the sequence composed of the enhanced feature corresponding to the first sample video. Based on this, the loss function Loss containing the weight loss can be represented by formula (10):

[0120] (10)

[0121] Wherein, the Loss obtained by formula (10) represents: after generating an initial loss value by training the target detection model using the ith training sample video group, the final loss value obtained by weighting based on the loss weight corresponding to the ith training sample video group.

[0122] ​​For example, the loss weight of the ith training sample video group can be obtained by formula (11) in the embodiment :

[0123] (11)

[0124] wherein, the in formula (11) is the first type of mark item added to the feature sequence corresponding to the first video, is the jth enhanced feature (the jth enhanced feature in the first enhanced sequence) corresponding to the first video, represents the similarity between the jth enhanced feature corresponding to the first video and the corresponding first type of mark item. Because the first type of mark item is the global representation of the entire first video, if the similarity between the enhanced feature corresponding to a video frame and the first type of mark item is high, it means that the video frame is strongly representative of the semantics of the entire first video, i.e. the key frame, and thus a higher loss weight is given by formula (11). The information entropy can be used to quantify the uncertainty of the enhanced feature. The lower the entropy value of the information entropy, the more explicit the enhanced feature. For example, the key frame usually corresponds to a low entropy value. Specifically, the information entropy of the jth enhanced feature corresponding to the first video can be obtained by formula (12) in the embodiment:

[0125] (12)

[0126] wherein, can be represented by formula (13):

[0127] (13)

[0128] wherein, is the jth enhanced feature in the first enhanced sequence, d represents the vector dimension of the enhanced feature, represents the kth element. In addition, can be set to 0.5. Wherein, can be represented by formula (14):

[0129] (14)

[0130] wherein, x in formula (14) is the in formula (11).

[0131] It can be seen that the target detection model realized by the loss weight in the embodiment can adaptively focus on the most discriminative video frame, suppress noise interference, and improve detection accuracy.

[0132] According to the technical solution, the method, device and electronic equipment for positioning a video segment are provided. For the first video and the second video which need to be positioned, the first feature sequence and the second feature sequence containing video features with time sequence characteristics are obtained. After the feature information interaction of the two feature sequences, the first enhanced sequence and the second enhanced sequence can be obtained. Each enhanced feature in the enhanced sequence contains a time sequence consistency regular term, which is used to represent the difference between the corresponding video frame and other video frames which meet the adjacent condition in time sequence. Thus, after the target similarity matrix is obtained according to the enhanced features, the video segment which meets the similarity condition in the first video and the second video can be positioned by processing the target similarity matrix through the target detection model. It can be seen that the time sequence characteristics are added to the feature sequence and the time sequence consistency regular term which represents the difference between the video frames which meet the adjacent condition in time sequence is added to the enhanced feature through the feature information interaction. Thus, the constraint between the video frames which meet the adjacent condition in time sequence can be set, so that the consistency of the two videos in time sequence is maintained. Thus, even if the first video or the second video is played in reverse or at a variable speed, the video segment can be positioned according to the video, thereby improving the reliability of the video segment positioning.

[0133] In an implementation manner, the step 103 can be implemented by the following manner, as shown in the following formula (15): Figure 3

[0134] Step 301: According to the dynamic temperature coefficient, the correlation calculation of the first enhanced sequence and the second enhanced sequence is performed to obtain an initial similarity matrix.

[0135] The dynamic temperature coefficient represents the difference between the first video and the second video.

[0136] Specifically, in the embodiment, the vector inner product results between each enhanced feature in the first enhanced sequence and each enhanced feature in the second enhanced sequence can be obtained first. For example, the vector inner product result between the i th enhanced feature in the first enhanced sequence and the j th enhanced feature in the second enhanced sequence can be represented by These vector inner product results form an m row n column result matrix, m is the number of enhanced features in the first enhanced sequence, and n is the number of enhanced features in the second enhanced sequence. Then, the dynamic temperature coefficient is used to weight the vector inner product results to obtain the initial similarity matrix. The initial similarity matrix The initial similarity matrix can be represented by the following formula (15):

[0137] (15)

[0138] Wherein, is the dynamic temperature coefficient, is an m row n column matrix.​ is fixed, which is difficult to adapt to the similarity distribution of different video pairs (e.g., a high-similarity video pair may need a smaller to enhance the distinguishability, and a low-similarity video pair may need a larger to avoid information compression), based on which, in the embodiment, a dynamic temperature coefficient is set , which can be obtained through the following process:

[0139] First, the vector inner product results between each enhanced feature in the first enhanced sequence and each enhanced feature in the second enhanced sequence are obtained, is the vector inner product result between the i-th enhanced feature in the first enhanced sequence and the j-th enhanced feature in the second enhanced sequence;

[0140] Then, the global average similarity between the first video and the second video is obtained according to the vector inner product results;

[0141] Finally, the dynamic temperature coefficient is obtained according to the global average similarity.

[0142] For example, in the embodiment, the dynamic temperature coefficient is obtained through formula (16):

[0143] (16)

[0144] Wherein, r is the number of enhanced features in the second enhanced sequence; q is the number of enhanced features in the first enhanced sequence. is the base temperature, such as 0.1; is the adjustment coefficient, characterizing the global average similarity between the first video and the second video. The first video and the second video with high global average similarity (e.g., a large number of repeated video frames) will obtain a smaller , and the first video and the second video pair with low global average similarity will obtain a larger , retaining potential weak similarity information, so that the initial similarity matrix is more suitable for different scenes.

[0145] Step 302: performing matrix normalization processing on the matrix elements in the initial similarity matrix to obtain a target similarity matrix.

[0146] Wherein, in the embodiment, the matrix elements in the initial similarity matrix can be normalized through formula (17) softmax operation in row and column dimensions to obtain the target similarity matrix :

[0147] (17)

[0148] wherein, represents taking each row out of and then doing a softmax operation separately, that is, each row gets a value, that is, processing into an n-dimensional vector. Similarly, represents taking each column out of and then doing a softmax operation, that is, each column gets a value, that is, processing into an m-dimensional vector; and then calculating the tensor product of the two vectors, and the tensor product is the target similarity matrix .

[0149] The above introduces a video segment positioning method provided by the embodiments of the present application, and the following will introduce a device for executing the video segment positioning method.

[0150] Please refer to Figure 4 , Figure 4 for a structural schematic diagram of a video segment positioning device provided by the embodiments of the present application. The device can include the following units:

[0151] The feature obtaining unit 401 is configured to obtain a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video respectively; each video feature of the first feature sequence and the second feature sequence is added with a time sequence feature, and the time sequence feature represents the time sequence of the corresponding video frame in the video;

[0152] The feature enhancement unit 402 is configured to perform feature information interaction on the first feature sequence and the second feature sequence to obtain a first enhanced sequence and a second enhanced sequence; each enhanced feature of the first enhanced sequence and the second enhanced sequence contains a time sequence consistency regularization term; and the time sequence consistency regularization term represents the difference degree between the corresponding video frame and other video frames that satisfy the adjacent condition in time sequence in the video;

[0153] The matrix obtaining unit 403 is configured to obtain a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; and the matrix element in the target similarity matrix represents the similarity degree between the corresponding video frame in the first video and the corresponding video frame in the second video;

[0154] The matrix processing unit 404 is configured to input the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result includes a probability value of a video segment meeting a similarity condition in the second video and the first video, a time point of the video segment in the first video, and a time point of the video segment in the second video.

[0155] According to the technical solution, for the first video and the second video that need to be positioned, the first feature sequence and the second feature sequence containing video features with time sequence characteristics are obtained. After the feature information of the two feature sequences is interacted, the first enhanced sequence and the second enhanced sequence are obtained, each enhanced feature of which contains a time sequence consistency regular term, which is used to represent the difference between the corresponding video frame and other video frames meeting the adjacent condition in time sequence in the video. Thus, after the target similarity matrix is obtained according to the enhanced features, the target similarity matrix is processed by the target detection model, and the video segment meeting the similarity condition in the first video and the second video can be positioned. It can be seen that, in the present application, the time sequence characteristics are added to the feature sequence, and the time sequence consistency regular term representing the difference between the video frames meeting the adjacent condition in time sequence is added to the enhanced feature through the feature information interaction. In this way, the constraint between the video frames meeting the adjacent condition in time sequence is set, so that the consistency of the two videos in time sequence is maintained. Thus, even if the first video or the second video is played in reverse or at a variable speed, the video segment can be positioned according to the same, thereby improving the reliability of the video segment positioning.

[0156] In an implementation manner, the feature obtaining unit 401 obtains the first feature sequence corresponding to the first video by the following manner:

[0157] video feature extraction is performed on each frame of video frame contained in the first video to obtain a first initial sequence; a corresponding time sequence feature in a first time coding sequence is added to each video feature in the first initial sequence to obtain the first feature sequence; each time sequence feature in the first time coding sequence is different;

[0158] In an implementation manner, the feature obtaining unit 401 obtains the second feature sequence corresponding to the second video by the following manner, which includes:

[0159] video feature extraction is performed on each frame of video frame contained in the second video to obtain a second initial sequence; a corresponding time sequence feature in a second time coding sequence is added to each video feature in the second initial sequence to obtain the second feature sequence.

[0160] In an implementation manner, the feature enhancement unit 402 performs feature information interaction between the first feature sequence and the second feature sequence in the following manner to obtain a first enhanced sequence:

[0161] inputting the first feature sequence into a self-attention layer to obtain a first initial enhanced sequence; and inputting the first initial enhanced sequence and the second feature sequence into a cross-attention layer to obtain the first enhanced sequence.

[0162] In an implementation manner, the feature enhancement unit 402 performs feature information interaction between the first feature sequence and the second feature sequence in the following manner to obtain a second enhanced sequence:

[0163] inputting the second feature sequence into the self-attention layer to obtain a second initial enhanced sequence; and inputting the second initial enhanced sequence and the first feature sequence into the cross-attention layer to obtain the second enhanced sequence.

[0164] The self-attention layer includes a multi-head attention part and a temporal consistency processing part; the multi-head attention part is configured to obtain semantic relationships between video frames in an input video; and the temporal consistency processing part is configured to obtain a difference degree between video frames that satisfy a proximity condition in a time sequence in the input video.

[0165] In an implementation manner, the matrix obtaining unit 403 is specifically configured to: perform correlation calculation on the first enhanced sequence and the second enhanced sequence according to a dynamic temperature coefficient to obtain an initial similarity matrix; wherein the dynamic temperature coefficient represents a difference degree between the first video and the second video; and perform matrix normalization processing on matrix elements in the initial similarity matrix to obtain a target similarity matrix.

[0166] The matrix elements in the initial similarity matrix are determined based on vector inner product results between each enhanced feature in the first enhanced sequence and each enhanced feature in the second enhanced sequence.

[0167] In an implementation manner, the matrix obtaining unit 403 obtains the dynamic temperature coefficient in the following manner:

[0168] obtains vector inner product results between each enhanced feature in the first enhanced sequence and each enhanced feature in the second enhanced sequence; obtains a global average similarity between the first video and the second video according to the vector inner product results; and obtains the dynamic temperature coefficient according to the global average similarity.

[0169] In an implementation manner, the loss function of the target detection model comprises a first loss part and a second loss part; wherein the first loss part is used for optimizing the probability value; and the second loss part is used for optimizing the time point bit.

[0170] In an implementation manner, the feature sequence corresponding to each of the first sample video and the second sample video in the training sample video group of the target detection model is added with a class label item; the class label item represents the video global feature of the sample video; wherein the loss function of the target detection model at least comprises a loss weight corresponding to the training sample video group, the loss weight representing the number of key frames in the first sample video and the information entropy corresponding to the first sample video; the key frame is a video frame in which the corresponding enhanced feature and the corresponding class label item in the training sample video group satisfy a similarity condition; and the information entropy is the information entropy of the sequence composed of the enhanced feature corresponding to the first sample video.

[0171] It should be noted that the specific implementation manners of the units in the present embodiment can refer to the corresponding contents in the foregoing, which will not be described in detail herein.

[0172] Reference Figure 5 A structural schematic diagram of an electronic device provided by the present embodiment can comprise at least one processor 501 and a memory 502 connected with the processor 501, wherein:

[0173] The memory 502 is configured to store a computer program;

[0174] The processor 501 is configured to execute the computer program, so that the electronic device can realize:

[0175] obtain a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video respectively; each video feature of the first feature sequence and the second feature sequence is added with a time sequence feature, and the time sequence feature represents the time sequence of the corresponding video frame in the video;

[0176] perform feature information interaction on the first feature sequence and the second feature sequence to obtain a first enhanced sequence and a second enhanced sequence; each enhanced feature of the first enhanced sequence and the second enhanced sequence comprises a time sequence consistency regularization term; and the time sequence consistency regularization term represents the difference degree between the corresponding video frame and other video frames which satisfy a proximity condition in the time sequence in the video;

[0177] obtain a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; a matrix element in the target similarity matrix represents the similarity degree between the corresponding video frame in the first video and the corresponding video frame in the second video.

[0178] inputting the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result includes: a probability value of a video segment in the first video and the second video that meets a similarity condition, a time point of the video segment in the first video, and a time point of the video segment in the second video.

[0179] By the above technical solution, in the electronic device provided by the embodiment of the present application, for the first video and the second video that need to be positioned, the first feature sequence and the second feature sequence containing video features with time sequence characteristics are obtained, and after the feature information interaction of the two feature sequences, the first enhanced sequence and the second enhanced sequence can be obtained, each enhanced feature in which contains a time sequence consistency regularization term, which is used to represent the difference between the corresponding video frame and other video frames that meet the adjacent condition in time sequence in the video. Thus, after obtaining the target similarity matrix according to the enhanced features, the target similarity matrix can be processed by the target detection model to locate the video segment that meets the similarity condition in the first video and the second video. As can be seen, in the present application, the time sequence characteristics are added to the feature sequence, and the time sequence consistency regularization term representing the difference between the video frames adjacent in time sequence is added to the enhanced feature through the feature information interaction. In this way, constraints can be set between the video frames that meet the adjacent condition in time sequence, so that the two videos remain consistent in time sequence. Therefore, even if the first video or the second video is played in reverse or at a variable speed, the video segment can be positioned accordingly, thereby improving the reliability of the video segment positioning.

[0180] In the embodiment of the present application, a computer program product is also provided, which includes computer readable instructions that, when executed on an electronic device, cause the electronic device to implement any of the video segment positioning methods provided by the embodiments of the present application.

[0181] In the embodiment of the present application, a computer readable storage medium is also provided, which carries one or more computer programs that, when executed by an electronic device, can cause the electronic device to implement any of the video segment positioning methods provided by the embodiments of the present application.

[0182] In the positioning scenario of the copied video segment in the query video and the reference video, the present application aims to construct a video segment positioning method based on the self-attention mechanism of time sequence consistency, the dynamic temperature coefficient and the loss function of attention mask, and solve the following problems:

[0183] 1. The manual comparison method requires a large amount of manpower, is costly and inefficient;

[0184] 2. Existing methods are not sensitive to segments with speed changes, reversals, etc.

[0185] 3. Existing methods are not sensitive to the different parts in high-similarity segments and the same parts in low-similarity segments;

[0186] 4. Existing methods do not distinguish between noisy frames in keyframes and are not accurate enough in locating the start and end times of copied segments.

[0187] To address the aforementioned issues, this application proposes a method for locating copied video segments by integrating three key technologies: temporal consistency self-attention, dynamic temperature coefficient, and attention mask loss function. The method comprises four steps: feature extraction, feature enhancement, similarity matrix calculation, and temporal point localization. The steps include: Figure 6 As shown:

[0188] S1, Reference Video (e.g.) ) and query videos (such as The feature sequence is generated. This includes added class markers and time-encoded sine and cosine values. The reference video refers to the original video, while the query video may be a modified or derivative work.

[0189] S2. Feature enhancement is achieved by exchanging feature information between feature sequences through self-attention layers and cross-attention layers, resulting in two enhanced sequences, such as... and .

[0190] S3. Calculate the target similarity matrix.

[0191] S4. Using an object detection model, perform time point localization from the object similarity matrix. This involves locating whether there are similar segments in the query video and the reference video, and identifying the time points of these similar segments within each video. In other words, determine which two time periods between the query video and the reference video contain the same content.

[0192] In summary, this application proposes a method for locating video copy segments by integrating three key technologies: temporal consistency self-attention, dynamic temperature coefficient, and attention mask loss function. This method involves four steps: feature extraction, feature enhancement, similarity matrix calculation, and time point localization. It can quickly, efficiently, and accurately retrieve and review massive amounts of video segments, greatly reducing the burden of video retrieval and copyright review, saving costs, and improving efficiency.

[0193] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiments provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0194] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and the necessary general hardware, and of course can also be realized by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily realized by corresponding hardware, and the specific hardware structure for realizing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products, which are stored in readable storage media, such as computer floppy disks, U disks, mobile hard disks, ROM, RAM, magnetic or optical disks, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0195] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part.

[0196] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A method of locating a video clip, characterized by, The method comprises the following steps: obtaining a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video respectively; each video feature in the first feature sequence and the second feature sequence comprises a time sequence feature, and the time sequence feature represents the time sequence of the corresponding video frame in the video; inputting the first feature sequence into a self-attention layer to obtain a first initial enhanced sequence, and inputting the first initial enhanced sequence and the second feature sequence into a cross-attention layer to obtain a first enhanced sequence; inputting the second feature sequence into the self-attention layer to obtain a second initial enhanced sequence, and inputting the second initial enhanced sequence and the first feature sequence into the cross-attention layer to obtain a second enhanced sequence; the self-attention layer comprises a multi-head attention processing part and a time sequence consistency processing part; the multi-head attention processing part is used to obtain the semantic relationship between video frames in the input video; the time sequence consistency processing part is used to obtain the difference degree between video frames that satisfy the adjacent condition in time sequence in the input video; each enhanced feature in the first enhanced sequence and the second enhanced sequence comprises a time sequence consistency regularization term; the time sequence consistency regularization term represents the difference degree between the corresponding video frame and other video frames that satisfy the adjacent condition in time sequence in the video; obtaining a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; the matrix element in the target similarity matrix represents the similarity degree between the corresponding video frame in the first video and the corresponding video frame in the second video; inputting the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result comprises: the probability value of the video segment that satisfies the similarity condition in the second video and the first video, the time point of the video segment in the first video, and the time point of the video segment in the second video.

2. The method of claim 1, wherein, obtaining a first feature sequence corresponding to a first video comprises: performing video feature extraction on each video frame included in the first video to obtain a first initial sequence; adding a corresponding time sequence feature in a first time coding sequence to each video feature in the first initial sequence to obtain a first feature sequence; each time sequence feature in the first time coding sequence is different; wherein obtaining a second feature sequence corresponding to a second video comprises: performing video feature extraction on each video frame included in the second video to obtain a second initial sequence; adding a corresponding time sequence feature in a second time coding sequence to each video feature in the second initial sequence to obtain a second feature sequence.

3. The method of claim 1, wherein, obtaining a target similarity matrix according to the first enhanced sequence and the second enhanced sequence comprises: performing correlation calculation on the first enhanced sequence and the second enhanced sequence according to a dynamic temperature coefficient to obtain an initial similarity matrix; wherein the dynamic temperature coefficient represents the difference degree between the first video and the second video; The matrix elements in the initial similarity matrix are subjected to matrix normalization to obtain a target similarity matrix.

4. The method of claim 3, wherein, The matrix elements in the initial similarity matrix are determined based on vector inner product results between each enhanced feature in the first enhanced sequence and each enhanced feature in the second enhanced sequence.

5. The method of claim 3, wherein, The dynamic temperature coefficient is obtained by: obtaining vector inner product results between each enhanced feature in the first enhanced sequence and each enhanced feature in the second enhanced sequence; obtaining a global average similarity between the first video and the second video according to the vector inner product results; obtaining the dynamic temperature coefficient according to the global average similarity.

6. The method of claim 1, wherein, The loss function of the target detection model comprises a first loss part and a second loss part; The first loss part is used to optimize the probability value, and the second loss part is used to optimize the time point bit.

7. The method of claim 1, wherein, The feature sequences corresponding to the first sample video and the second sample video in the training sample video group of the target detection model are added with class label items; the class label items represent the video global features of the sample video; The loss function of the target detection model comprises at least a loss weight corresponding to the training sample video group, wherein the loss weight represents the number of key frames in the first sample video and the information entropy corresponding to the first sample video; the key frame is a video frame in which the corresponding enhanced feature and the corresponding class label item in the training sample video group satisfy a similarity condition; and the information entropy is the information entropy of the sequence composed of the enhanced features corresponding to the first sample video.

8. An apparatus for locating a video clip, characterized in that The method comprises: a feature obtaining unit configured to obtain a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video; each video feature in the first feature sequence and the second feature sequence is added with a time sequence feature, and the time sequence feature represents the time sequence of the corresponding video frame in the video; a feature enhancement unit configured to input the first feature sequence into a self-attention layer to obtain a first initial enhanced sequence, and input the first initial enhanced sequence and the second feature sequence into a cross-attention layer to obtain a first enhanced sequence; input the second feature sequence into the self-attention layer to obtain a second initial enhanced sequence, and input the second initial enhanced sequence and the first feature sequence into the cross-attention layer to obtain a second enhanced sequence; wherein the self-attention layer comprises a multi-head attention processing part and a time sequence consistency processing part; the multi-head attention processing part is configured to obtain the semantic relationship between video frames in the input video; the time sequence consistency processing part is configured to obtain the difference degree between video frames that satisfy a proximity condition in the time sequence in the input video; each enhanced feature in the first enhanced sequence and the second enhanced sequence comprises a time sequence consistency regularization term; the time sequence consistency regularization term represents the difference degree between the corresponding video frame and other video frames that satisfy the proximity condition in the time sequence in the video; obtain a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; a matrix element in the target similarity matrix represents a similarity degree between a corresponding video frame in the first video and a corresponding video frame in the second video; input the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result includes a probability value of a video segment in the second video and the first video that meets a similarity condition, a time point of the video segment in the first video, and a time point of the video segment in the second video.

9. An electronic device, comprising: comprise at least one processor and a memory connected with the processor, wherein: the memory is configured to store a computer program; the processor is configured to execute the computer program, so that the electronic device can implement: obtain a first feature sequence corresponding to a first video and a second feature sequence corresponding to a second video respectively; each video feature of the first feature sequence and the second feature sequence is added with a time sequence feature, and the time sequence feature represents a time sequence of a corresponding video frame in a video to which the video frame belongs; input the first feature sequence into a self-attention layer to obtain a first initial enhanced sequence, input the first initial enhanced sequence and the second feature sequence into a cross-attention layer to obtain a first enhanced sequence, input the second feature sequence into the self-attention layer to obtain a second initial enhanced sequence, and input the second initial enhanced sequence and the first feature sequence into the cross-attention layer to obtain a second enhanced sequence; the self-attention layer comprises a multi-head attention processing part and a time sequence consistency processing part; the multi-head attention processing part is configured to obtain a semantic relationship between video frames in an input video; the time sequence consistency processing part is configured to obtain a difference degree between video frames that meet a proximity condition in a time sequence in the input video; each enhanced feature of the first enhanced sequence and the second enhanced sequence comprises a time sequence consistency regularization term; the time sequence consistency regularization term represents a difference degree between a corresponding video frame and other video frames that meet a proximity condition in a time sequence in a video to which the corresponding video frame belongs; obtain a target similarity matrix according to the first enhanced sequence and the second enhanced sequence; a matrix element in the target similarity matrix represents a similarity degree between a corresponding video frame in the first video and a corresponding video frame in the second video; input the target similarity matrix into a target detection model; the target detection model outputs a detection result; the detection result includes a probability value of a video segment in the second video and the first video that meets a similarity condition, a time point of the video segment in the first video, and a time point of the video segment in the second video.

Images