A video detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a video detection method and device, electronic equipment and storage medium, relating to the technical field of video processing. A video frame is obtained from a to-be-detected video as a to-be-detected video frame; an image feature of the to-be-detected video frame is extracted to obtain a first feature map; the first feature map is processed based on a mask corresponding to the first feature map to obtain a second feature map; for each to-be-matched video, a feature map of a to-be-matched video frame in the to-be-matched video is obtained as a third feature map; for each to-be-detected video frame, the similarity between the to-be-detected video frame and each to-be-matched video frame is calculated based on the second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame; based on the determined similarities, a target video matched with the to-be-detected video is determined from the preset to-be-matched videos. The accuracy of the detection result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to a video detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of the Internet video industry, a large number of long videos, such as TV series, movies, etc., are created by video creators. Some video creators will make secondary creation and publish on the long videos. For example, the video creators copy, edit, etc. the long videos to obtain short videos, and publish on public platforms.

[0003] In the related art, the image features of the video frames in the short video and the image features of the video frames in the long video can be extracted respectively, and then the extracted image features are matched to determine whether there is a long video matched with the short video, and then it can be determined whether the short video constitutes infringement.

[0004] However, in the process of creating short videos, the video creators may crop the video frames of the long videos, resulting in that the image content of the video frames in the short video is inconsistent with that of the long video, and then the matching based on the extracted image features of the video frames will result in low accuracy of the detection result. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a video detection method and device, electronic equipment and storage medium to improve the accuracy of the detection result. The specific technical solutions are as follows:

[0006] In a first aspect of the embodiments of the present application, a video detection method is provided. The method comprises: obtaining a video frame from a to-be-detected video as a to-be-detected video frame; extracting image features of the to-be-detected video frame to obtain a first feature map; processing the first feature map based on a mask corresponding to the first feature map to obtain a second feature map; wherein the mask corresponding to the first feature map comprises weights of elements in the first feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the first feature map are greater than the weights of elements located in an edge region; for each to-be-matched video in a preset, obtaining a feature map of a to-be-matched video frame in the to-be-matched video as a third feature map; wherein the third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map; the fourth feature map is obtained by extracting features of the to-be-matched video frame; the mask corresponding to the fourth feature map comprises weights of elements in the fourth feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the fourth feature map are greater than the weights of elements located in an edge region; for each to-be-detected video frame, calculating similarities between the to-be-detected video frame and each to-be-matched video frame based on the second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame; and determining a target video matched with the to-be-detected video from the preset to-be-matched videos based on the determined similarities.

[0007] In some embodiments, the extracting image features of the to-be-detected video frame to obtain a first feature map comprises: for each to-be-detected video frame, inputting the to-be-detected video frame into a convolutional neural network (CNN); and obtaining feature maps output by each of a plurality of specified network layers of the CNN to obtain a first feature map output by each specified network layer. The processing the first feature map based on a mask corresponding to the first feature map to obtain a second feature map comprises: for each specified network layer, processing the first feature map output by the specified network layer based on a mask corresponding to the specified network layer to obtain a fifth feature map corresponding to the specified network layer; and combining the fifth feature maps corresponding to the specified network layers to obtain the second feature map.

[0008] In some embodiments, the processing, for each specified network layer, the first feature map output by the specified network layer based on the mask corresponding to the specified network layer, to obtain the fifth feature map corresponding to the specified network layer, comprises: processing, for each first feature map output by each specified network layer, the first feature map based on the mask corresponding to the specified network layer, to obtain a sixth feature map; and processing, for each specified network layer, the sixth feature maps corresponding to the specified network layer based on an R-MAC (Region-Maximizing Convolution Activation) algorithm, to obtain an aggregated feature vector corresponding to the specified network layer as the fifth feature map.

[0009] In some embodiments, before the processing, for each first feature map output by each specified network layer, the first feature map based on the mask corresponding to the specified network layer, to obtain a sixth feature map, the method further comprises: determining a target network layer from the plurality of specified network layers; generating the mask corresponding to the target network layer based on a preset expectation, a preset variance, and a dimension of the first feature map output by the target network layer; and generating the masks corresponding to the other specified network layers except the target network layer based on interpolation operation on the mask corresponding to the target network layer.

[0010] In some embodiments, the combining the fifth feature maps corresponding to the respective specified network layers to obtain the second feature map comprises: performing binarization processing on the fifth feature maps corresponding to the respective specified network layers respectively to obtain binarization processing results; and splicing the binarization processing results to obtain the second feature map.

[0011] In some embodiments, the binarization processing on the fifth feature maps corresponding to the respective specified network layers respectively to obtain binarization processing results comprises: inputting the fifth feature maps corresponding to the respective specified network layers as input data into respective fully connected layers included in an input layer of a pre-trained JMLH (Just-Maximizing-Likelihood Hashing) model, to obtain output results of the fully connected layers as the binarization processing results.

[0012] In some embodiments, the determining, based on the determined similarities, a target video matching the to-be-detected video from the preset to-be-matched videos comprises: for each to-be-detected video frame, determining, based on the similarity corresponding to the to-be-detected video frame, a video frame matching the to-be-detected video frame from to-be-matched video frames included in the preset to-be-matched videos as a target video frame; and determining, respectively, a to-be-matched video to which each target video frame belongs as a target video.

[0013] In some embodiments, after the respective target video frame is determined to belong to a target video to be matched, as a target video, the method further comprises: for each target video, generating a similarity matrix corresponding to the target video based on the similarity between the video frame to be detected and the target video frame in the target video; wherein each row in the similarity matrix corresponds to the video frame to be detected one by one, and the order between the rows is determined according to the order of the corresponding video frame to be detected in the video to be detected; each column in the similarity matrix corresponds to the target video frame in the target video one by one, and the order between the columns is determined according to the order of the corresponding target video frame in the target video; the similarity element in the similarity matrix represents the similarity between the video frame to be detected corresponding to the row of the similarity element and the target video frame corresponding to the column of the similarity element; if the similarity between the video frame to be detected corresponding to the row of the similarity element and the target video frame corresponding to the column of the similarity element is less than a first threshold, the similarity element is invalid; in the order of row number from low to high, for the similarity element of each row in the similarity matrix, in the order of column number from high to low corresponding to the similarity element, each element position of the similarity matrix that is not invalid is added to a preset list in turn to obtain a position index list corresponding to the target video; the current processing element position is determined according to the order of the element position in the position index list; the current first element position is determined from each element position located before the current processing element position; wherein the column number corresponding to the current processing element position is greater than the column number corresponding to the current first element position, the time length between the timestamp of the video frame to be detected corresponding to the current processing element position and the timestamp of the video frame to be detected corresponding to the current first element position is less than a first time length, and the time length between the timestamp of the video frame to be matched corresponding to the current processing element position and the timestamp of the video frame to be matched corresponding to the current first element position is less than a second time length; the length value corresponding to the current processing element position is determined as the first length value of the current processing element position; wherein the first length value of the current processing element position is determined based on the second length value corresponding to the current first element position; the second length value corresponding to the current first element position represents the sum of the first length value of the current first element position and 1; the starting sequence number of the current processing element position is determined based on the starting sequence number of the current first element position; wherein the starting sequence number of the current first element position is determined based on the starting sequence number of each element position located before the current processing element position when the current first element position is the current processing element position; the starting sequence number of the first element position in the position index list is 0.determining the element position corresponding to the starting sequence number of the terminating element position in the position index list as a starting element position; determining the to-be-detected video frame corresponding to the starting element position as a first starting video frame, and determining the target video frame corresponding to the starting element position as a second starting video frame; and determining the first video segment in the to-be-detected video with the first starting video frame and the first terminating video frame as boundaries to be consistent with the second video segment in the target video with the second starting video frame and the second terminating video frame as boundaries.

[0014] In some embodiments, the determining of the length value corresponding to the current to-be-processed element position as a first length value of the current to-be-processed element position comprises: if the current first element position is multiple, determining the maximum value in the second length values corresponding to the current first element positions as the first length value of the current to-be-processed element position; or if the current first element position is one, determining the second length value corresponding to the current first element position as the first length value of the current to-be-processed element position.

[0015] In some embodiments, after the target video matching the to-be-detected video is determined from the preset to-be-matched videos based on the determined similarities, the method further comprises: if the target video is one, and the to-be-detected video is consistent with one video segment in the target video, determining that the matching type of the to-be-detected video and the target video is segment matching; if the target video is one, a plurality of first video segments in the to-be-detected video are respectively consistent with a plurality of second video segments in the target video, and the order of the plurality of first video segments in the to-be-detected video is consistent with the order of the plurality of second video segments in the target video, determining that the matching type of the to-be-detected video and the target video is single-video sequential matching; if the target video is one, a plurality of first video segments in the to-be-detected video are respectively consistent with a plurality of second video segments in the target video, and the order of the plurality of first video segments in the to-be-detected video is inconsistent with the order of the plurality of second video segments in the target video, determining that the matching type of the to-be-detected video and the target video is single-video non-sequential matching; if the target video is a plurality, and the plurality of target videos belong to the same video set, determining that the matching type of the to-be-detected video and the target video is multi-video non-sequential matching; if the target video is a plurality, and the plurality of target videos belong to different video sets, determining that the matching type of the to-be-detected video and the target video is multi-set non-sequential matching.

[0016] In some embodiments, the to-be-matched video frame in the to-be-matched video comprises a video frame obtained by sampling the to-be-matched video based on a first sampling frequency; the to-be-detected video frame comprises a video frame obtained by sampling the to-be-detected video based on a second sampling frequency; or, the to-be-detected video frame comprises a sampling video frame obtained by sampling the to-be-detected video based on a second sampling frequency, and a target key frame in the to-be-detected video frame; wherein the time sequence distance between the target key frame and the nearest sampling video frame is greater than a second threshold.

[0017] In a second aspect of the embodiments of the present application, a video detection device is further provided, which comprises: a to-be-detected video frame acquisition module, configured to acquire video frames from a to-be-detected video as to-be-detected video frames; a first feature map acquisition module, configured to extract image features of the to-be-detected video frames to obtain a first feature map; a second feature map acquisition module, configured to process the first feature map based on a mask corresponding to the first feature map to obtain a second feature map; wherein the mask corresponding to the first feature map comprises weights of elements in the first feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the first feature map are greater than the weights of elements located in an edge region; a third feature map acquisition module, configured to acquire, for each preset to-be-matched video, a feature map of a to-be-matched video frame in the to-be-matched video as a third feature map; wherein the third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map; the fourth feature map is obtained by performing feature extraction on the to-be-matched video frame; the mask corresponding to the fourth feature map comprises weights of elements in the fourth feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the fourth feature map are greater than the weights of elements located in an edge region; a similarity calculation module, configured to, for each to-be-detected video frame, calculate similarities between the to-be-detected video frame and each to-be-matched video frame based on the second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame; and a target video determination module, configured to determine, based on the determined similarities, a target video matched with the to-be-detected video from preset to-be-matched videos.

[0018] In some embodiments, the first feature map acquisition module comprises: a to-be-detected video frame input submodule, configured to, for each to-be-detected video frame, input the to-be-detected video frame into a convolutional neural network; and a first feature map output submodule, configured to acquire feature maps output by a plurality of specified network layers of the convolutional neural network respectively to obtain the first feature map output by each specified network layer; the second feature map acquisition module comprises: a fifth feature map acquisition submodule, configured to, for each specified network layer, process the first feature map output by the specified network layer based on a mask corresponding to the specified network layer to obtain a fifth feature map corresponding to the specified network layer; and a second feature map acquisition submodule, configured to obtain the second feature map by combining the fifth feature maps corresponding to the specified network layers.

[0019] In some embodiments, the fifth feature map obtaining submodule comprises: a sixth feature map obtaining unit configured to, for each first feature map output by each specified network layer, process the first feature map based on the mask corresponding to the specified network layer to obtain a sixth feature map; and a fifth feature map obtaining unit configured to, for each specified network layer, process each sixth feature map corresponding to the specified network layer based on a region maximum convolutional activation (R-MAC) algorithm to obtain an aggregated feature vector corresponding to the specified network layer as a fifth feature map.

[0020] In some embodiments, the device further comprises: a target network layer determining module configured to determine a target network layer from the plurality of specified network layers before processing each first feature map output by each specified network layer based on the mask corresponding to the specified network layer to obtain a sixth feature map; a mask generating module configured to generate the mask corresponding to the target network layer based on a preset expectation, a preset variance, and the dimension of the first feature map output by the target network layer; and a mask calculating module configured to obtain the masks corresponding to the other specified network layers except the target network layer based on interpolation operation on the mask corresponding to the target network layer.

[0021] In some embodiments, the second feature map obtaining submodule comprises: a binarization processing unit configured to perform binarization processing on each fifth feature map corresponding to each specified network layer respectively to obtain binarization processing results; and a splicing unit configured to splice the binarization processing results to obtain a second feature map.

[0022] In some embodiments, the binarization processing unit is specifically configured to: input each fifth feature map corresponding to each specified network layer into each fully connected layer included in an input layer of a pre-trained unique maximum likelihood hashing (JMLH) model as input data to obtain output results of the fully connected layers as the binarization processing results.

[0023] In some embodiments, the target video determining module comprises: a target video frame determining submodule configured to, for each to-be-detected video frame, determine a video frame matching the to-be-detected video frame from to-be-matched video frames included in each to-be-matched video based on the similarity corresponding to the to-be-detected video frame as a target video frame; and a target video determining submodule configured to determine a to-be-matched video to which each target video frame belongs as a target video respectively.

[0024] In some embodiments, the apparatus further comprises: a similarity matrix generating module configured to, after the determining of the target video to which each target video frame belongs, generate, for each target video, a similarity matrix corresponding to the target video based on similarities between the to-be-detected video frames and the target video frames in the target video; wherein each row in the similarity matrix corresponds to one of the to-be-detected video frames, and the order between the rows is determined according to the order of the corresponding to-be-detected video frames in the to-be-detected video; each column in the similarity matrix corresponds to one of the target video frames in the target video, and the order between the columns is determined according to the order of the corresponding target video frames in the target video; a similarity element in the similarity matrix represents a similarity between a to-be-detected video frame corresponding to a row to which the similarity element belongs and a target video frame corresponding to a column to which the similarity element belongs; if the similarity between the to-be-detected video frame corresponding to the row to which the similarity element belongs and the target video frame corresponding to the column to which the similarity element belongs is less than a first threshold, the similarity element is invalid; a position index list obtaining module configured to, in order of row number from low to high, for each similarity element in each row of the similarity matrix, in order of column number from high to low corresponding to the similarity element, sequentially add an element position of each non-invalid similarity element in the similarity matrix to a preset list to obtain a position index list corresponding to the target video; a to-be-processed element position determining module configured to determine a current to-be-processed element position according to the order of the element positions in the position index list; a first element position determining module configured to determine a current first element position from each element position located before the current to-be-processed element position; wherein the column number corresponding to the current to-be-processed element position is greater than the column number corresponding to the current first element position, the time length between the timestamp of the to-be-detected video frame corresponding to the current to-be-processed element position and the timestamp of the to-be-detected video frame corresponding to the current first element position is less than a first time length, and the time length between the timestamp of the to-be-detected video frame corresponding to the current to-be-processed element position and the timestamp of the to-be-detected video frame corresponding to the current first element position is less than a second time length; a first length value determining module configured to determine a length value corresponding to the current to-be-processed element position as a first length value of the current to-be-processed element position; wherein the first length value of the current to-be-processed element position is determined based on a second length value corresponding to the current first element position; the second length value corresponding to the current first element position represents a sum of the first length value of the current first element position and 1; and a start sequence number determining module configured to determine a start sequence number of the current to-be-processed element position based on a start sequence number of the current first element position; wherein the start sequence number of the current first element position is determined based on the start sequence number of each element position located before the current to-be-processed element position when the current first element position is the current to-be-processed element position.The starting sequence number of the first element position in the position index list is 0; a traversal module is configured to determine the next element position in the position index list as the current to-be-processed element position, and trigger the to-be-processed element position determining module until the first length value corresponding to the last element position in the position index list is obtained; a terminal element position determining module is configured to determine the element position corresponding to the largest numerical first length value as the terminal element position; a starting element position determining module is configured to determine the element position corresponding to the starting sequence number of the terminal element position in the position index list as the starting element position; a terminal video frame determining module is configured to determine the to-be-detected video frame corresponding to the terminal element position as the first terminal video frame, and determine the target video frame corresponding to the terminal element position as the second terminal video frame; a starting video frame determining module is configured to determine the to-be-detected video frame corresponding to the starting element position as the first starting video frame, and determine the target video frame corresponding to the starting element position as the second starting video frame; a video segment determining module is configured to determine the first video segment in the to-be-detected video with the first starting video frame and the first terminal video frame as boundaries, which is consistent with the second video segment in the target video with the second starting video frame and the second terminal video frame as boundaries.

[0025] In some embodiments, the first length value determining module comprises: a first determining submodule configured to, if the current first element position is multiple, determine the maximum value of the second length values corresponding to each current first element position as the first length value of the current to-be-processed element position; and a second determining submodule configured to, if the current first element position is one, determine the second length value corresponding to the current first element position as the first length value of the current to-be-processed element position.

[0026] In some embodiments, the apparatus further comprises: a first matching type determining module, configured to, after determining the target video matched with the to-be-detected video from the preset to-be-matched videos based on the determined similarities, if the target video is one and the to-be-detected video is consistent with one video segment in the target video, determining that the matching type of the to-be-detected video and the target video is segment matching; a second matching type determining module, configured to, if the target video is one, a plurality of first video segments in the to-be-detected video are respectively consistent with a plurality of second video segments in the target video, and the order of the plurality of first video segments in the to-be-detected video is consistent with the order of the plurality of second video segments in the target video, determining that the matching type of the to-be-detected video and the target video is single-video sequential matching; a third matching type determining module, configured to, if the target video is one, a plurality of first video segments in the to-be-detected video are respectively consistent with a plurality of second video segments in the target video, and the order of the plurality of first video segments in the to-be-detected video is inconsistent with the order of the plurality of second video segments in the target video, determining that the matching type of the to-be-detected video and the target video is single-video non-sequential matching; a fourth matching type determining module, configured to, if the target video is a plurality of videos and the plurality of target videos belong to the same video set, determining that the matching type of the to-be-detected video and the target video is multi-video non-sequential matching; and a fifth matching type determining module, configured to, if the target video is a plurality of videos and the plurality of target videos belong to different video sets, determining that the matching type of the to-be-detected video and the target video is multi-set non-sequential matching.

[0027] In some embodiments, the to-be-matched video frame in the to-be-matched video comprises a video frame obtained by sampling the to-be-matched video based on a first sampling frequency; the to-be-detected video frame comprises a video frame obtained by sampling the to-be-detected video based on a second sampling frequency; or the to-be-detected video frame comprises a sampling video frame obtained by sampling the to-be-detected video based on a second sampling frequency and a target key frame in the to-be-detected video frame; wherein the time sequence distance between the target key frame and the nearest sampling video frame is greater than a second threshold.

[0028] In a third aspect of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory is used to store a computer program; and the processor is used to execute the program stored in the memory to implement any of the above-mentioned video detection methods.

[0029] In yet another aspect of the embodiments of the present application, a computer readable storage medium is provided, in which a computer program is stored, and the computer program, when executed by a processor, implements any of the video detection methods described above.

[0030] In yet another aspect of the embodiments of the present application, a computer program product containing instructions, which, when executed on a computer, cause the computer to perform any of the video detection methods described above.

[0031] The video detection method provided by the embodiments of the present application can obtain a video frame from a to-be-detected video as a to-be-detected video frame, extract image features of the to-be-detected video frame to obtain a first feature map, process the first feature map based on a mask corresponding to the first feature map to obtain a second feature map, wherein the mask corresponding to the first feature map contains weights of elements in the first feature map, the weights of the elements conform to a Gaussian distribution, and the weight of an element at a center position of the first feature map is the largest, obtain, for each to-be-matched video, a feature map of a to-be-matched video frame in the to-be-matched video as a third feature map, wherein the third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map, the fourth feature map is obtained by performing feature extraction on the to-be-matched video frame, the mask corresponding to the fourth feature map contains weights of elements in the fourth feature map, the weights of the elements conform to a Gaussian distribution, and the weight of an element at a center position of the fourth feature map is the largest, calculate, for each to-be-detected video frame, a similarity between the to-be-detected video frame and each to-be-matched video frame based on the second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame, and determine, based on the determined similarities, a target video matched with the to-be-detected video from the preset to-be-matched videos.

[0032] Based on the above processing, each first feature map can be processed based on the weights of the elements in the first feature map, the weights of the elements conform to a Gaussian distribution, and the weight of the element at the center position of the first feature map is the largest, and thus the weight of the center region in the second feature map obtained by processing is relatively large. Similarly, the weight of the center region in the third feature map corresponding to the to-be-matched video frame is relatively large. Therefore, by comparing the second feature map and the third feature map, the attention to the center region can be improved, and the influence of the image edge on the detection result can be reduced. Since the image content of the center region of the video frame is not changed when the video frame of the long video is cropped, based on the above processing, the problem that the accuracy of the detection result is not high due to the cropping of the video frame can be avoided to some extent, so as to improve the accuracy of the detection result. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced.

[0034] Figure 1 A flowchart of a video detection method provided by an embodiment of the present application is shown in FIG. 1.

[0035] Figure 2 Another flowchart of a video detection method provided by an embodiment of the present application is shown in FIG. 2.

[0036] Figure 3 Still another flowchart of a video detection method provided by an embodiment of the present application is shown in FIG. 3.

[0037] Figure 4 A structure diagram of a JMLH model provided by an embodiment of the present application is shown in FIG. 4.

[0038] Figure 5 A diagram for determining a matching type of a video to be detected and a target video provided by an embodiment of the present application is shown in FIG. 5.

[0039] Figure 6 A flowchart for determining a matching type provided by an embodiment of the present application is shown in FIG. 6.

[0040] Figure 7 A flowchart of adaptive frame extraction provided by an embodiment of the present application is shown in FIG. 7.

[0041] Figure 8 Still another flowchart of a video detection method provided by an embodiment of the present application is shown in FIG. 8.

[0042] Figure 9 A structure diagram of a video detection device provided by an embodiment of the present application is shown in FIG. 9.

[0043] Figure 10 A structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0045] In order to improve the accuracy of video detection results, the present application provides a video detection method, device, electronic device and storage medium, which are described in detail below.

[0046] The embodiment of the present application provides a video detection method, which can be applied to an electronic device. For example, the electronic device can be a short video sharing platform. The electronic device can detect a short video uploaded by a user based on the video detection method provided by the present application to determine whether there is a long video matched with the short video. If there is a long video matched with the short video, the user can be informed that the short video may cause infringement, and the short video uploaded by the user can be deleted. That is, the short video uploaded by the user is the video to be detected in the embodiment of the present application, and the long video is the video to be matched in the embodiment of the present application.

[0047] For another example, when a user watches a short video through a short video sharing platform, the short video sharing platform can detect the short video and recommend a long video matched with the short video to the user. That is, the video to be detected can be the short video watched by the user, and the video to be matched can be the long video.

[0048] Referring to Figure 1 , Figure 1 A flowchart of the video detection method provided by the embodiment of the present application can include the following steps:

[0049] S101: Obtain a video frame from a video to be detected as a video frame to be detected.

[0050] S102: Extract image features of the video frame to be detected to obtain a first feature map.

[0051] S103: Process the first feature map based on a mask corresponding to the first feature map to obtain a second feature map.

[0052] The mask corresponding to the first feature map contains weights of elements in the first feature map; the weights of the elements conform to a Gaussian distribution, and the weight of an element located in a central region of the first feature map is greater than the weight of an element located in an edge region.

[0053] S104: For each video to be matched, obtain a feature map of a video frame to be matched in the video to be matched as a third feature map.

[0054] The third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map; the fourth feature map is obtained by extracting features of the video frame to be matched; the mask corresponding to the fourth feature map contains weights of elements in the fourth feature map; the weights of the elements conform to a Gaussian distribution, and the weight of an element located in a central region of the fourth feature map is greater than the weight of an element located in an edge region.

[0055] S105: For each to-be-detected video frame, based on the second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame, the similarity between the to-be-detected video frame and each to-be-matched video frame is calculated.

[0056] S106: Based on the determined similarities, the target video matched with the to-be-detected video is determined from the preset to-be-matched videos.

[0057] Based on the above processing, each first feature map can be processed based on the weight of each element in the first feature map, the weight of each element conforms to a Gaussian distribution, and the weight of the element at the center position of the first feature map is the largest, and thus the weight of the center region in the second feature map obtained by processing is larger. Similarly, the weight of the center region in the third feature map corresponding to the to-be-matched video frame is larger. Further, comparing the second feature map and the third feature map can improve the attention to the center region and reduce the influence of the image edge on the detection result. Since the video frame is cropped, the image content of the center region of the video frame is not changed. Therefore, based on the above processing, the problem of low accuracy of the detection result caused by cropping the video frame can be avoided to some extent, so as to improve the accuracy of the detection result.

[0058] For step S101, the to-be-detected video frame can include all video frames in the to-be-detected video; or it can also include part of the video frames in the to-be-detected video. For example, the video frames in the to-be-detected video can be sampled to obtain the to-be-detected video frame.

[0059] For step S102, the image of the to-be-detected video frame can be extracted based on a traditional algorithm. For example, the traditional algorithm can be an Oriented Fast and Rotated Brief (ORB) algorithm.

[0060] Alternatively, the image features of the to-be-detected video frame can be extracted using a deep neural network based on a deep learning method.

[0061] For step S103, the dimension of the mask corresponding to the first feature map is consistent with the dimension of the first feature map, and the dimension of the obtained second feature map is also consistent with the dimension of the first feature map.

[0062] The weights of the elements in the first feature map conform to a two-dimensional Gaussian distribution, that is, the weights of the elements in the first feature map can be obtained based on a two-dimensional Gaussian function. The central region of the first feature map can represent a region centered on the center point of the first feature map; the edge region of the first feature map represents a region away from the center point of the first feature map, that is, a region close to the edge of the first feature map. The point corresponding to the center position of the two-dimensional Gaussian function is located in the central region of the first feature map, and specifically, the point corresponding to the center position can be determined according to the mean of the two-dimensional Gaussian function. The smaller the distance between an element in the first feature map and the point corresponding to the center position of the two-dimensional Gaussian function, the larger the function value corresponding to the element, that is, the larger the weight of the element; the larger the distance between an element in the first feature map and the point corresponding to the center position of the two-dimensional Gaussian function, the smaller the function value corresponding to the element, that is, the smaller the weight of the element. Therefore, the attention to the central region of the video frame can be improved, and the influence of the edge on the detection result can be reduced.

[0063] For step S104, the dimension of the mask corresponding to the fourth feature map is consistent with the dimension of the fourth feature map, and the dimension of the obtained third feature map is also consistent with the dimension of the fourth feature map, and the weights of the elements in the fourth feature map conform to a two-dimensional Gaussian distribution. The relationship between the mask corresponding to the fourth feature map and the fourth feature map is similar to the relationship between the mask corresponding to the first feature map and the first feature map, and can be referred to the related introduction in step S103.

[0064] In addition, the electronic device can record the corresponding relationship between the third feature map and the to-be-matched video to which the third feature map belongs.

[0065] For example, the electronic device can record the third feature map and the identifier of the to-be-matched video to which the third feature map belongs in the database in the form of an inverted index. Correspondingly, based on the corresponding relationship, the electronic device can query the identifier of the to-be-matched video to which the third feature map belongs through the third feature map.

[0066] For step S105, the feature map can be represented as a feature vector, and correspondingly, the similarity between the video frames can be represented by the vector distance between the feature vectors of the corresponding feature maps. Wherein, the larger the vector distance, the smaller the similarity, and the smaller the vector distance, the greater the similarity. If the vector distance is greater than a preset threshold, the vector distance of the second feature map and the third feature map can be set to -1 or empty, indicating that the to-be-detected video frame and the to-be-matched video frame are inconsistent. Wherein, the vector distance can be represented by the Euclidean distance, cosine distance or Hamming distance between vectors.

[0067] In one embodiment, referring to Figure 2 , Figure 2 Another flowchart of the video detection method provided by the embodiment of the present application is shown in Figure 1On the basis of step S102, step S102 can include the following steps:

[0068] S1021: For each to-be-detected video frame, input the to-be-detected video frame into the convolutional neural network.

[0069] S1022: Obtain the feature maps respectively output by each of the plurality of specified network layers in the convolutional neural network, to obtain a first feature map output by each specified network layer.

[0070] Step S103 can include the following steps:

[0071] S1031: For each specified network layer, based on the mask corresponding to the specified network layer, process the first feature map output by the specified network layer to obtain a fifth feature map corresponding to the specified network layer.

[0072] S1032: Obtain a second feature map in combination with the fifth feature map corresponding to each of the specified network layers.

[0073] In the convolutional neural network, the number of feature maps output by each network layer is consistent with the output channel (Channel) number of the network layer. The specified network layer can be determined by a technician according to requirements. Each specified network layer outputs a feature map that can represent different features. That is, for each to-be-detected video frame, a corresponding first feature map can be obtained through each specified network layer.

[0074] The convolutional neural network can be a ResNeSt (Residual Neural Network with Group Attention Mechanism), an AlexNet network, or a LeNet network.

[0075] For example, the convolutional neural network can be a ResNeSt network, and the specified network layer can be layer2 (second layer), layer3 (third layer), and layer4 (fourth layer) in the ResNeSt network.

[0076] After the electronic device obtains the to-be-detected video frame, the to-be-detected video frame is input into the ResNeSt network to obtain a first feature map F l2 output by the second layer, a first feature map F l3 output by the third layer, and a first feature map F l4 output by the fourth layer, respectively. The first feature map F l2 may represent low-level features in the to-be-detected video frame, such as edges, corner points, and other features of objects in the image content; the first feature map F l3 may represent mid-level features in the image content corresponding to the to-be-detected video frame, such as the semantics represented by a certain region in the image content; and the first feature map F l4The high-level features in the image content corresponding to the video frame to be detected can be embodied, for example, the semantics represented by the image content.

[0077] The dimension of the mask corresponding to each specified network layer is consistent with the dimension of the first feature map output by the specified network layer. It can be understood that the mask corresponding to the specified network layer is also the mask corresponding to the output first feature map. Subsequently, the fifth feature map corresponding to the specified network layer is obtained by combining the plurality of first feature maps output by the specified network layer.

[0078] Based on the above processing, the weight of the elements in the center region of the second feature map can be greater than the weight of the elements in the edge region. Similarly, the weight of the elements in the center region of the third feature map corresponding to the video frame to be matched is greater than the weight of the elements in the edge region. Further, based on the second feature map and the third feature map for detection, the attention to the center region can be improved, the influence of the image edge on the detection result can be reduced, and the accuracy of the detection result can be improved.

[0079] In addition, different levels of features of the video frame to be detected can be obtained through the output results of different network layers in the convolutional neural network, so that the information contained in the obtained feature map can be enriched, and then when calculating the similarity between the video frames, the features of different levels can be compared, and the accuracy of the detection result can be improved.

[0080] For step S1031, in one implementation, for each specified network layer, after obtaining the plurality of first feature maps corresponding to the plurality of output channels output by the specified network layer, the result of multiplying each first feature map and the mask corresponding to the specified network layer is calculated to obtain the feature map corresponding to the first feature map (i.e., the sixth feature map hereinafter). Then, the sixth feature map corresponding to each first feature map can be combined to obtain the fifth feature map corresponding to the specified network layer.

[0081] For example, the sixth feature map corresponding to each first feature map can be fused or spliced to obtain the fifth feature map corresponding to the specified network layer.

[0082] In another implementation, in order to reduce the calculation amount during video detection, the result obtained by multiplying each first feature map and the mask corresponding to the specified network layer can be processed by dimension reduction after being calculated. Correspondingly, the above step S1031 comprises:

[0083] Step 1: For each first feature map output by each specified network layer, the first feature map is processed based on the mask corresponding to the specified network layer to obtain a sixth feature map.

[0084] Step 2: For each specified network layer, based on the R-MAC algorithm, the sixth feature map corresponding to the specified network layer is processed to obtain the aggregation feature vector corresponding to the specified network layer as the fifth feature map.

[0085] In an implementation manner, the first feature map is processed based on the mask corresponding to the specified network layer, that is, the product of the first feature map and the element at the corresponding position of the mask corresponding to the first feature map is calculated to obtain the sixth feature map. That is, the sixth feature map can reflect the weight of each position.

[0086] For example, the masks corresponding to the second layer, the third layer and the fourth layer in the ResNeSt network described above, that is, the masks corresponding to the first feature map F l2 , the first feature map F l3 and the first feature map F l4 . Each first feature map is multiplied by the mask corresponding to the first feature map to obtain the sixth feature map, as shown in formula (1):

[0087] F l2 ′=mask l2 *F l2

[0088] F l3 ′=mask l3 *F l3 (1)

[0089] F l4 ′=mask l4 *F l4

[0090] Wherein, F l2 ′ represents the sixth feature map obtained by processing the first feature map F l2 , F l3 ′ represents the sixth feature map obtained by processing the first feature map F l3 , and F l4 ′ represents the sixth feature map obtained by processing the first feature map F l4 . mask l2 represents the mask corresponding to the second layer in the ResNeSt network, mask l3 represents the mask corresponding to the third layer in the ResNeSt network, and mask l4 represents the mask corresponding to the fourth layer in the ResNeSt network.

[0091] If the second feature map is generated in the above manner of step 1- step 2, the fourth feature map can also be processed in a similar manner to generate the third feature map.

[0092] For example, for each specified network layer, after determining the sixth feature map corresponding to the specified network layer, the electronic device can determine, in the sixth feature map, M sampling regions R based on a preset scale L and a sampling stride, by using a sampling window.

[0093] The width and height of the sampling window can be obtained based on formula (2):

[0094] W R (H R )=2min(W F ,H F ) / (L+1) (2)

[0095] L∈[1,2,…,n]

[0096] wherein W R (H R ) represents the width and height of the sampling window, H F represents the height of the sixth feature map, W F represents the width of the sixth feature map, min(W F ,H F ) represents the minimum value between the height and the width of the sixth feature map, and n is preset by a technician, for example, n=7.

[0097] After determining H R and W R under a preset scale L1, for each sixth feature map, the electronic device can determine, in the sixth feature map, M sampling regions with a size of H R ×W R under the scale L1, by sliding with a sampling stride according to the size of the determined sampling window.

[0098] After determining the M sampling regions R contained in the sixth feature map, for each sampling region R o , the dimension of the sampling region R o is [H R *W R ], and the maximum value in the elements contained in the sampling region R i can be determined as the sampling value corresponding to the sampling region R i .

[0099] For each specified network layer, if the output channel number of the specified network layer is K, the number of the sixth feature maps corresponding to the specified network layer is K. That is, M×K sampling regions can be obtained through the specified network layer, and the M×K sampling regions can contain M sampling region groups, and each sampling region group contains K sampling regions with the same position.

[0100] Therefore, for one sampling region group, according to the sampling values corresponding to the sampling regions in the sampling region group, a K-dimensional vector (which can be referred to as a sampling vector, or an aggregated feature vector corresponding to the sampling region group) can be obtained. That is, M K-dimensional sampling vectors can be obtained through the specified network layer.

[0101] For example, reference can be made to formula (3) as shown:

[0102]

[0103] wherein, represents an aggregated feature vector corresponding to the i th sampling region group, represents a sampling value of a sampling region obtained by the j th output channel of the i th sampling region group; p represents the position of an element in a sampling region included in the i th sampling region group, and Ω includes the sampling region determined under the preset scale L1. F l,j (p) represents a value of an element in a sampling region obtained by the j th output channel in the i th sampling region group, represents the maximum value of an element in a sampling region obtained by the j th output channel in the i th sampling region group.

[0104] Then, the aggregated feature vectors corresponding to the respective sampling region groups can be superimposed, for example, the elements corresponding to the same channel in the aggregated feature vectors corresponding to the respective sampling region groups are added to obtain a K-dimensional vector corresponding to the specified network layer, that is, a fifth feature map, which can also be referred to as an aggregated feature vector corresponding to the specified network layer, as shown in formula (4).

[0105]

[0106] wherein, f R represents an aggregated feature vector corresponding to each specified network layer, represents a sampling value of a sampling region obtained by the j th output channel of the i th sampling region group; represents adding the sampling values corresponding to the M sampling regions corresponding to the j th output channel; f R,j represents the sum of the sampling values corresponding to the M sampling regions corresponding to the j th output channel.

[0107] Based on the above processing, for each preset scale, a K-dimensional vector corresponding to each specified network layer can be obtained. Correspondingly, for each specified network layer, the K-dimensional vectors of the specified network layer under each preset scale can be obtained, and the K-dimensional vectors are combined to obtain a fifth feature map corresponding to the specified network layer. For example, the weighted sum of the K-dimensional vectors can be calculated to obtain the fifth feature map corresponding to the specified network layer.

[0108] For example, the electronic device can obtain the sixth feature map Fl2 the sixth feature map F l3 the sixth feature map F l4 sampling. For each sixth feature map, the fifth feature map F the fifth feature map F the fifth feature map F

[0109] Based on the above processing, using the R-MAC algorithm, the local feature extraction of the feature map can be performed to obtain more fine local features, and the local features are aggregated to obtain the reduced global features (i.e., the fifth feature map), so that the data amount of the feature map is smaller. Further, the storage space occupied by the feature map corresponding to the video frame to be matched can be reduced, the calculation amount during video detection can be reduced, and the detection efficiency can be improved.

[0110] In some embodiments, before step S1031, the method further includes:

[0111] Step 3: determining a target network layer from the plurality of specified network layers.

[0112] Step 4: generating a mask corresponding to the target network layer based on a preset expectation, a preset variance, and a dimension of the first feature map output by the target network layer.

[0113] Step 5: performing interpolation operation on the mask corresponding to the target network layer to obtain a mask corresponding to each of the other specified network layers except the target network layer.

[0114] The convolutional neural network includes a plurality of specified network layers, and the target network layer can be determined according to the order of the plurality of specified network layers in the convolutional neural network.

[0115] In an implementation manner, the technician can select a specified network layer at a middle position as the target network layer from the plurality of specified network layers. For example, the convolutional neural network can be a ResNeSt network, the specified network layers can be layer2, layer3 and layer4 in the ResNeSt network, and the target network layer can be layer3.

[0116] Then, the mask corresponding to the target network layer can be generated based on a Gaussian function. Further, the mask corresponding to each of the other specified network layers except the target network layer can be calculated by an interpolation function and the mask corresponding to the target network layer. For example, the interpolation function can be a nearest neighbor interpolation function, a bilinear interpolation function, etc.

[0117] The dimension of the mask corresponding to the target network layer is consistent with the dimension of the first feature map output by the target network layer; the dimensions of the masks corresponding to the specified network layers can be inconsistent.

[0118] For example, the mask corresponding to the first feature map can be obtained based on formula (5):

[0119]

[0120] wherein mask ln (x1, x2) represents the weight of the element at position (x1, x2) in the first feature map output by the target network layer, μ1 and μ2 represent the expectation of the two-dimensional Gaussian distribution, and σ1 and σ2 represent the variance of the two-dimensional Gaussian distribution.

[0121] After obtaining the mask corresponding to the target network layer, the mask corresponding to other specified network layers except the target network layer can be calculated based on an interpolation function, as shown in formula (6):

[0122] mask lm (x1, x2) = interpolate lm (mask ln (x1, x2)) (6)

[0123] wherein mask lm (x1, x2) represents the weight of the element at position (x1, x2) in the first feature map output by the other specified network layer except the target network layer, and interpolate represents the interpolation function.

[0124] Since the dimensions of the masks corresponding to each specified network layer can be inconsistent, using the interpolation function to process the mask of the target network layer to obtain the mask of the other specified network layer can make the masks of the other specified network layers consistent with the weight distribution contained in the mask of the target network layer, and further ensure that the weight distribution of the elements in the fifth feature maps corresponding to each specified network layer is consistent, thereby improving the accuracy of the second feature map and further improving the accuracy of video detection.

[0125] In some embodiments, referring to Figure 3 , Figure 3 Another flowchart of the video detection method provided by the embodiments of the present application is provided, which is based on Figure 2 , and step S1032 comprises:

[0126] S10321: respectively performing binarization processing on the fifth feature maps corresponding to each specified network layer respectively to obtain binarization processing results.

[0127] S10322: splicing the binarization processing results to obtain the second feature map.

[0128] Based on the above processing, for a video frame to be detected, a binarized second feature map corresponding to that video frame can be obtained; that is, one second feature map corresponds to one video frame to be detected. Correspondingly, when extracting the feature map of each video frame to be matched, a binarized third feature map corresponding to that video frame can be obtained; that is, one third feature map corresponds to one video frame to be matched. Furthermore, the similarity between the video frame to be detected and the video frame to be matched can be obtained by comparing the second feature map and the third feature map corresponding to the video frame to be matched.

[0129] In related technologies, feature extraction from images typically yields results represented using floating-point numbers. Based on the above processing, the fifth feature maps corresponding to multiple specified network layers can be binarized and concatenated into a single feature map (i.e., the second feature map). This reduces the number of parameters and computational complexity, improving the efficiency of video detection. Furthermore, compared to feature fusion of the fifth feature maps corresponding to multiple specified network layers, this method maintains the discriminative power of the features output by each specified network layer, avoiding overfitting and thus improving detection accuracy.

[0130] In another embodiment, after acquiring the fifth feature map corresponding to each specified network layer, the electronic device performs binarization processing on the fifth feature map to obtain the binarized result, i.e., the second feature map corresponding to the fifth feature map. That is, for a video frame to be detected, each specified network layer can correspond to a second feature map. Correspondingly, when extracting the feature map of the video frame to be matched, each specified network layer can also correspond to a third feature map. Furthermore, the second and third feature maps corresponding to the same specified network layer can be compared to obtain a comparison result. Then, by combining the comparison results corresponding to each specified network layer, the similarity between the video frame to be detected and the video frame to be matched can be obtained.

[0131] In some embodiments, step S10321 includes:

[0132] The fifth feature map corresponding to each specified network layer is used as input data and input into each fully connected layer in the input layer of the pre-trained JMLH model to obtain the output results of each fully connected layer, which are used as the binarization results.

[0133] The input layer of the JMLH model includes multiple fully connected layers, which are used to binarize each fifth feature map to obtain the binarized feature vector corresponding to each specified network layer, i.e., the binarization result.

[0134] For example, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a JMLH model provided in an embodiment of the present invention.Figure 4 In some embodiments, the full connection layer in the JMLH model The fifth feature map can be binarized to obtain a binarized feature vector b l2 In some embodiments, the full connection layer in the JMLH model The fifth feature map can be binarized to obtain a binarized feature vector b l3 In some embodiments, the full connection layer in the JMLH model The fifth feature map can be binarized to obtain a binarized feature vector b l4 . The binarized feature vectors corresponding to the full connection layers are spliced into a binarized vector b (i.e., a second feature map). In addition, the one-hot label y' corresponding to the plurality of fifth feature maps can be calculated by the full connection layer The one-hot label y' can represent a preset category to which the video frame to be detected belongs.

[0135] Based on the above processing, the fifth feature maps corresponding to the plurality of specified network layers can be binarized and spliced into a feature map (i.e., a second feature map), which can reduce the parameter amount and the complexity of calculation, and improve the efficiency of video detection. In addition, compared with feature fusion of the fifth feature maps corresponding to the plurality of specified network layers, the discriminability of the features output by each specified network layer can be maintained, overfitting can be avoided, and thus the accuracy of detection can be improved.

[0136] In some embodiments, step S106 comprises:

[0137] Step 6: For each video frame to be detected, based on the similarity corresponding to the video frame to be detected, a video frame matching the video frame to be detected is determined from the preset video frames contained in each video to be matched as a target video frame.

[0138] Step 7: Determine the video to be matched to which each target video frame belongs as a target video.

[0139] For each video frame to be detected, the electronic device can obtain the second feature map corresponding to the video frame to be detected based on steps S101-S103, and calculate the vector distance between the second feature map corresponding to the video frame to be detected and each third feature map.

[0140] Then, the vector distances can be sorted in ascending order. Further, the third feature maps corresponding to the first preset number of vector distances are determined, and the video frames to be matched to which the first preset number of third feature maps belong are determined as target video frames.

[0141] ​​​​In some embodiments, after step 7, the method further comprises:

[0142] Step one: for each target video, generating a similarity matrix corresponding to the target video based on the similarity between the video frames to be detected and the target video frames in the target video.

[0143] Wherein each row in the similarity matrix corresponds to a video frame to be detected one by one, and the order between the rows is determined according to the order of the corresponding video frame to be detected in the video to be detected; each column in the similarity matrix corresponds to a target video frame in the target video one by one, and the order between the columns is determined according to the order of the corresponding target video frame in the target video; the similarity element in the similarity matrix represents the similarity between the video frame to be detected corresponding to the row to which the similarity element belongs and the target video frame corresponding to the column to which the similarity element belongs; if the similarity between the video frame to be detected corresponding to the row to which the similarity element belongs and the target video frame corresponding to the column to which the similarity element belongs is less than the first threshold, the similarity element is invalid.

[0144] Step two: in the order of row number from low to high, for the similarity elements in each row of the similarity matrix, in the order of column number from high to low, the element position of each non-invalid similarity element in the similarity matrix is added to the preset list in turn to obtain the position index list corresponding to the target video.

[0145] Step three: determine the current processing element position according to the order of the element position in the position index list.

[0146] Step four: determine the current first element position from each element position located before the current processing element position.

[0147] Wherein the column number corresponding to the current processing element position is greater than the column number corresponding to the current first element position, the time length between the timestamp of the video frame to be detected corresponding to the current processing element position and the timestamp of the video frame to be detected corresponding to the current first element position is less than the first time length, and the time length between the timestamp of the video frame to be matched corresponding to the current processing element position and the timestamp of the video frame to be matched corresponding to the current first element position is less than the second time length.

[0148] Step five: determine the length value corresponding to the current processing element position as the first length value of the current processing element position.

[0149] Wherein the first length value of the current processing element position is determined based on the second length value corresponding to the current first element position; the second length value corresponding to the current first element position represents the sum value of the first length value of the current first element position and 1.

[0150] Step six: determining the starting sequence number of the current to-be-processed element position based on the starting sequence number of the current first element position.

[0151] wherein the starting sequence number of the current first element position is determined based on the starting sequence number of each element position before the current to-be-processed element position when the current first element position is the current to-be-processed element position; and the starting sequence number of the first element position in the position index list is 0.

[0152] Step seven: determining the next element position in the position index list as the current to-be-processed element position, and returning to execute the above step four until the last element position in the position index list is obtained.

[0153] Step eight: determining the element position corresponding to the largest first length value as the terminal element position.

[0154] Step nine: determining the element position corresponding to the starting sequence number of the terminal element position in the position index list as the starting element position.

[0155] Step ten: determining the to-be-detected video frame corresponding to the terminal element position as the first terminal video frame, and determining the target video frame corresponding to the terminal element position as the second terminal video frame.

[0156] Step eleven: determining the to-be-detected video frame corresponding to the starting element position as the first starting video frame, and determining the target video frame corresponding to the starting element position as the second starting video frame.

[0157] Step twelve: determining that the first video segment in the to-be-detected video with the first starting video frame and the first terminal video frame as boundaries is consistent with the second video segment in the target video with the second starting video frame and the second terminal video frame as boundaries.

[0158] In an implementation manner, for each target video, all target video frames in the target video are determined, and a similarity matrix corresponding to the target video is generated based on the determined target video frames.

[0159] For example, the similarity matrix is shown in matrix (1):

[0160]

[0161] The number of the to-be-detected video frames is 3, and the number of the target video frames in the target video is 3. 120 represents the similarity between the first to-be-detected video frame in the to-be-detected video and the first target video frame in the target video; 367 represents the similarity between the second to-be-detected video frame in the to-be-detected video and the second target video frame in the target video; 192 represents the similarity between the second to-be-detected video frame in the to-be-detected video and the third target video frame in the target video; and 83 represents the similarity between the third to-be-detected video frame in the to-be-detected video and the third target video frame in the target video. -1 represents that the similarity element is invalid, that is, the to-be-detected video frame corresponding to the row of the similarity element is inconsistent with the target video frame corresponding to the column of the similarity element; or the similarity element can be set to empty, indicating that the similarity element is invalid.

[0162] In ascending order of the row number and descending order of the column number, for each similarity element that is not invalid, the element position (that is, the coordinate composed of the row number and the column number of the similarity element) corresponding to the similarity element is recorded in sequence to obtain a position index list.

[0163] For example, the position index list obtained based on the matrix (1) is shown in Table (1):

[0164] Table (1)

[0165] Element position (0,0) (1,2) (1,1) (2,2)

[0166] In ascending order of the row number and descending order of the column number, for each similarity element that is not invalid, the element position (that is, the coordinate composed of the row number and the column number of the similarity element) corresponding to the similarity element is recorded in sequence to obtain a position index list.

[0167] The position of an element position in the position index list can be referred to as a list position. For example, in order, the values of the list positions can be 0, 1, 2, 3, and so on. For example, if the list position of the current to-be-processed element position is list position i, for each list position j, the value of j is in the range of [0, i-1], if the column number of the element position at the list position j is less than the column number of the element position at the list position i, the value of j is in the range of [0, i-1], and if the column number of the element position at the list position j is greater than the column number of the element position at the list position i, the value of j is in the range of [0, i-1].

[0168] The column number of the element position at the list position i (that is, the current to-be-processed element position) is greater than the column number of the element position at the list position j.

[0169] The column number of the element position at the list position i (that is, the current to-be-processed element position) is greater than the column number of the element position at the list position j.

[0170] The timestamp of the to-be-detected video frame corresponding to the element position at the list position i is less than the timestamp of the to-be-detected video frame corresponding to the element position at the list position j, and the column number of the element position at the list position i is greater than the column number of the element position at the list position j.

[0171] The timestamp of the video frame to be detected corresponding to the element position at position i in the list is compared with the element position at position j in the list.

[0172] If the difference between the timestamps of the corresponding video frames to be detected is less than the second duration, then the element position at position j in the list can be determined as the current first element position. In other words, for the current element position to be processed, the determined first element position can be one or more. The first and second durations can be preset by technicians; the first duration can be 15 seconds, and the second duration can be 10 seconds.

[0173] If the current position of the element to be processed does not have a corresponding first element position, then the starting index of the current position of the element to be processed is determined as the value of the list position of the current position of the element to be processed. It is understandable that since the first element position in the position index list does not have a corresponding first element position, the starting index of the first element position can be determined to be 0.

[0174] If there are multiple positions for the current first element, then for each current first element position, obtain the list position of that first element position. Then, calculate the sum of the first length value corresponding to that list position and 1, and use this sum as the second length value corresponding to that first element position. If there is only one first element position corresponding to the current element to be processed, then the second length value corresponding to that first element position can be determined as the first length value of the current element to be processed. If there are multiple first element positions corresponding to the current element to be processed, then the maximum value among the second length values ​​corresponding to each first element position can be determined as the first length value of the current element to be processed.

[0175] If the current position of the element to be processed does not have a corresponding first element position, then the first length value of the current position of the element to be processed can be set to 1. It is understandable that the first element position in the position index list does not have a corresponding first element position; therefore, the first length value of the first element position can be determined to be 1.

[0176] The starting index of the current element to be processed is determined based on the starting index of the first element position corresponding to the current element to be processed. For example, if there is only one first element position corresponding to the current element to be processed, then the starting index of that first element position can be determined as the starting index of the current element to be processed. If there are multiple first element positions corresponding to the current element to be processed, then the first element position with the largest second length value can be determined, and the starting index of that first element position can be determined as the starting index of the current element to be processed.

[0177] Based on the above steps, the first length value and the starting sequence number corresponding to the current to-be-processed element position can be determined. That is, the value of the list position of the current to-be-processed element position in the length value list and the starting sequence number list is determined. Further, the next element position in the position index list can be determined as the current to-be-processed element position, and the first length value and the starting sequence number corresponding to the current to-be-processed element position are determined, until the first length value and the starting sequence number corresponding to all element positions in the position index list are determined.

[0178] Based on the above steps three to seven, the first length value and the starting sequence number corresponding to each element position in the position index list can be determined. Subsequently, the element position corresponding to the largest numerical first length value can be determined as the terminal element position, and the element position corresponding to the starting sequence number of the terminal element position in the position index list can be determined as the starting element position.

[0179] Further, the to-be-detected video frame corresponding to the starting element position can be determined as the first starting video frame, the target video frame corresponding to the starting element position can be determined as the second starting video frame, the to-be-detected video frame corresponding to the terminal element position can be determined as the first terminal video frame, and the target video frame corresponding to the terminal element position can be determined as the second terminal video frame.

[0180] Based on the above steps, the video segment bounded by the first starting video frame and the first terminal video frame in the to-be-detected video can be determined as the first video segment, and the video segment bounded by the second starting video frame and the second terminal video frame in the determined target video can be determined as the second video segment.

[0181] If there are multiple identical video segments in the to-be-detected video and the to-be-matched video, after a starting element position and a terminal element position are determined, the row (which can be referred to as a starting row) to which the similarity element corresponding to the starting element position belongs in the similarity matrix and the row (which can be referred to as a terminal row) to which the similarity element corresponding to the terminal element position belongs in the similarity matrix can be determined, and the elements of the starting row, the elements of the terminal row, and the elements of each row between the starting row and the terminal row in the similarity matrix are set to be invalid (for example, -1 or empty) to update the similarity matrix.

[0182] Then, steps eight to twelve are returned to be executed until all similarity elements in the similarity matrix are invalid.

[0183] For example, in table (1), the value of the list position of the element position (0, 0) is 0, the value of the list position of the element position (1, 2) is 1, the value of the list position of the element position (1, 1) is 2, and the value of the list position of the element position (2, 2) is 3.

[0184] According to the matrix (1), the position index list is [(0, 0), (1, 2), (1, 1), (2, 2)]. The length value list can be initialized as [1, 1, 1, 1], and the start index list can be initialized as [-1, -1, -1, -1]. In addition, the first time length is 25 seconds, the second time length is 15 seconds, the timestamps of the video frames corresponding to the 0th to 2nd rows of the matrix (1) are [10, 20, 30] respectively, and the timestamps of the video frames corresponding to the 0th to 2nd columns of the matrix (1) are [30, 40, 50] respectively.

[0185] When the element position (0, 0) is the current element position to be processed, the value of the list position of the element position (0, 0) is 0. Since there is no corresponding first element position, it can be determined that the start index of the element position (0, 0) is 0, and the first length value of the element position (0, 0) is 1. At this time, the length value list is [1, 1, 1, 1], and the start index list is [0, -1, -1, -1].

[0186] When the element position (1, 2) is the current element position to be processed, the value of the list position of the element position (1, 2) is 1. For each list position j, the value of j ranges from [0]. The column number of the element position (1, 2) is 2, the timestamp of the video frame corresponding to the element position (1, 2) is 50, and the timestamp of the video frame corresponding to the element position (1, 2) is 20.

[0187] When j is 0, the element position at the list position 0 is (0, 0). The column number of the element position (0, 0) is 0, the timestamp of the video frame corresponding to the element position (0, 0) is 30, and the timestamp of the video frame corresponding to the element position (0, 0) is 10. The column number of the element position (1, 2) is greater than the column number of the element position (0, 0), and the difference between the timestamp of the video frame corresponding to the element position (1, 2) and the timestamp of the video frame corresponding to the element position (0, 0) is less than the first time length; the difference between the timestamp of the video frame corresponding to the element position (1, 2) and the timestamp of the video frame corresponding to the element position (0, 0) is less than the second time length. It is determined that the element position (0, 0) is the current first element position. Then, the sum of the first length value of the element position (0, 0) and 1 is calculated to be 2, and the second length value corresponding to the element position (0, 0) is 2.

[0188] Since the first element position corresponding to the list position (1, 2) is one, the second length value corresponding to the list position (0, 0) is determined as the first length value of the list position (1, 2), that is, the first length value of the list position (1, 2) is 2, and the start serial number of the list position (0, 0) is determined as the start serial number of the list position (1, 2), that is, the start serial number of the list position (1, 2) is 0. At this time, the length value list is [1, 2, 1, 1], and the start serial number list is [0, 0, -1, -1].

[0189] When the element position (1, 1) is the current to-be-processed element position, the value of the list position of the element position (1, 1) is 2. For each list position j, the value range of j is [0, 1]. The column number of the element position (1, 1) is 1, the timestamp of the to-be-matched video frame corresponding to the element position (1, 1) is 40, and the timestamp of the to-be-detected video frame corresponding to the element position (1, 1) is 20.

[0190] When j is 0, the element position at the list position 0 is (0, 0). The column number of the element position (0, 0) is 0, the timestamp of the to-be-matched video frame corresponding to the element position (0, 0) is 30, and the timestamp of the to-be-detected video frame corresponding to the element position (0, 0) is 10. The column number of the element position (1, 1) is greater than the column number of the element position (0, 0), the difference between the timestamp of the to-be-matched video frame corresponding to the element position (1, 1) and the timestamp of the to-be-matched video frame corresponding to the element position (0, 0) is less than the first time length, and the difference between the timestamp of the to-be-detected video frame corresponding to the element position (1, 1) and the timestamp of the to-be-detected video frame corresponding to the element position (0, 0) is less than the second time length. The element position (0, 0) is determined as the current first element position. Then, the sum of the first length value of the element position (0, 0) and 1 is 2, and the second length value corresponding to the element position (0, 0) is 2.

[0191] When j is 1, the element position at the list position 1 is (1, 2). Since the column number of the element position (1, 1) is less than the column number of the element position (1, 2), the element position (1, 2) is not the current first element position.

[0192] Since the first element position corresponding to the list position (1, 1) is one, the second length value corresponding to the list position (0, 0) is determined as the first length value of the list position (1, 1), that is, the first length value of the list position (1, 1) is 2, and the start serial number of the list position (0, 0) is determined as the start serial number of the list position (1, 1), that is, the start serial number of the list position (1, 1) is 0. At this time, the length value list is [1, 2, 2, 1], and the start serial number list is [0, 0, 0, -1].

[0193] When the element position (2, 2) is the current element position to be processed, the value of the list position of the element position (2, 2) is 3. For each list position j, j takes a value in the range of [0, 1, 2]. The column number of the element position (2, 2) is 2, the timestamp of the video frame corresponding to the element position (2, 2) to be matched is 50, and the timestamp of the video frame corresponding to the element position (2, 2) to be detected is 30.

[0194] When j is 0, the element position at the list position 0 is (0, 0). The column number of the element position (0, 0) is 0, the timestamp of the video frame corresponding to the element position (0, 0) to be matched is 30, and the timestamp of the video frame corresponding to the element position (0, 0) to be detected is 10. The column number of the element position (2, 2) is greater than the column number of the element position (0, 0), the difference between the timestamp of the video frame corresponding to the element position (1, 2) to be matched and the timestamp of the video frame corresponding to the element position (0, 0) to be matched is less than the first time length, and the difference between the timestamp of the video frame corresponding to the element position (1, 2) to be detected and the timestamp of the video frame corresponding to the element position (0, 0) to be detected is greater than the second time length. Therefore, the element position (0, 0) is not the current first element position.

[0195] When j is 1, the element position at the list position 1 is (1, 2). Since the column number of the element position (2, 2) is equal to the column number of the element position (1, 2), the element position (1, 2) is not the current first element position.

[0196] When j is 2, the element position at the list position 2 is (1, 1), the column number of the element position (1, 1) is 1, the timestamp of the video frame corresponding to the element position (1, 1) to be matched is 40, and the timestamp of the video frame corresponding to the element position (1, 1) to be detected is 20. The column number of the element position (2, 2) is greater than the column number of the element position (1, 1), the difference between the timestamp of the video frame corresponding to the element position (2, 2) to be matched and the timestamp of the video frame corresponding to the element position (1, 1) to be matched is less than the first time length, and the difference between the timestamp of the video frame corresponding to the element position (2, 2) to be detected and the timestamp of the video frame corresponding to the element position (1, 1) to be detected is less than the second time length. It is determined that the element position (1, 1) is the current first element position. Then, the sum of the first length value of the element position (1, 1) and 1 is 3, and the second length value corresponding to the element position (1, 1) is 3.

[0197] Since the first element position corresponding to the list position (2, 2) is one, the second length value corresponding to the list position (0, 0) is determined as the first length value of the list position (2, 2), that is, the first length value of the list position (2, 2) is 2, and the starting serial number of the list position (1, 1) is determined as the starting serial number of the list position (2, 2), that is, the starting serial number of the list position (2, 2) is 0. At this time, the length value list is [1, 2, 2, 3], and the starting serial number list is [0, 0, 0, 0].

[0198] At this time, the first length value of each element position in the position index list has been determined, and the starting serial number of each element position has been determined. The length value list is [1, 2, 2, 3], and the starting serial number list is [0, 0, 0, 0]. Further, it can be determined that the first length value with the largest value is 3, the corresponding terminal element position is (2, 2), the starting serial number of the terminal element position (2, 2) is 0, and further, the starting element position can be determined as (0, 0).

[0199] Further, the first starting video frame corresponding to the 10th second of the to-be-detected video can be determined as the first starting video frame, and the first terminal video frame corresponding to the 30th second of the to-be-detected video can be determined as the first terminal video frame, to determine a first video segment in the to-be-detected video with the first starting video frame and the first terminal video frame as boundaries; the second starting video frame corresponding to the 30th second of the to-be-matched video can be determined as the second starting video frame, and the second terminal video frame corresponding to the 30th second of the to-be-matched video can be determined as the second terminal video frame, to determine a second video segment in the to-be-matched video with the second starting video frame and the second terminal video frame as boundaries.

[0200] In addition, the similarity matrix (1) can be updated, that is, all values in rows 0 to 2 in the similarity matrix (1) are set to -1. Subsequently, since there is no non-1 similarity element in the updated similarity matrix (1), the loop is no longer performed.

[0201] Based on the above processing, all video segments in the to-be-detected video consistent with the video segments contained in the target video can be determined, and the lengths and positions corresponding to the video segments can be recorded, and further, the matching type of the to-be-detected video and the to-be-matched video can be accurately determined subsequently.

[0202] In some embodiments, after step S106, the method further comprises:

[0203] If the target video is one, and the to-be-detected video is consistent with one video segment in the target video, it is determined that the matching type of the to-be-detected video and the target video is segment matching.

[0204] If the target video is one, the multiple first video segments in the video to be detected are respectively consistent with the multiple second video segments in the target video, and the order of the multiple first video segments in the video to be detected is consistent with the order of the multiple second video segments in the target video, it is determined that the matching type of the video to be detected and the target video is single video sequential matching.

[0205] If the target video is one, the multiple first video segments in the video to be detected are respectively consistent with the multiple second video segments in the target video, and the order of the multiple first video segments in the video to be detected is inconsistent with the order of the multiple second video segments in the target video, it is determined that the matching type of the video to be detected and the target video is single video non-sequential matching.

[0206] If the target video is multiple, and the multiple target videos belong to the same video set, it is determined that the matching type of the video to be detected and the target video is multiple video non-sequential matching.

[0207] If the target video is multiple, and the multiple target videos belong to different video sets, it is determined that the matching type of the video to be detected and the target video is multiple set non-sequential matching.

[0208] The first video segment represents a video segment in the video to be detected; and the second video segment represents a video segment in the target video.

[0209] The video set can also be referred to as a clip single. For example, the videos corresponding to each episode of a certain TV series can belong to the video set corresponding to the TV series; multiple movies belonging to the same theme can correspond to one video set.

[0210] In an implementation manner, as shown in FIG. 1, Figure 5 FIG. 1 is a schematic diagram of determining the matching type of the video to be detected and the target video provided by an embodiment of the present application. Figure 5

[0211] Figure 5 In the figure, (a) represents that the matching type of the video to be detected and the target video is segment matching. In (a), the video segment a1 is the video to be detected, the video segments a2, a3 and a4 constitute the target video, and the video segment a1 is consistent with the video segment a3. That is, the number of the target video is one, and the video to be detected is consistent with one video segment in the target video.

[0212] Figure 5 ​In the diagram, (b) indicates that the matching type between the video to be detected and the target video is single-video sequential matching. Here, video segments b1, b2, b3, and b4 constitute the video to be detected, and video segments b5, b6, b7, and b8 constitute the target video. Video segment b1 is identical to video segment b6, and video segment b3 is identical to video segment b8. That is, there is one target video, and the video to be detected has two first video segments, each identical to one of the two second video segments in the target video. Furthermore, the order of the two first video segments in the video to be detected is consistent with the order of the identical second video segments in the target video.

[0213] Figure 5 In the diagram, (c) indicates that the matching type between the video to be detected and the target video is single-video unordered matching. Here, video segments c1, c2, c3, and c4 constitute the video to be detected, and video segments c5, c6, c7, c8, and c9 constitute the target video. Video segment c1 is identical to video segment c8, and video segment c3 is identical to video segment c6. That is, there is one target video, and the video to be detected has two first video segments, each identical to one of the two second video segments in the target video. Furthermore, the order of the two first video segments in the video to be detected is different from the order of the identical second video segments in the target video.

[0214] Figure 5 In the diagram, (d) indicates that the matching type between the video to be detected and the target video is multi-video unordered matching. Specifically, video segments d1, d2, and d3 constitute the video to be detected; video segments d4, d5, and d6 constitute target video 1; and video segments d7 and d8 constitute target video 2. Video segment d1 matches video segment d5, and video segment d3 matches video segment d7. Target video 1 and target video 2 belong to the same video set. That is, there are multiple target videos, the video to be detected has two first video segments, each matching a second video segment in one of the two target videos, and the target videos belong to the same video set.

[0215] Figure 5 In the diagram, (e) indicates that the matching type between the video to be detected and the target video is a multi-set unordered matching. Specifically, video segments e1, e2, and e3 constitute the video to be detected; video segments e4 and e5 constitute target video 1; and video segments e6, e7, and e8 constitute target video 2. Video segment e1 matches video segment e4, and video segment e3 matches video segment e7. Target video 1 and target video 2 belong to different video sets. That is, there are multiple target videos, the video to be detected has two first video segments, each matching a second video segment in one of the two target videos, and the target videos belong to different video sets.

[0216] like Figure 6 As shown,Figure 6 A flowchart for judging the matching type is provided for the embodiments of the present application, as shown in Figure 6

[0217] S601: Retrieval result.

[0218] The target video frame determined based on each to-be-detected video frame. After determining the retrieval result, step S602 is performed.

[0219] S602: Classifying the retrieval result according to the target video ID (Identity document, identification).

[0220] Determine the target video to which each target video frame belongs, and perform step S603.

[0221] S603: Calculate the copy segment corresponding to each target video.

[0222] Determine all first video segments in the to-be-detected video and all second video segments in the target video, and perform step S604.

[0223] S604: Judge whether the target video is only one.

[0224] If the target video is only one, step S605 is performed; if the target video is multiple, step S610 is performed.

[0225] S605: Judge whether it is a single segment (i.e., the to-be-detected video is consistent with one video segment in the target video).

[0226] If it is a single segment, step S606 is performed; if it is not a single segment, step S607 is performed.

[0227] S606: Segment.

[0228] Determine that the matching type is segment matching.

[0229] S607: Judge whether it is sequential timing (i.e., the to-be-detected video has two first video segments, which are consistent with two second video segments in the target video respectively, and the order of the two first video segments in the to-be-detected video is the same as the order of the second video segments corresponding to the first video segments in the target video).

[0230] If it is sequential timing, step S608 is performed; if it is not sequential timing, step S609 is performed.

[0231] S608: Single video sequential clip.

[0232] Determine that the matching type is single video sequential matching.

[0233] ​S609: single video out-of-order clip.

[0234] determining that the matching type is single video out-of-order matching

[0235] S610: judging whether the target video is the same piece.

[0236] If the target video is the same piece, step S611 is performed; if the target video is not the same piece, step S612 is performed.

[0237] S611: multiple video out-of-order clip.

[0238] determining that the matching type is multiple video out-of-order matching.

[0239] S612: multiple piece out-of-order clip.

[0240] determining that the matching type is multiple set out-of-order matching.

[0241] In some embodiments, the to-be-matched video frames in the to-be-matched video include video frames obtained by sampling the to-be-matched video based on a first sampling frequency;

[0242] The to-be-detected video frames include video frames obtained by sampling the to-be-detected video based on a second sampling frequency; or the to-be-detected video frames include sampling video frames obtained by sampling the to-be-detected video based on the second sampling frequency, and target key frames in the to-be-detected video frames; wherein a time sequence distance between the target key frames and the nearest sampling video frame is greater than a second threshold.

[0243] The first sampling frequency can be 2 seconds. The second sampling frequency can be 2 seconds or 1 second.

[0244] In an implementation manner, based on the second sampling frequency, video frames with frame numbers P = {p1, p2, p3, …, p m} in the to-be-detected video can be obtained as the sampling video frames. In addition, the to-be-detected video can be decoded to obtain key frames, i.e., I frames, in the to-be-detected video, and the frame numbers of the key frames are Q = {q1, q2, q3, …, q n}.

[0245] For each key frame, a remainder obtained by taking the frame number of the key frame modulo a frame rate of the to-be-detected video is calculated, and if the remainder is less than a third threshold and greater than a fourth threshold, the key frame is a legal key frame, and the key frame can be determined as a target key frame. As shown in formula (7):

[0246]

[0247] Where I represents the indicator function; fps represents the frame rate of the video to be detected. The third threshold is fps-th1, and the fourth threshold is th1. The fourth threshold is preset by the technician; for example, when fps is 30, th1 is 3.

[0248] Based on formula (7), for each key frame, if the remainder of the frame number of the key frame after taking the modulo of the frame rate of the video to be detected is less than (fps-th1) and greater than th1, then I = 1, and the key frame can be determined as the target key frame; otherwise, I = 0, indicating that there is a time interval between a sampled video frame and the key frame that is less than the preset threshold, then there is no need to determine the key frame as the target key frame.

[0249] After identifying the target keyframes, these keyframes can be used as the video frames to be detected. The method described above for obtaining the video frames to be detected from the video can be called adaptive frame extraction.

[0250] Furthermore, the frame number V = {q1, p1, p2, ..., p} can be determined from the video to be detected. m ,q n The video frames to be detected are F = {f1, f2, ..., f} N-1 ,f N}

[0251] In one implementation, such as Figure 7 As shown, Figure 7 This is a schematic diagram of an adaptive frame extraction process provided in an embodiment of the present invention.

[0252] For a short video (i.e., the video to be detected), the electronic device can sample the video using a uniform frame sampling method to obtain sampled video frames. Alternatively, the electronic device can use I-frame sampling to obtain keyframes from the video to be detected. For each keyframe, the frame number of the keyframe is modulo the frame rate of the video to be detected. That is, the remainder after taking the frame number modulo the frame rate of the video to be detected is calculated. If the remainder is less than the frame rate minus a threshold (i.e., the third threshold) and greater than a threshold (i.e., the fourth threshold), then the keyframe is designated as the target keyframe. The frame number of the target keyframe is a valid frame number. The video to be detected includes sampled video frames and target keyframes.

[0253] Based on the above processing, the video frame to be detected can be determined from the video to be detected based on the uniform frame extraction manner, and the key frame meeting the preset condition can also be determined as the video frame to be detected, the meaningful image content can be supplemented, and the accuracy of video detection is improved. In addition, for the key frame which does not meet the preset condition, the key frame can be discarded, that is, for the key frame which has a small time sequence distance from the sampling video frame, since the image content difference between the key frame and the sampling video frame is small, the key frame does not need to be determined as the target key frame. Further, the redundant frame in the video frame to be detected can be reduced, the calculation amount of video detection is reduced, and the efficiency of video detection is improved.

[0254] In an implementation manner, as shown in Figure 8 , Figure 8 Another flowchart of a video detection method provided by an embodiment of the present application is shown.

[0255] Figure 8 In the embodiment, the video frame to be matched can be determined from the long video (i.e., the video to be matched) in the long video library based on the uniform frame extraction manner, and feature extraction is performed on the video frame to be matched. The electronic device can establish an index based on the features of the video frame to be matched, that is, store the features of each video frame to be matched and the video to be matched to which the video frame to be matched belongs in the index.

[0256] When the short video (i.e., the video to be detected) needs to be detected, the video frame to be detected in the video to be detected can be obtained based on the adaptive frame extraction manner, and feature extraction is performed on the video frame to be detected. Based on the features of the video frame to be matched stored in the index, the copy type is queried, that is, the video frame to be matched (i.e., the target video frame) matching the video to be detected is determined, and then the target video to which the target video frame belongs is determined, and the copy type (i.e., the matching type) is determined according to the video segment matching the video to be detected and the target video.

[0257] Based on the same inventive concept, an embodiment of the present application further provides a video detection device, which is shown in Figure 9 , Figure 9 A structural diagram of a video detection device provided by an embodiment of the present application is shown. The device comprises:

[0258] The video frame to be detected acquisition module 901 is configured to obtain a video frame from the video to be detected as the video frame to be detected.

[0259] The first feature map acquisition module 902 is configured to extract the image features of the video frame to be detected to obtain a first feature map.

[0260] The second feature map acquisition module 903 is configured to process the first feature map based on a mask corresponding to the first feature map, to obtain a second feature map; the mask corresponding to the first feature map contains weights of elements in the first feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the first feature map are greater than the weights of elements located in an edge region.

[0261] The third feature map acquisition module 904 is configured to acquire, for each to-be-matched video in the preset to-be-matched videos, a feature map of a to-be-matched video frame in the to-be-matched video as a third feature map; the third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map; the fourth feature map is obtained by performing feature extraction on the to-be-matched video frame; the mask corresponding to the fourth feature map contains weights of elements in the fourth feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the fourth feature map are greater than the weights of elements located in an edge region.

[0262] The similarity calculation module 905 is configured to, for each to-be-detected video frame, calculate similarities between the to-be-detected video frame and each to-be-matched video frame based on a second feature map corresponding to the to-be-detected video frame and a third feature map corresponding to each to-be-matched video frame.

[0263] The target video determination module 906 is configured to determine, from the preset to-be-matched videos, a target video matched with the to-be-detected video based on the determined similarities.

[0264] In some embodiments, the first feature map acquisition module 902 includes a to-be-detected video frame input sub-module configured to input, for each to-be-detected video frame, the to-be-detected video frame to a convolutional neural network; and a first feature map output sub-module configured to acquire feature maps output by a plurality of designated network layers of the convolutional neural network respectively, to obtain a first feature map output by each designated network layer; the second feature map acquisition module 903 includes a fifth feature map acquisition sub-module configured to, for each designated network layer, process the first feature map output by the designated network layer based on a mask corresponding to the designated network layer, to obtain a fifth feature map corresponding to the designated network layer; and a second feature map acquisition sub-module configured to obtain a second feature map in combination with the fifth feature maps corresponding to the designated network layers.

[0265] In some embodiments, the fifth feature map obtaining submodule comprises: a sixth feature map obtaining unit configured to, for each first feature map output by each specified network layer, process the first feature map based on the mask corresponding to the specified network layer to obtain a sixth feature map; and a fifth feature map obtaining unit configured to, for each specified network layer, process each sixth feature map corresponding to the specified network layer based on a region maximum convolutional activation (R-MAC) algorithm to obtain an aggregated feature vector corresponding to the specified network layer as a fifth feature map.

[0266] In some embodiments, the apparatus further comprises: a target network layer determining module configured to determine a target network layer from the plurality of specified network layers before processing each first feature map output by each specified network layer based on the mask corresponding to the specified network layer to obtain a sixth feature map; a mask generating module configured to generate the mask corresponding to the target network layer based on a preset expectation, a preset variance, and a dimension of the first feature map output by the target network layer; and a mask calculating module configured to obtain the masks corresponding to the specified network layers other than the target network layer based on interpolation operation on the mask corresponding to the target network layer.

[0267] In some embodiments, the second feature map obtaining submodule comprises: a binarization processing unit configured to perform binarization processing on each fifth feature map corresponding to each specified network layer respectively to obtain binarization processing results; and a splicing unit configured to splice the binarization processing results to obtain a second feature map.

[0268] In some embodiments, the binarization processing unit is specifically configured to input each fifth feature map corresponding to each specified network layer into each fully connected layer included in an input layer of a pre-trained unique maximum likelihood hashing (JMLH) model as input data to obtain output results of the fully connected layers as the binarization processing results.

[0269] In some embodiments, the target video determining module 906 comprises: a target video frame determining submodule configured to, for each to-be-detected video frame, determine a video frame matching the to-be-detected video frame from to-be-matched video frames included in each to-be-matched video based on the similarity corresponding to the to-be-detected video frame as a target video frame; and a target video determining submodule configured to determine a to-be-matched video to which each target video frame belongs as a target video respectively.

[0270] In some embodiments, the apparatus further comprises: a similarity matrix generating module configured to, after the determining of the target video to which each target video frame belongs, generate, for each target video, a similarity matrix corresponding to the target video based on similarities between the to-be-detected video frames and the target video frames in the target video; wherein each row in the similarity matrix corresponds to one of the to-be-detected video frames, and the order between the rows is determined according to the order of the corresponding to-be-detected video frames in the to-be-detected video; each column in the similarity matrix corresponds to one of the target video frames in the target video, and the order between the columns is determined according to the order of the corresponding target video frames in the target video; a similarity element in the similarity matrix represents a similarity between a to-be-detected video frame corresponding to a row to which the similarity element belongs and a target video frame corresponding to a column to which the similarity element belongs; if the similarity between the to-be-detected video frame corresponding to the row to which the similarity element belongs and the target video frame corresponding to the column to which the similarity element belongs is less than a first threshold, the similarity element is invalid; a position index list obtaining module configured to, in order of row number from low to high, for each similarity element in each row of the similarity matrix, in order of column number from high to low corresponding to the similarity element, sequentially add an element position of each non-invalid similarity element in the similarity matrix to a preset list to obtain a position index list corresponding to the target video; a to-be-processed element position determining module configured to determine a current to-be-processed element position according to the order of the element positions in the position index list; the to-be-processed element position determining module is configured to determine a current first element position from each element position located before the current to-be-processed element position; wherein the column number corresponding to the current to-be-processed element position is greater than the column number corresponding to the current first element position, the time length between the timestamp of the to-be-detected video frame corresponding to the current to-be-processed element position and the timestamp of the to-be-detected video frame corresponding to the current first element position is less than a first time length, and the time length between the timestamp of the to-be-detected video frame corresponding to the current to-be-processed element position and the timestamp of the to-be-detected video frame corresponding to the current first element position is less than a second time length; a first length value determining module configured to determine a length value corresponding to the current to-be-processed element position as a first length value of the current to-be-processed element position; wherein the first length value of the current to-be-processed element position is determined based on a second length value corresponding to the current first element position; the second length value corresponding to the current first element position represents a sum of the first length value of the current first element position and 1; the starting sequence number corresponding to the current first element position is determined based on the starting sequence number corresponding to each element position located before the current to-be-processed element position when the element position corresponding to the current first element position is the current to-be-processed element position; the starting sequence number corresponding to the first element position in the position index list is 0.The traversal module is configured to determine a next element position in the position index list as a current to-be-processed element position, and trigger the to-be-processed element position determination module until a first length value corresponding to a last element position in the position index list is obtained; the terminal element position determination module is configured to determine an element position corresponding to a maximum first length value as a terminal element position; the starting element position determination module is configured to determine an element position corresponding to a starting serial number of the terminal element position in the position index list as a starting element position; the terminal video frame determination module is configured to determine a to-be-detected video frame corresponding to the terminal element position as a first terminal video frame, and determine a target video frame corresponding to the terminal element position as a second terminal video frame; the starting video frame determination module is configured to determine a to-be-detected video frame corresponding to the starting element position as a first starting video frame, and determine a target video frame corresponding to the starting element position as a second starting video frame; and the video segment determination module is configured to determine a first video segment in the to-be-detected video, which is bounded by the first starting video frame and the first terminal video frame, to be consistent with a second video segment in the target video, which is bounded by the second starting video frame and the second terminal video frame.

[0271] In some embodiments, the first length value determination module comprises: a first determination sub-module configured to, if the current first element position is multiple, determine a maximum value among second length values corresponding to the current first element positions as a first length value of the current to-be-processed element position; and a second determination sub-module configured to, if the current first element position is one, determine a second length value corresponding to the current first element position as a first length value of the current to-be-processed element position.

[0272] In some embodiments, the apparatus further includes: a first matching type determination module, configured to, after determining a target video matching the video to be detected from a preset set of videos to be matched based on the determined similarities, if the target video is a single video and the video to be detected is consistent with a video segment in the target video, determine that the matching type between the video to be detected and the target video is segment matching; a second matching type determination module, configured to, if the target video is a single video, and multiple first video segments in the video to be detected are respectively consistent with multiple second video segments in the target video, and the order of the multiple first video segments in the video to be detected is consistent with the order of the multiple second video segments in the target video, determine that the matching type between the video to be detected and the target video is single video sequence matching; A third matching type determination module is used to determine the matching type between the video to be detected and the target video as single-video unordered matching if the target video is a single video, and multiple first video segments in the video to be detected are respectively consistent with multiple second video segments in the target video, and the order of the multiple first video segments in the video to be detected is inconsistent with the order of the multiple second video segments in the target video; a fourth matching type determination module is used to determine the matching type between the video to be detected and the target video as multi-video unordered matching if the target videos are multiple and the multiple target videos belong to the same video set; a fifth matching type determination module is used to determine the matching type between the video to be detected and the target video as multi-set unordered matching if the target videos are multiple and the multiple target videos belong to different video sets.

[0273] In some embodiments, the video frames to be matched in the video to be matched include: video frames obtained by sampling the video to be matched based on a first sampling frequency; the video frames to be detected include: video frames obtained by sampling the video to be detected based on a second sampling frequency; or, the video frames to be detected include: sampled video frames obtained by sampling the video to be detected based on a second sampling frequency, and target keyframes in the video frames to be detected; wherein the temporal distance between the target keyframe and the nearest sampled video frame is greater than a second threshold.

[0274] This invention also provides an electronic device, such as... Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0275] Memory 1003 is used to store computer programs;

[0276] The processor 1001 is configured to implement the steps of the video detection method in any of the above embodiments when the processor 1001 executes the program stored in the memory 1003.

[0277] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface is used for communication between the terminal and other devices. The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the processor. The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0278] In another embodiment provided by the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the video detection method in any of the above embodiments.

[0279] In another embodiment provided by the present application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute the video detection method in any of the above embodiments.

[0280] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program can be stored in any computer readable medium, and loaded into the computer system for execution. The computer readable medium includes: a computer storage medium and a computer communication medium. The computer storage medium includes: volatile media (such as random access memory (RAM) and others) and non-volatile media (such as read-only memory (ROM), floppy disks, CD-ROMs, optical disks, hard disks, etc.). The computer communication medium includes: computer networks and other media.

[0281] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In addition, the terms "comprise", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or other elements inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0282] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, electronic device, computer readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0283] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A video detection method, characterized by, The method comprises: obtaining a video frame from a to-be-detected video as a to-be-detected video frame; extracting image features of the to-be-detected video frame to obtain a first feature map; processing the first feature map based on a mask corresponding to the first feature map to obtain a second feature map; the mask corresponding to the first feature map contains weights of elements in the first feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the first feature map are greater than the weights of elements located in an edge region; for each to-be-matched video, obtaining a feature map of a to-be-matched video frame in the to-be-matched video as a third feature map; the third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map; the fourth feature map is obtained by performing feature extraction on the to-be-matched video frame; the mask corresponding to the fourth feature map contains weights of elements in the fourth feature map; the weights of the elements conform to a Gaussian distribution, and the weights of elements located in a central region of the fourth feature map are greater than the weights of elements located in an edge region; for each to-be-detected video frame, calculating similarities between the to-be-detected video frame and each to-be-matched video frame based on the second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame; based on the determined similarities, determining target videos matched with the to-be-detected video from the preset to-be-matched videos; for each target video, generating a similarity matrix corresponding to the target video based on similarities between the to-be-detected video frame and a target video frame in the target video; a similarity element in the similarity matrix represents a similarity between a to-be-detected video frame corresponding to a row to which the similarity element belongs and a target video frame in the target video corresponding to a column to which the similarity element belongs; if the similarity between the to-be-detected video frame corresponding to the row to which the similarity element belongs and the target video frame corresponding to the column to which the similarity element belongs is less than a first threshold, the similarity element is invalid; in order of row number from low to high, for each row of similarity elements in the similarity matrix, in order of column number from high to low corresponding to the similarity elements, each element position of each similarity element that is not invalid in the similarity matrix is sequentially added to a preset list to obtain a position index list corresponding to the target video; determining a current processing element position in the order of element positions in the position index list; determining a current first element position from each element position before the current processing element position; the column number corresponding to the current processing element position is greater than the column number corresponding to the current first element position, the time length between the timestamp of the to-be-detected video frame corresponding to the current processing element position and the timestamp of the to-be-detected video frame corresponding to the current first element position is less than a first time length, and the time length between the timestamp of the to-be-matched video frame corresponding to the current processing element position and the timestamp of the to-be-matched video frame corresponding to the current first element position is less than a second time length; Determine the length value corresponding to the current to-be-processed element position as the first length value of the current to-be-processed element position; wherein the first length value of the current to-be-processed element position is determined based on the second length value corresponding to the current first element position; the second length value corresponding to the current first element position represents the sum of the first length value of the current first element position and 1; Determine the starting sequence number of the current to-be-processed element position based on the starting sequence number of the current first element position; wherein the starting sequence number of the current first element position is determined based on the starting sequence number of each element position located before the current to-be-processed element position when the current first element position is taken as the current to-be-processed element position; the starting sequence number of the first element position in the position index list is 0; Determine the next element position in the position index list as the current to-be-processed element position, and return to execute the step of determining the current first element position from each element position located before the current to-be-processed element position until the first length value corresponding to the last element position in the position index list is obtained; Determine the element position corresponding to the first length value with the largest value as the terminal element position; Determine the element position corresponding to the starting sequence number of the terminal element position in the position index list as the starting element position; Determine the to-be-detected video frame corresponding to the terminal element position as the first terminal video frame, and determine the target video frame corresponding to the terminal element position as the second terminal video frame; Determine the to-be-detected video frame corresponding to the starting element position as the first starting video frame, and determine the target video frame corresponding to the starting element position as the second starting video frame; Determine the first video segment in the to-be-detected video with the first starting video frame and the first terminal video frame as boundaries to be consistent with the second video segment in the target video with the second starting video frame and the second terminal video frame as boundaries.

2. The method of claim 1, wherein, The extracting the image features of the to-be-detected video frames to obtain the first feature map comprises: For each to-be-detected video frame, input the to-be-detected video frame into the convolutional neural network; Respectively acquire the feature maps output by each of a plurality of specified network layers in the convolutional neural network to obtain the first feature map output by each specified network layer; The processing the first feature map based on the mask corresponding to the first feature map to obtain the second feature map comprises: For each specified network layer, process the first feature map output by the specified network layer based on the mask corresponding to the specified network layer to obtain the fifth feature map corresponding to the specified network layer; Combine the fifth feature maps corresponding to each of the specified network layers to obtain the second feature map.

3. The method of claim 2, wherein, The processing the first feature map output by each specified network layer based on the mask corresponding to the specified network layer to obtain the fifth feature map corresponding to the specified network layer comprises: For each first feature map output by each specified network layer, process the first feature map based on the mask corresponding to the specified network layer to obtain the sixth feature map; For each specified network layer, based on a region maximum convolutional activation (R-MAC) algorithm, each sixth feature map corresponding to the specified network layer is processed to obtain an aggregated feature vector corresponding to the specified network layer as a fifth feature map.

4. The method of claim 3, wherein, Before the processing of each first feature map output by each specified network layer based on a mask corresponding to the specified network layer to obtain a sixth feature map, the method further comprises: determining a target network layer from the plurality of specified network layers; generating a mask corresponding to the target network layer based on a preset expectation, a preset variance, and a dimension of a first feature map output by the target network layer; generating masks corresponding to other specified network layers except the target network layer based on interpolation operation on the mask corresponding to the target network layer.

5. The method of claim 2, wherein, The combination of the fifth feature map corresponding to each specified network layer to obtain a second feature map comprises: respectively performing binaryzation processing on the fifth feature map corresponding to each specified network layer to obtain a binaryzation processing result; splicing the binaryzation processing results to obtain the second feature map.

6. The method of claim 5, wherein, The binaryzation processing on the fifth feature map corresponding to each specified network layer to obtain a binaryzation processing result comprises: inputting the fifth feature map corresponding to each specified network layer as input data into each fully connected layer included in an input layer of a pre-trained unique maximum likelihood hashing (JMLH) model to obtain an output result of the fully connected layer as the binaryzation processing result.

7. The method of claim 1, wherein, The determination of a target video matching the to-be-detected video from the preset to-be-matched videos based on the determined similarities comprises: for each to-be-detected video frame, determining a video frame matching the to-be-detected video frame from to-be-matched video frames included in the preset to-be-matched videos based on a similarity corresponding to the to-be-detected video frame as a target video frame; respectively determining a to-be-matched video to which each target video frame belongs as a target video.

8. The method of claim 7, wherein, The similarity matrix comprises: each row of the similarity matrix corresponds to one to-be-detected video frame, and the order between the rows is determined according to the order of the corresponding to-be-detected video frame in the to-be-detected video; and each column of the similarity matrix corresponds to one target video frame in the target video, and the order between the columns is determined according to the order of the corresponding target video frame in the target video.

9. The method of claim 1, wherein, The determination of a length value corresponding to the current to-be-processed element position as a first length value of the current to-be-processed element position comprises: if the current first element position is multiple, determining a maximum value of the second length values corresponding to the current first element positions as the first length value of the current to-be-processed element position; if the current first element position is one, determining the second length value corresponding to the current first element position as the first length value of the current to-be-processed element position.

10. The method of claim 1, wherein, After the determination of the target video matching the to-be-detected video from the preset to-be-matched videos based on the determined similarities, the method further comprises: If the target video is one, and the to-be-detected video is consistent with one video segment in the target video, it is determined that the matching type of the to-be-detected video and the target video is segment matching. If the target video is one, a plurality of first video segments in the to-be-detected video are respectively consistent with a plurality of second video segments in the target video, and the order of the plurality of first video segments in the to-be-detected video is consistent with the order of the plurality of second video segments in the target video, it is determined that the matching type of the to-be-detected video and the target video is single-video sequential matching. If the target video is one, a plurality of first video segments in the to-be-detected video are respectively consistent with a plurality of second video segments in the target video, and the order of the plurality of first video segments in the to-be-detected video is inconsistent with the order of the plurality of second video segments in the target video, it is determined that the matching type of the to-be-detected video and the target video is single-video non-sequential matching. If the target video is a plurality, and the plurality of target videos belong to the same video set, it is determined that the matching type of the to-be-detected video and the target video is multi-video non-sequential matching. If the target video is a plurality, and the plurality of target videos belong to different video sets, it is determined that the matching type of the to-be-detected video and the target video is multi-set non-sequential matching.

11. The method of claim 1, wherein, The to-be-matched video frame in the to-be-matched video includes a video frame obtained by sampling the to-be-matched video based on a first sampling frequency. The to-be-detected video frame includes a video frame obtained by sampling the to-be-detected video based on a second sampling frequency, or the to-be-detected video frame includes a sampling video frame obtained by sampling the to-be-detected video based on a second sampling frequency and a target key frame in the to-be-detected video frame; wherein the time sequence distance between the target key frame and the nearest sampling video frame is greater than a second threshold.

12. A video detection apparatus characterized by comprising: The device comprises: A to-be-detected video frame acquisition module for acquiring a video frame from a to-be-detected video as a to-be-detected video frame; A first feature map acquisition module for extracting image features of the to-be-detected video frame to obtain a first feature map; A second feature map acquisition module for processing the first feature map based on a mask corresponding to the first feature map to obtain a second feature map; wherein the mask corresponding to the first feature map contains weights of elements in the first feature map; the weights of the elements conform to a Gaussian distribution, and the weight of an element located in a central region of the first feature map is greater than the weight of an element located in an edge region of the first feature map; The third feature map acquisition module is configured to acquire a feature map of a to-be-matched video frame in each to-be-matched video as a third feature map, for each to-be-matched video; the third feature map is obtained by processing a fourth feature map based on a mask corresponding to the fourth feature map; the fourth feature map is obtained by performing feature extraction on the to-be-matched video frame; the mask corresponding to the fourth feature map contains weights of elements in the fourth feature map; the weights of the elements conform to a Gaussian distribution, and a weight of an element located in a central region of the fourth feature map is greater than a weight of an element located in an edge region of the fourth feature map; The similarity calculation module is configured to calculate similarities between the to-be-detected video frame and each to-be-matched video frame, for each to-be-detected video frame, based on a second feature map corresponding to the to-be-detected video frame and the third feature map corresponding to each to-be-matched video frame; The target video determination module is configured to determine a target video matched with the to-be-detected video from the preset to-be-matched videos, based on the determined similarities; The similarity matrix generation module is configured to generate a similarity matrix corresponding to each target video, based on similarities between the to-be-detected video frame and a target video frame in the target video; a similarity element in the similarity matrix represents a similarity between the to-be-detected video frame corresponding to a row to which the similarity element belongs and the target video frame in the target video corresponding to a column to which the similarity element belongs; if the similarity between the to-be-detected video frame corresponding to the row to which the similarity element belongs and the target video frame corresponding to the column to which the similarity element belongs is less than a first threshold, the similarity element is invalid; The position index list acquisition module is configured to add, in order, element positions of each non-invalid similarity element in the similarity matrix to a preset list, to obtain a position index list corresponding to the target video, according to a column number corresponding to a similarity element from high to low, for each similarity element in each row in the similarity matrix, in a sequence from low to high according to a row number. The to-be-processed element position determination module is configured to determine a current first element position from each element position located before the current to-be-processed element position; the column number corresponding to the current to-be-processed element position is greater than the column number corresponding to the current first element position, a time length between a timestamp of the to-be-detected video frame corresponding to the current to-be-processed element position and a timestamp of the to-be-detected video frame corresponding to the current first element position is less than a first time length, and a time length between a timestamp of the to-be-matched video frame corresponding to the current to-be-processed element position and a timestamp of the to-be-matched video frame corresponding to the current first element position is less than a second time length. ​ The first length value determination module is configured to determine a length value corresponding to a current to-be-processed element position as a first length value of the current to-be-processed element position, wherein the first length value of the current to-be-processed element position is determined based on a second length value corresponding to a current first element position; and the second length value corresponding to the current first element position represents a sum of the first length value of the current first element position and 1. The start sequence number determination module is configured to determine a start sequence number of the current to-be-processed element position based on a start sequence number of the current first element position, wherein the start sequence number of the current first element position is determined based on start sequence numbers of each element position located before the current to-be-processed element position when the current first element position is taken as the current to-be-processed element position; and the start sequence number of the first element position in the position index list is 0. The traversal module is configured to determine a next element position in the position index list as the current to-be-processed element position, and trigger the to-be-processed element position determination module until a first length value corresponding to a last element position in the position index list is obtained. The terminal element position determination module is configured to determine an element position corresponding to a largest numerical value among the first length values as a terminal element position. The start element position determination module is configured to determine an element position corresponding to the start sequence number of the terminal element position in the position index list as a start element position. The terminal video frame determination module is configured to determine a to-be-detected video frame corresponding to the terminal element position as a first terminal video frame, and determine a target video frame corresponding to the terminal element position as a second terminal video frame. The start video frame determination module is configured to determine a to-be-detected video frame corresponding to the start element position as a first start video frame, and determine a target video frame corresponding to the start element position as a second start video frame. The video segment determination module is configured to determine a first video segment in the to-be-detected video, which is bounded by the first start video frame and the first terminal video frame, to be consistent with a second video segment in the target video, which is bounded by the second start video frame and the second terminal video frame.

13. An electronic device, comprising: The apparatus includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method steps of any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer program is stored in the computer-readable storage medium, and the computer program is executed by the processor to implement the method steps of any one of claims 1-11.

Images