Feature encoder training method, device and system and homologous video query method

Abstract

This application discloses a feature encoder training method, comprising: obtaining a video to be processed; selecting two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments including a first video segment and a second video segment; performing image editing processing on the second video segment using a pre-set image editing method to obtain a second target video segment similar to the second video segment after editing; inputting all frames contained in the first video segment and the second target video segment into a frame feature encoder to obtain a frame feature vector; and training the frame feature encoder based on the frame feature vector to obtain a target frame feature encoder. This method addresses the problems of insufficient robustness of existing feature encoders to complex image editing methods and limitations on retrieval performance.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a feature encoder training method, apparatus, electronic device, and storage device. This application also relates to a method and apparatus for querying source-same-source videos. This application further relates to a feature encoder training system. Background Technology

[0002] Currently, video files are frequently copied and shared on the internet, potentially resulting in a large number of plagiarized video clips. To protect the copyright of video files, it's necessary to identify which video files or clips infringe on copyright. When searching for plagiarized / copied video clips within a query video, video fingerprinting (i.e., the feature vector of a video file or clip) technology is typically used. Video fingerprinting is an emerging video identification technology that generates a unique "fingerprint" representing a video file or clip. To determine plagiarized / copied video clips from a query database based on video fingerprints, the videos need to be feature-encoded to generate both the video fingerprint of the video to be retrieved and the video fingerprint of the query video.

[0003] In current technologies, when generating video fingerprints through feature encoding, the encoders typically used are traditional DCT (Discrete Cosine Transform) encoders or ImageNet Pretrained Models. While these encoders are generally versatile, they don't consider specific requirements in video fingerprinting scenarios. For example, new images generated by editing images using certain image editing methods (such as scaling, cropping, picture-in-picture, etc.) are still considered plagiarized / copied images. These new images should have semantic information as similar as possible to the original image. However, these two encoders are not robust enough to these image editing methods. Therefore, when searching for plagiarized / copied images based on the original image, images edited using complex manual methods are not easily retrieved, thus limiting the final retrieval performance. Summary of the Invention

[0004] This application provides a feature encoder training method, apparatus, electronic device, and storage device to solve the problems of insufficient robustness of existing feature encoders to complex image editing methods and limitations on retrieval performance.

[0005] Firstly, this application provides a method for training a feature encoder, comprising:

[0006] Obtain the video to be processed;

[0007] Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment;

[0008] The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0009] Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0010] The frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0011] Optionally, also include:

[0012] All frame feature vectors are segmented to obtain the frame feature vectors contained in each segment;

[0013] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector;

[0014] A segment feature encoder is trained based on the segment feature vector to obtain the target segment feature encoder.

[0015] Optionally, the step of segmenting all frame feature vectors to obtain the frame feature vectors contained in each segment includes:

[0016] All frame feature vectors are segmented according to a preset time interval to obtain the frame feature vectors contained in each segment.

[0017] Optionally, training a frame feature encoder based on the frame feature vector to obtain a target frame feature encoder includes:

[0018] Take the frame feature vectors of the second target video segment and the frames with the same temporal location in the first video segment of any video to be processed as positive sample pairs;

[0019] The frame feature vectors corresponding to frames from other videos besides the video to be processed are used as negative samples for the calculation of contrastive loss and backpropagation to obtain the target frame feature encoder.

[0020] Optionally, training a segment feature encoder based on the segment feature vector to obtain a target segment feature encoder includes:

[0021] Use the segment feature vectors corresponding to the second target video segment and the first video segment of any video to be processed as positive sample pairs;

[0022] The segment feature vectors corresponding to video segments from other videos besides the video to be processed are used as negative samples for contrastive loss calculation and backpropagation to obtain the target segment feature encoder.

[0023] Optionally, the preset image editing methods include at least one of four editing methods: rotation editing, flip editing, cropping editing, and panning editing; and / or at least one of four editing methods: brightness editing, saturation editing, contrast editing, and hue editing.

[0024] This application also provides a feature encoder training apparatus, comprising:

[0025] The video acquisition unit is used to acquire the video to be processed.

[0026] A video segment selection unit is used to select two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment.

[0027] An image editing processing unit is used to perform image editing processing on the second video segment using a preset image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0028] The frame feature vector acquisition unit is used to input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0029] The frame feature encoder training unit is used to train the frame feature encoder based on the frame feature vector to obtain the target frame feature encoder.

[0030] This application also provides an electronic device, including:

[0031] Processor; and

[0032] The memory stores the program for the feature encoder training method. After the device is powered on and the program for the feature encoder training method is run by the processor, the following steps are performed:

[0033] Obtain the video to be processed;

[0034] Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment;

[0035] The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0036] Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0037] The frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0038] This application also provides a storage device storing a program for a feature encoder training method, which is executed by a processor to perform the following steps:

[0039] Obtain the video to be processed;

[0040] Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment;

[0041] The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0042] Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0043] The frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0044] This application also provides a method for querying videos from the same source, including:

[0045] Input all frames contained in the query video segment into the frame feature encoder to obtain the frame feature vector of the query video;

[0046] Based on the frame feature vector of the query video, determine the videos in the search video that are from the same source as the query video;

[0047] Wherein, the frame feature encoder is any of the frame feature encoders described in the feature encoder training method of this application.

[0048] Optionally, determining the videos from the same source as the query video in the searchable video based on the frame feature vector of the query video includes:

[0049] Based on the frame feature vector of the query video, generate the segment feature vector corresponding to the query video;

[0050] The feature vector of the segment corresponding to the query video is input into the vector search engine for retrieval, and video segments similar to the query video segment are retrieved; wherein, the index of the vector search engine is constructed based on the feature vector of the segment corresponding to the video segment to be retrieved.

[0051] Based on the video segments to be retrieved that are similar to the query video segment, identify the videos in the video segment to be retrieved that are from the same source as the query video.

[0052] Optionally, the vector search engine's index is created according to the following steps:

[0053] Input all frames contained in the search video segment of the video to be searched into the frame feature encoder to obtain the frame feature vector of the video to be searched.

[0054] Based on the frame feature vector of the video to be retrieved, generate the segment feature vector corresponding to the video to be retrieved;

[0055] An index for a vector search engine is constructed based on the feature vectors of the segments corresponding to the video to be retrieved.

[0056] Optionally, also include:

[0057] The video to be retrieved is sampled at preset intervals to obtain the retrieved video segments;

[0058] The query video is sampled at preset intervals to obtain the query video segments.

[0059] Optionally, also include:

[0060] The video to be retrieved is subjected to shot boundary detection and in-shot sampling to obtain the retrieved video segment;

[0061] The query video is processed by lens boundary detection and in-lens sampling to obtain the query video segment.

[0062] Optionally, generating the segment feature vector corresponding to the video to be retrieved based on the frame feature vector of the video to be retrieved includes:

[0063] The frame feature vector of the video to be retrieved is segmented to obtain the frame feature vector contained in each segment;

[0064] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector corresponding to the video to be retrieved; wherein, the segment feature encoder is any of the segment feature encoders described in claims 2-6.

[0065] Optionally, the step of inputting the feature vector of the segment corresponding to the query video into a vector search engine for retrieval to recall video segments similar to the query video segment includes:

[0066] The segment feature vector corresponding to the query video is input into the vector search engine for retrieval, and the distance between the segment feature vector corresponding to the query video and the segment feature vector of the video to be retrieved is obtained.

[0067] When the distance is less than a preset distance threshold, the video segment to be retrieved corresponding to the segment feature vector of the video segment to be retrieved at the distance is recalled.

[0068] Optionally, it also includes: locating the start and end positions of plagiarized segments in videos that are from the same source as the query video, based on video segments to be retrieved that are similar to the query video segment.

[0069] This application also provides a device for querying videos from the same source, characterized in that it includes:

[0070] The frame feature vector acquisition unit of the query video is used to input all the frames contained in the query video segment of the query video into the frame feature encoder to obtain the frame feature vector of the query video.

[0071] The same-origin video determination unit is used to determine, based on the frame feature vector of the query video, a video in the search video that is from the same origin as the query video;

[0072] Wherein, the frame feature encoder is any of the frame feature encoders described in the feature encoder training method of this application.

[0073] This application also provides a feature encoder training system, including: a platform server and a user terminal;

[0074] The platform server is configured to: receive videos to be processed sent by user terminals; select two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment; perform image editing processing on the second video segment using a pre-set image editing method to obtain a second target video segment similar to the second video segment after editing; input all frames contained in the first video segment and the second target video segment into a frame feature encoder to obtain frame feature vectors; train the frame feature encoder based on the frame feature vectors to obtain a target frame feature encoder; and send the target frame feature encoder to the user terminal.

[0075] The user terminal is used to send the video to be processed to the platform server and to obtain the target frame feature encoder sent by the platform server.

[0076] Compared with the prior art, this application has the following advantages:

[0077] This application provides a feature encoder training method, comprising: obtaining a video to be processed; selecting two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments including a first video segment and a second video segment; performing image editing processing on the second video segment using a pre-set image editing method to obtain a second target video segment similar to the second video segment after editing; inputting all frames contained in the first video segment and the second target video segment into a frame feature encoder to obtain a frame feature vector; and training the frame feature encoder based on the frame feature vector to obtain a target frame feature encoder. The feature encoder training method provided in this application, by performing image editing processing on the second video segment using a pre-set image editing method to obtain a second target video segment, achieves automatic editing of training samples to generate corresponding plagiarism samples (positive samples). Then, by using these samples to train the frame feature encoder, the frame feature encoder learns to reduce the impact of these image editing methods on the final encoding, thereby greatly enhancing robustness to manual editing methods and improving the final retrieval performance. Attached Figure Description

[0078] Figure 1a This is a scenario illustration provided in this application.

[0079] Figure 1 This is a flowchart of a feature encoder training method provided in the first embodiment of this application.

[0080] Figure 2 This is a schematic diagram of model training corresponding to a feature encoder training method provided in the first embodiment of this application.

[0081] Figure 3 This is a schematic diagram of a feature encoder training device provided in the second embodiment of this application.

[0082] Figure 4 This is a flowchart of a method for querying videos from the same source, provided in the fifth embodiment of this application.

[0083] Figure 5 This is a query diagram corresponding to a same-source video query method provided in the fifth embodiment of this application.

[0084] Figure 6 This is a schematic diagram of a same-source video query device provided in the sixth embodiment of this application.

[0085] Figure 7 This is a schematic diagram of a feature encoder training system provided in the seventh embodiment of this application. Detailed Implementation

[0086] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0087] To enable those skilled in the art to better understand the solution of this application, a specific application scenario embodiment of this application will be described in detail first.

[0088] The embodiments of this application provide a method for querying videos from the same source, which can be applied to scenarios involving interaction between a client and a server, such as... Figure 1a When searching for videos from the same source as the query video within the searched video, the client typically establishes a connection with the server first. After the connection is established, the client sends the query video to the server. Upon receiving the query video, the server first retrieves the query video segment (...). Figure 1a The frame feature encoder takes all frames from segment 1 to segment N in the query video as input to obtain the frame feature vector of the query video; then, based on the frame feature vector of the query video, it generates the segment feature vector corresponding to the query video. Figure 1a The process involves several steps: First, the segment feature vector corresponding to the query video is input into a vector search engine for retrieval, recalling similar video segments (recalling segments 1 to N). The vector search engine's index is constructed based on the segment feature vector corresponding to the video segment being retrieved. Then, based on the similar video segments, post-processing is used to determine videos originating from the same source as the query video. Finally, the server sends these videos to the client. The frame feature encoder is obtained through the following method:

[0089] Obtain the video to be processed;

[0090] Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment;

[0091] The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0092] Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0093] The frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0094] The first embodiment of this application provides a feature encoder training method, which is described below in conjunction with... Figure 1 , Figure 2 Let me introduce it.

[0095] like Figure 1 As shown, in step S101, the video to be processed is obtained.

[0096] The video to be processed refers to the video used to generate similar video clips. The similar video clips refer to two video clips, one of which is the original video clip, and the other is a video clip after image editing of the original video clip.

[0097] like Figure 2 Video 1 and Video 2 are the videos to be processed. Figure 2 This is just a simplified illustration; in actual processing, the number of videos to be processed is far greater than this figure.

[0098] like Figure 1 As shown, in step S102, two video segments are selected from at least one video to be processed, the two video segments at least partially overlap on the timeline; the two video segments include a first video segment and a second video segment.

[0099] The purpose of selecting two video segments from the video to be processed is to use the first video segment as the original video segment and the second video segment as a source of similar video segments from the first video segment.

[0100] like Figure 2 As shown, segments 1 and 3 are the first video segments, and segments 2 and 4 are the second video segments.

[0101] like Figure 1 As shown, in step S103, the second video segment is processed by a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0102] The preset image editing methods include at least one of four editing methods: rotation editing, flip editing, cropping editing, and panning editing; and / or at least one of four editing methods: brightness editing, saturation editing, contrast editing, and hue editing.

[0103] The second video segment is processed using a pre-set image editing method to obtain the second target video segment. Since the second target video segment is a similar video segment to the first video segment, the image frames in the second target video segment have similar semantic information to the corresponding image frames in the first video segment.

[0104] like Figure 1As shown, in step S104, all frames contained in the first video segment and the second target video segment are input into the frame feature encoder to obtain the frame feature vector.

[0105] For example, if the first video segment is 30 frames, inputting all the frames contained in the first video segment into the frame feature encoder will yield a frame feature vector of 30 frames. The second target video segment is also 30 frames, and inputting all the frames contained in the second target video segment into the frame feature encoder will also yield a frame feature vector of 30 frames.

[0106] Specifically, when inputting all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector, a ResNet50 (Residual Network) model + PCA (principal component analysis) layer can be used as the frame feature encoder, and the parameters of the frame feature encoder can be initialized using the ImageNet Pretrained Model.

[0107] like Figure 1 As shown, in step S105, a frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0108] The step of training a frame feature encoder based on the frame feature vector to obtain a target frame feature encoder includes:

[0109] Take the frame feature vectors of the second target video segment and the frames with the same temporal location in the first video segment of any video to be processed as positive sample pairs;

[0110] The frame feature vectors corresponding to frames from other videos besides the video to be processed are used as negative samples for the calculation of contrastive loss and backpropagation to obtain the target frame feature encoder.

[0111] For example, if the second target video segment consists of image frames from 1 to 30 seconds, and the first video segment consists of image frames from 1 to 30 seconds, then the frame feature vector of the first second image frame in the second target video segment and the frame feature vector of the first second image frame in the first target video segment are a positive sample pair.

[0112] The target frame feature encoder trained using the above method can output similar frame feature vectors for video frames (including unedited and edited video frames) that are from the same source as the input target frame feature encoder.

[0113] To improve efficiency and save storage space when using video fingerprints for retrieval, the first embodiment of this application can also train a segment feature encoder to generate segment feature vectors. Storing segment feature vectors in the retrieval database containing the videos to be retrieved can save storage space, and using segment feature vectors for retrieval can also improve retrieval efficiency.

[0114] As one implementation method, the first embodiment of this application may further include:

[0115] All frame feature vectors are segmented to obtain the frame feature vectors contained in each segment;

[0116] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector;

[0117] A segment feature encoder is trained based on the segment feature vector to obtain the target segment feature encoder.

[0118] Specifically, when inputting the frame feature vector contained in each segment into the segment feature encoder for processing to obtain the segment feature vector, a single-layer unidirectional GRU (Gated Recurrent Unit) model can be used as the segment feature encoder, and randomly initialized.

[0119] It's worth noting that compared to randomly selecting a frame feature vector from the frame feature vectors contained in a segment and taking the average of the frame feature vectors contained in the segment (i.e., downsampling and average pooling) as the segment's fragment feature vector, using a segment feature encoder to obtain the segment's fragment feature vector, and fusing information from multiple frames under the same encoding interval through an RNN, can significantly reduce information loss. For example, in short-to-long news search scenarios, the RNN (Recurrent Neural Network)-based model improves mAP by 1.7% compared to the downsampling approach and by 0.6% compared to the average pooling approach.

[0120] The process of segmenting all frame feature vectors to obtain the frame feature vectors contained in each segment includes:

[0121] All frame feature vectors are segmented according to a preset time interval to obtain the frame feature vectors contained in each segment.

[0122] The segment feature vector refers to the feature vector corresponding to a video segment.

[0123] For example, if the frame feature vector is the frame feature vector corresponding to image frames from 1 to 30 seconds, and if one second corresponds to one image frame, and the preset time interval is 5 seconds, then the frame feature vector can be segmented according to the 5-second time interval to obtain the frame feature vector contained in the segment. That is, the first segment contains the 1st, 2nd, 3rd, 4th, and 5th frames, and the frame feature vector of the first segment is the frame feature vector of the above 5 frames. The frame feature vector of the above 5 frames is input into the segment feature encoder for processing to obtain the segment feature vector of the first segment. Similarly, the segment feature vectors of the second segment to the sixth segment can be obtained. That is, the frame feature vector of 30 frames generates 6 segment feature vectors.

[0124] The step of training a segment feature encoder based on the segment feature vector to obtain a target segment feature encoder includes:

[0125] Use the segment feature vectors corresponding to the second target video segment and the first video segment of any video to be processed as positive sample pairs;

[0126] The segment feature vectors corresponding to video segments from other videos besides the video to be processed are used as negative samples for contrastive loss calculation and backpropagation to obtain the target segment feature encoder.

[0127] For example, the feature vector of the first segment of the second target video clip and the feature vector of the first segment of the first video clip form a positive sample pair.

[0128] This concludes the introduction of the first embodiment of this application. The feature encoder training method provided in the first embodiment of this application obtains a second target video segment by performing image editing processing on a second video segment using a pre-set image editing method. This achieves automatic editing of training samples to generate corresponding plagiarism samples (positive samples). Then, by using these samples to train a frame feature encoder, the frame feature encoder learns to reduce the impact of these image editing methods on the final encoding, thereby greatly enhancing robustness to manual editing methods and improving the final retrieval performance. In addition, in a preferred embodiment, training a segment feature encoder and using the segment feature encoder to generate segment feature vectors, storing the segment feature vectors in the retrieval base database of the video to be retrieved, can save storage space. Moreover, using segment feature vectors for retrieval can also improve retrieval efficiency. At the same time, using the segment feature encoder to obtain the segment feature vectors, and fusing information from multiple frames under the same encoding interval through RNN, can significantly reduce information loss.

[0129] Corresponding to the feature encoder training method provided in the first embodiment of this application, the second embodiment of this application provides a feature encoder training device.

[0130] like Figure 3 As shown, the feature encoder training device includes:

[0131] The video acquisition unit 301 is used to acquire the video to be processed.

[0132] The video segment selection unit 302 is configured to select two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment.

[0133] The image editing processing unit 303 is used to perform image editing processing on the second video segment using a preset image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0134] The frame feature vector acquisition unit 304 is used to input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0135] The frame feature encoder training unit 305 is used to train the frame feature encoder based on the frame feature vector to obtain the target frame feature encoder.

[0136] Optionally, the device further includes:

[0137] The segmentation processing unit is used to segment all frame feature vectors to obtain the frame feature vectors contained in the segment.

[0138] The segment feature vector acquisition unit is used to input the frame feature vector contained in each segment into the segment feature encoder for processing, and obtain the segment feature vector;

[0139] The fragment feature encoder training unit is used to train the fragment feature encoder based on the fragment feature vector to obtain the target fragment feature encoder.

[0140] Optionally, the segmentation processing unit is specifically used for:

[0141] All frame feature vectors are segmented according to a preset time interval to obtain the frame feature vectors contained in each segment.

[0142] Optionally, the frame feature encoder training unit is specifically used for:

[0143] Take the frame feature vectors of the second target video segment and the frames with the same temporal location in the first video segment of any video to be processed as positive sample pairs;

[0144] The frame feature vectors corresponding to frames from other videos besides the video to be processed are used as negative samples for the calculation of contrastive loss and backpropagation to obtain the target frame feature encoder.

[0145] Optionally, the fragment feature encoder training unit is specifically used for:

[0146] Use the segment feature vectors corresponding to the second target video segment and the first video segment of any video to be processed as positive sample pairs;

[0147] The segment feature vectors corresponding to video segments from other videos besides the video to be processed are used as negative samples for contrastive loss calculation and backpropagation to obtain the target segment feature encoder.

[0148] Optionally, the preset image editing methods include at least one of four editing methods: rotation editing, flip editing, cropping editing, and panning editing; and / or at least one of four editing methods: brightness editing, saturation editing, contrast editing, and hue editing.

[0149] It should be noted that for a detailed description of the apparatus provided in the second embodiment of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0150] Corresponding to the feature encoder training method provided in the first embodiment of this application, the third embodiment of this application provides an electronic device, including:

[0151] Processor; and

[0152] The memory stores the program for the feature encoder training method. After the device is powered on and the program for the feature encoder training method is run by the processor, the following steps are performed:

[0153] Obtain the video to be processed;

[0154] Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment;

[0155] The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0156] Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0157] The frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0158] Optionally, the electronic device also performs the following steps:

[0159] All frame feature vectors are segmented to obtain the frame feature vectors contained in each segment;

[0160] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector;

[0161] A segment feature encoder is trained based on the segment feature vector to obtain the target segment feature encoder.

[0162] Optionally, the step of segmenting all frame feature vectors to obtain the frame feature vectors contained in each segment includes:

[0163] All frame feature vectors are segmented according to a preset time interval to obtain the frame feature vectors contained in each segment.

[0164] Optionally, training a frame feature encoder based on the frame feature vector to obtain a target frame feature encoder includes:

[0165] Take the frame feature vectors of the second target video segment and the frames with the same temporal location in the first video segment of any video to be processed as positive sample pairs;

[0166] The frame feature vectors corresponding to frames from other videos besides the video to be processed are used as negative samples for the calculation of contrastive loss and backpropagation to obtain the target frame feature encoder.

[0167] Optionally, training a segment feature encoder based on the segment feature vector to obtain a target segment feature encoder includes:

[0168] Use the segment feature vectors corresponding to the second target video segment and the first video segment of any video to be processed as positive sample pairs;

[0169] The segment feature vectors corresponding to video segments from other videos besides the video to be processed are used as negative samples for contrastive loss calculation and backpropagation to obtain the target segment feature encoder.

[0170] Optionally, the preset image editing methods include at least one of four editing methods: rotation editing, flip editing, cropping editing, and panning editing; and / or at least one of four editing methods: brightness editing, saturation editing, contrast editing, and hue editing.

[0171] It should be noted that for a detailed description of the electronic device provided in the third embodiment of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0172] Corresponding to the feature encoder training method provided in the first embodiment of this application, the fourth embodiment of this application provides a storage device storing a program for the feature encoder training method. This program is executed by a processor to perform the following steps:

[0173] Obtain the video to be processed;

[0174] Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment;

[0175] The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing.

[0176] Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector;

[0177] The frame feature encoder is trained based on the frame feature vector to obtain the target frame feature encoder.

[0178] It should be noted that for a detailed description of the storage device provided in the fourth embodiment of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0179] The fifth embodiment of this application provides another method for querying videos from the same source, which is described below in conjunction with... Figure 4 , Figure 5 Let me introduce it.

[0180] like Figure 4 As shown, in step S401, all frames contained in the query video segment of the query video are input into the frame feature encoder to obtain the frame feature vector of the query video, wherein the frame feature encoder is the frame feature encoder trained in the first embodiment of this application.

[0181] For example, such as Figure 5 The query video is the query video, and segments 1, 2, ..., N are the query video segments. All frames contained in the query video segments are input into the frame feature encoder for frame feature encoding to obtain the frame feature vector of the query video.

[0182] As one implementation method, the fifth embodiment of this application may further include:

[0183] The query video is sampled at preset intervals to obtain the query video segments.

[0184] For example, if the query video consists of image frames ranging from 1 to 30 seconds, and each second corresponds to one image frame, with a preset interval of 5 seconds, then the image frames can be sampled at 5-second intervals to obtain the query video segments. That is, the first query video segment contains frames 1, 2, 3, 4, and 5; the second query video segment contains frames 6, 7, 8, 9, and 10; and so on, with the sixth query video segment containing frames 26, 27, 28, 29, and 30.

[0185] As one implementation method, the fifth embodiment of this application may further include:

[0186] The query video is processed by lens boundary detection and in-lens sampling to obtain the query video segment.

[0187] By performing shot boundary detection and in-shot sampling on the query video to obtain the query video segment, information loss during segment encoding can be further reduced.

[0188] like Figure 4 As shown, in step S402, the segment feature vector corresponding to the query video is generated based on the frame feature vector of the query video.

[0189] Specifically, generating the segment feature vector corresponding to the query video based on the frame feature vector of the query video includes:

[0190] The frame feature vector of the query video is segmented to obtain the frame feature vector contained in each segment;

[0191] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector corresponding to the query video; wherein, the segment feature encoder is any of the segment feature encoders described in the first embodiment of this application.

[0192] Specifically, the query video can be segmented based on its sampling interval to obtain the frame feature vectors contained in each segment. For example, if the sampling interval of the query video is 5 frames, then each segment will also contain the frame feature vectors of 5 frames.

[0193] like Figure 4 As shown, in step S403, the feature vector of the segment corresponding to the query video is input into the vector search engine for retrieval to recall video segments similar to the query video segment; wherein, the index of the vector search engine is constructed based on the feature vector of the segment corresponding to the video segment to be retrieved.

[0194] The index of a vector search engine can be created using the following steps:

[0195] Input all frames contained in the search video segment of the video to be searched into the frame feature encoder to obtain the frame feature vector of the video to be searched.

[0196] Based on the frame feature vector of the video to be retrieved, generate the segment feature vector corresponding to the video to be retrieved;

[0197] An index for a vector search engine is constructed based on the feature vectors of the segments corresponding to the video to be retrieved.

[0198] The step of generating a segment feature vector corresponding to the video to be retrieved based on the frame feature vector of the video to be retrieved includes:

[0199] The frame feature vector of the video to be retrieved is segmented to obtain the frame feature vector contained in each segment;

[0200] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector corresponding to the video to be retrieved; wherein, the segment feature encoder is any of the segment feature encoders described in the first embodiment of this application.

[0201] Specifically, building an index for a vector search engine based on the feature vectors of the segments corresponding to the video to be retrieved can refer to building an inverted index for a vector search engine based on the feature vectors of the segments corresponding to the video to be retrieved.

[0202] As one implementation method, the fifth embodiment of this application may further include:

[0203] The video to be retrieved is sampled at preset intervals to obtain the retrieved video segments.

[0204] It should be noted that the video to be retrieved is sampled at a preset interval, similar to the sampling process for the query video.

[0205] For example, if the sampling interval for the query video is 5 frames, then the sampling interval for the video to be retrieved is also 5 frames.

[0206] As one implementation method, the fifth embodiment of this application may further include:

[0207] The video to be retrieved is subjected to shot boundary detection and in-shot sampling to obtain the retrieved video segments.

[0208] By performing shot boundary detection and in-shot sampling on the video to be retrieved, the retrieved video segments can be obtained, which can further reduce information loss during segment encoding.

[0209] The step of inputting the feature vector of the segment corresponding to the query video into a vector search engine for retrieval, and recalling video segments similar to the query video segment, includes:

[0210] The segment feature vector corresponding to the query video is input into the vector search engine for retrieval, and the distance between the segment feature vector corresponding to the query video and the segment feature vector of the video to be retrieved is obtained.

[0211] When the distance is less than a preset distance threshold, the video segment to be retrieved corresponding to the segment feature vector of the video segment to be retrieved at the distance is recalled.

[0212] For example, if a query video corresponds to 6 segment feature vectors, these 6 segment feature vectors are input into a vector search engine for retrieval. The distance between the segment feature vector corresponding to each query video and the segment feature vector of the video to be retrieved is obtained. When the distance is less than a preset distance threshold, the video segment corresponding to the segment feature vector of the video to be retrieved is recalled. Figure 5 For the query video segment 1, recall segment 1, recall segment 2, ... recall segment N were retrieved.

[0213] like Figure 4 As shown, in step S404, based on the video segments to be retrieved that are similar to the query video segments, videos from the same source as the query video are determined.

[0214] As one implementation method, the fifth embodiment of this application further includes: locating the start and end positions of plagiarized segments in videos that are from the same source as the query video based on video segments to be retrieved that are similar to the query video segments.

[0215] This concludes the introduction of the fifth embodiment of this application. The same-source video query method provided in the fifth embodiment of this application obtains the second target video segment by performing image editing processing on the second video segment using a pre-set image editing method. This achieves automatic editing of training samples to generate corresponding plagiarism samples (positive samples). Then, by using these samples to train the frame feature encoder, the frame feature encoder learns to reduce the impact of these image editing methods on the final encoding, thereby greatly enhancing robustness to manual editing methods and improving the final retrieval performance. In addition, using the segment feature encoder to generate segment feature vectors and storing the segment feature vectors in the retrieval base database storing the videos to be retrieved can save storage space, and using segment feature vectors for retrieval can also improve retrieval efficiency. At the same time, using the segment feature encoder to obtain the segment feature vectors and fusing information from multiple frames under the same encoding interval through RNN can significantly reduce information loss and improve retrieval performance.

[0216] Corresponding to the same-source video query method provided in the fifth embodiment of this application, the sixth embodiment of this application provides a same-source video query device.

[0217] like Figure 6 As shown, the same-source video query device includes:

[0218] The frame feature vector acquisition unit 601 of the query video is used to input all the frames contained in the query video segment of the query video into the frame feature encoder to obtain the frame feature vector of the query video.

[0219] The same-origin video determination unit 602 is used to determine, based on the frame feature vector of the query video, a video in the search video that is from the same origin as the query video;

[0220] The frame feature encoder is the frame feature encoder described in the first embodiment of this application.

[0221] Optionally, the same-source video determination unit includes:

[0222] The query video segment feature vector generation subunit is used to generate the segment feature vector corresponding to the query video based on the frame feature vector of the query video.

[0223] The similar video segment recall subunit is used to input the segment feature vector corresponding to the query video into the vector search engine for retrieval, and recall video segments similar to the query video segment to be retrieved; wherein, the index of the vector search engine is constructed based on the segment feature vector corresponding to the video to be retrieved;

[0224] The same-source video determination subunit is used to determine videos that are from the same source as the query video based on video segments to be retrieved that are similar to the query video segment;

[0225] Optionally, the apparatus further includes an index creation unit for a vector search engine, used for

[0226] Input all frames contained in the search video segment of the video to be searched into the frame feature encoder to obtain the frame feature vector of the video to be searched.

[0227] Based on the frame feature vector of the video to be retrieved, generate the segment feature vector corresponding to the video to be retrieved;

[0228] An index for a vector search engine is constructed based on the feature vectors of the segments corresponding to the video to be retrieved.

[0229] Optionally, the device further includes:

[0230] The video segment retrieval unit is used to sample the video to be retrieved at preset intervals to obtain the retrieved video segment.

[0231] The query video segment acquisition unit is used to sample the query video at preset intervals to obtain the query video segment.

[0232] Optionally, the device further includes:

[0233] The retrieved video segment unit is used to perform shot boundary detection and in-shot sampling processing on the video to be retrieved, thereby obtaining the retrieved video segment.

[0234] The query video segment acquisition unit is used to perform shot boundary detection and in-shot sampling processing on the query video to obtain the query video segment.

[0235] Optionally, the apparatus further includes an index creation unit for a vector search engine, specifically used for:

[0236] The frame feature vector of the video to be retrieved is segmented to obtain the frame feature vector contained in each segment;

[0237] The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector corresponding to the video to be retrieved; wherein, the segment feature encoder is any of the segment feature encoders described in claims 2-6.

[0238] Optionally, the similar video clip recall subunit is specifically used for:

[0239] The segment feature vector corresponding to the query video is input into the vector search engine for retrieval, and the distance between the segment feature vector corresponding to the query video and the segment feature vector of the video to be retrieved is obtained.

[0240] When the distance is less than a preset distance threshold, the video segment to be retrieved corresponding to the segment feature vector of the video segment to be retrieved at the distance is recalled.

[0241] Optionally, the device further includes: a start and end position positioning unit.

[0242] This tool is used to locate the start and end positions of plagiarized segments in videos that share the same origin as the query video, based on video segments to be retrieved that are similar to the query video segment.

[0243] It should be noted that for a detailed description of the apparatus provided in the sixth embodiment of this application, please refer to the relevant description of the fifth embodiment of this application, which will not be repeated here.

[0244] Corresponding to the feature encoder training method provided in the first embodiment of this application, the seventh embodiment of this application also provides a feature encoder training system.

[0245] like Figure 7As shown, the feature encoder training system includes: a platform server 701 and a user terminal 702;

[0246] In the seventh embodiment of this application, the so-called platform server 701 refers to a computing device that provides services to the software platform or application platform installed on the user terminal 702. The software platform or application platform is used to execute the feature encoder training method provided in this application, and in specific implementations, it is generally a server or server cluster. The so-called user terminal 702 refers to a computing device that interacts with the platform server 701. The computing device has a software platform or application platform installed on it, and in specific implementations, it is generally a smartphone, tablet computer, personal computer, etc. After the user terminal 702 uploads the video to be processed to the platform server 701 through the software platform or application platform installed on it, the platform server 701 obtains the target frame feature encoder using the SaaS (Software-as-a-Service) working mode and provides it to the user terminal 702.

[0247] The platform server 701 is configured to: receive a video to be processed sent by a user terminal; select two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment; perform image editing processing on the second video segment using a pre-set image editing method to obtain a second target video segment similar to the second video segment after editing; input all frames contained in the first video segment and the second target video segment into a frame feature encoder to obtain a frame feature vector; train the frame feature encoder based on the frame feature vector to obtain a target frame feature encoder; and send the target frame feature encoder to the user terminal.

[0248] The user terminal 702 is used to send the video to be processed to the platform server and obtain the target frame feature encoder sent by the platform server.

[0249] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0250] In a typical configuration, a computing device includes one or more processors (CPUs), memory-mapped input / output interfaces, network interfaces, and memory.

[0251] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0252] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0253] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A method for training a feature encoder, characterized in that, include: Obtain the video to be processed; Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment; The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing. Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector; Train a frame feature encoder based on the frame feature vector to obtain the target frame feature encoder; The method further includes: encoding the video segment using a target segment feature encoder to obtain a segment feature vector of the target segment; and storing the segment feature vector of the target segment, wherein the segment feature vector of the target segment is used to retrieve the target segment.

2. The method according to claim 1, characterized in that, Also includes: All frame feature vectors are segmented to obtain the frame feature vectors contained in each segment; The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector; The segment feature encoder is trained based on the segment feature vector to obtain the target segment feature encoder.

3. The method according to claim 2, characterized in that, The process of segmenting all frame feature vectors to obtain the frame feature vectors contained in each segment includes: All frame feature vectors are segmented according to a preset time interval to obtain the frame feature vectors contained in each segment.

4. The method according to claim 1, characterized in that, The step of training a frame feature encoder based on the frame feature vector to obtain a target frame feature encoder includes: Take the frame feature vectors of the second target video segment and the frames with the same temporal location in the first video segment of any video to be processed as positive sample pairs; The frame feature vectors corresponding to frames from other videos besides the video to be processed are used as negative samples for the calculation of contrastive loss and backpropagation to obtain the target frame feature encoder.

5. The method according to claim 2, characterized in that, The step of training a segment feature encoder based on the segment feature vector to obtain a target segment feature encoder includes: Use the segment feature vectors corresponding to the second target video segment and the first video segment of any video to be processed as positive sample pairs; The segment feature vectors corresponding to video segments from other videos besides the video to be processed are used as negative samples for contrastive loss calculation and backpropagation to obtain the target segment feature encoder.

6. The method according to claim 1, characterized in that, The preset image editing methods include at least one of four editing methods: rotation editing, flip editing, cropping editing, and panning editing; and / or at least one of four editing methods: brightness editing, saturation editing, contrast editing, and hue editing.

7. A feature encoder training device, characterized in that, include: The video acquisition unit is used to acquire the video to be processed. A video segment selection unit is used to select two video segments from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment. An image editing processing unit is used to perform image editing processing on the second video segment using a preset image editing method to obtain a second target video segment that is similar to the second video segment after editing. The frame feature vector acquisition unit is used to input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector; A frame feature encoder training unit is used to train a frame feature encoder based on the frame feature vector to obtain a target frame feature encoder. The device is further configured to: encode the video segment using a target segment feature encoder to obtain a segment feature vector of the target segment; and store the segment feature vector of the target segment, wherein the segment feature vector of the target segment is used to retrieve the target segment.

8. An electronic device, characterized in that, include: processor; as well as The memory stores the program for the feature encoder training method. After the device is powered on and the program for the feature encoder training method is run by the processor, the following steps are performed: Obtain the video to be processed; Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment; The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing. Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector; Train a frame feature encoder based on the frame feature vector to obtain the target frame feature encoder; The video segment is encoded using a target segment feature encoder to obtain a segment feature vector of the target segment; the segment feature vector of the target segment is stored, wherein the segment feature vector of the target segment is used to retrieve the target segment.

9. A storage device, characterized in that, A program containing the training method for the feature encoder is executed by the processor, performing the following steps: Obtain the video to be processed; Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment; The second video segment is processed using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing. Input all frames contained in the first video segment and the second target video segment into the frame feature encoder to obtain the frame feature vector; Train a frame feature encoder based on the frame feature vector to obtain the target frame feature encoder; The video segment is encoded using a target segment feature encoder to obtain a segment feature vector of the target segment; the segment feature vector of the target segment is stored, wherein the segment feature vector of the target segment is used to retrieve the target segment.

10. A method for querying videos from the same source, characterized in that, include: Input all frames contained in the query video segment into the frame feature encoder to obtain the frame feature vector of the query video; Based on the frame feature vector of the query video, determine the videos in the search video that are from the same source as the query video; The frame feature encoder is any one of the frame feature encoders described in claims 1 to 6.

11. The method according to claim 10, characterized in that, The step of determining, based on the frame feature vector of the query video, videos from the same source as the query video in the searchable video includes: Based on the frame feature vector of the query video, generate the segment feature vector corresponding to the query video; The feature vector of the segment corresponding to the query video is input into the vector search engine for retrieval, and video segments similar to the query video segment are retrieved; wherein, the index of the vector search engine is constructed based on the feature vector of the segment corresponding to the video segment to be retrieved. Based on the video segments to be retrieved that are similar to the query video segment, identify the videos in the video segment to be retrieved that are from the same source as the query video.

12. The method according to claim 11, characterized in that, The vector search engine's index is created according to the following steps: Input all frames contained in the search video segment of the video to be searched into the frame feature encoder to obtain the frame feature vector of the video to be searched. Based on the frame feature vector of the video to be retrieved, generate the segment feature vector corresponding to the video to be retrieved; An index for a vector search engine is constructed based on the feature vectors of the segments corresponding to the video to be retrieved.

13. The method according to claim 11, characterized in that, Also includes: The video to be retrieved is sampled at preset intervals to obtain the retrieved video segments; The query video is sampled at preset intervals to obtain the query video segments.

14. The method according to claim 11, characterized in that, Also includes: The video to be retrieved is subjected to shot boundary detection and in-shot sampling to obtain the retrieved video segment; The query video is processed by lens boundary detection and in-lens sampling to obtain the query video segment.

15. The method according to claim 12, characterized in that, The step of generating a segment feature vector corresponding to the video to be retrieved based on the frame feature vector of the video to be retrieved includes: The frame feature vector of the video to be retrieved is segmented to obtain the frame feature vector contained in each segment; The frame feature vector contained in each segment is input into the segment feature encoder for processing to obtain the segment feature vector corresponding to the video to be retrieved; wherein, the segment feature encoder is any of the segment feature encoders described in claims 2-6.

16. The method according to claim 11, characterized in that, The step of inputting the feature vector of the segment corresponding to the query video into a vector search engine for retrieval, and recalling video segments similar to the query video segment, includes: The segment feature vector corresponding to the query video is input into the vector search engine for retrieval, and the distance between the segment feature vector corresponding to the query video and the segment feature vector of the video to be retrieved is obtained. When the distance is less than a preset distance threshold, the video segment to be retrieved corresponding to the segment feature vector of the video segment to be retrieved at the distance is recalled.

17. The method according to claim 11, characterized in that, Also includes: Based on the video segments to be retrieved that are similar to the query video segment, locate the start and end positions of the plagiarized segments in the videos that are from the same source as the query video.

18. A device for querying videos from the same source, characterized in that, include: The frame feature vector acquisition unit of the query video is used to input all the frames contained in the query video segment of the query video into the frame feature encoder to obtain the frame feature vector of the query video. The same-origin video determination unit is used to determine, based on the frame feature vector of the query video, a video in the search video that is from the same origin as the query video; The frame feature encoder is any one of the frame feature encoders described in claims 1 to 6.

19. A feature encoder training system, characterized in that, include: Platform server and user terminal; The platform server is used to obtain the video to be processed sent by the user terminal; Two video segments are selected from at least one video to be processed, the two video segments at least partially overlapping on the timeline; the two video segments include a first video segment and a second video segment; the second video segment is subjected to image editing processing using a pre-set image editing method to obtain a second target video segment that is similar to the second video segment after editing; all frames contained in the first video segment and the second target video segment are input into a frame feature encoder to obtain a frame feature vector; the frame feature encoder is trained based on the frame feature vector to obtain a target frame feature encoder; the target frame feature encoder is sent to the user terminal; The user terminal is used to send the video to be processed to the platform server; and to obtain the target frame feature encoder sent by the platform server. The platform server is further configured to encode the video segment using a target segment feature encoder to obtain a segment feature vector of the target segment; and to store the segment feature vector of the target segment, wherein the segment feature vector of the target segment is used to retrieve the target segment.

Images